<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Carlos Andrade on Medium]]></title>
        <description><![CDATA[Stories by Carlos Andrade on Medium]]></description>
        <link>https://medium.com/@carlosviansi?source=rss-173d7a1ac00------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hgJkCapKNfCnGQyQTy_Rfw.jpeg</url>
            <title>Stories by Carlos Andrade on Medium</title>
            <link>https://medium.com/@carlosviansi?source=rss-173d7a1ac00------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:03:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@carlosviansi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Análise de Dados com o Orçamento Público no Brasil]]></title>
            <link>https://medium.com/@carlosviansi/an%C3%A1lise-de-dados-com-o-or%C3%A7amento-p%C3%BAblico-no-brasil-81ce984f96e1?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/81ce984f96e1</guid>
            <category><![CDATA[brasil]]></category>
            <category><![CDATA[governo]]></category>
            <category><![CDATA[orcamento]]></category>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Thu, 03 Dec 2015 20:21:02 GMT</pubDate>
            <atom:updated>2015-12-03T20:21:02.150Z</atom:updated>
            <content:encoded><![CDATA[<p>Esta é uma lista de referências que tem sido repassadas no projeto <a href="http://gastosabertos.org/">Gastos Abertos</a> para entender sobre o orçamento público.</p><p>Considere por exemplo os dados disponíveis sobre orçamento e execução do município de São Paulo:</p><p><a href="http://orcamento.prefeitura.sp.gov.br/orcamento/">Balancetes Receitas Próprias de Impostos - Portal da Prefeitura da Cidade de São Paulo</a></p><p>Se você possuir quase nenhum conhecimento sobre o vocabulário e tem interesse em entender para utilizar estes dados, estamos no mesmo ponto de partida.</p><p>Um livro inicial no assunto é o de Odilon Guedes com apenas 100 páginas e um dos mais baratos no assunto.</p><p><a href="http://www.extra.com.br/livros/AdministracaoNegocios/LiderancaGestaodeEquipes/Livro-Orcamento-Publico-e-Cidadania-Odilon-Guedes-2394311.html">Livro - Orçamento Público e Cidadania - Odilon Guedes - Liderança e Gestão de Equipes no Extra.com.br</a></p><p>Segunda recomendação são as aulas sobre Orçamento Público por Paulo Marin. O livro conforme mencionado no vídeo em que se baseia a aula é o de James Giacomoni, citado logo abaixo. Existe mais de 1 parte, basta buscar na lista de vídeos adicionados no canal.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FdFsDMoQNhDg%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DdFsDMoQNhDg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FdFsDMoQNhDg%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b13a7755f93376969e546591bcaae4b/href">https://medium.com/media/8b13a7755f93376969e546591bcaae4b/href</a></iframe><p><a href="http://www.extra.com.br/livros/AdministracaoNegocios/FinancasdeMercadodeCapitais/Orcamento-Publico-James-Giacomoni-1737117.html">Livro - Orçamento Público - James Giacomoni - Finanças de Mercado de Capitais no Extra.com.br</a></p><p>Também foi recomendado o livro:</p><p><a href="http://www.americanas.com.br/produto/7416856/livro-financas-publicas-teoria-e-pratica-no-brasil">Livro - Finanças Públicas: Teoria e Prática no Brasil</a></p><p>A quantidade de conceitos pode ser excessiva para uma primeira exposição. Mas após brincar com os dados um pouco e observar os códigos, as vídeo-aulas partes 1 a 5 são o suficiente para começar a entender boa parte dos códigos fornecidos e como são agregados.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81ce984f96e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualizing More With R — Interactive Graphs and Maps]]></title>
            <link>https://medium.com/@carlosviansi/visualizing-more-with-r-interactive-graphs-and-maps-ae66308f52bb?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae66308f52bb</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Sun, 29 Nov 2015 05:09:50 GMT</pubDate>
            <atom:updated>2015-11-29T05:15:20.009Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5VD3IiAVJJqmuDlxBVHimA.png" /></figure><p>On one discussion on the <a href="http://gastosabertos.org/">Gastos Abertos</a> project I have been participating, someone suggested an interactive graph of the static plot title of this post. Since I am only used to ggplot2, I tried to get as close to as what was suggested: Being able to turn invisible some of the lines by clicking them.</p><p>However, little I knew I was so very close to the plot. Accidentally I stumbled on ggplotly, from plot.ly which literally only required me to pass the ggplot2 graph as a parameter. The result is as follows:</p><p><a href="https://plot.ly/~carlosviansi/9.embed">Ritmo vs Meses | line chart made by Carlosviansi | plotly</a></p><p>A more detailed article comparing other libs for data visualization that I came across can be found at:</p><p><a href="http://ouzor.github.io/blog/2014/11/21/interactive-visualizations.html">Interactive visualizations with R - a minireview</a></p><p>Comparing several basic plots across all those libraries to show on R Markdown.</p><p>Other kind of visualization I have been musing on is ggmap, which appears to be a breeze to generate map plots.</p><p>The author page is here:</p><p><a href="https://sites.google.com/site/davidkahle/ggmap">ggmap - David Kahle&#39;s Homepage</a></p><p>A simple tutorial showing its power and easiness is here:</p><p><a href="http://amunategui.github.io/ggmap-example/">Mapping The United States Census With {ggmap}</a></p><p>And a more detailed explanation can be found on the following article:</p><iframe src="https://drive.google.com/viewerng/viewer?url=http%3A//stat405.had.co.nz/ggmap.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a326c97d56761a47505f1c15bb42068/href">https://medium.com/media/2a326c97d56761a47505f1c15bb42068/href</a></iframe><p>There has also been some discussion on integrating them with R Markdown for data reports:</p><p><a href="http://stackoverflow.com/questions/28173647/interactive-r-markdown-document-with-ggmap">Interactive R Markdown Document with ggmap</a></p><p>Last but not least, I also came across this Social Network Analysis lib for much better (interactive) plots. Some are also useful for hierarchical clustering dendograms:</p><p><a href="http://christophergandrud.github.io/networkD3/">http://christophergandrud.github.io/networkD3/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae66308f52bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to IPython Workshop]]></title>
            <link>https://medium.com/@carlosviansi/introduction-to-ipython-workshop-b1126d1798d4?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1126d1798d4</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ipython]]></category>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Sat, 10 Oct 2015 09:06:52 GMT</pubDate>
            <atom:updated>2015-10-10T09:06:52.494Z</atom:updated>
            <content:encoded><![CDATA[<p>These are some notes I took while attending to an IPython Notebook Workshop. If you are interested on learning about them and perhaps has some interest on how this cost-free framework relates to MATLAB read on.</p><h3>Notes</h3><p>The workshop is made by the following group:</p><p><a href="https://www.soihub.org/">https://www.soihub.org</a></p><p>These notes were ran on a Macbook. It is interesting to note that a static version of the notebook is rendered on Github if you try to open them there. Also on the notebook you can find a link to a NASA notebook online that include interactive menu bars that update the plot on the fly. Noteworthy the report is running on someones else computer on that report, which is interesting to showcase results.</p><ol><li>Install Continumm Anaconda</li></ol><p><a href="https://store.continuum.io/cshop/anaconda/">Anaconda</a></p><p>2. Download the Ipython Notebook.</p><p><a href="https://github.com/diego898/ipython_intro_hawaii">diego898/ipython_intro_hawaii</a></p><p>Open the terminal and type ‘ipython notebook’. Run it from the home directory or you may have issues to navigate to the right folder.</p><p>3. Open Introduction to IPython.ipynb</p><p>Run the first 2 blocks to check if the libraries are install after the conda installation. If any is missing run on the terminal:</p><blockquote>conda install seaborn</blockquote><p>Ipython is organized in cells. There are two type of cells. Code and Markdown (you can see by clicking on them and checking the dropdown box to the right side of the play,stop,loop button).</p><p>You can double click a cell to see the internals of the cell.</p><p>On the cell with images and video, remove the “ ‘ “ and it will render the image or videos.</p><p>For seeing the documentation directly from the notebook you can add a question mark to the end. For example:</p><blockquote>np.random?</blockquote><p>The notebook is just a text file so it plays well with version control. Before github, nbviewer was used to render. Host could be used for example on gist.</p><p>Ipython was renamed to jupyter. See jupyster.uscd.edu for an example of a server.</p><p>Spyder provides an IDE just like MATLAB and RStudio.</p><p>The notebook then moves to Panda Library which works like a mini-excell and provides the dataframe capability to Python. Essentiality this was motivated with some exploratory analysis examples on the Iris Dataset. The screenshot plot matrix should go here.</p><p>It was also pointed out the .describe() function on one of the cells of the notebook that is equivalent to the R summary() function.</p><p>It was pointed out on the Machine Learning block that the methods are very similar across all the environments. Some prediction functions were shown but not ran on the interest of time, plots were shown emphasizing the little amount of code to get work done. There is a block with a interactive piece on the notebook on cell [80].</p><p>On Resources there is a link for a github gallery of Ipython Notebooks.</p><p>Probabilistic Methods for hackers is a book which was created entirely on ipython notebook.</p><p>Plots can get pretty slow (see the 3D cube on the Python noteboook). To get around this, we can see the IPython notebook called “Plotly Examples” shows how to use plotly to create the plots using javascript. The interaction it is much more smooth. An advantage of using plotly is that even if the notebook is static the graph is still interactive where the person is accessing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gVQgydHNf1sBE0wLvtjgxA.png" /></figure><p>Resource sections will be updated after the workshop.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1126d1798d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seeing Patterns with Good Data Visualization. Tales of ggplot2 in R]]></title>
            <link>https://medium.com/@carlosviansi/seeing-patterns-with-good-data-visualization-tales-of-ggplot2-in-r-b81717659828?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/b81717659828</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Sat, 10 Oct 2015 09:00:01 GMT</pubDate>
            <atom:updated>2015-10-10T09:32:55.826Z</atom:updated>
            <content:encoded><![CDATA[<p>Among several other nice things my Masters exposed me to were data visualization problems. Curiously enough, this was a personal need, as my advisor often preferred to “code up a perl script to do the job for him”. I instead always preferred to think of a way to see it rather than having a routine do the job and say yes or no. In fact, this dates back to my anguish with statistical correlations when a plot could just be made. But this is a digression and rant for other post and I will move on.</p><p>The purpose of this post is just to reunite some data problems I had while working on my thesis, and show how I tackled it with a visualization. Perhaps it is not the best approach to it, and if you happen to notice this, please do comment with a better or an alternative way :)</p><h3>Some Context about the following plots</h3><p>Like I mentioned, the plots are all related to my Ms Thesis. The work is about solar forecasting for renewable energy applications in Hawaii, and more specific on Oahu island. Here is a Google Maps I made by hand mapping the stations across the island.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.google.com%2Fmaps%2Fd%2Fembed%3Fmid%3DzhfCWfFL_lfY.kHBo1bVPXCxY%26hl%3Den_US&amp;url=https%3A%2F%2Fwww.google.com%2Fmaps%2Fd%2Fviewer%3Fmid%3DzhfCWfFL_lfY.kHBo1bVPXCxY%26hl%3Den_US&amp;image=https%3A%2F%2Fwww.google.com%2Fmaps%2Fd%2Fthumbnail%3Fmid%3DzhfCWfFL_lfY.kHBo1bVPXCxY%26hl%3Den_US&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=google" width="600" height="450" frameborder="0" scrolling="no"><a href="https://medium.com/media/367dbdbd173de59dbd0b01cf1423b10e/href">https://medium.com/media/367dbdbd173de59dbd0b01cf1423b10e/href</a></iframe><p>You don’t need to worry about the icons, but if you click on any one of them you will get something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*vgSroIaYxCZ1Ey2lPhxlng.png" /></figure><p>This shows some of the metadata I was interested per station. A station is just a collection of weather sensors on the ground. For SCBH1, there were sensors to collect the temperature (TMP), dew point (DWP), relative humidity (RELH), solar radiation (SOLR), and precipitation (PREC).</p><p>Another very important piece of information is how much data of previous years is available there, SCBH1 has it since 2002. Again, my interest is on solar forecasting across different sites on the island, and of course the site location, amount of data available for training a model, and weather variables are potential factors that can affect the forecasting performance. Most of the stations sample each weather variable hourly throughout the year.</p><h3>Visualization 1: Missing Data</h3><p><strong>Problem:</strong> Let’s say that I am interested on comparing something across different sites of the island (which would be done using different stations data). While at first we could just go by the metadata from MesoWest specifying the time range the data is available, missing data due to problems on a given sensor (thus a particular weather variable or all the weather variables of the station) can still occur. If we then compare two different sites by some criteria using stations that have big gaps on different consecutive years, we may be misled on our conclusions.</p><p><strong>Solution: </strong>To easily compare which stations have different patterns, the following plot was created in R ggplot2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*zADMrKw2LKNgNxM4Dg3eMA.png" /></figure><p>It shows the weather variables of interest of each station of interest (here the weather variable is just solar) and colors in red all the days (or hours depending on the granularity of interest) there is missing data. This makes it now very easy to decide which stations to choose which very minimal memory effort.</p><h3>Visualization 2: Group of Models</h3><p><strong>Problem: </strong>By just using a simple probability model to verify non-linear correlations among weather variables using forecast errors can become quite complicated as more and more weather variable combinations are tested if you are relying solely on bar plots.</p><p><strong>Solution:</strong> I found it easier to use a heat map of forecast errors to more easily compare several models and investigate the non-linear correlations. One visual cue that will attract our eyes will be very different colors, and that is great!</p><p>A second one will be clusters of cells with the same color.This will be dependent on row organization. For my particular interest, this is also fine. Specifically, there are some hypothesis we would be interest on seeing all at the same time. The rows 1 to 7 for example on the heatmap are verifying correlations of solar energy and any other weather variable. The rows 8 to 12 are verifying cross-site forecasting using an individual weather variable. The following every 5 rows do so testing different cross-sites.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*pUDxRHig0bWU-u0r6s-hUQ.png" /></figure><p>This heat map therefore provides a quick glance over several hypothesis which otherwise would be compared across many, many bar plots. Furthermore, due to the first visual cue, we may be able to identify non-linear correlations across our different original hypothesis which could suggest other re-orderings.</p><h3>Visualization 3: Non-Linear Temporal Correlations</h3><p><strong>Problem: </strong>Understand non-linear pairwise correlations across weather variables. Typically mutual information applications across probability distributions are summarized in a single number, such as would be a linear correlation.</p><p><strong>Solution. </strong>An extension of it to account for the temporal dimension is shown on the plot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BadF98dWM1CrykjhkXZqNg.png" /></figure><p>Basically we are seeing how other weather variable relates to solar radiation as we increase the length of days (e.g. precipitation yesterday and solar radiation today ; precipitation 2 days ago and solar radiation today, etc). Had we considered only one point in time we would be left with a value below 0.1 which wouldn’t really tell us much (given M.I. is not as intuitive as a correlation which ranges from 0 to 1). We would miss the fact the M.I. reaches its peak on 0.1 not on the closest day, and that beyond a 25 days it start increasing again.</p><h3><strong>Other Visualizations</strong></h3><p>I will edit this post with other visualizations as the work evolves and time permits. Although the ones above originated from my own ideas or peer discussions to be applied under this context, I am also planning on reusing other more basic plots available on the web:</p><p><a href="http://www.r-bloggers.com/my-favorite-graphs/">My Favorite Graphs</a></p><p>This particular one called my attention because of the blue dots overlay to indicate the distribution of points across the quartiles, and therefore how confidently we can draw conclusions on the variation across quartiles.</p><p>Another style of visualization I used in the past and I find interesting although not immediately understood by some are scatterplot with axial histograms as shown here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S-LWf2ALtxeIHNNsIkH0Gw.png" /></figure><p>I originally came across the concept for the first time on a StackOverflow post:</p><p><a href="http://stackoverflow.com/questions/8545035/scatterplot-with-marginal-histograms-in-ggplot2">Scatterplot with marginal histograms in ggplot2</a></p><p>As you will notice, the aesthetics of mine are much better than the one on stackoverflow. I find it that aesthetics slightly help on visual cues. Luckly after you set your preferences on a function, it is as simples as adding it to any ggplot2 graph call. My function is a combination of the notes of the following post:</p><p><a href="http://www.noamross.net/blog/2013/11/20/formatting-plots-for-pubs.html">How to format plots for publication using ggplot2 (with some help from Inkscape)</a></p><p>plus font styling:</p><p><a href="http://www.r-bloggers.com/change-fonts-in-ggplot2-and-create-xkcd-style-graphs/">Change fonts in ggplot2, and create xkcd style graphs</a></p><p>You can find the code altogether here (without the font piece):</p><p><a href="http://rpubs.com/carlosandrade/fb_flickr">RPubs - Comparison of Flickr and Facebook Amount of Comments</a></p><h3>Other Partially Related Stuff</h3><p>If your interest goes a bit beyond visual cues on graphs and extends to everyday design such as Don Normann’s book you may be interested on my Pinterest of photos I took of good and bad design:</p><p><a href="https://www.pinterest.com/carlosviansi/the-design-vagrant/">https://www.pinterest.com/carlosviansi/the-design-vagrant/</a></p><p>or my rants on poor templates of data reports:</p><p><a href="https://medium.com/p/1684baee40ca"></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b81717659828" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rethinking Probability Theory for Data Analysis]]></title>
            <link>https://medium.com/@carlosviansi/rethinking-probability-theory-for-data-analysis-7a688a0c49c6?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a688a0c49c6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[mooc]]></category>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Sat, 10 Oct 2015 06:57:15 GMT</pubDate>
            <atom:updated>2015-10-10T07:01:40.618Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Course</h3><p>I am currently following up one interesting probability theory course on Coursera:</p><p><a href="https://www.coursera.org/course/probability">Probability - University of Pennsylvania | Coursera</a></p><p>This is particularly great because I can find other like-minded folks discussing the subject :) In fact, already on the first week I got a good set of references from the professor himself on another student’s post. The course was described by the student as taking the following approach:</p><blockquote>Observation →experiment →general definitions/theorems →proof provided</blockquote><p>I also liked the Lecturer reply to the student’s question on material with rigorously definition of the subject:</p><blockquote>Any of the standard graduate level mathematics textbooks in probability theory will contain formal material with an emphasis on proof along the lines of what you’re looking for. Ultimately, which one(s) you select are a matter of accident (your professor happens to use a particular book) and taste (your preferences on, say, writing style, pedagogical exposition, and so on). In addition to my own book there are several other books I like very much as well. Here is a short list of my personal favorites: it’s headed by the two volume sequence of the immortal Feller; other books I have enjoyed very much are those by Billingsley, Williams, and Chung. All of these are really graduate-level books though the first volume of Feller’s can be tackled by an ambitious senior undergraduate who has a strong mathematics background. And all of them come with an attitude and a style: which one(s) you like is where personal taste comes into the equation.</blockquote><blockquote>In this course the emphasis is not on theorem-proving so much as in the logical development of concepts and their link to application. But we do end with three of the fundamental theorems in probability: Poisson approximation (the law of small numbers), the law of large numbers, and the central limit theorem. The heart of the demonstration of Poisson approximation is in the lectures (though I don’t make a song and dance about getting all the i’s dotted and t’s crossed); we actually see a complete proof of an elementary version of the law of large numbers on the way to understanding why polls work; and I finesse the proof of the central limit theorem altogether (it’s too much for a first course).</blockquote><h3>Rethinking Probability Theory for Data Analysis</h3><p>My first serious exposition to Probability Theory came as a freshman in an undergraduate computer science majors. It came, however, within a Statistics class, which really didn’t help making it clear how one related to the other. Furthermore, more than often I see probability classes embedded within Statistics and this may lead to a lot of confusion.</p><p>Data analysis is the most immediate thought after talking about Statistics nowadays with the Data Science hype. Probability, however, at least on my undergraduate days reminded me of those “annoying” exercises to count how many colored balls are in the box, coin flips, card games (which I often had to ask my professor about the assumed details since I didn’t play it) or those “weird formulas” of probability distributions.</p><p>The year of 2015 was really fruitful to revise my understanding on the matter. First of course, because of this amazing course. Second because of a class currently running on my school at the University of Hawaii at Manoa going by the name of <strong>Pattern Recognition</strong> using a classic textbook on the subject:</p><p><a href="http://www.amazon.com/gp/product/0471056693/ref=pd_lpo_sbs_dp_ss_1?pf_rd_p=1944687582&amp;pf_rd_s=lpo-top-stripe-1&amp;pf_rd_t=201&amp;pf_rd_i=1118456688&amp;pf_rd_m=ATVPDKIKX0DER&amp;pf_rd_r=0FYYDDRYTGKESBWMRRW3">Pattern Classification (Pt.1)</a></p><p>The book motivates on Chapter 1, and discuss on Chapters 2,3 and 4 a more advanced take on Probability within Pattern Recognition. While the book is relatively old (2000), and these very same chapters are met with some skepticism by some Amazon reviewers, e.g.:</p><blockquote>A disproportionate number of pages are devoted to topics like density estimation despite the fact that it has been well established in recent years, through the work of Vapnik and others, that when working with limited data, trying to solve the problem of pattern classification through density estimation (which turns out to be, in a well-defined sense of the term, a much harder problem than pattern classification) is rather futile. When modern techniques for learning pattern classifiers from limited data sets (e.g., support vector classifiers) are touched on in the book, the treatment is disappointingly superficial and in some cases, misleading. (December 27, 2000).</blockquote><p>I found it refreshing to see a more general framework of probability theory that isn’t necessarily focusing on more rigorous math alone, but leading towards a notion of data analysis.</p><p>Lastly, my masters is currently using basic probability models for solar forecasting on renewable energy applications for identifying temporal and spatial patterns. I was surprised to see how just simple R code routines for joint conditional probability distributions on time-series could lead to so much insight, discussion and even published work on a decent conference. On that note, after the Probability course I am currently following to polish my understand on the basics, and before the Pattern Recognition class, I think this particular EdX course is a good glue to both:</p><p><a href="https://www.edx.org/course/introduction-probability-science-mitx-6-041x-0">Introduction to Probability - The Science of Uncertainty</a></p><p>As it introduces Random Processes, something my work walks around quite a lot since it deals with time series.</p><h3>Information Theory</h3><p>Probability Theory also lies on the first steps of a staircase I am interested to climb for my PhD: Thermodynamics of Information. But before going there, a background on Information Theory is necessary. I only played around little with it on my Ms on this realm. A good starting point MOOC appears to be on Coursera:</p><p><a href="https://www.coursera.org/course/informationtheory">Information Theory - The Chinese University of Hong Kong | Coursera</a></p><p>Although the video lectures are no longer available at Coursera, after some digging around I found them on the lecturer’s own website:</p><p><a href="http://www.inc.cuhk.edu.hk/InformationTheory/lectures.html">Information Theory - Prof. Raymond W. Yeung</a></p><p>The course has a book by the same author, but a textbook I am constantly referred to is:</p><p><a href="http://www.amazon.com/Elements-Information-Edition-Telecommunications-Processing/dp/0471241954/ref=dp_ob_title_bk">Elements of Information Theory 2nd Edition (Wiley Series in Telecommunications and Signal Processing)</a></p><p>A companion textbook is also:</p><p><a href="http://www.amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/0521642981">Information Theory, Inference and Learning Algorithms</a></p><p>In my case, the discussion presented here is more interesting to grasp basic concepts of the Information Theory realm, such as Entropy, Conditional Entropy, Mutual Information, KL-Divergence ,etc. There are also companion video-lectures to this book. See this youtube listing for example:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBCiZc0n6COY%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBCiZc0n6COY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBCiZc0n6COY%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8500df7a54b5677de29385b50c098d17/href">https://medium.com/media/8500df7a54b5677de29385b50c098d17/href</a></iframe><p>Most of the material above will try to introduce you notions of Information Theory from an Engineering / Signal Processing perspective. Somehow this is hard to sink in my head with my probability theory “way to see things” if you will.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*X9XMeRXQeJBcvCs_qa9QmQ.png" /><figcaption>Mutual Information Venn. Diagram on Wikipedia and also used on the presentation.</figcaption></figure><p>A more interesting visualization to first come across Information Theory concepts is by one well known on Probability Theory since highschool: Venn Diagrams.</p><p><a href="http://www.nimbios.org/wordpress-training/entropy/2015/03/25/introductory-talk/">Introductory Talk</a></p><iframe src="https://drive.google.com/viewerng/viewer?url=http%3A//math.ucr.edu/home/baez/nimbios/nimbios_baez.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd58f2d51a9c94f7c2a537f2e1cc585d/href">https://medium.com/media/bd58f2d51a9c94f7c2a537f2e1cc585d/href</a></iframe><p>I only saw this approach being used as a “refresh” lecture on a <a href="http://www.nimbios.org/wordpress-training/entropy/2015/03/25/introductory-talk/">workshop</a> for an introductory explanation. So you understand the context, this video is an attempt to normalize vocabulary to the following presentations that will occur on the workshop. Nonetheless, it is a good summary of the entropy metrics.</p><p>Shannon’s Entropy start being introduced on 3:25.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FGEv2u9zSCdQ%3Fstart%3D206%26feature%3Doembed%26start%3D206&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGEv2u9zSCdQ%26feature%3Dyoutu.be%26t%3D3m26s&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FGEv2u9zSCdQ%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e75edb3eb030f59f5737976e448fad6/href">https://medium.com/media/3e75edb3eb030f59f5737976e448fad6/href</a></iframe><p>Lastly these lecture notes were another good summary of the concepts and in fact a good presentation of them (there are so many names for the same information theoretic metric, and those notes do a good job on mentioning them).</p><iframe src="https://drive.google.com/viewerng/viewer?url=http%3A//www.cims.nyu.edu/%7Echou/notes/infotheory.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ca2d75a65f72bd7a1b13bfab6c0eab6/href">https://medium.com/media/9ca2d75a65f72bd7a1b13bfab6c0eab6/href</a></iframe><h3>Thermodynamics of Information and Prediction</h3><p>The next chapter on this story starts on an older blog post of mine:</p><p><a href="https://medium.com/p/fb31757d095a"></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a688a0c49c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thermodynamics of Prediction]]></title>
            <link>https://medium.com/@carlosviansi/thermodynamics-of-prediction-fb31757d095a?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb31757d095a</guid>
            <category><![CDATA[entropy]]></category>
            <category><![CDATA[thermodynamics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Sun, 30 Aug 2015 22:35:32 GMT</pubDate>
            <atom:updated>2015-08-30T22:58:53.656Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sKfEyckrLJd9B5uwIArqYA.png" /><figcaption>Image Credit: <a href="https://en.wikipedia.org/wiki/Maxwell%27s_demon#/media/File:Maxwell%27s_demon.svg">Wikipedia</a></figcaption></figure><p>Disclaimer: These notes were made while learning about the subject. Therefore they may contain mistakes. Please add a comment on any paragraph you find confusing and/or wrong and I will do my best to fix it!</p><h3>Objective</h3><p>My current interest in this area is the concept of <strong>Non-Predictive Information</strong>. Take one given learning algorithm and associated with it, a data set. The question I am interested in is then, after the <strong>learning step </strong>has occurred, how much of the <strong>information acquired by the learning algorithm is useful (predictive-information)? And therefore how much is useless (non-predictive information)?</strong></p><iframe src="https://drive.google.com/viewerng/viewer?url=http%3A//tdlc.ucsd.edu/SV2013/Bell_etal_2012_thermodynamics_of_prediction.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/49dd1e445f3e10ddb0c8749285e6f59a/href">https://medium.com/media/49dd1e445f3e10ddb0c8749285e6f59a/href</a></iframe><p>According to the <strong>Thermodynamics of Prediction</strong>, non-predictive information is defined as the difference between instantaneous memory and predictive power, Imem(t) − Ipred(t). It represents useless memory and provides a measure for the ineffectiveness of the model.</p><p>One of the results of the work is then that <strong>Non-predictive Information is tied to Dissipated Work</strong>. I.e. there is a way to understand the boundaries of learning by looking at thermodynamics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*UYXuVqgYuUlgPazCQdvg0w.png" /><figcaption>Equation 14.</figcaption></figure><p>That means now, being a computer scientist who has been exposed to the right side of the equation, finding myself trying to understand the framework underlying the left side. These notes are the footsteps I am leaving behind so I don’t get lost in the woods on my first encounter with this vast field.</p><p>Specifically, Equation 14 takes place within the realm of Thermodynamics. And this first post reflects some basic concepts in this field while walking inside the woods to fully understand the equality proposed on the equation.</p><blockquote>The first goal here is to understand how Information Theory relates to Thermodynamics.</blockquote><p>This first understanding goal maps back to Szilard’s Engine. Sizlard’s Engine is actually a “answer” to a apparent paradox proposed by Maxell, which was in turn known as “Maxwell’s Demon”. Finally, the apparent paradox posed by Maxwell is a violation of <strong>The Second Law of Thermodynamics</strong>. As such, it is clear that to fully understand equation 14, an understanding of a Thermodynamics Framework is essential.</p><p>Discussions with Elan Stopnitzky and Lisa Miller, together with EdX Thermodynamics (see References at the end of the post) helped me decide what material of Thermodynamics was more important to focus on for the purpose of reaching, at least partially, this ‘framework’, or way of thinking about the presented problem.</p><p>The remainder of these notes are reminders of concepts that more than often I came across on the papers talking about the relation between Information and Thermodynamics, or simple <strong>Thermodynamics of Information</strong>. If they do not immediate relate to that, they either helped me form a better understanding of the framework, or were preliminary concepts to understand them. Without further ado, let’s jump to them.</p><h3>Thermodynamics Introduction</h3><p>One fundamental idea surrounding all the work in this area is the notion of <strong>system</strong>, <strong>heat</strong>, <strong>work</strong> <strong>extracted</strong> and so forth. While intuitively these concepts make sense, often I felt like I was understanding the more advanced material as <em>computer scientist</em>, rather than someone from <em>physics</em> or <em>engineering</em>. In other words, having the wrong framework in my head, often I found myself thinking: “I understand how it is done, but not why it was done this way”.</p><p>Let’s start from the word ‘Thermodynamics’ itself.</p><blockquote>Thermodynamics is essentially going to look at stuff which interacts doing or absorbing work and which interacts doing and absorbing heat. And these could be machines, these could be gadgets, this could be anything. In thermodynamics, we just call these thermodynamics systems.</blockquote><blockquote>[Thermodynamics EdX, Lecture 1, Block 2]</blockquote><p>In a nutshell, thermodynamic <strong>systems</strong> are defined by their <strong>boundaries</strong><em>.</em></p><p>If there are <strong>boundaries</strong><em> </em>across these <strong>systems</strong>, we can expect <strong>interactions</strong><em> </em>across these <strong>boundaries</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fZ484m-7idh1SS0r9b0ChA.png" /><figcaption>This is a draft of the EdX lecture showing the interplay of the concepts. A system A and B may have an interaction, with or not energy transfer, which can lead to a change of states on both. This change of states is defined by process.[Thermodynamics EdX, Lec. 4 — Block 1]</figcaption></figure><p>These interactions may or not exchange <strong>energy </strong>(e.g. think of how someone would put money in a bank). Furthermore, often one system is of interest, “<strong>the system</strong>”, and the other is secondary. The secondary is sometimes called <strong>surroundings</strong>, or <strong>environment. </strong>A system is also called <strong>closed</strong> if no mass can flow across the system boundaries (e.g. a closed water bottle), or <strong>open </strong>otherwise. Both<strong> Open </strong>and<strong> Closed systems </strong>can still have <strong>energy transactions</strong><em> </em><strong><em>(</em></strong>e.g. heat from hands transferring to the cold water inside the closed bottle). If neither mass or energy can flow, the system is called <strong>isolated</strong>.</p><h3>System States</h3><p>Another recurring concept is that of the <strong>system states</strong>. Specifically, when two systems <strong>interact </strong>they may <em>change</em> the <strong>state </strong>of one another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BNS2zeriWZ2W2UTu5TWS9Q.png" /><figcaption>[Thermodynamics EdX, Block 9]</figcaption></figure><p>The relation between interaction and the change of states is also fundamental for the <strong>Thermodynamics of Prediction</strong>. Finally what describes a state is also a concept that may appear too vague at a first glance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DvktILMVwk2SECTY4zXSZQ.png" /><figcaption>[Thermodynamics EdX, Block 9]</figcaption></figure><p>To define a state of a system, we need (a) the list of relevant characteristics and (b) quantify them. The list of relevant characteristics and associated values comes from expertise of the actual system. Nonetheless, having defined the list of relevant characteristics and measured them, a state can be defined by which values the chosen characteristics take, as shown by Gas [ … ].</p><h3>Macroscopic and Microscopic Approach</h3><p>Another point of important terminology surrounding systems is wether the approach is Macroscopic or Microscopic.</p><p>Microscopic approach define a system as a (often very large) amount of <strong>N</strong> particles, where each particle is defined by a vector of position <strong>x(t) </strong>and a vector of velocity <strong>v(t), </strong>both parameterized by time <strong>t</strong>. This leads to statistical thermodynamics and statistical mechanics.</p><p>The other approach is the Macroscopic. In this case the system is assumed continuous, which leads to fewer parameters, laws, etc. This is also called the “Classic” approach.</p><p>Later on, to understand Maxwell’s Demon and Szilard Engine, it will be useful to see how <strong>The Second Law of Thermodynamics can be redefined from Macroscopic to Microscopic. For now, the remaining of the notes will be using theMacroscopic level. It will be noted when we are changing to the Microscopic for the Second Law.</strong></p><h3>Properties</h3><p>Characteristics are formally known as properties, and can be defined on <strong>Primitive, Basic Thermodynamics, </strong>and<strong> Derived Properties. Gibbs function </strong>for instance is a derived property<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9cvUjmfgQwAK5UzLjMmufA.png" /><figcaption>[Thermodynamics EdX, Lecture 1, Block 11]</figcaption></figure><p>This basically provides a framework to reason about the used formulas later on the discussion and how they all relate to one another.</p><h3><strong>Interactions</strong></h3><p>There are two types of thermodynamics <em>interactions </em>across their <em>boundaries</em>:</p><ul><li>Work</li><li>Heat</li></ul><h4>Work Interaction</h4><p>There are several examples that can be used to introduce the notion of work. One of particular interest for the following discussion of Maxwell’s Demon, it’s alternative version by Szilárd, and also which a lot of systems in real world map to after having defined it’s boundaries, is the definition of a <strong>Cylinder-Piston Arrangement</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hY2z-oORwKWFcom0OwTGhg.png" /><figcaption>The Cylinder-Piston Arrangement. [Thermodynamics EdX, Lec. 4 — Block 2]</figcaption></figure><p>On this particular system, what minimally interest us is that the compressed gas applies a force <strong>F</strong> over the <em>piston </em>whose degrees of freedom are <strong>pressure</strong> and <strong>area</strong>. In doing so, if the opposing force to the piston is relaxed, the gas will move the piston by <strong>dx</strong>, therefore performing <strong>work over the piston.</strong></p><p>The <strong>Thermodynamical Definition</strong> of <strong>Work</strong> is given as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*2x5oKzbqbabNaQk7XlVYqg.png" /><figcaption>[Thermodynamics EdX, Lec 5, Block 1].</figcaption></figure><p>First, it is an energy interaction between two systems, second this interaction can be redefined into an abstract system where the work raises a weight of mass <strong>m</strong> under gravity <strong>g.</strong></p><p><strong>The Operational Definition of Work </strong>extends this notion for two systems. If neither of the 2 systems interacting can be setup so the definition is met, there will be a heat interaction (and perhaps partially work interaction).</p><p>While verifying wether or not the interaction is work or not or involve heat is not necessary at this point, this clarifies why most examples later on uses a setup of a weight being raised as result of work.</p><p>If there is only <strong>Work</strong> interaction, then we use the adjective <strong>adiabatic. </strong>The adjective applies <em>boundaries</em>, a <em>system </em>and a <em>process.</em></p><h4>Heat Interaction Preliminares</h4><p>Before it can be defined, the interplay of <strong>Energy </strong>and <strong>Adiabatic Processes </strong>must be defined.</p><p>Count’s Rumford experiments lead to understand that <strong>Work</strong> and <strong>Heat</strong> are two forms of <strong>Energy</strong>. Following by Joule’s experiment, the ideas of <strong>conservation of energy </strong>was later defined, which led to the <strong>First Law of Thermodynamics</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*laNYLkJ6KL_zs-TQ2qzIFw.png" /><figcaption>[EdX Thermodynamics, Lecture 7, Block 3]</figcaption></figure><p>As will be noted on Szilárd engine discussion, the redefinition of the <strong>Laws of Thermodynamics </strong>is quite common as a matter of convenience to discuss a topic or proof. Above are some of them.</p><p>The <strong>First Law of Thermodynamics </strong>states that if the initial and final states are fixed, and the process is adiabatic, then the work done will be the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*usNcJv642AJyzNCOgPeTKQ.png" /><figcaption>[EdX Thermodynamics Lecture 8, Block 2].</figcaption></figure><p>We can only evaluate the integral over the path to obtain the amount of work if the process (change of states from 1 to 2) is <strong>semi-quasi static.</strong> Nonetheless, we know under these conditions that after the change of states, <strong>some property will be changed. This property will be the thermodynamic energy (E) of the system. </strong>Specifically, a difference of energy equals the negative evaluation of work performed to the <em>adiabatic </em>change from state 1 to state 2.</p><h4>Heat Interaction</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*swlCaf-fwMpkb7vOpyh-aA.png" /><figcaption>[EdX Thermodynamics, Lecture 8, Block 4]</figcaption></figure><p>Consider now the two process 1-&gt;2 where one is adiabatic (ad) and the other is not. In general, W12 != W12,ad. Again, remember an adiabatic process dictates only work interaction exist, so it is plausible to expect “there is something more” on the non-adiabatic process by definition. The Heat Interaction Q is defined as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q3zAcSryQClmMDcjs73_eg.png" /><figcaption>[EdX Thermodynamics, Lecture 8, Block 4]</figcaption></figure><p>This tells us a equation from the First Law of Thermodynamics. In particular,</p><p>DeltaE = Q — W.</p><p>notice also that DeltaE is a change of property, while Q and W are interactions. Since they are interactions, there are <strong>two systems involved and DeltaE is energy in transit</strong>. Q and W are not contained in a system. It is always <strong>work done by a system, or heat absorbed by a system.</strong></p><p>Also, if state 1 &lt;=&gt;(equivalent to) state 2, i.e. we have a <strong>cycle</strong>, then DeltaE = 0 and therefore from the First Law of Thermodynamics equation, Qcycle = Wcycle.</p><p>It is also useful to keep in mind the expansion form of DeltaE and DeltaW when considering heat, specifically because this makes explicit the <strong>thermal energy of the system, DeltaU</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_m8Z-XRq_-9z7XtJAHoaA.png" /><figcaption>[EdX Thermodynamics, Lecture 8, Block 5]</figcaption></figure><p>Finally notice this can be observed <strong>as the law of conservation of energy</strong>, given the transfer of energy can only occur due to work or heat, and can’t be created from nothing.</p><h3><strong>Thermal Equilibrium</strong></h3><p>Having defined both types of interaction, and the equation of the <strong>First Law of Thermodynamics</strong>, we move to the <strong>Zeroth Law.</strong></p><p>Specifically, the first part of <strong>Zeroth Law </strong>guarantees that there will be state <strong>a</strong> of system A, and state <strong>b </strong>of system B, in which no heat transfer <strong>Q </strong>occurs. In such configuration, the systems A and B are said to be in <strong>Thermal Equilibrium</strong>. This concept is also very important, given the <strong>Thermodynamics of Prediction </strong>work focus mostly on the properties of systems when moved <strong>outside </strong>Thermal Equilibrium!</p><h3>Second Law of Thermodynamics</h3><p>The first and zeroth law do not fully state what is possible or impossible on a thermodynamical system. The intuition behind the second law is associated to energy spent to perform work is associated to heat loss. This loss of energy, in turn is associated to entropy.</p><p>Another definition is that in an isolated system, entropy <strong>S </strong>always increase until it reaches a maximum value.[Complexity Explorer]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XQ7KBWm_cHC_DnAda9Z3Zg.png" /><figcaption>[EdX Thermodynamics, Lecture 28, Block 3]</figcaption></figure><p>The entropy principle provide us with a equation. In particular, for adiabatic process, Q = 0, and S2-S1 &gt;= 0 results on the principle of increase of entropy, as aforementioned. Noteworthy, the equality 0 occurs when the system is reversible.</p><h3>From Macrospace to Microspace</h3><p>Aside from a single citation, the definitions reviewed here are mostly from the EdX course on Thermodynamics, which is targeted for a Mechanical Engineering. The focus of the class was on the Macrostate.</p><p>The following notes shift attention to the <strong>Complexity Explorer </strong>course, since more details are provided on the <strong>Microspace</strong>. Some of the formulae used later on relies on an understanding of what is discussed here.</p><h3>Thermodynamics Entropy and Statistical Mechanics Entropy</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBnsSBRQT-qo%3Ffeature%3Doembed%26start%3D186%26start%3D186&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Ft%3D186%26v%3DBnsSBRQT-qo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBnsSBRQT-qo%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e71921d785988f21dd8992a2aa8a0158/href">https://medium.com/media/e71921d785988f21dd8992a2aa8a0158/href</a></iframe><p>As pointed out on the video lectures, the discussion insofar presented of entropy as defined by the Second Law of Thermodynamics was introduced based on <strong>Clausius </strong>definition and it is associated to heat loss.</p><p>The one that interests us more is the one defined by Boltzmann, as it ties microstates and macrostates.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fuwif3RhQRec%3Ffeature%3Doembed%26start%3D416%26start%3D416&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Ft%3D416%26v%3Duwif3RhQRec&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fuwif3RhQRec%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/66ba79cff3ced15e5aa79ba900797200/href">https://medium.com/media/66ba79cff3ced15e5aa79ba900797200/href</a></iframe><p><strong>Second Law of Thermodynamics (Statistical Mechanics Version):</strong></p><p>In an isolated system, the system will always progress to a Macrostate that corresponds to the maximum number of Microstates.</p><p>S = k log W</p><p>The entropy S of a macro state is k times the natural logarithm of the number W of microstates corresponding to the macrostate. The Boltzmann Constant k also recurrently appears on later work discussion.</p><p>Since k and log are just to adjust the units, the proportion between the system entropy and the number of microstates W is clear. Now, note that a set of microstates leads to a single macrostate. Therefore, by the new equation of statistical mechanics, the macrostate which has a higher number of microstates, will have a higher S, i.e. entropy.</p><p>A second version of the law can them be derived:</p><p><strong>Second Law of Thermodynamics (Statistical Mechanics Version) V2:</strong></p><p>In an isolated system, the system will tend to progress to the most probable Macrostate.</p><p>Notice on this definition, the probability character of the law is explicit, as cited on the 2015 Thermodynamics of Information work referred by the class e-mail.</p><p><strong>Information Theory</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FvLhul9CgAOQ%3Ffeature%3Doembed%26start%3D263%26start%3D263&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Ft%3D263%26v%3DvLhul9CgAOQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FvLhul9CgAOQ%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/77ec321d8310f45a4303164d9f9d1d1e/href">https://medium.com/media/77ec321d8310f45a4303164d9f9d1d1e/href</a></iframe><p>It is interesting to note Shannon derived the formula based on Boltzmann’s. We can think of this initial formula:</p><p>H = k log M ; where M is the number of messages. Furthermore the Macrostate can be compared to the Channel Source, while the Microstate to each word.</p><p>Given the probability of each word being sent is not the same, we can substitute M by the associated probabilities of each word being sent, which derives the usual definition of entropy H.</p><p>We are now fully equipped to understand Maxwell’s Demon apparent paradox and Szilard Engine.</p><h3>Maxwell’s Demon</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FP_P13S3J1hA%3Fstart%3D272%26feature%3Doembed%26start%3D272&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Ft%3D272%26v%3DP_P13S3J1hA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FP_P13S3J1hA%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7256a5c4e98abe7e77d6d22a9e6b333a/href">https://medium.com/media/7256a5c4e98abe7e77d6d22a9e6b333a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*Msu4s1hhKvBOSylZuREx8w.png" /><figcaption>Image Credit: <a href="https://en.wikipedia.org/wiki/Maxwell%27s_demon#/media/File:Maxwell%27s_demon.svg">Wikipedia</a></figcaption></figure><p>In short, the idea behind Maxwell Demon was that the Second Thermodynamic Law, i.e. the entropy of a system will always stay the same or increase, but never decrease, wasn’t really a Law (i.e. there was no counter proof case). Rather, he believed it was just statistically unlikely that particles could rearrange themselves on a configuration that would reduce entropy.</p><p>To try to make a point on this, he defined two rooms of gas, with a wall with a trap door to open it. A “Demon” would then have control over this opening and would only allow the fast particles to pass to one direction, and the slow particles to pass to the other direction. In that way, the entropy of the system would reduce, while no work was done to the system, defying the second law of thermodynamics.</p><h3>Szilard Engine</h3><blockquote><strong><em>Article</em></strong><em>: On the Decrease of Entropy in a Thermodynamics System by the Intervention of Intelligent Beings</em></blockquote><p>Szilard casted light over this apparent paradox, by pointing out <strong>the act of obtaining “information” by a measurement.</strong> Specifically, when the Demon would perform a measurement of each particle to know if it was fast or slow (so that the demon could decide wether to open the trap door or not to a particular particle), this process would take energy, even if performed inside the brain of the demon. Therefore the Second Law of Thermodynamics was not being violated.</p><p>Notice it is from this work that a connection between entropy and information is first put in place. Specifically, answering if a particle is “fast or slow” can be translated to a “yes/no” answer to wether the particle is fast. A “yes/no” question in turn can be represented by a a single bit of information (digits 0 or 1). Bits in turn are tied to Information Theory, and computer science as memory for computers.</p><p>Following from Szilard work, Launderer and Bennett pioneered the <strong>Physics of Information.</strong></p><blockquote>“Information itself is a physical property”. <strong>Article:</strong> The physical nature of information — Rolf Landauer</blockquote><blockquote>Article: The Fundamental Physical Limits of Computation — Charles H. Bennett and Rolf Landauer</blockquote><p>Biology Reference: Life’s Demon: information and order in biology (see reference on the video).</p><h3>References</h3><p>[Thermodynamics EdX]</p><p><a href="https://www.edx.org/course/thermodynamics-iitbombayx-me209-1x-0">Thermodynamics</a></p><p>[Complexity Explorer]</p><p><a href="http://www.complexityexplorer.org/online-courses/27-introduction-to-complexity-summer-2015/segments/3016">Complexity Explorer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb31757d095a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ink CSS for knitR]]></title>
            <link>https://medium.com/@carlosviansi/ink-css-for-knitr-1684baee40ca?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/1684baee40ca</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[knitr]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Sat, 30 May 2015 08:11:27 GMT</pubDate>
            <atom:updated>2015-05-30T08:20:29.238Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pIu7UUl2669NAvCOsx2aqw.png" /><figcaption>Ink CSS for knitR. <a href="http://rpubs.com/carlosandrade/uhsmp2rInk">Click here to see a demo on RPubs.</a> You can also see the same report on the default template <a href="http://rpubs.com/carlosandrade/uhsmp2r">by clicking here</a>.</figcaption></figure><h2>InkCSS for knitR</h2><h4>A better way to explore and tell your data story.</h4><p>Over my days as a data scientist research assistant, I have continuously needed to crunch data, generate various plots, and draw conclusions to report <strong>to myself</strong> and <strong>to other people</strong>. This work <strong>also involved seeing others analysis reports</strong>.<em> And there lied my problem.</em></p><p>Being exposed to concepts from Edward Tufte, Clevand, Alberto Cairo and Don Normann about design and visualization, some reports were just too hard to deal with. If you don’t believe me, here is an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yDOEUsoTepaRa5Wjzg9M_A.png" /><figcaption>This data analysis “report” was loaded on a big screen for the group appreciation and discussion. The code is embedded with annotations (rather than the other way around!) explaining how the analysis was done. The code was cut and paste from a console to the group’s Google page journal.</figcaption></figure><p>Here are some problems with this kind of report:</p><ul><li>It is hard to tell apart text from code.</li><li>It is hard to navigate around (there are no headings or menu).</li><li>A consequence of lack of headings is the inability of a reader to see the main chunks (or method) used to simplify his / her mental model of the report.</li><li>Can only be reused after some (considerable) work (given output and commands are mixed someone will have to manually distinguish them).</li></ul><p>I will go ahead and remember this is not the worst case scenario, most of the times no code, notes or even memory of what was done can be found. Then again, maybe you are better off without any if your advisor tells you to “use as a reference” these reports.</p><h3>knitR to the rescue</h3><p>If one of these post tags lured you here, then you are most likely familiar with knitR and RStudio (if not, I urge you to Google those terms and learn more about them! ☺).</p><p>knitR offers a solution to these problems which, in my opinion, has an extremely low learning cost to someone who can code in R. Here is what I mean:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-26hfk2u1YwfPFhVbXVO4Q.png" /><figcaption>Default knitR template. The table of contents is generated automatically by the usage of Markdown on the raw source file. A reader can quickly click on a given section of the TOC to “jump” to the section.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OAT1eUAjSYVCzsIDpgnR_g.png" /><figcaption>Graphs and and code on knitrR default template provides a clear separation of analysis results and code snippets. The .Rmd source file also enable a programmer to quickly re-run the analysis on any computer.</figcaption></figure><p>You can also see this particular class project report on <a href="http://rpubs.com/carlosandrade/uhsmp2r">Rpubs by clicking here</a>. Although knitR using it’s default template address all the previous problems, a couple more came to surface as more iterations happened on my research project using it:</p><ul><li><strong>“Stuck on the top” table of contents</strong>: Over time, as the raw report grew in size (as opposed to the one you would submit to a conference or refer outsiders after a talk), navigating it became cumbersome. A continuous need to scroll up and down made finding and comparing information extremely frustrating. This was even more true when the group discussions had no story to be told (yet) and the flow of the analysis was ,at best, a guess.</li><li><strong>“Big picture is still big!”: </strong>For large reports, even with a TOC to understand what lies in there, by the time the group was done discussing one topic, the big picture was long gone.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*eRIc76b9bmSp7ZHarNxmJA.png" /><figcaption>Sometimes to meet weekly deadlines and in order to quickly navigate over all the information that may be required during a meeting without shuffling over many tabs, a report can grow considerable large!</figcaption></figure><ul><li><strong>“Which data set is this analysis associated to again?”: </strong>Another continuous problem was the headers for some of my group mates. Having only the size to distinguish from H1 to H5 header and some bold and italics here and there, it was easy to get confused which analysis was under what data set (or other hierarchy for the report) as shown below:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*Ch6Rw8_XuABwsbFC5j52bg.png" /><figcaption>H1 and H2 header on the default template. The difference is sometimes too subtle after reading these all day long.</figcaption></figure><h3>InkCSS for KnitR</h3><p>While looking for a new template for <a href="http://carlosandrade.co/">my personal website more portfolio-wise</a> (still on the making, current is the old one), I stumbled across a very crisp and clean <strong>Manual </strong>of one of them: <a href="http://www.alessioatzeni.com/blog/brushed-template/"><strong><em>B</em>rushed, by Alessio Atzeni</strong></a>. As you may expect, the manual was found by surprise within the zip folder. Here’s how it looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xz4AZI-xi4cqjiVZ2x8yMA.png" /><figcaption>The manual of a free website template found by luck after attempting to use the template on my local machine.</figcaption></figure><p>If you noted the first image of this post, you may have guessed right: <strong>This looks like a perfect template for knitR reports!</strong></p><p>Although motivated to use it right away, that introduced a new problem: I am a data analyst, not a front-end developer, and this falls off my knowledge domain. Furthermore, most documentation and questions found surrounding the usage of custom.css for knitR (at least in a way it would be easy to reuse with RStudio) were little to non-existent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*ccrtNPYlOCVHweDKhWAwWw.png" /><figcaption>Changing the default template requires only the top section of the report to add the line ‘css: ~/ink.css’. Other available solutions required additional pieces of R code.</figcaption></figure><p>Just have a quick look on RPubs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pHeznhGEY-bEbtnjvXF3mA.png" /><figcaption>RPubs main page. Since all reports uploaded to RPubs are public, it’s main page provide a continuous feed of the most recent uploaded reports.</figcaption></figure><p>And you will see nothing but the same, old and default template.</p><p>After <em>by accident realizing </em>the location of where the <em>index.html</em> file that uses the CSS in RStudio is located while generating a report and a lot of accidental tweaks with the manual page InkCSS was born:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pIu7UUl2669NAvCOsx2aqw.png" /><figcaption>InkCSS on it’s full glory. Notice this is the same project used to illustrate the problems on the start of “knitR to the rescue” section!</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rD-9__qS94X7EyVkDOTQug.png" /><figcaption>InkCSS menu will remain still to the left side regardless of the section you are, providing quickly navigation and a reminder of the big picture. This menu is an extension of <a href="http://rpubs.com/stevepowell99/floating-css">floating-css</a>.</figcaption></figure><p>You can open the same project now using InkCSS <a href="http://rpubs.com/carlosandrade/uhsmp2rInk">clicling here</a>. In summary, InkCSS addresses the previous issues with the following features:</p><ul><li>Implements <a href="http://rpubs.com/stevepowell99/floating-css">floating-css</a> to provide a permanent side-bar addressing the big picture and navigation problems. It also provides a lot of room on the left menu TOC for deep nested long-reports in preparation for self usage or group discussion.</li><li>Code snippets using <strong>zenburn</strong> code highlight blend with the menu colors, providing an even stronger distinction between results and code, while being aesthetically pleasing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6mIhVvoHqGvrUv_JMCuqpw.png" /></figure><ul><li>Since the left menu is large, there may be large empty spaced click areas between the analysis and a small heading (e.g. CODING below). In this case, hovering the cursor on the empty space will provide appropriate highlight to identify which header will be skipped upon clicking the empty region.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*v5ICYhdWMLIDKZNv5uk9cg.png" /></figure><ul><li><strong>Header colors</strong>: Colors with varying opacity provides an additional visual clue among the top headers.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*nJtpOob7tmGs_Vkb1HIsew.png" /></figure><ul><li>And last but not least, a centralized title for the report☺.</li></ul><h3>Conclusion</h3><p>I hope this post inspire you to come up with other templates for yourself, and to draw inspiration from and rethink the way reports are done, even within the knitR community. I also invite you to rethink how people react to your reports, and try to improve your everyday template in curious new ways.</p><p>I am still thinking how to improve InkCSS, please leave a comment if you have ideas! I am planning on making the code open-source under CC once I can clean the .css file properly.</p><p>p.s.: Did I mention how much it’s design call attention on RPubs?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U7102BPquSd-XJ34-zrZow.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1684baee40ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning About Machine Learning]]></title>
            <link>https://medium.com/@carlosviansi/learning-about-machine-learning-44c056b6d64c?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/44c056b6d64c</guid>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Sun, 03 Aug 2014 02:49:45 GMT</pubDate>
            <atom:updated>2014-08-03T03:02:08.563Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a re-blog (2010) of an <a href="http://web.archive.org/web/20110620134659/http://measuringmeasures.com/blog/2010/3/12/learning-about-machine-learning-2nd-ed.html">post only available in an weird to read and perhaps to vanish archive page</a> which has been pointed on <a href="http://stackoverflow.com/questions/3539657/what-are-the-prerequisites-for-studying-machine-learning">stackoverflow discussions</a>. If you are the owner of this post and have made it available elsewhere please do let me know :-)</p><p>This is not a pure copy-paste replica, I brushed it a bit so it fits Medium post style. I hope if you stumbled across this page you appreciate the rework as much as I do, given I constantly found myself visiting the original post.</p><h2>Original Post</h2><p>In my <a href="http://web.archive.org/web/20110620134659/http://measuringmeasures.com/blog/2010/1/15/learning-about-statistical-learning.html">previous post</a> on learning about statistical learning, I took a shot at creating a guide for those who want to learn about statistical learning. I received a lot of great feedback from friends in the field, the comments on my blog, and discussions around the web like the <a href="http://web.archive.org/web/20110620134659/http://news.ycombinator.com/item?id=1055042">hackernews thread</a>. This revised edition of the post takes these discussions and feedback into account.</p><p>I humbly submit this guide, upon material over which I do not have full mastery, in the case that it may be of use to others trying to find their way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*eCsYvWFG_03fi8Aj01CQNg.jpeg" /><figcaption>Student working in Statistics Machine Room. source: <a href="http://web.archive.org/web/20110620134659/http://www.flickr.com/photos/lselibrary/3926513542/">flickr</a></figcaption></figure><h3><strong>Fragmentation and Ambiguity: what’s in a name?</strong></h3><p>I renamed this second edition of my post to “Learning about Machine Learning” from “Learning about Statistical Learning.” What is the better name? I don’t know. Things are a bit fragmented. There are some nuances and some of the fragmentation makes sense. In my opinion, most of it doesn’t. You have “computational learning,” “computational statistics,” “statistical learning,” “machine learning” and “learning theory”. You have <a href="http://web.archive.org/web/20110620134659/http://anyall.org/blog/2008/12/statistics-vs-machine-learning-fight/">Statistics vs. Machine Learning</a>, the <a href="http://web.archive.org/web/20110620134659/http://www.inherentuncertainty.org/2010/01/algorithmstheory-culture-vs-machine.html">culture of algorithms vs. machine learning</a>, the <a href="http://web.archive.org/web/20110620134659/http://hunch.net/?p=318">stats handicap</a>, and <a href="http://web.archive.org/web/20110620134659/http://nlpers.blogspot.com/2010/01/machine-learners-apology.html">COMPUTATIONAL statistics vs. computational STATISTICS</a>.</p><p>It can be confusing. I bring this up to help you preempt the confusion. When you see these different terms, don’t worry about it, just focus on the topics you need to learn. In this guide I will try to help point out where you will see the same thing by different names.</p><h3><strong>Straight into the Deep End</strong></h3><p>In the previous list, I thought it would be good to recommend some lighter texts as introductions to topics like probability theory and machine learning. Based upon subsequent discussions and feedback, I’ve changed my view. Straight into the deep end is the way to go.</p><p>The caveat is that you need to be honest with yourself about when you encounter something that you do not understand and you are unable to work through despite trying your best. At this point you’ve found your way to some material that you do not have the foundation for. Put the book down, insert a bookmark. Close the book. Walk away. Go find a resource to build the foundation. Come back later. Repeat.</p><p>Don’t be afraid to admit you don’t understand something, that is the only way you are going to move forward and really understand the material. As <a href="http://web.archive.org/web/20110620134659/http://norvig.com/21-days.html">Peter Norvig</a> says about programming, be prepared to make a serious commitment of time and effort if you want to be a researcher.</p><h3><strong>The Books</strong></h3><p>I am making a number of changes which I mention along the way. The overall list is now structured by topic with inline notes and there are a number of new topics so that you can pick and choose based upon your areas of interest.</p><blockquote>Now on to the books.</blockquote><p>I’m moving the <strong>foundation in proofs and analysis </strong>up to be a the top priority, as well as adding some new references from the <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Analysis/lm/R1E106JMYLAHOX">analysis list</a>. You won’t get far if you are intimidated by Greek characters and unable to follow proofs. If you build an intuition for programming as proof writing, that will also bend your mind in the right direction. I’m including a link to one of the <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Problems-and-Puzzles/lm/RPP9VLMX10OYI">fun puzzle books</a> that I find helpful — logic puzzles can help get you into the flow of proofs. I’m not including a book on functional analysis here, but I have a few you can choose from on the <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Analysis/lm/R1E106JMYLAHOX">analysis list</a> if you are interested.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Introduction-Analysis-Maxwell-Rosenlicht/dp/0486650383/ref=cm_lmf_tit_2">Introduction to Analysis</a> — Maxwell Rosenlicht<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Mathematical-Analysis-Second-Tom-Apostol/dp/0201002884">Mathematical Analysis</a> — Apostol<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/How-Prove-Structured-Daniel-Velleman/dp/0521675995/ref=cm_lmf_tit_4">How to Prove It: A Structured Approach</a> — Daniel J. Velleman<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/How-Solve-Mathematical-Princeton-Science/dp/069111966X/">How to Solve it</a> — Gyorgy Polya<br><a href="http://web.archive.org/web/20110620134659/http://homepages.inf.ed.ac.uk/wadler/papers/frege/frege.pdf">Proofs are Programs [pdf]</a> — Philip Wadler<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Puzzles-Including-Mathematical-Features-Discovery/dp/0812921178">The Lady and the Tiger</a> — Raymond Smullyan</blockquote><p>You will be <strong>working with algorithms </strong>constantly, so work through a couple books from the <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Algorithms-and-Data-Structures/lm/R1M4CQKKSDTT35">algorithms and data structures list</a>. We are also in an age of rapidly increasing data volume, so I am also recommending a book to help you learn how decompose algorithms and work with Map Reduce style parallelism — a great read which is not published yet but you can get the freePDF online.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Introduction-Algorithms-Thomas-Cormen/dp/0072970545/ref=cm_lmf_tit_1">Introduction to Algorithms</a> by Charles E. Leiserson<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1848000693">The Algorithms Design Manual</a> — Steven Skiena<br><a href="http://web.archive.org/web/20110620134659/http://www.umiacs.umd.edu/~jimmylin/book.html">Data-Intensive Text Processing with MapReduce</a> — Jimmy Lin and Chris Dyer<br><a href="http://web.archive.org/web/20110620134659/http://www-cs.stanford.edu/~ang/papers/nips06-mapreducemulticore.pdf"><em>Map-Reduce </em>for Machine Learning on Multicore</a> — from a group in the Stanford CS department</blockquote><p>Linear algebra is critical — <strong>matrices are your friends. </strong>A lot of what you will see is represented and computed naturally in matrix notation — it is convenient for both theory and practice. You can find a variety of items in the <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Matrix-Fu/lm/R138PYL5XLBQK3">matrix fu list</a>, and you can also find <a href="http://web.archive.org/web/20110620134659/http://ocw.mit.edu/OcwWeb/Mathematics/18-06Spring-2005/VideoLectures/index.htm">Gilbert Strag’s lectures online</a> from MIT courseware. Matrix algebra, like algorithms, is a fundamental building block that you will work with on a daily basis.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Introduction-Linear-Algebra-Fourth-Gilbert/dp/0980232716/ref=cm_lmf_tit_3">Introduction to Linear Algebra, Fourth Edition</a> by Gilbert Strang<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Matrix-Algebra-Computations-Applications-Statistics/dp/1441924248">Matrix Algebra</a> — James Gentle</blockquote><p>In this second edition of my post, I’m adding a <strong>strong focus on linear models</strong>, as they are the foundation for a lot of models that you will encounter in machine learning. The books in the machine learning section below will help to put linear models in a broader setting. For example Chapter 2 of <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Elements-Statistical-Learning-Prediction-Statistics/dp/0387848576/ref=cm_lmf_tit_1">The Elements of Statistical Learning</a> has a nice overview section of least squares and nearest neighbors that helps you understand how many other techniques are based upon these two simple building blocks. Some sources also have nice discussions of the relationship between linear models and other approaches like neural networks. Starting with a strong foundation in linear models is wise. I had not done so properly when jumping into machine learning, and it has required a lot of backtracking.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Linear-Models-Chapman-Statistical-Science/dp/1584884258">Linear Models With R</a> — Julian James Faraway<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Introduction-Generalized-Chapman-Statistical-Science/dp/1584889500">An Introduction to Generalized Linear Models</a> — Annette J. Dobson<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Applied-Linear-Statistical-Models-Michael/dp/007310874X">Applied Linear Statistical Models</a> — John Neter</blockquote><p>There are many titles to choose from the<a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Probability/lm/R3U28NIW8WKGGW">probability list</a>, in order to <strong>build a base in probability theory. </strong>I’m also adding a reference for looking at probability from the Bayesian perspective. Probability can be very counter-intuitive. A great way to get around this is to train yourself to understand how we misinterpret things. Read one of theKaneman and Tversky books from my <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Finance-Economics-and-Trading/lm/RVS439HBBUCCD">finance, trading and economics list</a>, and work through some <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Problems-and-Puzzles/lm/RPP9VLMX10OYI">puzzle books</a>. Aside from psycological biases, a lot of the trickiness of empirical probability, especially conditional probabilities, is knowing what and how to count. If you want to explore counting, there a number of good books on the <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Probability/lm/R3U28NIW8WKGGW">probability list</a>.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Course-Probability-Theory-Revised-Second/dp/0121741516/ref=cm_lmf_tit_1">A Course in Probability Theory, Revised Edition, Second Edition</a> by Kai Lai Chung<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/First-Look-Rigorous-Probability-Theory/dp/9812703713/ref=cm_lmf_tit_3">First Look at Rigorous Probability Theory</a> by Jeffrey S. Rosenthal<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Probability-Theory-Logic-Science-Vol/dp/0521592712">Probability Theory: The Logic of Science</a> — E. T. Jaynes</blockquote><p>When you are reading to build <strong>an understanding of statistics on top of probability theory</strong>, move on to <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Statistics/lm/R3B0YGCMUYS5FT">the statistics list</a>. Statistics and inference are critical in machine learning, and it is also nice to understand how information theory fits into this picture. I haven’t included an information theory book here, but you can find a nice one that fits information theory with statistics on <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Statistics/lm/R3B0YGCMUYS5FT">the statistics list</a>. A lot of bits and pieces on statistics will have been covered already with probability theory and linear models. I’ve added a reference on<a href="http://web.archive.org/web/20110620134659/http://en.wikipedia.org/wiki/Statistical_hypothesis_testing">hypothesis testing</a> and multivariate statistics. I’ve also created a new new section below on robust and non-parametric statistics.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/All-Statistics-Statistical-Inference-Springer/dp/0387402721/ref=cm_lmf_tit_4">All of Statistics: A Concise Course in Statistical Inference (Springer Texts in Statistics</a>) by Larry Wasserman<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Testing-Statistical-Hypotheses-Springer-Statistics/dp/0387988645">Testing Statistical Hypothesis</a> — E. L. Lehmann<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Introduction-Bayesian-Statistics-William-Bolstad/dp/0470141158">Bayesian Statistics</a> — William M. Bolstad<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Modern-Multivariate-Statistical-Techniques-Classification/dp/0387781889">Modern Multivariate Statistical Techniques</a> — Alan Julian Izenman</blockquote><p><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Nonparametric-and-Robust-Statistics/lm/RDE0OWXR9E0M3"><strong>Non-parametric and robust statistics</strong></a><strong> </strong>are very neat topics. <a href="http://web.archive.org/web/20110620134659/http://en.wikipedia.org/wiki/Non-parametric_statistics">Non-parametric statistics</a>, can mean a few things but the core intuition is that the techniques do not assume a parametrized model for the probability distribution underlying the observed data. The notion of <a href="http://web.archive.org/web/20110620134659/http://en.wikipedia.org/wiki/Robust_statistics">robust statistics</a>stems from the desired to create statistical methods that are 1) resistant to the deviation from assumptions that empirical data often exhibit, and 2) tolerant to outliers as can be measured by the<a href="http://web.archive.org/web/20110620134659/http://en.wikipedia.org/wiki/Breakdown_point#Breakdown_point">breakdown point</a> — the point at which infinitely large outliers causes an estimator to yield an infinitely large result.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Robust-Statistics-Wiley-Probability/dp/0470129905">Robust Statistics</a> — Peter J. Huber<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Robust-Statistics-Influence-Functions-Probability/dp/0471735779">Robust Statistics: The Approach Based on Influence Functions</a> — Frank R. Hampel<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Regression-Outlier-Detection-Probability-Statistics/dp/0471488550">Robust Regression</a> — Peter J. Rousseeuw<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/All-Nonparametric-Statistics-Springer-Texts/dp/0387251456">All of Nonparametric Statistics</a> — Larry Wasserman</blockquote><p>I’ve replaced <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Machine-Learning-Mcgraw-Hill-International-Edit/dp/0071154671">Mitchell</a> with Bishop as a companion for <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Elements-Statistical-Learning-Prediction-Statistics/dp/0387848576">EoSL</a>. There are more to choose from on the<a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Machine-Learning/lm/R1ABXZWXRZBFD6">machine learning list</a> if you’re feeling motivated, but the three below give a very solid coverage of most of the core material. MacKay’s book is a great help with information theory if you haven’t yet gotten a solid understanding by the time you get to this point.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Pattern-Recognition-Learning-Information-Statistics/dp/0387310738">Pattern Recognition and Machine Learning</a> — Christopher Bishop<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Information-Theory-Inference-Learning-Algorithms/dp/0521642981">Information Theory, Inference and Leaning Algorithms</a> — David J. C. MacKay<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Elements-Statistical-Learning-Prediction-Statistics/dp/0387848576/ref=cm_lmf_tit_1">The Elements of Statistical Learning</a> by Robert Tibshirani, et al.</blockquote><p><strong>Understand features (</strong>or predictors, input variables, whatever you prefer to call them) <strong>and feature selection. </strong>Know your different strategies for dealing with continuous vs. discrete features. Feature selection (especially “embedded methods”), “sparsification” and “shrinkage methods” are all more or less different sides of the same hyper-coin. These ideas seek to eliminate the impact of irrelevant features upon your model by either marginalizing them out entirely or at least reducing their influence by reducing the parameter values for less relevant predictors. The two books below are focused on this problem, but you will also find a lot about regularized regression and other such techniques in other machine learning and statistical learning texts.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Feature-Extraction-Foundations-Applications-Fuzziness/dp/3540354875">Feature Extraction</a> — Isabelle Guyon, et al.<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Computational-Methods-Selection-Knowledge-Discovery/dp/1584888784">Computational Methods of Feature Selection</a> — Huan Liu, Hiroshi Motoda</blockquote><p><strong>Heuristic search and optimization </strong>are important topics, and fortunately I have found a great survey text that I am working my way through right now, which is included below from the <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/gp/richpub/listmania/fullview/R3NTKE6BHE8ZM9">heuristic search and optimization list</a>. I am also including a link that a friend pointed out to a book and lecture slides on convex optimization that you can get online.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Introduction-Stochastic-Search-Optimization-James/dp/0471330523/ref=cm_lmf_tit_1">Introduction to Stochastic Search and Optimization</a> by James C. Spall<br><a href="http://web.archive.org/web/20110620134659/http://stanford.edu/~boyd/cvxbook/">Convex Optimization (online</a>) — Stephen Boyd</blockquote><p>I’ve added two texts on <strong>Bayesian networks and graphical models. </strong><a href="http://web.archive.org/web/20110620134659/http://en.wikipedia.org/wiki/Graphical_model">Graphical models</a> are a cool family of techniques that let you model rich dependence structures between input variables by relying on a graph structure to represent conditional independence.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Modeling-Reasoning-Bayesian-Networks-Darwiche/dp/0521884381">Modeling and Reasoning with Bayesian Networks</a>- Adnan Darwiche<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Probabilistic-Graphical-Models-Principles-Computation/dp/0262013193">Probabilistic Graphical Models</a> — Nir Friedman</blockquote><p>For those <strong>working with text</strong>, I’ve added a books from the <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/IR-amp-NLP/lm/R115VJSJN8I6O0">IR and NLP list</a> — each of which I think represents the strongest single introductory text and reference. Information Retrieval (IR) and Natural Language Processing (NLP) are active areas of study that continue to become and increasingly important as a result of the internet. We use IR every day in search engines, but I think new forms of retrieval systems using statistical NLP are just around the corner.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Introduction-Information-Retrieval-Christopher-Manning/dp/0521865719">Introduction to Information Retrieval</a> — Christopher D. Manning (<a href="http://web.archive.org/web/20110620134659/http://nlp.stanford.edu/IR-book/information-retrieval-book.html">available free online</a>)<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Foundations-Statistical-Natural-Language-Processing/dp/0262133601">Foundations of Statistical Natural Language Processing</a> — Christopher D. Manning</blockquote><p><strong>Network Theory </strong>is another booming area accelerated by the Internet. Social, information and transportation networks all present very challenging and important problems. As especially ripe and tricky topic is the evolution of networks over time, or what is sometimes called “longitudinal analysis” by the social networks folks. Network theory is extremely useful for determining the importance of information or people within networks — for example, link analysis algorithms like <a href="http://web.archive.org/web/20110620134659/http://en.wikipedia.org/wiki/PageRank">PageRank</a> are based upon network theory. I’ve tossed in a book on graph theory in case you need to brush up — you will need it for networks.</p><blockquote><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Introductory-Graph-Theory-Gary-Chartrand/dp/0486247759">Introductory Graph Theory</a> — Gary Chartrand<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Social-Network-Analysis-Applications-Structural/dp/0521387078">Social Network Analysis</a> — Stanley Wasserman<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Statistical-Analysis-Network-Data-Statistics/dp/038788145X">Statistical Analysis of Network Data</a> — Eric D. Kolaczyk<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Network-Flows-Theory-Algorithms-Applications/dp/013617549X">Network Flows</a> — Ravindra K. Ahuja<br><a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Fundamentals-Queueing-Theory-Probability-Statistics/dp/047179127X">Fundamentals in Queuing Theory</a> — Donald Gross, et al.</blockquote><p>You will have to <strong>write a lot of code. </strong>Learning how to do good engineering work is more important than people realize. Learn some<a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Programming-Languages/lm/R195F9SN6I3YQH">programming languages</a>; python (<a href="http://web.archive.org/web/20110620134659/http://numpy.scipy.org/">numpy</a>/<a href="http://web.archive.org/web/20110620134659/http://www.scipy.org/">scipy</a>),<a href="http://web.archive.org/web/20110620134659/http://www.r-project.org/">R</a>, and at least one nice functional language (probably <a href="http://web.archive.org/web/20110620134659/http://www.haskell.org/">Haskell</a>, <a href="http://web.archive.org/web/20110620134659/http://clojure.org/">Clojure</a>, or <a href="http://web.archive.org/web/20110620134659/http://caml.inria.fr/">OCaml</a>). Learn how to <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Linux/lm/R3T3RC1MJSYIB9">work with Linux</a>. Learn about <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Software-Design-and-Testing/lm/R2ZN6IECG2SYUQ">design and testing</a>. Picking up some things on the <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Lean-Software/lm/R1LDWJMITIA39L">lean and agile</a> software development approach will likely help — I use it for research projects just as much as software. As I mentioned earlier, <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Algorithms-and-Data-Structures/lm/R1M4CQKKSDTT35">algorithms and data structures</a> are critical, and you may also want to learn about <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Parallel-and-Distributed-Systems/lm/RLHD9OSWUQ9CR">parallelism and distributed systems</a>if you work with large data sets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*Orjd1te6BJGOQSOwJl5Xgw.jpeg" /><figcaption>Students working with large data sets. source:<a href="http://web.archive.org/web/20110620134659/http://www.flickr.com/photos/lselibrary/3925726761/">flickr</a></figcaption></figure><p>All along the way, you might enjoy watching lectures. I have found <a href="http://web.archive.org/web/20110620134659/http://videolectures.net/">videolectures.net</a> and <a href="http://web.archive.org/web/20110620134659/http://video.google.com/">google video</a> to be the best overall sources. I am very happy to hear others if you have any recommendations.</p><p>You can find a lot more on my <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/gp/richpub/listmania/byauthor/A1JKHQFC9WMPN5">Amazon lists</a> and ever-expanding <a href="http://web.archive.org/web/20110620134659/http://www.amazon.com/Bradford-N-Cross/wishlist/3RB4REDIKE28I">Amazon wish lists</a>. If you are involved in theoretical or applied research in academia or industry, and you disagree with some of these book selections, have alternate recommendations, or you think I am missing some topics, I’d love to hear.</p><p>Lastly, here are <a href="http://web.archive.org/web/20110620134659/http://www.eecs.berkeley.edu/Faculty/Homepages/jordan.html">Michael Jordan</a>’s recommended readings via the <a href="http://web.archive.org/web/20110620134659/http://news.ycombinator.com/item?id=1055042">hackernews thread</a> that sprung up following my last post. He knows a lot more about machine learning than I do, so perhaps you should just ignore the above recommendations and listen to him.</p><p><strong>I personally think that everyone in machine learning should be (completely) familiar with essentially all of the material in the following intermediate-level statistics book:</strong></p><p><em>1.) Casella, G. and Berger, R.L. (2001). “Statistical Inference” Duxbury Press.</em></p><p>For a slightly more advanced book that’s quite clear on mathematical techniques, the following book is quite good:</p><p><em>2.) Ferguson, T. (1996). “A Course in Large Sample Theory” Chapman &amp; Hall/CRC.</em></p><p>You’ll need to learn something about asymptotics at some point, and a good starting place is:</p><p><em>3.) Lehmann, E. (2004). “Elements of Large-Sample Theory” Springer.</em></p><p>Those are all frequentist books. You should also read something Bayesian:</p><p><em>4.) Gelman, A. et al. (2003). “Bayesian Data Analysis” Chapman &amp; Hall/CRC.</em></p><p>and you should start to read about Bayesian computation:</p><p><em>5.) Robert, C. and Casella, G. (2005). “Monte Carlo Statistical Methods” Springer.</em></p><p>On the probability front, a good intermediate text is:</p><p><em>6.) Grimmett, G. and Stirzaker, D. (2001). “Probability and Random Processes” Oxford.</em></p><p>At a more advanced level, a very good text is the following:</p><p><em>7.) Pollard, D. (2001). “A User’s Guide to Measure Theoretic Probability” Cambridge.</em></p><p>The standard advanced textbook is Durrett, R. (2005). “Probability: Theory and Examples” Duxbury.</p><p>Machine learning research also reposes on optimization theory. A good starting book on linear optimization that will prepare you for convex optimization:</p><p><em>8.) Bertsimas, D. and Tsitsiklis, J. (1997). “Introduction to Linear Optimization” Athena.</em></p><p>And then you can graduate to:</p><p><em>9.) Boyd, S. and Vandenberghe, L. (2004). “Convex Optimization” Cambridge.</em></p><p>Getting a full understanding of algorithmic linear algebra is also important. At some point you should feel familiar with most of the material in</p><p><em>10.) Golub, G., and Van Loan, C. (1996). “Matrix Computations” Johns Hopkins.</em></p><p>It’s good to know some information theory. The classic is:</p><p><em>11.) Cover, T. and Thomas, J. “Elements of Information Theory” Wiley.</em></p><p>Finally, if you want to start to learn some more abstract math, you might want to start to learn some functional analysis (if you haven’t already). Functional analysis is essentially linear algebra in infinite dimensions, and it’s necessary for kernel methods, for nonparametric Bayesian methods, and for various other topics. Here’s a book that I find very readable:</p><p><em>12.) Kreyszig, E. (1989). “Introductory Functional Analysis with Applications” Wiley.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44c056b6d64c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A semester as a Federal University Professor]]></title>
            <link>https://medium.com/@carlosviansi/a-semester-as-a-federal-university-professor-da4bf06d824d?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/da4bf06d824d</guid>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Sun, 03 Aug 2014 00:37:30 GMT</pubDate>
            <atom:updated>2014-10-20T08:49:05.463Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3wp9DryTeBUTXrl39PmwZQ.png" /><figcaption>Teaching feels like writing drafts of your own ideas and finishing the fine grain details during execution. It&#39;s a strange trip to the student&#39;s mind all at the same time.</figcaption></figure><h4>How it feels like to be one when two days ago you were just a student</h4><p>Let me tell you a story that you may find it trivial from the outside, but it actually made a huge change on how I perceived the profession of being a professor.</p><p>Everything started soon after I graduated. It all happened too fast. I started my Computer Science majors in 2009. In 2012 I was abroad under a scholarship for a year, and by 2013 I graduated. Few days before the ceremony, I had noticed my school was lacking a professor and three classes of about 52 students were about to close.</p><p>Being waiting for the response for my PhD application , and having just taking away the title of undergraduate, I expected it to be just a job, like any other else I took as an undergraduate. In fact, as a T.A. and having given few classes, I wasn’t expecting nothing special at all. I was plain wrong.</p><p>Being the only person applying on the last and third call before the classes would close due to about two months of delay, I was formally accepted as a professor in the institution I graduated after a proper evaluation. Everything was normal until the first day of class.</p><p>Monday came, and with it the first day of two of the three classes. The first one, being of advanced undergraduates took me less than 6 hours to put something together and present it. Having most of the class being my former classmates, it felt rather just an individual presentation in class with some ghost professor sitting by.</p><p>It was with the class just afterwards that everything changed. Most students were on their second semester, and I knew little to no one among those 30 young people. I started explaining the material I created, and everything seem to be running fine until I realized there were many confused faces. I went with Plan B on the explanation and yet the expressions didn’t change.</p><p>Halfway the class I sat down and realized that wouldn’t work out. I wrote a summation symbol over the board and a student finally took the courage to say “What is this strange symbol?”. Other students then whispered “How doesn’t he knows that?!” and suddenly I felt that the tension towards what to do about the class was directed towards that single poor student. Few minutes later, the class would end, and the next day would be the second and last class day of the week. I had less than 24 hours to take action.</p><blockquote>What to do?</blockquote><p>At this point I realized that there would be no boss to tell me I was at wrong, or what should I do. More importantly, there would be no one to fix anything I did wrong along the line. At most, I would get more confused faces. I could teach something wrong, and perhaps most would just accept it.</p><blockquote><strong>I could just</strong> <strong>go on</strong>.</blockquote><p>Later I could just complain students nowadays don’t try hard enough, make about 90% fail and say “well, it is a core course of this curriculum and it is known to be hard”.</p><p>To make matters worse, the next day a veteran approached me and said:</p><blockquote>“they are different from you, you need to go easy on them”.</blockquote><p>While I had realized during class I was the one at fault, others wouldn’t. As a matter of fact, I was known by my classmates to be a complete bookworm during undergraduate years. Yes, I had the “smart kid label”. It made therefore “more sense” to blame the students.</p><p>I could do that, right? They are just “not smart enough”. It is just a job anyway. Really?</p><p>Rather than taking an arrogant point of view I found it quite challenging and rewarding how to turn those confused faces into positive replies. Took me the day without sleep to figure it out, and just like a bookworm happy to get a grade 10 in an exam, I felt completed happy when by the end of the next day of class I would ask constantly students to answer questions and they were able to. Even better, when by the end of the day one said</p><blockquote>“Thank you professor, I understood everything!”</blockquote><p>with a smile on her face.</p><p>Now let me ask you: How many professors along your way actually felt this way? I couldn’t use two hands to count mine. Well maybe they don’t express it, I don’t know. But I can assure you how many classes were a completely torture for myself with boring classes, classmates playing on their cellphones, and so on and none even realized or bothered to. I lost the count how many students drop out the course early on saying</p><blockquote>I can just try the next semester</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*77C-AoyZ42w5vAKVb4ly4w.jpeg" /><figcaption>In class section on my student days. To the left, an e-mail account. To the right, one student is paying attention and the second is looking **something else** on the computer screen. To be fair, I had a hard time dealing with this class as well.</figcaption></figure><p>(Public schools in Brazil are free, so well..).</p><p>I also got to see, enough times to be scared of, teachers and professors in my life saying</p><blockquote>Well by the end of the semester there will be just a few student’s left in class anyway.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VSO7ZAU3cyhgL3jVulzsmA.jpeg" /><figcaption>In class session on my student days. One of the class I usually heard many people complaining for not adding anything to them.</figcaption></figure><p>The bottom line message is: Being responsible for so many people education is something you only feel the weight once you really step in as one and one of your old professors look at you and say &#39;here are the class code and student emails&#39; and smiling at you leave all under your care.</p><p>You realize there is no one teaching you or telling you what is right or wrong to be taught. At most, there is a minimum set of words that very broadly compromise the bare bones of it but all else is entirely up to you.</p><blockquote>If you teach it wrong, it will remain wrong in their heads for who knows how long.</blockquote><p>In the end of the day, it is up to you, and only you, to shape whatever will lie within their heads on those few hours you spend in the classroom with them. It is a very heavy responsibility that as a student I never realized, and that perhaps even not many professors realize themselves.</p><p>Maybe a professor who started teaching formally after a PhD after his 30s thinks differently. Perhaps they also feel different about this. I don’t know. However, I must say that being a Substitute Professor being a 23 year old does broad your horizons in a rather unique way. Besides, I spent 4 years being taught how to fish. It doesn’t hurt teaching others the same by myself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da4bf06d824d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MOOCs and Books for Data Analysts ]]></title>
            <link>https://medium.com/@carlosviansi/moocs-and-books-for-data-analysts-1c0fa2922ba9?source=rss-173d7a1ac00------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c0fa2922ba9</guid>
            <dc:creator><![CDATA[Carlos Andrade]]></dc:creator>
            <pubDate>Tue, 25 Jun 2013 04:25:12 GMT</pubDate>
            <atom:updated>2013-06-25T17:41:34.420Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*fmniooYWvZmwYp_L.jpeg" /></figure><h4>A list of free online courses and books for those interested in dealing with Data</h4><p>This is a list of <strong>free </strong>online courses or books I have either browsed over or taken when I was an undergraduate. I sub-divided them below by field, and in case you are new to this whole thing of data analysis, I am also adding in a very informal description of where I have applied what I learned during my undergraduate days.</p><p>As with any MOOC, I did not become any expert in those sub-fields (in fact picking one to go really deeper was one of my interests on browsing all this). However, knowing those sub-fields and what they are all about helped me interface with researchers who knew what they were doing, and actually apply them for a real problem under their supervision, rather than just delegating the work of the research group to them.</p><p>Like one statistician once told me:</p><blockquote>You won’t be able to reason about a solution to a problem if you don’t even know what a given method, it’s constraints and it’s solution are.</blockquote><p>Although the following Data Science diagram does not include all the sub-fields below, you should probably find it a good overall picture for the courses below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/0*CbSpDr9U4lTffgev.png" /><figcaption><a href="http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram">Drew Conway Data Science Venn Diagram</a></figcaption></figure><h3>Machine Learning</h3><h4><a href="https://www.coursera.org/course/ml">Stanford Machine Learning</a></h4><p>This course gave me a practical start in machine learning using Octave/MATLAB. Since by this time I took this course I was in need of applying machine learning to my problems, it helped me considerably. It also had many practical advice, that is usually forgotten in more theoretical focused courses. It did not delve much in theoretical aspects.</p><h4><a href="http://work.caltech.edu/telecourse.html">Caltec Learning From Data</a></h4><p>Caltec course is by far the best available lectures to get you started in Machine Learning. The lecturer’s book is amazing in how easy you can understand very complicated concepts. It was for me a very good complement for <a href="https://www.coursera.org/course/ml">stanford machine learning course</a>.</p><h4><a href="https://www.coursera.org/course/neuralnets">Neural Networks for Machine Learning</a></h4><p>This courses covers much more deeply neural nets, which is just covered in a week on the other machine learning courses. In particular, they even cover the <a href="http://en.wikipedia.org/wiki/Recurrent_neural_network">RNN</a> types.</p><h3>Probabilistic Graphic Models</h3><h4><a href="https://www.coursera.org/course/pgm">Probabilistic Graphic Models</a></h4><p>PGM’s stand their own ground apart from Machine Learning. In this course introduction, it is mentioned that they are better over certain scenarios where machine learning is weak. I’ve seen mostly applied to computer vision, but I have not delve deep on this one (yet). I’ve taken this sub-field as an intersection between statistics and graph theory.</p><h3>Data Science</h3><h4><a href="https://www.coursera.org/course/datasci">Introduction to Data Science</a></h4><p>As the name suggest, this covers very broadly a bit of each part of what a data scientist does. I enjoyed the hands-on part of this course in respect to the sentiment analysis of twitter, map-reduce focus and so on. It was the ‘closest to reality’ assignments I’ve seen.</p><h4><a href="http://harvarddatascience.com/2013/05/05/harvard-stat-221-statistical-computing-and-visualization-all-lectures-online/">Harvard Data Science</a></h4><p>This was a class about data science given in Harvard that all the material was made available for free online. They suggest a framework called VMC (Visualizing, Modelling and Computing) and also include other contents I haven’t seen around the other courses in a broad data science view like HMM’s and Monte Carlo. They also support interactive visualization, which is a different take from other courses, since they use <a href="http://d3js.org/">D3,</a> instead of <a href="http://ggplot2.org/">ggplot</a> or <a href="http://cran.r-project.org/web/packages/lattice/index.html">lattice</a>.</p><h4><a href="http://datascienc.es/spring-2011-course/">Berkley Data Science</a></h4><p>Another course in data science that made available the slides (also note that a previous semester lecture slides is available). They enforce the use of R <a href="http://www.inside-r.org/packages/cran/plyr/docs/ddply">ddply</a> for pre-processing.</p><h4><a href="http://columbia-applied-data-science.github.io/categories.html">Columbia Data Science</a></h4><p>This was a course whose material was made available as well. The professor of this course is publishing <a href="http://www.amazon.com/Doing-Data-Science-Cathy-ONeil/dp/1449358659/ref=sr_1_1?ie=UTF8&amp;qid=1372140501&amp;sr=8-1&amp;keywords=doing+data+science">a book</a> entirely about data science which is only available as pre-order at this point. Their take in</p><blockquote>Data scientists should be able to craft production code</blockquote><p>is also very interesting, and they point to a collection of free video lectures and slides that aids in doing so: <a href="http://software-carpentry.org/">Software Carpentry</a>.</p><h3>Statistics</h3><h4><a href="https://www.coursera.org/course/compdata">Computing for Data Analysis</a></h4><p>This was the second course I’ve taken when trying to make sense of data analysis. It covers a lot of ground in <a href="http://en.wikipedia.org/wiki/R_%28programming_language%29">R</a>. The focus on statistical analysis is given in a <a href="https://www.coursera.org/course/dataanalysis">subsequent course</a>. Do notice I am not a bio-statistician, yet sometimes you learn even more by seeing how statistics is applied in other fields than trying to learn only from your own.</p><h4><a href="https://www.coursera.org/course/dataanalysis">Data Analysis</a></h4><p>Subsequent course from <a href="https://www.coursera.org/course/compdata">Computing for Data Analysis</a>. Here the focus is stronger in statistics, and weaker in R. I’ve benefit immensely from this course. Many data-sets are used from all across the internet that are not related to bio-statistics, which gives a very broad picture of what statistics can do for you.</p><h4><a href="https://www.coursera.org/course/stats1">Statistics One</a></h4><p>This was the very first course I’ve taken in statistics, before even considering data analysis in general. The majority of examples were focused in psychology, but easy to understand. It has a strong focus on validity. Even being targeted for a broad audience, I was able to notice many common slips people do in applying statistics thanks to this course. It is focused more in theory than in applying statistics, but it provides R scripts for each week’s theoretical material.</p><h3>Latent Variable Models</h3><h4><a href="https://www.coursera.org/course/sem">Latent Variable Models</a></h4><p>This is a very advanced course that is yet to start and is within Statistics, but I am adding here separately for reference. Latent variable models covers a lot of models from statistics, not to mention the idea of hidden variables which I have only seem briefly in probabilistic graphic models. I have been learning mostly about this subject from a local research group of my university.</p><h3>Text Mining</h3><h4><a href="https://www.coursera.org/course/nlp">Stanford Natural Language Processing</a></h4><p>Not much to say about this course, yet to explore text mining. In most cases I have only come across the need of performing exploratory clustering of words by frequency and what is also called context mining.</p><h4><a href="https://www.coursera.org/course/nlangp">Toronto Natural Language Processing</a></h4><p>Not much to say about this course, other than it seem harder than the Stanford version on Coursera.</p><h3>Social Network Analysis</h3><h4><a href="https://www.coursera.org/course/sna">Michigan Social Networks Analysis</a></h4><p>A good introduction for SNA, helped me get some idea of what it is about. I have mostly used Gelphi after taken this course for analyzing some patterns in of my projects in respect to education.</p><h4><a href="https://www.coursera.org/course/networksonline">Stanford Social Network Analysis</a></h4><p>This is a more advanced version of SNA, although it covers the basics as well.</p><h3>Visualization</h3><p>Visualization are very rare courses to be found. Those two were found by accident, and their focus are not quite of exploratory data analysis, but the creation of infograms using a myriad of tools.</p><h4><a href="http://ivmooc.cns.iu.edu/">Information Visualization</a></h4><p>I didn’t go much further in this course, but you should expect to learn about how those infographics are made in this course among other things.</p><h4><a href="https://www.youtube.com/watch?v=QinBJpjsm6E">Data Visualization</a></h4><p>This course was offered once in a moodle environment that is no longer available. I found the video lectures on youtube. The link leads to the first video, but the other lectures should probably be available on youtube.</p><h3>Databases</h3><h4><a href="https://class2go.stanford.edu/db/Winter2013/preview/">Stanford Database</a></h4><p>Covers a lot of ground on all kinds of databases. I took parts of it as complement for my undergraduate databases classes.</p><h4><a href="https://education.10gen.com/courses/10gen/M101P/2013_June/about">10gen MongoDB</a></h4><p>Covers pretty much everything I could need in handling JSON data and structuring it in a MongoDB database. Given that a lot of websites are providing data easily as JSON files, knowing how to store them in a MongoDB makes life much easier in exploring and preparing .csvs for tools like R and Octave.</p><h4><a href="http://www.neo4j.org/learn/books">Graph Database</a></h4><p>For the sake of including all databases I’ve used for data analysis, this free available book contains a lot of ground for Neo4j graph databases. They’ve been to me particularly useful for social network analysis.</p><p>There are courses being made available every day from different platforms, and perhaps it can be the case I missed one or more (feel free to point them out and I will add them here). A full list of MOOC courses I usually check is available on the link below.</p><p><a href="http://www.mooc-list.com/">http://www.mooc-list.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c0fa2922ba9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>