<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vít Gabrhel on Medium]]></title>
        <description><![CDATA[Stories by Vít Gabrhel on Medium]]></description>
        <link>https://medium.com/@vitgabrhel?source=rss-6e9400412127------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*d8c-Y_lVyQ172ew3ehKo6A.png</url>
            <title>Stories by Vít Gabrhel on Medium</title>
            <link>https://medium.com/@vitgabrhel?source=rss-6e9400412127------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:49:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vitgabrhel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[See you on The Dark Side of the Moon]]></title>
            <link>https://medium.com/analytics-vidhya/see-you-on-the-dark-side-of-the-moon-a61497819813?source=rss-6e9400412127------2</link>
            <guid isPermaLink="false">https://medium.com/p/a61497819813</guid>
            <category><![CDATA[spotify]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pink-floyd]]></category>
            <dc:creator><![CDATA[Vít Gabrhel]]></dc:creator>
            <pubDate>Sun, 05 Mar 2023 15:55:41 GMT</pubDate>
            <atom:updated>2023-03-05T16:08:29.688Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6vHoioT7DKUi55O5.png" /><figcaption><em>“Pink Floyd, The Dark Side of the Moon, devilcore cyberpunk vaporwave pixel diffusion, unholy geometry, radiating –stylize 1250”</em></figcaption></figure><h3>Introduction or Any Madness You Like</h3><p>On <strong>March 1st 1973</strong>, exactly <strong>50 years ago</strong>, <em>The Dark Side of the Moon</em> by <em>Pink Floyd</em> was released. It is hard to overstate the impact this album had on music and art in general.</p><p>For me, it is one of the key pieces of my adolescent soundscape. I remember <em>On The Run</em> blasting from speakers in a classmate’s car, accompanied by his verbalized doubts whether I have gone insane. The melancholic sentiment associated with the human perception of the passage of <em>Time</em> whenever the cacophony of the clocks starts. The unnerving confrontation with death and cessation of consciousness in <em>The Great Gig in The Sky</em>.</p><p>So, I decided to prepare a small tribute upon realizing the anniversary. And being the data nerd that I am, I used this opportunity to explore the data behind the album. To be more specific, the data available at Spotify. Also, I employed DALL-E to create several visual pieces based on the original artwork since I am fond of generative art. So, you may find them throughout the text along with the command prompts used in the process. Finally, you may also find trivia like breadcrumbs scattered here and there.</p><p>Enjoy!</p><h3>Packaging Unlike Hipgnosis</h3><p>The name of this section refers to the fact that the original artwork for the album was <a href="https://en.wikipedia.org/wiki/The_Dark_Side_of_the_Moon">created</a> by Hipgnosis, a design studio founded by Storm Thorgerson and Aubrey Powell. At the same time, it is a wordplay on the fact that the text is written using Python packages like <a href="https://plotly.com/python/">plotly</a> or <a href="https://spotipy.readthedocs.io/en/2.22.1/">spotipy</a>.</p><p>At any rate, the album’s artwork <a href="https://en.wikipedia.org/wiki/The_Dark_Side_of_the_Moon#Packaging">depicts</a> light refracting from a triangular dispersive prism. A spicy detail is that this physical phenomenon has found itself lost in the <a href="https://twitter.com/shane_elliott/status/1616606801323073536/photo/2">culture wars</a>.</p><pre># Load Python packages<br>import plotly.graph_objs as go<br>import plotly.express as px<br>import numpy as np<br>import pandas as pd<br><br># Credentials import <br>from credentials import client_id, client_secret<br><br># Spotify connection<br>import spotipy<br>from spotipy.oauth2 import SpotifyClientCredentials<br>from spotipy.oauth2 import SpotifyOAuth</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mcayw3FWP-KIzzDlItxMcQ.png" /><figcaption>“<em>Create an abstract, geometric image inspired by the Suprematism movement and Kazimir Malevich’s work from 1913. The image should incorporate elements that are reminiscent of Pink Floyd’s album ‘The Dark Side of the Moon’, such as a rainbow spectrum or a prism. The image should be predominantly dark in color, with shades of black, blue, and purple. The image should also include a prominent circular</em>”</figcaption></figure><h3>Spotify or Everything under the sun is in tune</h3><p>The first step is to create a developer account at Spotify and obtain the credentials. Then, one can connect to the Spotify API using the <a href="https://spotipy.readthedocs.io/en/2.22.1/">spotipy</a> package. There are several articles covering the process of connecting to the API, so check (e.g.) <a href="https://python.plainenglish.io/building-a-cli-spotify-playlist-generator-using-python-spotipy-3b32b63a25da"><em>Building a CLI Spotify playlist generator using Python &amp; Spotipy</em></a> out if you want to know more.</p><p>After connecting to the API, I extracted information about Pink Floyd as an artist (like &#39;genres&#39;), but also about their albums (like &#39;release_date&#39;). Finally, I downloaded information about individual songs (like &#39;popularity&#39;).</p><p>This approach allows me to focus on the album in detail, but also in comparison to other albums by Pink Floyd. But let us start with Pink Floyd as an artist.</p><pre># Connect to spotipy and return data on the album<br># Replace the values below with your own credentials<br>client_id = client_id<br>client_secret = client_secret<br><br># Authenticate with the Spotify API using your credentials<br>client_credentials_manager = SpotifyClientCredentials(client_id, client_secret)<br>sp = spotipy.Spotify(client_credentials_manager=client_credentials_manager)<br><br># Search for the &quot;Dark Side of the Moon&quot; album by Pink Floyd<br>results = sp.search(q=&#39;artist:Pink Floyd album:&quot;Dark Side of the Moon&quot;&#39;, type=&#39;album&#39;)<br><br># Get the album&#39;s ID<br>album_id = results[&#39;albums&#39;][&#39;items&#39;][0][&#39;id&#39;]<br><br># Get the audio features for each track on the album<br>tracks = sp.album_tracks(album_id)[&#39;items&#39;]<br><br>features = [sp.audio_features(track[&#39;id&#39;])[0] for track in tracks]<br><br>album_tracks = sp.album_tracks(album_id)[&#39;items&#39;]<br>album_track_ids = [track[&#39;id&#39;] for track in album_tracks]<br><br># Get the artist&#39;s ID<br>results = sp.search(q=&#39;artist:Pink Floyd&#39;, type=&#39;artist&#39;)<br>artist_id = results[&#39;artists&#39;][&#39;items&#39;][0][&#39;id&#39;]</pre><h3>Popular songs or It’s a hit!</h3><p>It is no secret that Pink Floyd albums are mainly conceptual. One of the consequences is that listeners are invited to experience songs in the order suggested by the band — from the beginning to the end, as a whole. The Dark Side of the Moon is definitely in this scope.</p><p>According to the popularity of the songs by Pink Floyd on Spotify, the listeners are keen to listen to multiple songs from the album. However, not necessarily in the order suggested by the band.</p><p>All in all, half of the Top 10 songs of Pink Floyd on Spotify are from The Dark Side of the Moon, namely <em>Money</em>, <em>Time</em>, <em>Brain Damage</em>, <em>Breathe</em>, and <em>The Great Gig in the Sky</em>.</p><pre># Show the top 10 songs of Pink Floyd per albums on Spotify<br><br># Get the top tracks for a given artist<br>results = sp.artist_top_tracks(artist_id)<br><br># Create a list of dictionaries, each containing track name and album name<br>tracks = []<br>for track in results[&#39;tracks&#39;]:<br>    tracks.append({&#39;Track Name&#39;: track[&#39;name&#39;], &#39;Album Name&#39;: track[&#39;album&#39;][&#39;name&#39;]})<br><br># Create a dataframe from the list of track and album dictionaries<br>df = pd.DataFrame(tracks)<br><br># Create a dataframe with columns &quot;Album Name&quot; and &quot;Count&quot; based on the number of occurences of albums in the top 10 songs<br>df_count = df.groupby(&#39;Album Name&#39;).size().reset_index(name=&#39;Count&#39;) <br># | label: fig-albums<br># | fig-cap: &quot;The top 10 albums of Pink Floyd on Spotify&quot;<br><br># Create a bar plot of the top 10 albums<br><br># Define the color scale<br>colors = [&#39;rgb(0, 0, 255)&#39;, &#39;rgb(255, 127, 0)&#39;, &#39;rgb(255, 0, 0)&#39;, &#39;rgb(0, 255, 255)&#39;, &#39;rgb(255, 0, 0)&#39;]<br><br>fig = px.bar(df_count, x=&#39;Album Name&#39;, y=&#39;Count&#39;,<br>             hover_data=[&#39;Album Name&#39;, &#39;Count&#39;],<br>             color=&#39;Album Name&#39;,<br>             color_discrete_sequence=colors,<br>             labels={&#39;Count&#39;: &#39;Number of times in Top 10&#39;}, height=400,<br>             template=&quot;simple_white&quot;)<br><br>fig.update_traces(showlegend=False)<br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*osW858duQW57tDrvMUvbiw.png" /><figcaption>The top 10 songs of Pink Floyd per albums on Spotify</figcaption></figure><h3>Album features or All that you taste</h3><p>There are several properties or <a href="https://developer.spotify.com/documentation/web-api/reference/#/operations/get-several-audio-features">audio features</a> of the songs on the album that can be used to describe the album as a whole. For example, there are:</p><ul><li>danceability - how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity.</li><li>acousticness - a confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.</li><li>instrumentalness - predicts whether a track contains no vocals. &quot;Ooh&quot; and &quot;aah&quot; sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly &quot;vocal&quot;. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.</li></ul><p>Let’s calculate the average of danceability, accousticness, and instrumentalness for each album and see how each of them compares to the other selected albums! But there is a disclaimer - I decided to plot only 6 selected ones out of a total of 20. The reason is that including all would - in my view - defeat the purpose. Anyway, let&#39;s see how The Dark Side of The Moon fares!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*5B6EkzyIzhXnZ0WCgQrTqA.png" /><figcaption>The average values of selected audio properties per album</figcaption></figure><p>So, it seems that the Dark Side of the Moon is the most instrumental album of the selected ones. It is also perhaps the least <strong>danceable</strong> album within the selection. So, perhaps the best way how to enjoy is to warm our bones beside the fire?</p><p>But what about the <strong>popularity</strong> of the songs on the albums? And what is it in the first place? Spotify defines it as follows:</p><p>Now, let’s see!</p><pre># Extract the popularity scores of the Pink Floyd albums’ tracks and plot them<br><br># Create an empty list to store the popularity data<br>popularity_data_list = []<br><br># Loop through the album IDs<br>for album_id in album_ids:<br>    # Get the album name<br>    album_name = sp.album(album_id)[&#39;name&#39;]<br><br>    # Get the track IDs for the album<br>    tracks = sp.album_tracks(album_id)<br>    track_ids = [track[&#39;id&#39;] for track in tracks[&#39;items&#39;]]<br><br>    # Get the popularity values for each track<br>    popularity_values = sp.tracks(track_ids)[&#39;tracks&#39;]<br><br>    # Append the popularity data to the list<br>    popularity_data_list.append(pd.DataFrame([{&#39;Album&#39;: album_name, &#39;Song&#39;: track[&#39;name&#39;], &#39;Popularity&#39;: track[&#39;popularity&#39;]} for track in popularity_values]))<br><br># Concatenate all DataFrames in the list<br>popularity_data = pd.concat(popularity_data_list, ignore_index=True)<br><br># Define the regex pattern to match substrings inside parentheses or square brackets<br>pattern = r&quot;[\(\[].*?[\)\]]&quot;<br><br># Use the str.replace() method to remove the substrings and trim whitespaces<br>popularity_data[&#39;Album&#39;] = popularity_data[&#39;Album&#39;].str.replace(pattern, &quot;&quot;, regex=True).str.strip()<br><br># Create a sample of 6 albums with all their songs - The Dark Side of the Moon, The Wall, Wish You Were Here, Animals, The Final Cut and The Division Bell<br>df_box_plot = popularity_data[popularity_data[&#39;Album&#39;].isin([&#39;The Dark Side Of The Moon&#39;, &#39;The Wall&#39;, &#39;Wish You Were Here&#39;, &#39;Animals&#39;, &#39;The Final Cut&#39;, &#39;The Division Bell&#39;])]<br><br># Keep only album and popularity columns<br>df_box_plot = df_box_plot[[&#39;Album&#39;, &#39;Popularity&#39;, &#39;Song&#39;]]<br><br>colors = [&#39;blue&#39;, &#39;red&#39;, &#39;green&#39;, &#39;orange&#39;, &#39;purple&#39;, &#39;yellow&#39;]<br><br># Plotly boxplot on popularity_data - grouped by Album<br>fig = px.box(df_box_plot, x=&#39;Album&#39;, y=&#39;Popularity&#39;, title=&#39;Popularity of songs by Pink Floyd&#39;,<br>             template=&quot;simple_white&quot;, <br>             color=&#39;Song&#39;, <br>             color_discrete_sequence=colors,<br>             points=&quot;all&quot;)<br><br># Hide legend<br>fig.update_layout(showlegend=False)<br>                  <br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*IY8WJdV6VEbRISnurISU4g.png" /></figure><p>Overall, The Dark Side of the Moon is the most popular album of the selected ones. In addition, the popularity scores of the individual songs are quite similar. This means that the album is relatively homogenous popularity-wise. Or, in other words, there seem to be no quite unpopular songs on the album.</p><h3>Tribute Playlist or Different tunes</h3><p>Now, Spotify API has the capacity to recommend songs based on a given song (or a list of songs). So, let’s try it out on <em>The Dark Side of the Moon</em>!</p><pre># Create a playlist of songs that are similar to the songs on the album The Dark Side of the Moon<br><br># Return album_id of The Dark Side of the Moon<br>album_id = sp.search(q=&#39;The Dark Side of the Moon&#39;, type=&#39;album&#39;)[&#39;albums&#39;][&#39;items&#39;][0][&#39;id&#39;]<br><br># Get the track IDs for the album<br>tracks = sp.album_tracks(album_id)<br>track_ids = [track[&#39;id&#39;] for track in tracks[&#39;items&#39;]]<br><br># Get song recommendations based on track_ids of The Dark Side of the Moon<br>recommendations = sp.recommendations(seed_tracks=[track_ids[1], track_ids[3], track_ids[5], track_ids[6], track_ids[8]], limit=50)<br><br># Create a list of dictionaries, each containing track name, artist name, album name, release date, and track uri<br>tracks = []<br>for track in recommendations[&#39;tracks&#39;]:<br>    tracks.append({&#39;Track Name&#39;: track[&#39;name&#39;], &#39;Artist Name&#39;: track[&#39;artists&#39;][0][&#39;name&#39;], &#39;Album Name&#39;: track[&#39;album&#39;][&#39;name&#39;], &#39;Release Date&#39;: track[&#39;album&#39;][&#39;release_date&#39;], &#39;Track URI&#39;: track[&#39;uri&#39;]})<br><br># Create a dataframe from the list of track dictionaries<br>df = pd.DataFrame(tracks)<br><br># Parse year from release date and create a new column &quot;Release Year&quot;<br>df[&#39;Release Year&#39;] = df[&#39;Release Date&#39;].str[:4]<br><br># Create a dataframe with columns &quot;Release Date&quot; and &quot;Count&quot; based on the number of occurences of release dates<br>df_count = df.groupby(&#39;Release Year&#39;).size().reset_index(name=&#39;Count&#39;)<br><br># Create a Plotly table object<br>table = go.Table(<br>    header=dict(values=list(df[[&#39;Artist Name&#39;, &#39;Track Name&#39;]].columns),<br>                fill_color=&#39;#18bc9c&#39;,<br>                align=&#39;left&#39;),<br>    cells=dict(values=[df[col] for col in df[[&#39;Artist Name&#39;, &#39;Track Name&#39;]].columns],<br>               fill_color=&#39;whitesmoke&#39;,<br>               align=&#39;left&#39;))<br><br># Define the layout for the table - 100% width and height<br>layout = go.Layout(<br>    width=800,<br>    height=1000,<br>    autosize=True<br>)<br><br># Combine the table and layout objects into a Figure object<br>fig = go.Figure(data=[table], layout=layout)<br><br># Display the table in the notebook<br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*6jw7F2Nz1-maV6nVhV0EtQ.png" /><figcaption>Tribute Playlist</figcaption></figure><h3>Inspired or inspiring? Or both?</h3><p>The generated list of songs is quite interesting. It contains songs across decades and genres. There are songs from the albums preceding as well as following <em>The Dark Side of the Moon</em>.</p><p>Some might consider thinking about this as an “inspirational lineage”. So, let’s take a look at the number of songs from the list released per year.</p><pre># Create plotly bar chart of the number of songs released per year. Add color one bar.<br>fig = px.bar(df_count, x=&#39;Release Year&#39;, y=&#39;Count&#39;, title=&#39;Number of songs released per year&#39;,<br>             template=&quot;simple_white&quot;, color_discrete_sequence=[&#39;blue&#39;, &#39;red&#39;, &#39;green&#39;, &#39;orange&#39;, &#39;purple&#39;, &#39;yellow&#39;],<br>             color=&#39;Release Year&#39;)<br><br># Update the layout<br>fig.update_layout(<br>    title_text=&#39;Number of songs released per year&#39;,<br>    xaxis_title=&quot;Release Year&quot;,<br>    yaxis_title=&quot;Number of Songs&quot;,<br>    xaxis_tickangle=-45,<br>    xaxis_tickfont_size=14,<br>    yaxis=dict(<br>        tickmode=&#39;linear&#39;,<br>        tick0=0,<br>        dtick=5<br>    ),<br>    bargap=0.15,<br>    bargroupgap=0.1,<br>    template=&quot;simple_white&quot;<br>)<br><br># Hide legend<br>fig.update_layout(showlegend=False)<br><br># Display the bar chart in the notebook<br>fig.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*rgGJF6YvitByY_Be79iDGQ.png" /></figure><p>All right, Spotify recommends quite evenly across decades, but the distribution of values is denser towards the 70s.</p><p>Another neat capacity of the API is the option to create a playlist programmatically. Let’s try it out! A small tip — in case you want to run your own “app”, don’t forget to set up the redirect_uri in the Spotify Developer Dashboard.</p><pre># Programaticaly create a playlist named See you on The Dark Side of the Moon<br><br># Get the client ID and client secret from the environment variables<br>sp_oauth = SpotifyOAuth(<br>    client_id=client_id,<br>    client_secret=client_secret,<br>    redirect_uri=&quot;http://localhost:8000/callback&quot;,<br>    scope=&quot;playlist-modify-public&quot;<br>)<br><br># Token generation: # access_token = sp_oauth.get_access_token()<br>access_token = sp_oauth.get_cached_token()<br><br>sp = spotipy.Spotify(auth=access_token[&quot;access_token&quot;])<br><br># Get the user&#39;s Spotify username<br>username = sp.current_user()[&#39;id&#39;]<br><br># Create a new playlist<br>playlist = sp.user_playlist_create(username, &#39;See you on The Dark Side of the Moon&#39;)<br><br># Get the playlist ID<br>playlist_id = playlist[&#39;id&#39;]<br><br># Get the track URIs<br>track_uris = df[&#39;Track URI&#39;].tolist()<br><br># Add the tracks to the playlist<br>sp.user_playlist_add_tracks(username, playlist_id, track_uris)<br><br># Add the description of the playlist<br>sp.user_playlist_change_details(username, playlist_id, description=&#39;A playlist of songs that sound like &quot;Time&quot; by Pink Floyd&#39;)</pre><p>So, the playlist should be ready. Let’s check it out <a href="https://open.spotify.com/playlist/0Ruho36fiZPFQy7MJRUmhU?si=170a96b313be4870">here</a>.</p><h3>Honorary mentions or All you create</h3><p>Before I wrap up this post, I would like to turn your attention to some of The Dark Side of the Moon-related projects.</p><p>First of all, Hans Zimmer used Eclipse in the trailer for Dune. It is a nice touch on the megalomaniac <a href="https://en.wikipedia.org/wiki/Jodorowsky%27s_Dune">project</a> by Alejandro Jodorowsky as Pink Floyd was supposedly on his short-list for the movie soundtrack. Full circle.</p><ul><li><a href="https://en.wikipedia.org/wiki/Jodorowsky%27s_Dune">Wikipedia</a></li><li><a href="https://open.spotify.com/track/3bHH1y8RyqfQGMYiixQJe8?si=d71b5d58aeb8446c">Spotify</a></li></ul><p>Secondly, if you like rock music, don’t miss out on The Doom Side of The Moon as it is a rewarding take on the album from a more metal-ish perspective.</p><ul><li><a href="https://metalinjection.net/reviews/doom-side-of-the-moon-doom-side-of-the-moon">Article</a></li><li><a href="https://open.spotify.com/album/3i32lYJBZv3YOR1oS16U6S?si=tWSQUkTgT4aJqvE8qklFTg">Spotify</a></li></ul><p>Finaly, Darkside or Tom Stoppard’s excellent mashup of the album and topics such as the trolley problem, the tragedy of the commons, and the prisoner’s dilemma. Remember Iwan Rheon as Ramsay Bolton in Game of Thrones? He is one of the characters and let’s just say that the wheels are not turning well for him.</p><ul><li><a href="https://en.wikipedia.org/wiki/Darkside_(radio_play)">Wikipedia</a></li><li><a href="https://vimeo.com/77882285">Vimeo</a></li></ul><h3>Closing words or There is no dark side in the moon, really</h3><p>Great, that’s it. If you have made it this far, I bow to you and thank you for reading! I hope you enjoyed the post. If you have any questions or comments, please feel free to reach out to me via <a href="https://www.vitgabrhel.com/contact/contact.html">Contact</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C90sfHXz6acUHj6kOzl2dg.png" /><figcaption><em>“Pink Floyd, The Dark Side of the Moon, abstract, full hd render + 3d octane render +4k UHD + immense detail + dramatic lighting + well lit + black, purple, blue, pink, cerulean, teal, metallic colours, + fine details + octane render + 8k</em>”</figcaption></figure><p><em>Originally published at </em><a href="https://vitgabrhel.netlify.app/posts/dsotm/">https://vitgabrhel.com/posts/dsotm/</a> <em>(</em><a href="https://vitgabrhel.netlify.app/posts/dsotm/">https://vitgabrhel.netlify.app/posts/dsotm/</a><em>).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a61497819813" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/see-you-on-the-dark-side-of-the-moon-a61497819813">See you on The Dark Side of the Moon</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to version control with Git for a more efficient collaboration]]></title>
            <link>https://medium.com/analytics-vidhya/introduction-to-version-control-with-git-for-a-more-efficient-collaboration-316762706ce4?source=rss-6e9400412127------2</link>
            <guid isPermaLink="false">https://medium.com/p/316762706ce4</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[academia]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Vít Gabrhel]]></dc:creator>
            <pubDate>Sat, 28 Jan 2023 19:32:04 GMT</pubDate>
            <atom:updated>2023-01-28T19:32:26.297Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kziNFMVBoSoxyp2o54Pdeg.jpeg" /></figure><h3>What is a version?</h3><p>Contents of a file at a given point in time</p><p><strong>Metadata</strong> (information associated with the file):</p><ul><li><em>The author of the file</em></li><li><em>Location</em></li><li><em>Type</em></li><li><em>Timestamp of the last save</em></li><li>…</li></ul><h3>What is version control?</h3><p><strong>Version control</strong> is a group of systems and processes to manage changes made to documents, programs, and directories</p><p>Version control is <strong>useful</strong> for anything that:</p><ul><li><strong>changes over time</strong>, or</li><li><strong>needs to be shared</strong></li></ul><h3>Git</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qdSP64cjgExVDMXwwu0Tmw.png" /></figure><p><a href="https://git-scm.com/">Git</a> is a popular <strong>version control system</strong> for computer programming and other projects</p><ul><li><a href="https://en.wikipedia.org/wiki/Open-source_software"><strong>Open source</strong></a></li><li><strong>Scalable</strong> (= in terms of projects, collaborators, or resources)</li></ul><p><strong>Benefits</strong> of Git:</p><ul><li>Git <strong>stores contents and metadata</strong>, so nothing is lost</li><li>Git <strong>notifies</strong> users when there is <strong>conflicting content</strong> in files</li><li>Git <strong>synchronizes</strong> across different people and computers</li></ul><blockquote><strong>Action</strong></blockquote><ul><li>Install Git or check whether you have it already for your respective operating system: <a href="https://github.com/git-guides/install-git?fbclid=IwAR1kihhEjz295LQ6s65BlIzxdTamThQBcVzBnBs1ZDQ0pKT4HOJLZF9aWT8#install-git-on-linux">Linux</a>, <a href="https://github.com/git-guides/install-git?fbclid=IwAR1kihhEjz295LQ6s65BlIzxdTamThQBcVzBnBs1ZDQ0pKT4HOJLZF9aWT8#install-git-on-windows">macOS</a>, and <a href="https://github.com/git-guides/install-git?fbclid=IwAR1kihhEjz295LQ6s65BlIzxdTamThQBcVzBnBs1ZDQ0pKT4HOJLZF9aWT8#install-git-on-windows">Windows</a>.</li></ul><p>In case you are <strong>Windows</strong> users, <strong>consider installing a Unix shell</strong> along the way. Alternatively, look for the <a href="https://gist.github.com/jonlabelle/e8ba94cd29b8f63fd7dd3c4f95c1d210">equivalent command</a> in your respective shell.</p><h3>Using Git</h3><p>Git commands are run on the <strong>shell</strong>, also known as the <strong>terminal</strong></p><p>The <strong>shell</strong> (<em>Zsh</em>, <em>PowerShell</em>,…)</p><ul><li>is a <strong>program for executing commands</strong> (e.g., create a folder)</li><li>can be used to easily preview, modify, or inspect files and directories (= folders)</li></ul><blockquote><strong>Action</strong></blockquote><p>Open your terminal — either inside another tool like <a href="https://support.posit.co/hc/en-us/articles/115010737148-Using-the-RStudio-Terminal">RStudio</a> or as a <a href="https://towardsdatascience.com/a-quick-guide-to-using-command-line-terminal-96815b97b955">standalone app</a>.</p><h3>Useful shell commands</h3><ul><li><em>Locate your current working directory</em> — pwd</li><li><em>List all of the items in the current working directory</em> — ls</li><li><em>Change a directory</em> — cd &lt;directory_name&gt;</li><li><em>Checking Git version</em> — git --version</li><li><em>Create or edit a file</em> — echo</li></ul><p>For more commands, see (e.g.) the following <a href="https://towardsdatascience.com/how-to-become-a-command-line-wizard-5d78d75fbf0c">article</a>.</p><p>Please note that the <a href="https://www.makeuseof.com/tag/a-beginners-guide-to-the-windows-command-line"><strong>default command prompt</strong></a> on <strong>Windows uses different syntax</strong>.</p><h3>GitHub</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Gs83fNgoinpq7cUaMhRqQ.png" /></figure><p>What is GitHub?</p><ul><li>Cloud-based hosting <strong>service</strong></li><li>Allows users to store and track their work (= version control)</li><li><strong>On-demand resources</strong> over the internet, such as <strong>storage</strong></li></ul><p>There are plenty of sites on the Internet where you can upload git repositories with code — e.g. <a href="https://about.gitlab.com/">GitLab</a> or <a href="https://bitbucket.org/product">BitBucket</a>.</p><p><strong>Use-cases</strong></p><ul><li><em>Storing projects</em></li><li><em>Keeping track of projects and files</em></li><li><em>Collaborating with others</em></li><li><em>Social network</em></li><li><em>Open-source projects</em></li></ul><p><strong>GitHub enhances Git, but cannot be used without it.</strong></p><blockquote><strong>Action</strong></blockquote><p>If you don’t already have a user account on github.com, go there and create one.</p><h3>Clone a repository (git clone)</h3><p>To get started, let’s try working with a repository created for the purposes of this workshop.</p><blockquote><strong>Action</strong></blockquote><p>Type the following command in your command line:</p><pre>git clone https://github.com/VGabrhel/git-scientific-communication.git</pre><p>A new repository will be created - a directory with the name git-scientific-communication that contains folders and files.</p><blockquote><strong>Action</strong></blockquote><p>You can look at the <strong>URL</strong> you used in this example in your <strong>browser</strong>. There, you’ll see a <strong>list of files</strong> and lots of <strong>links</strong> to information about the repository (for example, under <strong>“commits”</strong> is the history).</p><blockquote><strong>Action</strong></blockquote><p>Switch to the new directory (cd git-scientific-communication) and try looking at the history (gitk or git log). It may be short, but it’s important that there is some. You have a copy of a project on your computer that someone else created!</p><p><strong>Your contribution to this project</strong> will be a <strong>write to the repository</strong>: specifically, adding a file with your name on it.</p><ul><li>The <strong>name</strong> (or distinct identificator) is there <strong>to avoid clashes</strong>: we need the contributions from all the people going through this course to be different.</li></ul><p>However, your post will be <strong>publicly displayed on the internet</strong>. If you don’t want to expose your real name, feel free to use a <strong>nickname</strong>, a favorite food, or a few random letters instead (be original to avoid conflicts).</p><ul><li>Whatever you decide to use will be referred to as &lt;your_name&gt; further down the line.</li></ul><h3>Creating a branch</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xJI1xHbnZNN7I-q7N1y7QA.jpeg" /></figure><blockquote><strong>Action</strong></blockquote><p>Use git branch to find out what branch you are currently on. It should be the master / main <a href="https://www.theserverside.com/feature/Why-GitHub-renamed-its-master-branch-to-main">branch</a>.</p><p>This master / main branch is only good to use for commits that the whole team has already <strong>agreed</strong> on. So when you want to contribute to a project, the <strong>first step is to make a new branch for your contribution and switch to that branch</strong>. For example:</p><pre>git branch &lt;your_branch&gt;<br>git checkout &lt;your_branch&gt;</pre><h3>Pushing changes (git push)</h3><blockquote><strong>Action</strong></blockquote><p>In R, create a data file to be shared with others.</p><pre># Excplicitedly call the source package<br>library(&quot;datasets&quot;)<br><br># Import the USArrests dataset from the package<br>data(&quot;USArrests&quot;, package = &quot;datasets&quot;)<br><br># Save the object as a data frame<br>USArrests &lt;- as.data.frame(USArrests)<br><br># Export the data frame as a .csv file with &lt;your_name&gt;<br>write.csv(USArrests, file = &quot;data/USArrests_&lt;your_name&gt;.csv&quot;, row.names = FALSE)</pre><blockquote><strong>Action</strong></blockquote><p>Add a file named after your name (or nickname):</p><pre>git add USArrests_&lt;your_name&gt;.csv</pre><p>Use the git status to observe the indicated changes:</p><pre>git status</pre><p>Commit the added file to git (git commit):</p><pre>git commit -m &quot;Add the USArrests data for the respective contributor&quot;</pre><p>Again, use the git status and compare the outcome with the previous status:</p><pre>git status</pre><p>Now all that’s left to do is “just” <strong>merge the change into the originally</strong> shared repository. But that’s not easy: the repository you cloned <a href="https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/managing-a-branch-protection-rule">cannot be changed</a> by just anyone.</p><p>A lot of places on the Internet (<strong>blogs, news sites, e-shops</strong>) work in such a way that a <strong>select group of peopl</strong>e, “editors”, have the <strong>right to change the content</strong> as they like.</p><ul><li>Such editors must be <strong>trusted by the project administrator</strong> before they are allowed access.</li></ul><p>With <strong>Git</strong>, a slightly <strong>different mechanism</strong> is used: you <strong>upload changes to your own shared repository</strong>, which only you have the right to change. — You then <strong>write a pull request</strong> to the <strong>owner of the original project</strong>. This could be an email saying <em>“Hey, I have some changes at so-and-so address that you might find useful! Add them to your project!”</em></p><p>The advantage is that <strong>anyone can</strong> join the project — if it’s <a href="https://docs.github.com/en/repositories/creating-and-managing-repositories/about-repositories"><strong>public</strong></a>. <strong>You don’t have to ask beforehand</strong>, you don’t have to prove you’re a trustworthy person, just change something and send it.</p><ul><li>Whether the authors of the project like the change or not is another matter — but they can judge the change itself, not the credibility of the author.</li></ul><p><strong>Services</strong> like github.com allow you to make your own <strong>shared repository</strong> (which will be <strong>available on the web</strong>) and make it easy to <strong>incorporate changes</strong> (instead of sending an email, just press a button). Let’s see how to do it.</p><blockquote><strong>Action</strong></blockquote><p><strong>Log in to GitHub</strong> and then go to the <strong>address</strong> you used for git clone. Find the <strong>“Fork”</strong> button on the top left and click it. This will create your own copy of the repository on GitHub. The address should be something like this: https://github.com/&lt;your_github_name&gt;/git-scientific-communication.git.</p><blockquote><strong>Action</strong></blockquote><p>Now, how do you upload changes from your computer to GitHub? For each repository on your computer, Git remembers the addresses where you can download and push changes. The git remote -v command will show you a list of those addresses. For example:</p><pre>git remote -v<br>origin https://github.com/VGabrhel/git-scientific-communication.git (fetch)<br>origin https://github.com/VGabrhel/git-scientific-communication.git (push)</pre><p>This output means that “origin” is the address from which you cloned the repository.</p><blockquote><strong>Action</strong></blockquote><p>Add a similar shortcut for your own repository on GitHub. Don’t forget to replace your name with the name of the account you have on GitHub. (Note that your name appears twice in the command!)</p><p>git remote add &lt;your_name&gt; https://github.com/&lt;your_github_name&gt;/git-scientific-communication.git and check that it worked:</p><pre>git remote -v<br>origin git@github.com:kpsych-fss-mu/git-scientific-communication.git.git (fetch)<br>origin git@github.com:kpsych-fss-mu/git-scientific-communication.git.git (push)<br>&lt;your_name&gt; https://github.com/&lt;your_name&gt;/git-scientific-communication (fetch)<br>&lt;your_name&gt; https://github.com/&lt;your_name&gt;/git-scientific-communication (push)</pre><blockquote><strong>Action</strong></blockquote><p>So much for the setup — just do git remote add once for each repository. Then you can upload changes using:</p><pre>git push --set-upstream origin &lt;your_branch&gt;</pre><p>Which means: push the &lt;your_branch&gt; branch to the address stored under &lt;your_name&gt;.</p><blockquote><strong>Action</strong></blockquote><p>Does it work? Look at https://github.com/&lt;your_github_name&gt;/git-scientific-communication in your browser and make sure your changes are there.</p><h3>Pull request</h3><p>Now all you have to do is ask the authors of the original project to add the changes from your shared repository to their copy. GitHub has a mechanism for doing this called a <a href="https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests"><strong>pull request</strong></a>.</p><blockquote><strong>Action</strong></blockquote><p>Go to the original project’s page (the address you used for git clone in the beginning). There you should see a notification of your newly uploaded branch with a big green <strong>Compare &amp; pull request button</strong>. <strong>Click</strong> on it. If you want, <strong>add/change the description</strong> of what this change entails. Then hit the next button.</p><p>If you don’t see the Compare &amp; pull request button, go to the <strong>address of your copy of the repository</strong> and hit the <strong>New pull request</strong> button. <strong>Select</strong> what you want to include, <strong>add/change</strong> the label, then press <strong>Create pull request</strong>.</p><p><strong>You’re done</strong>! Now it’s up to the <strong>project authors</strong> to look at the changes and <strong>accept</strong> them — or start a <strong>discussion</strong> about how to make them even better. (You can discuss this on the pull request page or via email.)</p><p>This probably won’t happen for adding a name to the attendance list, but if you needed to work on the change a bit more before incorporating it (even after a few days of discussion), it wouldn’t be a problem. <strong>Switch to the</strong> &lt;your_branch&gt; <strong>branch on your machine</strong>, make <strong>further commits</strong> and use <strong>git push to update</strong> your &lt;your_branch&gt; <strong>pull request</strong>.</p><h3>Update (git pull)</h3><p>Once your <strong>changes</strong> — and those of others — have been <strong>merged</strong>, you can <strong>update your local repository</strong>.</p><p>This is the <strong>one on your computer</strong>.</p><blockquote><strong>Action</strong></blockquote><p>First, <strong>switch back</strong> to the master branch. Now you won’t be working on &lt;your_branch&gt;; that branch is already committed.</p><p>This is done with git pull origin master (pull changes from the “master” branch from the address under “origin”). You can use gitk --all or git log to see how the project has evolved in the meantime.</p><p>It’s always a good idea to do this git pull <strong>before you start working on a new change/branch</strong>. This will ensure that the project you are changing is <strong>“fresh”</strong>.</p><ul><li>Congratulations! You just went through the “round” that most programmers do every day: making a change, pushing it to colleagues for review and incorporation, and pulling changes from others.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*hltFqZHd-Mw3-2LxfV0Q9w.png" /></figure><h3>Bug reporting (issues)</h3><p>Occasionally, you <strong>find a bug</strong> in a GitHub project, but you <strong>don’t have the time or knowledge to fix it</strong>.</p><ul><li>In this case, you’ll often find a list of reported <a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/about-issues"><strong>issues</strong></a> on the GitHub project page under the <strong>Issues tab</strong>.</li><li>If you don’t find “your” bug among them, you can report it — just click on <strong>New Issue</strong> and you can post when the bug occurs, what the program is doing wrong, and what it should be doing instead.</li></ul><p>Some projects don’t use Issues on GitHub. If you can’t find the Issues tab, check the project documentation to see if there’s a link to the bug list.</p><h3>README: Information for others</h3><p>If you’re creating a project and want others to contribute to it, you need them to know what your project does, what it’s good for, how it’s used, etc.</p><p>The <a href="https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-readmes"><strong>README</strong></a> file is used for <strong>basic information about the project/repository</strong>. This file can include:</p><ul><li><em>project name</em>,</li><li><em>a brief description of the project (one or two sentences)</em>,</li><li><em>a short instruction on how to install the project</em>,</li><li><em>short instructions on how to run the project</em>,</li><li><em>short instructions on how to use the project, or a link to more extensive documentation</em>,</li><li><em>if the project has tests, information on how to run them</em>,</li><li><em>information on how to get involved in the development of the project</em>,</li><li><em>information about the authors of the project</em>,</li><li><em>licensing information (more on licensing later)</em>.</li></ul><p>The README should be <strong>structured</strong> and shouldn’t take the user an hour to read, <strong>usually short punchy information with a possible link somewhere else is enough</strong>.</p><ul><li>For example, you don’t need to explain in each project how to install Python. Just say that Python is needed (and in which version) and refer the user to the appropriate instructions.</li><li>You also need to take into account the <strong>intended audience(s)</strong> or <strong>who will be reading</strong> the README.</li></ul><p>GitHub (and lots of other similar services) allows you to <strong>use a markup language</strong> such as <strong>Markdown</strong> for the README. It’s then possible to use headings, images, etc.</p><p>And last but not least: <strong>open-source projects tend to be in English</strong> so that anyone from around the world can participate. <strong>Variable names, comments, documentation</strong> — everything is <strong>primarily in English</strong>.</p><h3>License</h3><p>To make <strong>sharing work for legal purposes</strong>, it is <strong>not enough to upload a piece of code to the Internet</strong>. You also have to <strong>officially announce</strong> that others can play with it.</p><ul><li>This is because <strong>you have a copyright</strong> on your code that says <strong>others cannot use your program</strong>, let alone improve it <strong>unless you give them permission.</strong></li><li>To formally grant this permission, they use licenses written by lawyers.</li></ul><p>The issue of licenses can, unfortunately, be quite complex. But if you simplify it to the minimum, you just want to ensure that everyone can use your creation, learn from it, pass it on, and improve it. In that case, choose, for example, the <a href="https://en.wikipedia.org/wiki/MIT_License"><em>MIT license</em></a>.</p><p>In addition, if you want to prevent someone from taking your code and “improving” it and profiting from it without sharing the improvements with others, try the <a href="https://en.wikipedia.org/wiki/Affero_General_Public_License"><em>AGPL</em></a>.</p><p>Code is most often licensed by putting the license text into a file called LICENSE and adding it to Git. It is a good idea to mention the license in the README file as well.</p><p>If you want to read more about licenses, I refer you to choosealicense.com, or creativecommons.org, and opensource.org.</p><h3>Next: Hands-on!</h3><blockquote><strong>Action</strong></blockquote><p><strong>Consider a project</strong> where you want to use Git (and GitHub; <em>~15 min</em>)</p><ul><li>What is the <strong>nature of the collaboration</strong> in this project?</li><li>Who are the <strong>collaborators</strong>?</li></ul><p><strong>Create</strong> the respective <strong>repository</strong></p><p>Secure the main branch from pushing directly to it</p><p>Set <strong>tokens</strong> and other <strong>security</strong> elements</p><p><strong>Create</strong> the <strong>README</strong> and <strong>license</strong> files</p><p><strong>Push</strong> all of the related <strong>materials to the repository</strong> and <strong>create a pull request</strong></p><p>Let the <strong>collaborator review</strong> the pushed code.</p><p><strong>After</strong> the changes are <strong>merged</strong>, check out to the local main branch and pull the updated remote main branch.</p><h3>Resources</h3><p>You may find multiple resources for learning git and/or services like GitHub available for free (although sometimes you need to register).</p><h3>Courses</h3><ul><li><a href="https://app.datacamp.com/learn/courses/introduction-to-version-control-with-git">Introduction to Version Control with Git</a></li><li><a href="https://app.datacamp.com/learn/courses/introduction-to-git">Introduction to Git</a></li><li><a href="https://app.datacamp.com/learn/courses/github-concepts">GitHub Concepts</a></li><li><a href="https://www.udemy.com/course/git-expert-4-hours">Git: Become an Expert in Git &amp; GitHub in 4 Hours</a></li><li><a href="https://www.coursera.org/learn/introduction-git-github">Introduction to Git and GitHub</a></li><li><a href="https://naucse.python.cz/course/pyladies/sessions/foss/">Spolupráce a Open-Source</a> (Czech)</li></ul><h3>Documentation</h3><ul><li><a href="https://git-scm.com/doc">Git — Documentation</a></li><li><a href="https://docs.github.com/en">GitHub Docs</a></li></ul><h3>Cheat-sheets</h3><ul><li><a href="https://pyvec.github.io/cheatsheets/basic-git/basic-git-cs.pdf">git</a> (Czech)</li><li><a href="https://www.markdownguide.org/cheat-sheet/">Markdown</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=316762706ce4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/introduction-to-version-control-with-git-for-a-more-efficient-collaboration-316762706ce4">Introduction to version control with Git for a more efficient collaboration</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combining R {tidyverse, leaflet, DT, flexdashboard} and Python {pandas, GeoPy} to create a…]]></title>
            <link>https://medium.com/data-science/combining-r-tidyverse-leaflet-dt-flexdashboard-and-python-pandas-geopy-to-create-a-c25c5ef7d2d9?source=rss-6e9400412127------2</link>
            <guid isPermaLink="false">https://medium.com/p/c25c5ef7d2d9</guid>
            <dc:creator><![CDATA[Vít Gabrhel]]></dc:creator>
            <pubDate>Mon, 05 Apr 2021 09:59:56 GMT</pubDate>
            <atom:updated>2021-04-05T13:35:50.821Z</atom:updated>
            <content:encoded><![CDATA[<h3>Combining R {tidyverse, leaflet, DT, flexdashboard} and Python {pandas, GeoPy} to create a serverless psychotherapy dashboard</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mY7xlY_JEPPCYa3ogcZoEQ.jpeg" /><figcaption>Photo by <a href="https://www.data-must-flow.com/projects/photos/photos.html">Vit Gabrhel</a></figcaption></figure><h3>Introduction</h3><p>In the Czech Republic, only a <strong>fraction of psychotherapeutic care</strong> is <strong>covered</strong> by <strong>health insurance</strong> companies. So, it was a welcomed step when VZP, one of the largest such companies, offered its clients an allowance of 7 000 CZK (or circa 280 EUR) for <a href="https://www.vzp.cz/pojistenci/vyhody-a-prispevky/dospeli/prispevek-na-podporu-zvyseni-dostupnosti-psychosocialni-podpory">psychotherapy</a>.</p><p>However, the way of finding yourself a psychotherapist was not the most convenient one as you had to scroll through a .pdf <a href="https://media.vzpstatic.cz/media/Default/dokumenty/seznam-terapeutu.pdf?ver=0303">file</a>. Luckily, <a href="https://www.linkedin.com/in/dominika-%C4%8Dechov%C3%A1-m-a-b6b95724/">Dominika Čechová</a>, a board member of the <a href="https://czap.cz/eng"><em>Czech Association for Psychotherapy</em></a>, negotiated additional information from VZP like which of the psychotherapists can take new clients.</p><p>My part was to combine these inputs into a more user-friendly way of finding yourself a therapist and thus make the service more <strong>accessible</strong>. The focus was the speed of delivery and ease of deployment, so I decided to follow through <strong>R</strong> with <a href="https://rmarkdown.rstudio.com/flexdashboard/"><strong>flexdashboard</strong></a>, a package allowing for creating <strong>interactive</strong> yet <strong>serverless</strong> dashboards.</p><p>I had most of the building blocks covered from previous <a href="https://www.data-must-flow.com/post/">projects</a> — R implementations of <a href="https://rstudio.github.io/leaflet/"><strong>Leaflet</strong></a> for maps and <a href="https://rstudio.github.io/DT/"><strong>DataTables</strong></a> for tables. <a href="https://www.tidyverse.org/"><strong>Tidyverse</strong></a> for loading and wrangling the data.</p><p>However, the input table did not contain latitude and longitude, two elements needed for displaying the therapists’ location on a map. Previously, I had adjusted data using <a href="https://github.com/dkahle/ggmap"><strong>ggmap</strong></a>. However, the changes in the Google API requirements have made this somewhat inconvenient.</p><p>So, I decided to look for an alternative. Then I came across the article <a href="https://towardsdatascience.com/geocode-with-python-161ec1e62b89"><em>Geocode with Python</em></a> by <a href="https://shakasom.medium.com/">Abdishakur</a>, a great introduction to Python’s package <a href="https://geopy.readthedocs.io/en/stable/"><strong>GeoPy</strong></a>.</p><p>Now, although R is my special data friend, I also like to <strong>integrate</strong> other languages and tools like <strong>Python</strong> and <a href="https://pandas.pydata.org/"><strong>pandas</strong></a> in my workflow where beneficial.</p><p>In this post, I would like to review how I proceed when creating the desired outcome.</p><p>You may find the final <strong>dashboard</strong> <a href="https://www.data-must-flow.com/posts/psychotherapy_vzp/"><strong>here</strong></a> and the <strong>original article</strong> with interactive elements <a href="https://www.data-must-flow.com/posts/python_r_therapy/"><strong>here</strong></a>.</p><h3>R packages</h3><p>Let us begin by loading the required packages:</p><pre><em># Load required packages</em><br>library(flexdashboard) <em># dashboard wrapping</em><br>library(tidyverse) <em># data wrangling</em><br>library(crosstalk) <em># interactivity</em><br>library(broom) <em># output of built-in functions cleanup</em><br>library(DT) <em># Table formatting</em><br>library(htmltools) <em># widgets</em><br>library(reshape2) <em># data taransformations</em><br>library(leaflet) <em># interactive maps</em><br>library(leaflet.extras) <em># interactive features</em></pre><h3>Input data</h3><p>You may find the input dataset on <a href="https://github.com/VGabrhel/datamustflow/blob/master/public/posts/python_r_therapy/vzp_data_geo.csv"><strong>GitHub</strong></a>.</p><pre><em># Initial dataset</em><br>vzp_data = read.csv(&quot;vzp_data_geo.csv&quot;) %&gt;%<br>    select(name, surname, alias, website, address,<br>           city, region, psc, phone, email, Kapacita, remote_therapy)</pre><h3>Python</h3><h4>Activation</h4><p>First, you need to turn on the Python interface in R. I work with <a href="https://rstudio.github.io/reticulate/index.html"><strong>reticulate</strong></a>.</p><pre><em># Python interoperability in R</em><br>library(reticulate)</pre><pre><em># Specifying which version of python to use.</em><br>use_python(&quot;/home/vg/anaconda3/bin/python3.7&quot;,<br>           required=T)  <em># Locate and run Python</em></pre><h4>Packages and data wrangling</h4><p>Even though it is possible to run all of the following chunks of code at once, let us follow the <a href="https://blog.codinghorror.com/curlys-law-do-one-thing/">do one thing</a> principle and separate these according to their functions.</p><p>Python packages first:</p><pre>from geopy.geocoders import Nominatim<br>from geopy.extra.rate_limiter import RateLimiter<br>import pandas as pd</pre><p>Activate the geocoder:</p><pre>locator = Nominatim(user_agent=&quot;myGeocoder&quot;)</pre><p>Link parts of an address to one variable — will be used for geocoding:</p><pre>df = r[&#39;vzp_data&#39;]</pre><pre>df[&#39;state&#39;] = &quot;CZ&quot;</pre><pre>df[&quot;ADDRESS&quot;] = df[&quot;address&quot;] + &quot;,&quot; + df[&quot;city&quot;] + &quot;,&quot; + df[&quot;state&quot;]</pre><pre>df[&quot;Adresa&quot;] = df[&quot;address&quot;] + &quot;, &quot; + df[&quot;city&quot;]</pre><h4>Geocoding using GeoPy</h4><p>To mitigate the Too Many Requests error, use RateLimiter. Adding a delay “<em>between geocoding calls to reduce the load on the Geocoding service</em>”, as the <a href="https://geopy.readthedocs.io/en/stable/">documentation</a> of GeoPy puts it.</p><pre><em># 1 - conveneint function to delay between geocoding calls</em><br>geocode = RateLimiter(locator.geocode, min_delay_seconds=1)</pre><pre><em># 2- create location column</em><br>df[&#39;location&#39;] = df[&#39;ADDRESS&#39;].apply(geocode)</pre><pre><em># 3 - create longitude, laatitude and altitude from location column (returns tuple)</em><br>df[&#39;point&#39;] = df[&#39;location&#39;].apply(<strong>lambda</strong> loc: tuple(loc.point) <strong>if</strong> loc <strong>else</strong> None)</pre><pre><em># 4 - split point column into latitude, longitude and altitude columns</em><br>df[[&#39;latitude&#39;, &#39;longitude&#39;, &#39;altitude&#39;]] = pd.DataFrame(df[&#39;point&#39;].tolist(), index=df.index)</pre><h4>Export the geocoded data</h4><p>Now, you might be asking — why I am exporting the table? Am I not going to proceed with the data further as a Python/R object?</p><p>Unfortunately, flexdashboard <a href="https://community.rstudio.com/t/python-code-inside-flexdashboard-framework/44873/2">cannot yet be knitted</a> while containing Python code. So, when working on the dashboard, I had to create two separate workflows — one for the data geocoding and the second for compiling the dashboard. However, I wanted to make the dashboard coding more reproducible while keeping all of the important components in one workflow.</p><p>Fortunately, other R Markdown outputs like R notebook are happy wrapping Python code. That is why you see this step in code. Surely, it is not the most efficient workflow, but you know what they say — done (and working) is better than perfect.</p><p>Of course, if the goal would be an R Markdown document containing some Python chunks, you can simply add vzp_data = py[df] in an R chunk to convert a pandas DataFrame into R Data Frame and proceed smoothly to the next part.</p><pre><em># Convert pandas DataFrame into R Data Frame (tibble)</em><br>vzp_data_geocoded = py[&#39;df&#39;]</pre><pre>vzp_data_geo &lt;- vzp_data_geocoded %&gt;%<br>    select(name, surname, alias, website, address,<br>           city, region, psc, phone, email, `Kapacita`, remote_therapy,<br>           state, Adresa, latitude, longitude)</pre><pre>vzp_data_geo &lt;- apply(vzp_data_geo,2,as.character)<br>write.csv(vzp_data_geo, file = &quot;vzp_data_geo.csv&quot;)</pre><h3>R</h3><h4>Data wrangling in tidyverse</h4><p>Select only the columns that will be used in the output and format them:</p><pre><em># Data import and wrangling</em><br>vzp_data = read.csv(&quot;vzp_data_geo.csv&quot;) %&gt;%<br>  select(name,<br>         surname,<br>         alias,<br>         website,<br>         address,<br>         city,<br>         phone,<br>         email,<br>         Kapacita,<br>         remote_therapy,<br>         latitude,<br>         longitude) %&gt;%<br>  mutate_all(list(~str_trim(.,side = &quot;both&quot;))) %&gt;%<br>  mutate(`Jméno a příjmení` = paste(name,surname),<br>         latitude = as.numeric(latitude),<br>         longitude = as.numeric(longitude)) %&gt;%<br>  rename(&quot;Email&quot; = email,<br>         &quot;Telefon&quot; = phone,<br>         &quot;Web&quot; = website,<br>         &quot;Online nebo telefonické konzultace?&quot; = remote_therapy,<br>         &quot;Kapacita?&quot; = Kapacita,<br>         &quot;Město&quot; = city,<br>         &quot;Ulice&quot; = address) %&gt;%<br>  mutate(`Kapacita?` = case_when(`Kapacita?` == &quot;volno&quot; ~ &quot;Volno&quot;,<br>                              `Kapacita?` == &quot;naplněno&quot; ~ &quot;Naplněno&quot;,<br>                              `Kapacita?` == &quot;Naplněno&quot; ~ &quot;Naplněno&quot;,<br>                              TRUE ~ `Kapacita?`),<br>         `Online nebo telefonické konzultace?` = case_when(`Online nebo telefonické konzultace?` == &quot;Ano&quot; ~ &quot;Ano&quot;,<br>                                                           `Online nebo telefonické konzultace?` == &quot;Ano/ možnost konzultací v anglickém jazyce&quot; ~ &quot;Ano&quot;,<br>                                                           `Online nebo telefonické konzultace?` == &quot;Ne&quot; ~ &quot;Ne&quot;,<br>                                                           `Online nebo telefonické konzultace?` == &quot;ne&quot; ~ &quot;Ne&quot;,<br>                                                           TRUE ~ `Online nebo telefonické konzultace?`)<br>         )</pre><pre><em># Replace &quot;NaN&quot; with &quot;Neuvedeno&quot;</em><br>vzp_data[vzp_data==&quot;NaN&quot;] &lt;- &quot;Neuvedeno&quot;</pre><h3>Outputs</h3><h4>List of available psychotherapists</h4><p>The following table is a snapshot of the final <a href="https://www.data-must-flow.com/posts/psychotherapy_vzp/">look-up table</a>, done using DataTable package. This package means a number of interactive elements at your disposal. You can add search and filter across columns, sorting, pagination, and so forth. In addition, it could be very easily linked with filters using crosstalk.</p><p>As the output is in Czech, I suggest the Translate to English feature of your browser to get more value.</p><pre>vzp_data_table &lt;- vzp_data %&gt;%<br>  select(`Kapacita?`, `Online nebo telefonické konzultace?`,<br>        Město,<br>        Ulice,<br>         `Jméno a příjmení`, <br>         Email, Telefon, Web, Ulice,<br>         `Online nebo telefonické konzultace?`) </pre><pre>test_shared &lt;- crosstalk::SharedData$new(vzp_data_table)</pre><pre>DT::datatable(test_shared,<br>    extensions = c(<br>      &quot;Responsive&quot;<br>    ),<br>    rownames = FALSE,  <em># remove rownames</em><br>    style = &quot;bootstrap&quot;,<br>    class = &#39;cell-border display&#39;,<br>    options = list(<br>      pageLength = 10,<br>      dom = &#39;t&#39;,<br>      deferRender = TRUE,<br>      scroller = TRUE,<br>      columnDefs = list(list(className = &#39;dt-center&#39;, targets = &quot;_all&quot;))<br>        )<br>      ) %&gt;% formatStyle(<br>  &quot;Kapacita?&quot;,<br>  target = &#39;row&#39;,<br>  backgroundColor = styleEqual(c(&quot;Naplněno&quot;, &quot;Volno&quot;), c(&#39;#ffcccb&#39;, &#39;#d2e9af&#39;)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*ud5cgdiayrlz8nuE13T3NA.png" /></figure><h4>Map of available psychotherapists</h4><p>As the search might be more convenient via a map, I decided to add one using leaflet. Similar to DataTable, leaflet allows for multiple interactive elements. Including search based on a string or different map layers.</p><p>Again, as the output is in Czech, I suggest the Translate to English feature of your browser to get more value.</p><pre><em># prepare a palette - manual colors according to branch column</em><br>pal &lt;- leaflet::colorFactor(palette = c(&quot;Naplněno&quot; = &quot;#8b0000&quot;,<br>                                        &quot;Neuvedeno&quot; = &quot;#A9A9A9&quot;,<br>                                        &quot;Volno&quot; = &quot;#006400&quot;<br>                                        ), <br>                               domain = vzp_data$`Kapacita?`)</pre><pre>points_fin &lt;- SharedData$new(vzp_data)</pre><pre>map1 &lt;- leaflet(data = points_fin, width = &#39;100%&#39;, height = 800) %&gt;%<br>          addProviderTiles(&quot;CartoDB.Positron&quot;, group = &#39;Základní&#39;) %&gt;%<br>          addProviderTiles(&quot;Esri.WorldImagery&quot;, group = &#39;Letecká&#39;) %&gt;%<br>          addProviderTiles(&quot;OpenStreetMap.Mapnik&quot;, group = &#39;Uliční&#39;) %&gt;%<br>          addProviderTiles(&quot;OpenTopoMap&quot;, group = &#39;Zeměpisná&#39;) %&gt;%<br>          addScaleBar(&#39;bottomright&#39;) %&gt;%<br>          setView(15.4129318, 49.7559455, zoom = 8.2) %&gt;%<br>          addCircleMarkers(<br>                   group = &#39;Obor&#39;, <br>                   stroke = FALSE, <br>                   opacity = 0.9,<br>                   fillOpacity = 0.9,<br>                   fillColor = ~sapply(`Kapacita?`, <strong>switch</strong>, USE.NAMES = FALSE,<br>                                        &quot;Volno&quot; = &quot;#006400&quot;, <br>                                        &quot;Naplněno&quot; = &quot;#8b0000&quot;,<br>                                        &quot;Neuvedeno&quot; = &quot;#A9A9A9&quot;<br>                                     ),<br>                   popup = ~paste0(&#39;&lt;h2&gt;Detail&lt;/h2&gt; &lt;br&gt;&#39;,<br>                                   &#39;&lt;b&gt;Město&lt;/b&gt;: &#39;, Město, &#39;&lt;br&gt;&#39;,<br>                                   &#39;&lt;b&gt;Ulice&lt;/b&gt;: &#39;, Ulice, &#39;&lt;br&gt;&#39;,<br>                                   &#39;&lt;b&gt;Jméno a příjmení&lt;/b&gt;: &#39;, `Jméno a příjmení`, &#39;&lt;br&gt;&#39;,<br>                                   &#39;&lt;b&gt;Online nebo telefonické konzultace&lt;/b&gt;: &#39;, `Online nebo telefonické konzultace?`, &#39;&lt;br&gt;&#39;,<br>                                   &#39;&lt;b&gt;Telefon&lt;/b&gt;: &#39;,`Telefon`, &quot;&lt;br&gt;&quot;,<br>                                   &#39;&lt;b&gt;Email&lt;/b&gt;: &#39;, Email, &#39;&lt;br&gt;&#39;,<br>                                   &#39;&lt;b&gt;Web&lt;/b&gt;: &#39;, Web, &#39;&lt;br&gt;&#39;,<br>                                   &#39;&lt;b&gt;Kapacita&lt;/b&gt;: &#39;, `Kapacita?`, &#39;&lt;br&gt;&#39;)<br>                   ,<br>                   clusterOptions = markerClusterOptions(showCoverageOnHover = FALSE,<br>                    iconCreateFunction=JS(&quot;function (cluster) {    <br>    var childCount = cluster.getChildCount(); <br>    var c = &#39; marker-cluster-&#39;;  <br>    if (childCount &lt; 100) {  <br>      c += &#39;small&#39;;  <br>    } else if (childCount &lt; 1000) {  <br>      c += &#39;medium&#39;;  <br>    } else { <br>      c += &#39;large&#39;;  <br>    }    <br>    return new L.DivIcon({ html: &#39;&lt;div&gt;&lt;span&gt;&#39; + childCount + &#39;&lt;/span&gt;&lt;/div&gt;&#39;, className: &#39;marker-cluster&#39; + c, iconSize: new L.Point(40, 40) });</pre><pre>  }&quot;))) %&gt;%<br>  addLegend(position = &quot;topright&quot;,<br>            values = ~`Kapacita?`,<br>            opacity = .7,<br>            pal = pal,<br>            title = &quot;Kapacita?&quot;) %&gt;%<br>  leaflet.extras::addResetMapButton() %&gt;%<br>   addLayersControl(<br>                baseGroups = c(&quot;Základní&quot;, &quot;Letecká&quot;, &quot;Uliční&quot;, &quot;Zeměpisná&quot;),<br>                options = layersControlOptions(collapsed = TRUE)<br>              ) %&gt;%<br>      addSearchOSM()</pre><pre>map1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*eMLMTWpp5BQuGZ5TqikDIQ.png" /></figure><h3>Closing remarks</h3><p>I believe that some will roll their eyebrow when integrating R and Python the way presented above. “<em>Why not choose just one tool and use it?</em>” you may ask yourself. In my case, I needed to add a piece to my template. Although its parts had been written in R, I did not want to limit myself to one tool only.</p><p>At the same time, by looking at the current limits, we can promote future ease of integration. Here, I have to acknowledge that <a href="https://www.rstudio.com/solutions/r-and-python/">RStudio</a>, in my view one of the best tools for working with data, has taken several significant steps in integrating Python. However, in particular use-cases, it is still a somewhat bumpy ride.</p><p>Ultimately, I wanted to experiment and test the limits. I believe that both Python and R have their pros and cons when working with data, so why not explore where you can go if you combine them?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c25c5ef7d2d9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/combining-r-tidyverse-leaflet-dt-flexdashboard-and-python-pandas-geopy-to-create-a-c25c5ef7d2d9">Combining R {tidyverse, leaflet, DT, flexdashboard} and Python {pandas, GeoPy} to create a…</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python done the R (Markdown) way]]></title>
            <link>https://medium.com/data-science/python-done-the-r-markdown-way-d03bec4b96b?source=rss-6e9400412127------2</link>
            <guid isPermaLink="false">https://medium.com/p/d03bec4b96b</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[rstudio]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Vít Gabrhel]]></dc:creator>
            <pubDate>Mon, 24 Aug 2020 19:11:39 GMT</pubDate>
            <atom:updated>2020-08-26T18:08:04.604Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xIvHbmO_rjdGgpHIWw5LaA.jpeg" /><figcaption>Photo by <a href="https://www.data-must-flow.com/projects/photos/photos.html">Vit Gabrhel</a></figcaption></figure><h3>Introduction</h3><p>Starting to work with a new tool may not be easy, especially when that <em>“tool”</em> means a new programming language. At the same time, there might be an opportunity to build upon something already known and so make the transition smoother and less painful.</p><p>In my case, it was the transition from <strong>R</strong> to <strong>Python</strong>. Unfortunately, my former colleagues, pythonians, did not share my genuine fondness of R. Also, whether R enthusiasts like it or not, <strong>Python</strong> is a widely used tool in data analysis/engineering/science and beyond. So, I concluded that learning at least some Python is a reasonable thing to do.</p><p>For me, the first steps were maybe the most difficult ones. Residing in the comfort of <em>RStudio</em>, IDEs like <em>Pycharm</em> or <em>Atom</em> did not feel familiar. This experience led to the decision to begin in the well-known environment and test its limits when it comes to working with Python.</p><p>To tell the truth, I did not end up using <em>RStudio</em> as the weapon of choice for using Python in a general setting. Hopefully, the following text will deliver the message why. However, I am convinced that for some use-cases, like integrating R and Python in an ad hoc analysis R Markdown way, <em>RStudio</em> still represents a viable way to go.</p><p>More importantly, it could be a convenient <strong>starting line</strong> for people with the <strong>primary background in R</strong>.</p><p>So, what did I find?</p><h3>Analysis</h3><h3>Packages and environment</h3><p>First of all, let us set the environment and load the required packages.</p><ul><li><strong>Global environment:</strong></li></ul><pre><em># Globally round numbers at decimals</em><br><strong>options</strong>(digits=2)</pre><pre><em># Force R to use regular numbers instead of using the e+10-like notation</em><br><strong>options</strong>(scipen = 999)</pre><ul><li><strong>R libraries:</strong></li></ul><pre><em># Load the required packages. </em><br><em># If these are not available, install them first.</em><br>ipak &lt;- <strong>function</strong>(pkg){<br>  new.pkg &lt;- pkg[!(pkg %in% <strong>installed.packages</strong>()[, &quot;Package&quot;])]<br>  <strong>if</strong> (<strong>length</strong>(new.pkg)) <br>    <strong>install.packages</strong>(new.pkg, <br>                     dependencies = TRUE)<br>  <strong>sapply</strong>(pkg, <br>         require, <br>         character.only = TRUE)<br>}</pre><pre>packages &lt;- <strong>c</strong>(&quot;tidyverse&quot;, <em># Data wrangling</em><br>              &quot;gapminder&quot;, <em># Data source</em><br>              &quot;knitr&quot;, <em># R Markdown styling</em><br>              &quot;htmltools&quot;) <em># .html files manipulation</em></pre><pre><strong>ipak</strong>(packages)</pre><pre>tidyverse gapminder     knitr htmltools <br>     TRUE      TRUE      TRUE      TRUE</pre><p>In this analysis, I will be working with the reticulate <a href="https://rstudio.github.io/reticulate/">library</a>, a package developed at RStudio. However, feel free to look for <a href="https://www.datacamp.com/community/tutorials/using-both-python-r">alternatives</a>.</p><p>Also, I am going to import the reticulate package separately for the purposes of a clearer flow.</p><ul><li><strong>Python activation:</strong></li></ul><pre><em># The &quot;weapon of choice&quot;</em><br><strong>library</strong>(reticulate)</pre><pre><em># Specifying which version of python to use.</em><br><strong>use_python</strong>(&quot;/home/vg/anaconda3/bin/python3.7&quot;, <br>           required = T) <em># Locate and run Python</em></pre><ul><li><strong>Python libraries:</strong></li></ul><pre>import pandas as pd <em># data wrangling</em><br>import numpy as np <em># arrays, matrices, math</em><br>import statistics as st <em># functions for calculating statistics</em><br>import plotly.express as px <em># plotting package</em></pre><p>One of the limits of working with <strong>Python</strong> in <strong>R</strong> <em>(Studio)</em> is that in some cases you do not receive the Python traceback by default. That is a problem because, if I oversimplify things, a traceback helps you to identify where is the problem. Meaning it helps you fix it. So, consider it as an <strong>error message</strong> (or the lack of it).</p><ul><li>For example, when calling a library that you do not have installed, the Python chunk in R Markdown gives you green lights (so everything looks up and running), but this does not mean that the code ran the way you would expect (e.g. it imported a library).</li></ul><p>To deal with that, I suggest making sure you have your libraries installed in Terminal. If they are not, you can always install them and import afterwards.</p><p>For example, I will first import the json package which is already installed on my machine. I will do so by using <strong>Terminal</strong> here in <em>RStudio</em>. In addition, let me try to import the TensorFlow <a href="https://www.tensorflow.org/">package</a>.</p><p>From the following picture, you can see that there is no package TensorFlow. So, let me switch back to <strong>bash</strong> and install the package:</p><p>Then go to the directory where is the newly installed package TensorFlow installed, switch to <strong>Python</strong> and import the package once again:</p><p>All right. For more information, take a look at the <a href="https://docs.python.org/3.4/installing/index.html">Installing Python Modules</a> page.</p><h3>Data</h3><p>As we now have <strong>R</strong> and <strong>Python</strong> set, let us import some data to play with. I will be working with a sample from the <a href="https://www.gapminder.org/">Gapminder</a>, an intriguing project related to socio-demography of the world’s population. To be more specific, I will be working with the latest available data within the <a href="https://cran.r-project.org/web/packages/gapminder/README.html">GapMinder</a> library.</p><ul><li><strong>Data import in R:</strong></li></ul><pre><em># Let us begin with the most recent set of data</em><br>gapminder_latest &lt;- gapminder %&gt;%<br>          <strong>filter</strong>(year == <strong>max</strong>(year))</pre><p>So, now we have a data loaded in <strong>R</strong>. Unfortunately, it is not possible to access the <strong>R objects</strong> (e.g. <em>vectors</em> od <em>tibbles</em>) directly by <strong>Python</strong>. So, we need to <strong>convert</strong> the R object(s) first.</p><ul><li><strong>Data import in Python:</strong></li></ul><pre><em># Convert R Data Frame (tibble) into the pandas Data Frame</em><br>gapminder_latest_py = r[&#39;gapminder_latest&#39;]</pre><p>One important thing to realise when working with <em>Python objects</em> (e.g. <em>arrays</em> or pandas <em>Data Frame</em>) is that they are <strong>not</strong> <a href="https://community.rstudio.com/t/rstudio-ide-global-environment-variable-when-running-python/17999">explicitly <strong>stored</strong> in the environment</a> as the <em>R objects</em> are.</p><ul><li>In other words, if we want to know what is stored in the workspace, we must call functions like dir(), globals() or locals():</li></ul><pre>[&#39;R&#39;, &#39;__annotations__&#39;, &#39;__builtins__&#39;, &#39;__doc__&#39;, &#39;__loader__&#39;, &#39;__name__&#39;, &#39;__package__&#39;, &#39;__spec__&#39;, &#39;contact_window_days&#39;, &#39;contact_window_days_style&#39;, &#39;fig&#39;, &#39;gapminder_latest_count_py&#39;, &#39;gapminder_latest_max_lifeExp_py&#39;, &#39;gapminder_latest_mean_lifeExp_py&#39;, &#39;gapminder_latest_median_lifeExp_py&#39;, &#39;gapminder_latest_min_lifeExp_py&#39;, &#39;gapminder_latest_py&#39;, &#39;gapminder_latest_shape&#39;, &#39;gapminder_latest_stdev_lifeExp_py&#39;, &#39;lifeExpHist&#39;, &#39;np&#39;, &#39;pd&#39;, &#39;px&#39;, &#39;r&#39;, &#39;st&#39;, &#39;sys&#39;, &#39;variable&#39;, &#39;variable_grouping&#39;, &#39;variable_name&#39;]</pre><p>Great, among the present objects, we can clearly see the data (gapminder_latest_py) or libraries (e.g. px).</p><p>So, let us explore the data a bit!</p><h3>Life Expectancy</h3><p>For the demonstration purposes, I will focus on the <a href="https://en.wikipedia.org/wiki/Life_expectancy">life expectancy</a> or the average number of years a person is expected to live.</p><h3>Descriptive statistics</h3><p>Let’s begin with calculating some <strong>descriptive statistics</strong> like <em>mean</em>, <em>median</em> or the <em>number of rows</em> in the data using <strong>Python</strong>:</p><pre><em># Descriptive statistics for the inline code in Python</em></pre><pre><em>## Data Frame Overview</em></pre><pre><em>### Number of rows</em><br>gapminder_latest_shape = gapminder_latest_py.shape[0] <br><em>### Number of distinct values within the life expectancy variable</em><br>gapminder_latest_count_py = gapminder_latest_py[&#39;lifeExp&#39;].nunique()</pre><pre><em>## Life Expectancy</em></pre><pre><em>### Median (Life Expectancy)</em><br>gapminder_latest_median_lifeExp_py = st.median(gapminder_latest_py[&#39;lifeExp&#39;]) <br><em>### Mean</em><br>gapminder_latest_mean_lifeExp_py = st.mean(gapminder_latest_py[&#39;lifeExp&#39;])<br><em>### Minimum</em><br>gapminder_latest_min_lifeExp_py = min(gapminder_latest_py[&#39;lifeExp&#39;]) <br><em>### Maximum</em><br>gapminder_latest_max_lifeExp_py = max(gapminder_latest_py[&#39;lifeExp&#39;])<br><em>### Standard deviation</em><br>gapminder_latest_stdev_lifeExp_py = st.stdev(gapminder_latest_py[&#39;lifeExp&#39;])</pre><p>Nice. Unfortunately, we are not able to use the <strong>Python objects</strong> for <strong>inline coding</strong>, one of the key features of <a href="https://en.wikipedia.org/wiki/Literate_programming#:~:text=Literate%20programming%20is%20a%20programming,source%20code%20can%20be%20generated."><strong>literate coding</strong></a> in <a href="https://rmarkdown.rstudio.com/lesson-4.html"><em>R Markdown</em></a>. So, if we want to use the results for inline codes, we need to transform the <strong>Python objects</strong> back to <strong>R</strong>:</p><pre><em># Descriptive statistics for the inline code in Python - transformed to R</em></pre><pre><em>## Data Frame Overview</em></pre><pre><em>## Number of rows</em><br>gapminder_latest_nrow_r = py$gapminder_latest_shape<br><em>### Number of distinct values within the life expectancy variable</em><br>gapminder_latest_count_r = py$gapminder_latest_count_py</pre><pre><em>## Life Expectancy</em></pre><pre><em>### Median (Life Expectancy)</em><br>gapminder_latest_median_lifeExp_r = py$gapminder_latest_median_lifeExp_py<br><em>### Mean</em><br>gapminder_latest_mean_lifeExp_r = py$gapminder_latest_mean_lifeExp_py<br><em>### Minimum</em><br>gapminder_latest_min_lifeExp_r = py$gapminder_latest_min_lifeExp_py<br><em>### Maximum</em><br>gapminder_latest_max_lifeExp_r = py$gapminder_latest_max_lifeExp_py<br><em>### Standard deviation</em><br>gapminder_latest_stdev_lifeExp_r = py$gapminder_latest_stdev_lifeExp_py</pre><p>So, what can we say about <strong>life expectancy</strong> in <strong>2007</strong>?</p><p>First of all, there were <strong>142 countries</strong> on the list. The <strong>minimum</strong> value of life expectancy was <strong>39.61</strong> years, the maximum <strong>82.6</strong> years.</p><p>The <strong>average</strong> value for life expectancy was <strong>67.01</strong> years and <strong>50%</strong> or <strong>median</strong> hope to live <strong>71.94</strong> years or more. Lastly, the standard deviation was <strong>12.07</strong> years.</p><h3>Graphs (using Plotly)</h3><p>Okay, let’s move to something else, like graphs.</p><p>For example, we can take a look at <em>how is the life expectancy distributed</em> across the globe using Plotly:</p><pre>fig = px.histogram(gapminder_latest_py, <em># package.function; Data Frame</em><br>                   x=&quot;lifeExp&quot;, <em># Variable on the X axis</em><br>                   range_x=(gapminder_latest_min_lifeExp_py, <br>                            gapminder_latest_max_lifeExp_py), <em># Minimum and maximum values for the X axis</em><br>                   labels={&#39;lifeExp&#39;:&#39;Life expectancy - in years&#39;}, <em># Naming of the interactive part</em><br>                   color_discrete_sequence=[&#39;#005C4E&#39;]) <em># Colour of fill </em></pre><pre>lifeExpHist = fig.update_layout(<br>  title=&quot;Figure 1. Life Expectancy in 2007 Across the Globe - in Years&quot;, <em># The name of the graph</em><br>  xaxis_title=&quot;Years&quot;, <em># X-axis title</em><br>  yaxis_title=&quot;Count&quot;, <em># Y-axis title</em><br>  font=dict( <em># &quot;css&quot;</em><br>    family=&quot;Roboto&quot;,<br>    size=12,<br>    color=&quot;#252A31&quot;<br>  ))</pre><pre>lifeExpHist.write_html(&quot;lifeExpHist.html&quot;) <em># Save the graph as a .html object</em></pre><p>Unfortunately, it is not possible to print interactive Plotly graphs in <strong>R Markdown</strong> via <strong>Python</strong>. Or, to be more precise, you will receive a Figure object by printing (e.g. print(lifeExpHist)) it:</p><pre>Figure({<br>    &#39;data&#39;: [{&#39;alignmentgroup&#39;: &#39;True&#39;,<br>              &#39;bingroup&#39;: &#39;x&#39;,<br>              &#39;hovertemplate&#39;: &#39;Life expectancy - in years=%{x}&lt;br&gt;count=%{y}&lt;extra&gt;&lt;/extra&gt;&#39;,<br>              &#39;legendgroup&#39;: &#39;&#39;,<br>              &#39;marker&#39;: {&#39;color&#39;: &#39;#005C4E&#39;},<br>              &#39;name&#39;: &#39;&#39;,<br>              &#39;offsetgroup&#39;: &#39;&#39;,<br>              &#39;orientation&#39;: &#39;v&#39;,<br>              &#39;showlegend&#39;: False,<br>              &#39;type&#39;: &#39;histogram&#39;,<br>              &#39;x&#39;: array([43, 76, 72, 42, 75, 81, 79, 75, 64, 79, 56, 65, 74, 50, 72, 73, 52, 49,<br>                          59, 50, 80, 44, 50, 78, 72, 72, 65, 46, 55, 78, 48, 75, 78, 76, 78, 54,<br>                          72, 74, 71, 71, 51, 58, 52, 79, 80, 56, 59, 79, 60, 79, 70, 56, 46, 60,<br>                          70, 82, 73, 81, 64, 70, 70, 59, 78, 80, 80, 72, 82, 72, 54, 67, 78, 77,<br>                          71, 42, 45, 73, 59, 48, 74, 54, 64, 72, 76, 66, 74, 71, 42, 62, 52, 63,<br>                          79, 80, 72, 56, 46, 80, 75, 65, 75, 71, 71, 71, 75, 78, 78, 76, 72, 46,<br>                          65, 72, 63, 74, 42, 79, 74, 77, 48, 49, 80, 72, 58, 39, 80, 81, 74, 78,<br>                          52, 70, 58, 69, 73, 71, 51, 79, 78, 76, 73, 74, 73, 62, 42, 43]),<br>              &#39;xaxis&#39;: &#39;x&#39;,<br>              &#39;yaxis&#39;: &#39;y&#39;}],<br>    &#39;layout&#39;: {&#39;barmode&#39;: &#39;relative&#39;,<br>               &#39;font&#39;: {&#39;color&#39;: &#39;#252A31&#39;, &#39;family&#39;: &#39;Roboto&#39;, &#39;size&#39;: 12},<br>               &#39;legend&#39;: {&#39;tracegroupgap&#39;: 0},<br>               &#39;margin&#39;: {&#39;t&#39;: 60},<br>               &#39;template&#39;: &#39;...&#39;,<br>               &#39;title&#39;: {&#39;text&#39;: &#39;Figure 1. Life Expectancy in 2007 Across the Globe - in Years&#39;},<br>               &#39;xaxis&#39;: {&#39;anchor&#39;: &#39;y&#39;, &#39;domain&#39;: [0.0, 1.0], &#39;range&#39;: [39.613, 82.603], &#39;title&#39;: {&#39;text&#39;: &#39;Years&#39;}},<br>               &#39;yaxis&#39;: {&#39;anchor&#39;: &#39;x&#39;, &#39;domain&#39;: [0.0, 1.0], &#39;title&#39;: {&#39;text&#39;: &#39;Count&#39;}}}<br>})</pre><p>So, we <strong>import</strong> the previously created <em>.html</em> file instead (e.g. using the includeHTML function from the htmltools package):</p><pre>htmltools::<strong>includeHTML</strong>(&quot;lifeExpHist.html&quot;) <em># Render the graph</em></pre><p>So, here comes a basic, yet <a href="https://www.data-must-flow.com/posts/python-in-r/lifeexphist">interactive histogram </a>made in the Python version of Plotly.</p><p>However, producing this graph in <strong>RStudio</strong> required quite a workaround. At the same time, a size of the graphs produced this way could easily be <strong>tens of MBs</strong>.</p><ul><li>Consequently, a .html report containing such graphs would require <strong>a lot of data to download</strong> for a reader and it would take <strong>more time to render the page</strong>.</li></ul><h3>Summary tables (using pandas)</h3><p>One of the common use-cases for pandas is to provide a data description. The respective code runs just fine. However, the output cannot be styled as you are used from styling in pandas.</p><ul><li>When speaking of pandas, we can easily create a summary table for the commonly used statistics like mean or standard deviation of <strong>life expectancy</strong> for <strong>individual continents</strong>:</li></ul><pre><em># Create a pandas Data Frame object containing the relevant variable,</em><br><em># conduct formatting.</em><br>  <br>  gapminder_latest_py[&#39;continent&#39;] = gapminder_latest_py[&#39;continent&#39;].astype(str)<br>  variable = &#39;continent&#39;<br>  variable_name = &#39;Continents&#39;<br>  <br>  gapminder_latest_py[&#39;lifeExp&#39;] = gapminder_latest_py[&#39;lifeExp&#39;].astype(int)<br>  variable_grouping = &#39;lifeExp&#39;<br>  <br>  contact_window_days = gapminder_latest_py.groupby([<br>                          pd.Grouper(key=variable)])\<br>                          [variable_grouping]\<br>                          .agg([&#39;count&#39;,<br>                                &#39;min&#39;,<br>                                &#39;mean&#39;,<br>                                &#39;median&#39;,<br>                                &#39;std&#39;,<br>                                &#39;max&#39;])\<br>                          .reset_index()<br>  <br>  contact_window_days_style = contact_window_days\<br>                          .rename({&#39;count&#39;: &#39;Count&#39;,<br>                                   &#39;median&#39;: &#39;Median&#39;,<br>                                   &#39;std&#39;: &#39;Standard Deviation&#39;,<br>                                   &#39;min&#39;: &#39;Minimum&#39;, <br>                                   &#39;max&#39;: &#39;Maximum&#39;,<br>                                   &#39;mean&#39;: &#39;Mean&#39;,}, axis=&#39;columns&#39;)</pre><p>Output:</p><pre>continent  Count  Minimum       Mean  Median  Standard Deviation  Maximum<br>  0    Africa     52       39  54.326923    52.0            9.644100       76<br>  1  Americas     25       60  73.040000    72.0            4.495183       80<br>  2      Asia     33       43  70.151515    72.0            7.984834       82<br>  3    Europe     30       71  77.100000    78.0            2.916658       81<br>  4   Oceania      2       80  80.500000    80.5            0.707107       81</pre><p>Also, note that the <strong>Python environment settings override those from R</strong>. For example, take a look at the number of digits for Mean or Standard Deviation. There are six digits instead of two set at the beginning.</p><h3>Closing remarks</h3><p>Okay, that’s enough for now. If you are hungry for more advanced things like unsupervised learning using scikit-learn <a href="https://scikit-learn.org/stable/">package</a> in <strong>R</strong>, take a <a href="https://www.r-bloggers.com/r-and-python-using-reticulate-to-get-the-best-of-both-worlds/">look</a>.</p><p>However, before closing this post, let me just say that if you think about switching to <strong>Python</strong> as such and using it often, consider IDE alternatives to <em>RStudio</em>.</p><p>Many analysts swear on <a href="https://jupyter.org/"><strong>Jupyter notebooks</strong></a> for the interactivity, integration of markdown or option to run code in various languages like R, Julia or JavaScript. <a href="https://jupyter.org/hub"><strong>JupyterHub</strong></a> is a platform based on Jupyter notebooks, adding version control. Usually, users run analyses in a containerised environment). Another take on interactivity and collaboration could be <a href="https://colab.research.google.com/notebooks/intro.ipynb#recent=true"><strong>Colab</strong></a>, basically, Jupyter notebooks running on Google Cloud.</p><p>Last but not least, there is a great piece of software called <a href="https://code.visualstudio.com/"><strong>Visual Studio Code</strong></a>. It not only allows you to create and run code in a plethora of languages or seamless flow between pure <strong>Python</strong> code and interactive <strong>Jupyter notebooks</strong>. And maybe even more importantly, it provides you with very efficient version control management (like Git integration and extensions). If you choose this IDE, you can set up <a href="https://stevenmortimer.com/setting-up-vs-code-for-python-development-like-rstudio/#settings-json-file">VS Code for Python development like RStudio</a>.</p><p>But no matter what path to Python you choose, don’t forget that it is a tool suitable for some situations and maybe not so suitable to others. Just like R. Try to leverage the best of it while being aware of the pros of different tools.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d03bec4b96b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/python-done-the-r-markdown-way-d03bec4b96b">Python done the R (Markdown) way</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with generative art in R]]></title>
            <link>https://medium.com/data-science/getting-started-with-generative-art-in-r-3bc50067d34b?source=rss-6e9400412127------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bc50067d34b</guid>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[art]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[r-programming-language]]></category>
            <dc:creator><![CDATA[Vít Gabrhel]]></dc:creator>
            <pubDate>Tue, 30 Jun 2020 16:42:38 GMT</pubDate>
            <atom:updated>2021-05-10T18:43:37.512Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting Started with Generative Art in R</h3><h4>Data can be not only useful but also beautiful. This article points to some of the starting points for those who want to explore generative art in R.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w5QrXOz6M_i-WTNvgsY_SQ.jpeg" /><figcaption>Rijksmuseum. Photo by <a href="https://www.data-must-flow.com/projects/photos/photos.html">Vit Gabrhel</a></figcaption></figure><h3>Introduction</h3><p>Generative art represents the involvement of a human-independent system (such as an algorithm) in a creative process. Such a system is a key element within the creative process, including its outputs.</p><p>Unsurprisingly, there are many “tools” you can use if you want to delve into generative art. In case you want to have <em>“hands-on”</em> experience in R, there are two basic directions you can go.</p><p>If you are a person proficient in working with code, you may write your code straight away. I dare to argue that this is <strong>the way</strong> of the <em>“individual”</em> expression as it allows for more flexibility, and, well, freedom.</p><p>However, getting to that point takes time. Luckily, you can begin experimenting with your pieces of generative art by using one of the existing packages. Not only it could be quite rewarding, but it also works perfectly for grasping the basics of the <em>“trade”</em>.</p><p>In this article, I am going to present to you some of the <em>“ready-to-use”</em> packages for making generative art. Also, I will tap into some of the examples of using general packages or libraries to arrive at the same goal.</p><h3>generativeart</h3><p>The first package I would like to present is called simply generativeart.</p><p>One of its key features is the way how it structures the output files. In order to get the output files, you have to specify the location where they will be stored. Also, it iterates on the provided formula that doesn’t need to be supported with a seed. In this scenario, each of the outcome files is unique.</p><p>Also, it is worth mentioning that the generativeart package is well <a href="https://github.com/cutterkom/generativeart">documented</a> and easy to use.</p><pre><strong>library</strong>(generativeart) <em># devtools::install_github(&quot;cutterkom/generativeart&quot;)</em><br><strong>library</strong>(ambient)<br><strong>library</strong>(dplyr)</pre><pre><em># set the paths</em><br>IMG_DIR &lt;- &quot;img/&quot;<br>IMG_SUBDIR &lt;- &quot;everything/&quot;<br>IMG_SUBDIR2 &lt;- &quot;handpicked/&quot;<br>IMG_PATH &lt;- <strong>paste0</strong>(IMG_DIR, <br>                   IMG_SUBDIR)</pre><pre>LOGFILE_DIR &lt;- &quot;logfile/&quot;<br>LOGFILE &lt;- &quot;logfile.csv&quot;<br>LOGFILE_PATH &lt;- <strong>paste0</strong>(LOGFILE_DIR, <br>                       LOGFILE)</pre><pre><em># create the directory structure</em><br>generativeart::<strong>setup_directories</strong>(IMG_DIR, <br>                                 IMG_SUBDIR, <br>                                 IMG_SUBDIR2, <br>                                 LOGFILE_DIR)<br><em># include a specific formula, for example:</em><br>my_formula &lt;- <strong>list</strong>(<br>  x = <strong>quote</strong>(<strong>runif</strong>(1, -1, 10) * x_i^2 - <strong>sin</strong>(y_i^2)),<br>  y = <strong>quote</strong>(<strong>runif</strong>(1, -1, 10) * y_i^3 - <strong>cos</strong>(x_i^2) * y_i^4)<br>)</pre><pre><em># call the main function to create five images with a polar coordinate system</em><br>generativeart::<strong>generate_img</strong>(formula = my_formula, <br>                            nr_of_img = 5, <em># set the number of iterations</em><br>                            polar = TRUE, <br>                            filetype = &quot;png&quot;, <br>                            color = &quot;#c1a06e&quot;, <br>                            background_color = &quot;#1a3657&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vj0rJD5-ITEvwDYI.png" /></figure><h3>jasmines</h3><p>Danielle Navarro, the author of the Learning Statistics with R <a href="https://learningstatisticswithr.com/">Learning Statistics with R</a> handbook, is a well-known person among psychology students.</p><p>Besides her teaching and research activities at the University of Adelaide, she also released the <a href="https://github.com/djnavarro/jasmines">jasmines</a> package, a tool allowing for creating a generative art in R.</p><p>This package gives you an opportunity to play with simulation parameters (e.g. grain or interaction), shapes (e.g. entity_circle or scene_discs) and their modifications (like style_ribbon) or colours (palette or alpha). And noise, linked to the <a href="https://github.com/thomasp85/ambient">ambient</a> library:</p><pre><strong>library</strong>(dplyr) <em># or install.packages(&quot;dplyr&quot;) first</em><br><strong>library</strong>(jasmines) <em># or devtools::install_github(&quot;djnavarro/jasmines&quot;)</em></pre><pre>p0 &lt;- <strong>use_seed</strong>(100) %&gt;% <em># Set the seed of R‘s random number generator, which is useful for creating simulations or random objects that can be reproduced.</em><br>  <strong>scene_discs</strong>(<br>    rings = 10, <br>    points = 50000, <br>    size = 50<br>  ) %&gt;%<br>  <strong>mutate</strong>(ind = 1:<strong>n</strong>()) %&gt;%<br>  <strong>unfold_warp</strong>(<br>    iterations = 10,<br>    scale = .5, <br>    output = &quot;layer&quot; <br>  ) %&gt;%<br>  <strong>unfold_tempest</strong>(<br>    iterations = 5,<br>    scale = .01<br>  ) %&gt;%<br>  <strong>style_ribbon</strong>(<br>    color = &quot;#E0542E&quot;,<br>    colour = &quot;ind&quot;,<br>    alpha = <strong>c</strong>(1,1),<br>    background = &quot;#4D7186&quot;<br>  )</pre><pre><strong>ggsave</strong>(&quot;p0.png&quot;, p0, width = 20, height = 20, units = &quot;in&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MIM5rO9p01Rz6j5l.png" /><figcaption>Image by Author</figcaption></figure><h3>mathart + ggart</h3><p>The mathart package wraps many commonly used algorithms into functions. One of such examples is the <a href="https://github.com/marcusvolz/mathart/blob/master/R/k_nearest_neighbour_graph.R">nearest neighbour graph</a>, a visualisation of the <a href="https://en.wikipedia.org/wiki/K-d_tree">k-d tree</a>, a data structuring procedure of multidimensional space:</p><pre><strong>library</strong>(mathart) <em># devtools::install_github(&quot;marcusvolz/mathart&quot;)</em><br><strong>library</strong>(ggart) <em># devtools::install_github(&quot;marcusvolz/ggart&quot;)</em><br><strong>library</strong>(ggforce)<br><strong>library</strong>(Rcpp)<br><strong>library</strong>(tidyverse)</pre><pre>points &lt;- mathart::points</pre><pre>result &lt;- <strong>kdtree</strong>(points)<br>p1 &lt;- <strong>ggplot</strong>() +<br>  <strong>geom_segment</strong>(<strong>aes</strong>(x, y, xend = xend, yend = yend), result) +<br>  <strong>coord_equal</strong>() +<br>  <strong>xlim</strong>(0, 10000) + <strong>ylim</strong>(0, 10000) +<br>  <strong>theme_blankcanvas</strong>(bg_col = &quot;#fafafa&quot;, margin_cm = 0)</pre><pre><em># save plot</em><br><strong>ggsave</strong>(&quot;kdtree.png&quot;, p1, width = 20, height = 20, units = &quot;in&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VKnzuL0dNho5QR0s.png" /><figcaption>Image by Author</figcaption></figure><p>mathart is a well-documented package. This fact widens the scope for experimenting. Maybe more importantly, by looking <em>“under the hood”</em>, the package improves understanding of data simulation.</p><h3>General-purpose packages</h3><p>In many instances, previously mentioned packages built on more generally used packages such as magritr (the pipeline operator comes in handy even in the art creation), ggplot, dplyr, or purr.</p><p>The <strong>k-d tree</strong> from the mathart package is no different. As we can see from the example of <a href="https://github.com/marcusvolz/metropolis/blob/master/metropolis.R">Marcus Volz</a>, the original k-d tree algorithm can be tweaked using even base R:</p><pre><em># Metropolis: Generative city visualisations</em></pre><pre><em># Packages</em><br><strong>library</strong>(ggart)<br><strong>library</strong>(tidyverse)<br><strong>library</strong>(tweenr)<br><strong>library</strong>(viridis)</pre><pre><em># Make reproducible</em><br><strong>set.seed</strong>(10001)</pre><pre><em># Parameters</em><br>n &lt;- 10000 <em># iterations</em><br>r &lt;- 75 <em># neighbourhood</em><br>width &lt;- 10000 <em># canvas width</em><br>height &lt;- 10000 <em># canvas height</em><br>delta &lt;- 2 * pi / 180 <em># angle direction noise</em><br>p_branch &lt;- 0.1 <em># probability of branching</em><br>initial_pts &lt;- 3 <em># number of initial points</em><br>nframes &lt;- 500 <em># number of tweenr frames</em></pre><pre><em># Initialise data frames</em><br>points &lt;- <strong>data.frame</strong>(x = <strong>numeric</strong>(n), y = <strong>numeric</strong>(n), dir = <strong>numeric</strong>(n), level = <strong>integer</strong>(n))<br>edges &lt;-  <strong>data.frame</strong>(x = <strong>numeric</strong>(n), y = <strong>numeric</strong>(n), xend = <strong>numeric</strong>(n), yend = <strong>numeric</strong>(n), level = <strong>integer</strong>(n))</pre><pre><strong>if</strong>(initial_pts &gt; 1) {<br>  i &lt;- 2<br>  <strong>while</strong>(i &lt;= initial_pts) {<br>    points[i, ] &lt;- <strong>c</strong>(<strong>runif</strong>(1, 0, width), <strong>runif</strong>(1, 0, height), <strong>runif</strong>(1, -2*pi, 2*pi), 1)<br>    i &lt;- i + 1<br>  }<br>}</pre><pre>t0 &lt;- <strong>Sys.time</strong>()</pre><pre><em># Main loop ----</em><br>i &lt;- initial_pts + 1<br><strong>while</strong> (i &lt;= n) {<br>  valid &lt;- FALSE<br>  <strong>while</strong> (!valid) {<br>    random_point &lt;- <strong>sample_n</strong>(points[<strong>seq</strong>(1:(i-1)), ], 1) <em># Pick a point at random</em><br>    branch &lt;- <strong>ifelse</strong>(<strong>runif</strong>(1, 0, 1) &lt;= p_branch, TRUE, FALSE)<br>    alpha &lt;- random_point$dir[1] + <strong>runif</strong>(1, -(delta), delta) + (branch * (<strong>ifelse</strong>(<strong>runif</strong>(1, 0, 1) &lt; 0.5, -1, 1) * pi/2))<br>    v &lt;- <strong>c</strong>(<strong>cos</strong>(alpha), <strong>sin</strong>(alpha)) * r * (1 + 1 / <strong>ifelse</strong>(branch, random_point$level[1]+1, random_point$level[1])) <em># Create directional vector</em><br>    xj &lt;- random_point$x[1] + v[1]<br>    yj &lt;- random_point$y[1] + v[2]<br>    lvl &lt;- random_point$level[1]<br>    lvl_new &lt;- <strong>ifelse</strong>(branch, lvl+1, lvl)<br>    <strong>if</strong>(xj &lt; 0 | xj &gt; width | yj &lt; 0 | yj &gt; height) {<br>      <strong>next</strong><br>    }<br>    points_dist &lt;- points %&gt;% <strong>mutate</strong>(d = <strong>sqrt</strong>((xj - x)^2 + (yj - y)^2))<br>    <strong>if</strong> (<strong>min</strong>(points_dist$d) &gt;= 1 * r) {<br>      points[i, ] &lt;- <strong>c</strong>(xj, yj, alpha, lvl_new)<br>      edges[i, ] &lt;- <strong>c</strong>(xj, yj, random_point$x[1], random_point$y[1], lvl_new)<br>      <em># Add a building if possible</em><br>      buiding &lt;- 1<br>      valid &lt;- TRUE<br>    }<br>  }<br>  i &lt;- i + 1<br>  <strong>print</strong>(i)<br>}<br>edges &lt;- edges %&gt;% <strong>filter</strong>(level &gt; 0)</pre><pre>sand &lt;- <strong>data.frame</strong>(alpha = <strong>numeric</strong>(0), x = <strong>numeric</strong>(0), y = <strong>numeric</strong>(0))<br>perp &lt;- <strong>data.frame</strong>(x = <strong>numeric</strong>(0), y = <strong>numeric</strong>(0), xend = <strong>numeric</strong>(0), yend = <strong>numeric</strong>(0))</pre><pre><em># Create plot</em><br>p2 &lt;- <strong>ggplot</strong>() +<br>  <strong>geom_segment</strong>(<strong>aes</strong>(x, y, xend = xend, yend = yend, size = -level), edges, lineend = &quot;round&quot;) +<br>  <em>#geom_segment(aes(x, y, xend = xend, yend = yend), perp, lineend = &quot;round&quot;, alpha = 0.15) +</em><br>  <em>#geom_point(aes(x, y), points) +</em><br>  <em>#geom_point(aes(x, y), sand, size = 0.05, alpha = 0.05, colour = &quot;black&quot;) +</em><br>  <strong>xlim</strong>(0, 10000) +<br>  <strong>ylim</strong>(0, 10000) +<br>  <strong>coord_equal</strong>() +<br>  <strong>scale_size_continuous</strong>(range = <strong>c</strong>(0.5, 0.5)) +<br>  <em>#scale_color_viridis() +</em><br>  <strong>theme_blankcanvas</strong>(bg_col = &quot;#fafafa&quot;, margin_cm = 0)</pre><pre><em># print plot</em><br><strong>ggsave</strong>(&quot;plot007w.png&quot;, p2, width = 20, height = 20, units = &quot;cm&quot;, dpi = 300)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YeLJmbLfm-8KYlQ9.png" /><figcaption>Image by Author</figcaption></figure><p>You don’t need to stop here. Some time spent playing with the code can easily end up by creating your distinct <a href="https://www.r-bloggers.com/generative-art-let-your-computer-design-you-a-painting/">wall poster</a>.</p><h3>Closing remarks</h3><p>Generative art demonstrates that code can be beautiful. If you are still not persuaded, go and see more elaborated works like <a href="https://www.data-imaginist.com/art">these</a>.</p><p>Not only can generative art result in something astonishing, but it also helps you to understand the code itself. Interestingly, getting to the code underlying generative art may not be as easy as with other open-source areas. Some <em>“generative artists”</em> conclude not to share the code used in the creative process:</p><blockquote>“I do not share the code I use to create my pieces. The main reason for this is that I don’t think it would be beneficial to anyone. People interested in getting started with generative art would become to focused on my ideas instead of developing their own. Knowing the answer is the killer of creativity.”</blockquote><blockquote>Thomas Lin Pedersen, <a href="https://www.data-imaginist.com/art">Data Imaginist</a></blockquote><p>Regardless if we agree or not, generative art confronts us with ideas that are not that common in the world of coding.</p><p>So, what are your ideas on the matter? Do you have any favourite generative artists? Or maybe your pieces of art you would like to share? Feel free to discuss this on <a href="https://www.data-must-flow.com/posts/intro-generative-art-r/"><strong>Twitter</strong></a>.</p><p><em>Originally published at </em><a href="https://www.data-must-flow.com/about/about.html"><em>https://www.data-must-flow.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bc50067d34b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/getting-started-with-generative-art-in-r-3bc50067d34b">Getting started with generative art in R</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Must Flow]]></title>
            <link>https://vitgabrhel.medium.com/data-must-flow-83d57052a324?source=rss-6e9400412127------2</link>
            <guid isPermaLink="false">https://medium.com/p/83d57052a324</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Vít Gabrhel]]></dc:creator>
            <pubDate>Tue, 23 Jun 2020 06:29:10 GMT</pubDate>
            <atom:updated>2020-06-23T06:29:10.216Z</atom:updated>
            <content:encoded><![CDATA[<p>My name is Vit Gabrhel, and I am a <strong>data enthusiast</strong>.</p><p>I started my data journey as a researcher in the mobility field, but nowadays, I envision myself on the road of becoming a <em>“full-stack data scientist”</em>.</p><p>I believe that data could serve not only as a way to <strong>understand the world</strong> but also to <strong>appreciate its beauty</strong>. Just take a look at the following example of the generative art located bellow. It is based solely on a data simulation, a <a href="https://www.data-must-flow.com/posts/intro-generative-art-r/#mathart_+_ggart">k-d tree</a> algorithm.</p><p>Also, I am convinced that <strong>insights from data can be transformed into action</strong> and so benefit many. May it be an <em>evidence-based policy</em> in public health, or an <em>application</em> like mobility manager making lives easier. For that to be possible, data must be available; <strong>data must flow</strong>. Hence the name of the page.</p><p>This web serves as a space to present my work and ideas. Here, you may find <strong>blog posts</strong> on <strong>data projects</strong>, be that applications, analyses from various domains such as <em>mobility</em> or <em>culture</em>, or tools ( <strong>R</strong> and <strong>Python</strong>). Also, feel free to refresh yourself by some of my takes on <a href="https://www.data-must-flow.com/projects/generativeart/generativeart.html"><strong>generative art</strong></a> or urban and landscape <a href="https://www.data-must-flow.com/projects/photos/photos.html"><strong>photography</strong></a>.</p><p>Have questions or suggestions? Feel free to <a href="https://github.com/VGabrhel">open an issue on <strong>GitHub</strong></a> or <a href="https://twitter.com/GabrhelVit">ask me on <strong>Twitter</strong></a>.</p><p>Thanks for reading!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M_5gesZ85MxIsusq.png" /></figure><p><em>Originally published at </em><a href="https://www.data-must-flow.com/about/about.html"><em>https://www.data-must-flow.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83d57052a324" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>