<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by AlfonsoRReyes on Medium]]></title>
        <description><![CDATA[Stories by AlfonsoRReyes on Medium]]></description>
        <link>https://medium.com/@oilgains?source=rss-5c397fc2ba37------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*zOU9Zd1stR7L4fOL.png</url>
            <title>Stories by AlfonsoRReyes on Medium</title>
            <link>https://medium.com/@oilgains?source=rss-5c397fc2ba37------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:39:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@oilgains/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Transforming [Petroleum] Engineers in Data Science Wizards. Update 2020]]></title>
            <link>https://oilgains.medium.com/transforming-petroleum-engineers-in-data-science-wizards-update-2020-b4c6dd631593?source=rss-5c397fc2ba37------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4c6dd631593</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reproducibility]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[AlfonsoRReyes]]></dc:creator>
            <pubDate>Wed, 30 Dec 2020 19:08:01 GMT</pubDate>
            <atom:updated>2020-12-31T15:30:52.798Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cFxt-kzGtbC2boAlXH5NYg.jpeg" /></figure><p><strong>Note</strong>. This is an update of the original article I published in 2017 and 2019. So many things have changed, and so many new things I have learned, that this article needs some refreshing to reflect those new experiences. This year, 2020, I set a new frontier: learning about <a href="https://www.britannica.com/technology/artificial-intelligence">Artificial Intelligence</a>. But from books, not the internet. Building artificial intelligence agents requires understanding of what AI is.</p><p>First of all, on the title of the article. I am enclosing “Petroleum” in brackets because that was the original title of the article. But I came to realize that it goes beyond a narrow engineering field; it may be useful to other engineers as well.</p><p>These are not absolute truths written in stone. I am sharing experiences, recommendations of something that worked for me and others. Part of them are collection of notes or posts that I have shared with colleagues when asked me:</p><ul><li>“Alfonso, how do I start with Data Science?”,</li><li>“Is there any value in learning Data Science?”,</li><li>“What impact could Data Science bring to my engineering job to improve processes, increase revenues, or reduce costs?”</li><li>“Should I attend a Data Science career in an established university?”</li></ul><p>It is difficult to find out if you don’t apply Data Science in the the real world. I was an skeptic until I took the decision to seriously learn it. I don’t mean one hour a week or a day; I meant living it, and making it part of my engineering process at work. It is funny, because my team was a bit reluctant at changing the classic way of solving well models to optimize well production. Then after, we wouldn’t dare to analyze an oil field without first applying statistics, creating datasets from input files, simulation, or solver outputs, and look at the resulting data with a new perspective. Later on, our group, <em>Wells and Network Modeling</em>, was made one of the pillars, in the overall company strategy, at bringing extra revenue in challenging times.</p><p>In many ways, what I will be describing you may resemble a scientific approach to things. Mind you that the word “science” is not gratuitously in “data science”. Data science comes from the world of Statistics when Computer Science made it a practical tool at making discoveries from data. We owe Data Science, first and foremost, to Statisticians, to scientists. This new industrial revolution, based on data, we are living in, requires a new set of lenses to understand, and discover things that are not immediately evident using the classical methods. Of course it will be difficult, no doubt you will meet resistance, but what human endeavor that is worth, doesn’t.</p><p>I don’t particularly buy in this new concept of “physics-driven” vs “data-driven” methods. I’ll tell you why. First, you will never hear a scientist say “I will apply a data-driven methodology to explain this phenomena”. Or, “I will try this time with a huge amount of data to discover things”. You would be laughed out of the department. Second, data is a manifestation of natural phenomena; you cannot separate data from the physics. They are both faces of the same coin. Third, go look at the book classics on physics, fluid mechanics, chemistry, geology, [petroleum] engineering, simulation, biology, etc. You will not find an instance at which the authors say “with more data I would get a better explanation of the phenomena.” More data doesn’t necessarily mean new data, ground breaking data, or unique data. It just means that you have more. It is up to you to filter out the noise and hear what the physical phenomena is trying to tell you.</p><p>You may take these as some recipes to start your transformation in a <strong>Data Science</strong> wizard:</p><ol><li>Complete any of the Python or R <strong>online</strong> courses on Data Science. My favorites are the ones from Johns Hopkins and the University of Michigan in Coursera (<a href="https://www.coursera.org/specializations/jhu-data-science">Data Science Specializations in R</a> or Introduction to <a href="https://www.coursera.org/learn/python-data-analysis">Data Science in Python</a>). Don’t be mistaken: the data science specialization in R is a high quality course, and would make you feel sometimes like going through a <strong>PhD</strong> program. You will need a firm commitment and set aside some time for lectures, quizzes and project assignments. You could complement it with DataCamp short workshops. For instance, I started, few years ago, with the two-hour “Introduction to R” quick course. And then never stopped. There are other online institutions such as edX, Udacity, Udemy, etc. You will also be able to find online courses from reputable universities such as Stanford, MIT, or Harvard. If you don’t have previous programming experience, start with <strong>Python</strong>; if you feel confident about your programming skills, and would like breaking the barrier between engineering and science, go full throttle with <strong>R</strong>. If you are a MSc or PhD, you know I will immediately recommend you to pick R.</li><li>Start using <strong>Git</strong> as much as possible in all your projects. It is useful for sharing and maintaining code, text, papers, working in teams, synchronize projects, work in different computers, bring<strong> reproducibility</strong> to your data science projects, etc. To access Git in the cloud you may use GitHub, Bitbucket, or GitLab. Don’t be frustrated if you don’t get it or understand it at first; <strong>everybody</strong> struggles with Git. <strong>Even PhDs</strong>, who, by the way, have written the best tutorials on Git. So, you are not alone. Think of Git as a <strong>time machine</strong> that will let you go backward and forward in the history of your document.</li><li>In parallel with Git you will need to learn the basics of the <strong>Unix</strong> terminal. This is useful for many things that Windows doesn’t do -and might never do; it is even useful for Linux and Macs which are Unix based as well. You can do <strong>automated scripting</strong> using the Unix terminal that can serve you in many data oriented activities, such as operations on huge datasets, deployment, backups, file transfer, manage remote computers, secure transfer, low level settings of your work environment, version control with Git, reproduce tasks, find keywords in text and PDF documents, etc. If you are a Windows guy, there are ways to get familiar with the Unix terminal. There are few hybrid Windows applications such as <strong>Git-Bash</strong>, <strong>MSYS2</strong>, or <strong>Cygwin</strong>, that let you bring a Unix terminal to Windows. There is no question about that you have to know the Unix terminal. It makes your data science much more powerful and <strong>reproducible, </strong>giving you also avenues for easier deployment. Additionally, it may be your secret weapon when dealing with huge datasets. I am finding more frequently articles where engineers have managed to read and transform <strong>terabyte-</strong>size datasets, in laptops, using combination of Unix command line tools such as <em>grep</em>, <em>awk</em>, <em>sed</em>, etc., along with data.frame and data.table structures. No need of big-data computer clusters with Hadoop or Spark, which are more more difficult to handle. I know there is also <a href="https://docs.microsoft.com/en-us/windows/wsl/about">Windows Subsystem for Linux</a> (WSL) but may require a little bit of hackery. I have tested it a couple of times and is kind of hard to benchmarking it to other Linuxes when you run a data science or machine learning project.</li><li>One of the best tools for reproducibility is <a href="https://rmarkdown.rstudio.com/"><strong>Rmarkdown</strong></a>. As soon as you have installed R, Rtools, and RStudio in your computer, start using <a href="https://www.markdownguide.org/"><strong>Markdown</strong></a>. In R is called <strong>Rmarkdown</strong>, which is widely used in science for generating documentation, papers, citations, booklets, manuals, tutorials, schematics, diagrams, web pages, blogs, slides, etc. Make a habit in using Markdown. If possible, during engineering work, avoid Word -which generates mostly binary files. Working with Markdown makes easier to do revision control and it is <strong>reproducible</strong>, both, key to reliable, testable, traceable, repeatable data science. With markdown, you can also embed Latex equations with text, code and calculations. Besides you gain an additional ecosystem to run code and tools from the <strong>Latex</strong> universe, which is enormous. My favorite application for Markdown is <a href="https://typora.io/">Typora</a>. It works under Windows, Linux and macOS. I use it every day.</li><li>Strive to publish your engineering results using Markdown. It will complement your efforts of batch automation, data science and machine learning. Combine <strong>calculations with code</strong> and text (called <a href="http://www.literateprogramming.com/index.html">literate programming</a>) using the Rmarkdown notebooks in R. Essentially, any document can be written mixing code, text, graphics and calculations with R or Python. Even though I am originally a Python guy (15+ years), I am not strongly recommending the Python notebooks, or Jupyter, because they are not 100% human readable text (it uses JSON), that you may find difficult to apply version control and apply reproducible practices, or using it with Git. I have possible built more than a thousand Jupyter notebooks but when I learned Rmarkdown, it was like stepping in another dimension. In case you haven’t heard, <em>R</em> and <em>RStudio</em> have update packages, such as <em>reticulate</em>, that allows you to write <em>Python</em> code in <em>Rmarkdown</em>. I have published several examples of data science books completely coded in Python using Rmarkdown. This is for <a href="https://github.com/f0nzie/rmarkdown-python-plotnine">Python plotnine</a>, this is <a href="https://github.com/f0nzie/matplotlib-with-rmarkdown">introduction to Matplotlib written in Rmarkdown</a>, and <a href="https://github.com/f0nzie/yongks-python-rmarkdown-book">this other</a> is a tutorial by Keh-Soo Yong, that I modified; also entirely written in Python using Rmarkdown notebooks.</li><li>Start bringing your data into <strong>datasets</strong> with assistance of R or Python. Build your favorite collections of datasets. Share with colleagues in the office and discuss the challenges in converting raw data into tidy data. Generate tables, plots and statistical reports to come up with discoveries. Use markdown to document the variables or features (columns). If you want to share the data, keeping the confidentiality, learn how to anonymize your data with R or Python cryptographic or scrambling packages. There is another way to anonymize confidential data and that is <a href="https://en.wikipedia.org/wiki/Generative_adversarial_network">Generative Adversarial Networks </a>(GANs). I am still working on a package that will use <a href="https://pytorch.org/">PyTorch</a> to transform a data structure with 2D tensors into a new dataset with similar statistical properties but completely hard to trace to the original source.</li><li>Data Science requires continuous deliberate practice. Start solving daily <strong>engineering</strong> problems with R or Python incorporating them in your daily workflow. If you can, avoid Excel or Excel-VBA, if possible. VBA original purpose was never to deal with version control, or reproducibility, or data science; much less, machine learning. Sticking to Office tools may keep you stuck to outdated practices, or being unable to perform a much richer and productive data science. There is one more thing you may have possible noticed, and that is Excel plots are very simplistic; they go back to 30 years old techniques. You may run the risk of dumbing down your analysis, or prevent you of making discoveries from your data, or showing a compelling story, which is the purpose of data science anyway.</li><li>Learn and apply <strong>statistics</strong> everywhere; every time you can, on all engineering activities you perform. Find what no other person can by using math, physics and statistics. Data Science is about making discoveries and answering questions on the data. Data Science was invented by statisticians; who at that time they called it “data analysis”. An article I never get tired of read and re-read is this “<a href="https://www.tandfonline.com/doi/pdf/10.1080/10618600.2017.1384734?needAccess=true">50 years of Data Science by David Donohoe</a>”. Please, read it. It will explain statistics and its tempestuous, albeit tight, relationship with data science.</li><li>Read what other <strong>disciplines outside</strong> yours are doing in data science and machine learning. Look at <strong>bioscience</strong>, <strong>biostatistics</strong>, <strong>genetics</strong>, robotics, medicine, medical imaging, cancer research, pharmaco-kinetics, psychology, <strong>biology</strong>, ecology, hydrology, high performance computing, spatial data, automotive, <strong>finance</strong>, etc. You may want to take a brief look by browsing the <a href="https://cran.r-project.org/">R packages catalog</a> at CRAN.</li><li><strong>Read</strong> articles in the net on data science and try to reproduce those that resemble solving a problem you might find at work. It doesn’t matter if it is Python or R. You just have to learn what <strong>data science</strong> is about; how it could bring value to your everyday engineering workflow. They may give you ideas of applications involving data in your area of expertise. They may not even be data science per-se now, but they most likely could be your next stepping stone. Furthermore, most of the articles are <strong>free</strong> as well, and so, hundreds of books, booklets, tutorials and papers. We never had the chance to learn so much for so little. Somebody has call this the era of <em>democratization of knowledge and information</em>. <strong>What you have to invest is time</strong>.</li><li>The next stepping stone while learning Data Science is <strong>Machine Learning</strong>. Start inquiring about what machine learning is about. There are many online resources to learn it, and lots of open source libraries that you could use. In Python you have NumPy, ScikitLearn, TensorFlow, PyTorch, mxNet, NLTK, and others. In R you also have plenty of options: carrot, TensorFlow, <a href="https://github.com/f0nzie/rTorch">rTorch</a>, mxNet, e1071, gbm, glmnet, h2o, kernlab, rpart, and more. Although, there are several machine learning algorithms that are used in data science, machine learning is a discipline by its own. Most of the successes that are attributed to <em>artificial intelligence</em> today are really <em>machine learning</em> applications. What I recommend you here is to make <strong>collections</strong> of <strong>ML algorithms</strong> to give you a rapid intuition on when an algorithm should be used. Not all problems in the world will be solved with Neural Networks or Deep Learning. You would be amazed how classic algorithms still outperform them. Keep the <strong>hype</strong> at arm’s length.</li><li>Review <strong>C++</strong> and <strong>Fortran</strong> scientific code. I don’t mean to say that you need to learn another programming language, but knowing what they can do will add power to your toolbox. Sooner or later you will need Fortran, C, or C++ for reasons of efficiency and speed; most likely <strong>deployment</strong>. Not for nothing the best in class simulators and optimizers of today have plenty of Fortran routines under the hood.</li><li>Learn how to read from different <strong>file formats</strong>. It is amazing the enormous variety of file formats in what you may find raw data. There is a lot of value that you could bring to your daily activities by automating your data analysis workflow using R or Python. Also, ask what are the different data formats that are used in your <strong>company</strong> for storing data. Get familiar with them. If you are in [petroleum] engineering, try reading some chunks of that data: try with logs, seismic, well tests, buildups, drilling reports, deviation surveys, geological data, process data, simulation output, etc. Create tidy datasets out of them. Explore the data. Embark in finding and discover things.</li><li>Something that is more challenging is learning how to read and transform <strong>unstructured</strong> data, meaning, data that is not in row-column (rectangular) format. The typical and close cases to us are the text output from simulators, optimizers, stimulation or well design, etc. This is one of the most difficult data to operate with, and when learning or knowing “<strong>regex</strong>” really pays off. There is a more complex side of <strong>unstructured</strong> data, and that is video, sound, and images. Today there are plenty of algorithms available that deal with that kind of data either with Matlab, Python or R. Images is one of the areas that has been widely explored with Neural Networks, Deep Learning, Convolutional Neural Networks, and GANs. They are quite effective at showing visible and compelling results.</li><li>Virtualization. Learn something about <strong>Virtual Machines</strong> with VirtualBox (free) or Vmware. It is very useful to have several operating systems working at the same time in your PC: Windows, Linux, MacOS. There is a lot of good data science and machine learning stuff in Linux packaged as VMs, which could be run under Windows very easily. These are applications that are ready to run without the need of installing anything on the physical machine. Some time ago, I was able to download a couple of <strong>Linux VM</strong> with whole bunch of machine learning and <strong>artificial intelligence</strong> applications, and test them with minimum effort. I have other VMs from <em>Cloudera</em> and <em>Horton-Works</em> where I was able to run <strong>big-data</strong> applications such as <em>Hadoop</em>, <em>Spark</em>, etc. Another virtualization tool that you may want to learn is Docker <strong>containers</strong>. The concept is similar to that of virtual machines but lighter and less resource intensive. Another tool you may want to explore in virtualization is <strong>Vagrant</strong>, which is an advanced combination of virtual machines and containers. These tools will make your data science even more reproducible and stand the test of time.</li><li>Dare to explore the world of <strong>artificial intelligence</strong>. You may not need it today but will be the indispensable tool of the <strong>5th Industrial Revolution</strong>. There is nothing better than knowing, at least, the fundamentals of what others are <strong>trying to sell you</strong>. There is so much noise, and snake-oil marketing nowadays surrounding the words “machine learning” and “artificial intelligence”. To start, three books I would recommend, out of the top of my head, on <strong>artificial intelligence</strong>, are: <em>“Artificial Intelligence: A New Synthesis”</em> by Nils Nilsson; <em>“Computational Intelligence. A logical approach”</em> by David Poole, Alan Mackworth and Randy Goebel; and <em>“Artificial Intelligence: A Modern Approach”</em> by Russell and Norvig. You will find that <strong>AI</strong> is not what you read in newspapers, magazines or articles in the web. An elucidating article about Machine Learning vs Artificial Intelligence is “<a href="https://hdsr.mitpress.mit.edu/pub/wot7mkc1/release/9">Artificial Intelligence:The Revolution Hasn’t Happened Yet” by Prof. Michael I. Jordan,</a> which in one sentence tells you all the hype around AI is not artificial intelligence but machine learning. Indeed, almost all of the successful applications in use today are ML. Couple of days ago, I woke up with the idea of separating the field of Artificial Intelligence in two branches: <strong>Internet AI</strong> vs <strong>Scientific AI</strong>. Internet AI would be the “astrology” side of Scientific AI. The hype, “AI did this”, “AI did that”, all the pseudo-science articles. The robot Sophia would inhabit this fantasy world. <strong>Scientific AI</strong> would be the AI we inherited from Alan Turing, John McCarthy, Alan Newell, Herbert Simon, Marvin Minsky, Noam Chomsky, John Hopfield, et al. In this world you cite your sources and are not afraid of being peer reviewed or criticized.</li><li>For those who have asked me if I recommend a formal data science degree at a university, what I tell them is try first with online courses and see if it is for you.</li></ol><p>Alfonso R. Reyes</p><p>Houston, Texas. 2020</p><p>Copyright Alfonso. R. Reyes 2020 Oil Gains Analytics LLC.</p><h3>References</h3><ul><li>My GitHub repository with open source code: <a href="https://github.com/f0nzie">link</a></li><li>My Rmarkdown blog: <a href="http://blog.oilgainsanalytics.com/">link</a></li><li>Original article 2019 in LinkedIn: <a href="https://www.linkedin.com/pulse/transforming-petroleum-engineers-data-science-wizards-reyes-1d/">link</a></li><li>Original article 2017 in LinkedIn: <a href="https://www.linkedin.com/pulse/transforming-petroleum-engineers-data-science-wizards-reyes">link</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4c6dd631593" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Machine Learning is not Artificial Intelligence?]]></title>
            <link>https://oilgains.medium.com/why-machine-learning-is-not-artificial-intelligence-61b174a3c9a2?source=rss-5c397fc2ba37------2</link>
            <guid isPermaLink="false">https://medium.com/p/61b174a3c9a2</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[AlfonsoRReyes]]></dc:creator>
            <pubDate>Fri, 18 Dec 2020 03:34:10 GMT</pubDate>
            <atom:updated>2020-12-18T16:16:52.486Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*ZQ4KO4K6fuQRvGYBgIYlaQ.png" /></figure><p>It is simple: because <strong>ML</strong> is only a contributing branch of <strong>AI</strong>. If we follow Norvig and Russell book -and other authors-, saying that <strong>machine learning</strong> is equivalent to <strong>artificial intelligence</strong> is grossly misleading. <strong>ML</strong> is a contributing discipline of <strong>AI</strong>, just like search, reasoning, planning, decision making, natural language processing, vision, and robotics.</p><p>For instance, <strong>ML</strong> by itself cannot be intelligent because lacks reasoning, planning, logic, and doesn’t interact with the environment. <strong>ML</strong> detects patterns and performs predictions based on statistical analysis of data using <em>math based algorithms</em>. These <strong>algorithms</strong> are not intelligent per se.</p><p>Intelligence is much more than that.</p><p>Stating that <strong>ML</strong> is part of <strong>AI</strong> dramatically lowers the bar of what John McCarthy meant by <strong>AI</strong>. Chapter two of the AIAMA (<a href="http://aima.cs.berkeley.edu/">Artificial Intelligence: A Modern Approach</a>) book -which counts five editions- contains examples and schematics of what an <strong>AI</strong> agent is. In essence, an <strong>AI agent</strong> perceives its environment by means of <strong>sensors</strong>, and acts upon the <strong>environment</strong> through <strong>actuators</strong>; in between these two, there is logic, reasoning, decision making that makes the agent act autonomously, correcting itself with no human intervention.</p><p>The figure is my crude attempt of explaining what the AI contributors are (brown).</p><h3>The view of AI by Peter Norvig and Stuart Russell (2010)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*_xwYVHQxUvkR8I-1.png" /></figure><h3>Background</h3><p>Our first instinct would bring us to say Machine Learning is a <strong>subset</strong> of Artificial intelligence. Then, starts getting fuzzy and nebulous when we find papers, articles, posts claiming to be on Artificial intelligence. You start reading them and turns out they are about Machine Learning or Data Science. At this point we just don’t know, if the author(s) made the mistake by <strong>ignorance</strong> (didn’t research or read enough literature), or purposely tried to <strong>deceive</strong> readers. <strong>Commercialism</strong> also plays a huge part in all this. Although, <strong>Machine Learning</strong> has had -still has- <strong>tremendous achievements</strong> to show, <strong>marketing</strong> and PR departments prefer to call it Artificial intelligence.</p><p>Books on artificial intelligence DO NOT do that. They are written by scientists -generally- with strong ethical codes.</p><p>So, today’s Artificial intelligence, Machine Learning, and Data Science atmosphere is charged with <strong>false</strong> stories, <strong>inflated</strong> achievements. That’s bad for all of us. Because in the end what we receive is pseudo-science.</p><p><strong>Science is hard</strong> — it’s supposed to be. Artificial intelligence, at the <strong>top</strong> of the food chain, was one of the ultimate frontiers. I’m sure <a href="http://jmc.stanford.edu/">John McCarthy</a> would not agree to qualify a linear regression, a neural network, or a bare robot, as being Artificial intelligence. Of course, they could be AI with the right combination of AI components. Machine Learning is only one of them.</p><h3>“You are spreading misinformation …”</h3><p>If I may invite you to read <a href="https://hdsr.mitpress.mit.edu/pub/wot7mkc1/release/8">“Artificial Intelligence — The Revolution Hasn’t Happened Yet” by Prof. Michael I. Jordan</a>. With that clarity that comes from a mixed background of engineering and science, Prof. Jordan diagnose the current state of AI. These are my main takeaways:</p><p>1- Clearly, there is a <strong>misunderstanding</strong> of the word Artificial Intelligence, not only by the general public but also <strong>scientists</strong>.</p><p>2- It is difficult to grasp the concept of AI because <strong>we are living it</strong>. But it’s time that universities come up with a career on <strong>AI Engineering</strong>.</p><p>3- Most of the <strong>successes</strong> we attribute to AI are actually <em>Machine Learning</em>. All technological leading companies use ML throughout all their business; read Google, Netflix, Facebook, Amazon, Twitter, Microsoft, IBM, etc.</p><p>4- AI originally was meant to focus on the <strong>cognitive capability</strong> of humans. After almost 70 years still remains elusive.</p><p>5- What we call AI today <strong>arose from the fields</strong> of pattern recognition, movement control and statistics.</p><p>6- Although document retrieval, text classification, fraud detection, recommendation systems, personalized search, social network analysis, planning, diagnostics, and A/B testing are not AI but <em>Machine Learning</em>, they are <strong>very successful</strong>. And, still we insist in calling them AI.</p><p>There is also eye an opening interview by Lex Friedman to Prof. Jordan in YouTube. Here is the <a href="https://www.youtube.com/watch?v=EYIKy_FM9x0">link</a>, in which Lex and Yann LeCun refer to Prof. Jordan as “the Miles Davis of Machine Learning”.</p><h3>The view of Artificial Intelligence by Prof. John McCarthy (2007)</h3><p>It is interesting to note how the personal computer and computerized electronics completely warped the AI landscape. If we compare Prof. McCarthy to the one by Norvig and Russell, the latter gives way to Robotics, Object Recognition (or Vision), Machine Learning, and Natural Language Processing. Several disciplines that Prof. McCarthy considered as branches of AI have merged by 2010.</p><p>It is mind boggling to see <em>Machine Learning</em>, having brought most of the successes that are attributed to AI, public and general, and even scientists insist in calling it AI. It is not because of the word “artificial”; we are irremediably fascinated and in continuous awe by anything labeled with “intelligence”.</p><p>And sadly to say, good part of it is commercialism and competition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/0*IzU2p-i8kcscwKYD.png" /></figure><h3>The view of AI by Poole, Mackworth and Goebel (1998)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*8o-lKgYqiXY20iIF.png" /></figure><h3>The view of AI by Nils Nilsson (1998)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/0*NTufFHi_SoSDhj1S.png" /></figure><h3>The view of AI by Edward A. Bender (1986)</h3><p>Although Professor Bender classifies Artificial Intelligence, from the viewpoint of Goals, in six groups, he start with two main groups: <em>Goals as Results</em> (Reasoning, Planning, and Learning), and <em>Specific Goals</em> (Robotics, Vision, and Language). He dedicates most of the book to reasoning, search, and learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/0*Xfgzpr0rs7mqYEGN.png" /></figure><p>Prof. Bender covers robotics, vision and language succinctly in the final chapter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/0*6WxFH-Bu0Fxam5KQ.png" /></figure><h3>What Artificial Intelligence is not</h3><h4>Machine Learning gluttony</h4><p>Definitely the following diagrams do NOT represent what Artificial Intelligence is.</p><p>The main flaw is that each of the figures makes it look like <em>Machine Learning</em> is the biggest contributor of AI and makes it a subset of AI, when ML is only a contributing branch of AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N7Epdr0TK8Km_Gbe.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/0*GLBH-UxewZ07Lw3Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/0*Uo761IS7odrArYpK.jpeg" /></figure><p>The <strong>interaction with the environment</strong>, called here, “technologies physical enablement” does not belong to <em>Machine Learning</em>, it is part of an AI agent. Sensors and actuators (or effectors) are used by an Artificial Intelligence agent to interact with the environment; they are not contributors. Interaction with the environment is another dimension of an AI agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/0*SLwn-ifG0V4RBzeV.jpeg" /></figure><h4>Robotics, Vision and NLP are contributing branches of AI</h4><p>In the next figure, I would move the <em>Machine Learning</em> circle outside of AI. The other smaller circles (vision, robotics, NLP), should be surrounding AI, just as Machine Learning. Autonomous Vehicles is an application of AI, not a contributing branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/0*KbaFxb_upNg6L433.png" /></figure><h4>The most successful techniques come from Machine Learning not AI</h4><p>These all are Machine Learning techniques; they are not AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/0*Qps2LkOpjYnu14JZ" /></figure><h4>Data Science is not Artificial Intelligence</h4><p>Here we see the two disciplines, <em>Machine Learning</em> and <em>Data Science</em>. They both complement each other but not necessarily intersect with Artificial Intelligence. Data Science is a reciprocal main contributor of Machine Learning but also minor contributor of Natural Language Processing and Object Recognition. ML and DS are fundamental to get a basic understanding of AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/0*f54lMN6ZkWhwFUpS.jpeg" /></figure><h4>Deep Learning and Neural Networks are part of Machine Learning not AI</h4><p>The following diagram is wrong twice. Deep Learning is a subset of Neural Networks. Machine Learning is not a subset of Artificial Intelligence; it is a contributing branch. ML by itself cannot be intelligent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VHEVs4hcsDogMvh8.png" /></figure><p>On the figure below. Some AI authors think Cognitive Computing should split from Artificial Intelligence as the former uses psychology, neurobiology to search for human intelligence.</p><p>Neural Networks and Deep Learning are not branches of Artificial Intelligence but rather of <em>Machine Learning</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KR_p1JW7uNVeXhux.jpg" /></figure><h3>Artificial Intelligence representations</h3><p>Disciplines such as operations research, problem solving, machine learning, reasoning, knowledge representation, are all enabling branches of an <strong>artificial intelligence agent</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/0*S1NO8AJ_nPLpWdYX.png" /></figure><p><em>Fuzzy Logic</em> and <em>Neural Networks</em> could have been included in <em>Machine Learning</em>. I am not so sure about Fuzzy Logic. I think it should be in Process Control or Optimization, though. There is some debate there. Same with Genetic Algorithms; not AI but a method also used in ML and Optimization.</p><p>Originally, Prof. McCarthy had a branch for Genetic Algorithms under an independent branch he called Genetic Programming. Today, some AI authors consider it part of Search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*oeGyIUSQvQ8oHCOU.png" /></figure><p>Separate is better. <em>Deep Learning</em> should be included in Machine Learning. The two disciplines <em>Machine Learning</em> and <em>Data Science</em> complement each other. ML, DS, computer science, mathematics are necessary to get a fundamental understanding of AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/0*E5o6_G8OJFSeOxjg.jpeg" /></figure><p>The arrows should point the other way around. The disciplines around enable AI. I don’t know what the author meant by “big data”. <em>Big data</em> is a misnomer. Again, neural networks and big data (which is a wrong term), should be in Machine Learning. Speech recognition should be under <em>Natural Language Processing</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/0*brvr3iu74A9mZA7m.jpeg" /></figure><p>Neural Networks and “big data” -whatever the heck means -, should be part of Machine Learning. Arrows should point in direction of AI and no the other way around.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*gnBpiED7dFyoEa0c.jpeg" /></figure><p>This diagram slightly follows the book “Artificial Intelligence: A Modern Approach” by Peter Norvig and Russell. <em>Neural Networks</em>, though, has to be moved to Machine Learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*hw5aq-8x_JqykfV9.png" /></figure><p><em>Computational Intelligence</em> is another preferred term for Artificial Intelligence. At some point, Professor McCarthy wished he had called the discipline “Computational Intelligence” according to Prof. Noel Sharkey.</p><p>This diagram is focused to algorithms that do not necessarily belong to Machine Learning but to optimization and process control: Fuzzy Logic, Evolutionary Computing, and Swarm Intelligence. They could be grouped in a common branch but not ML if they don’t learn from data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*As-e1RU9nlgrqb5v.png" /></figure><p>We see something new here: “physical algorithms” and “immune algorithms”. They would not have a branch in my diagram; they are not ML if they don’t learn from data. I will do some research and come up with a common branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/0*wakW9qRz1EKI1lbL.png" /></figure><p>The following figure needs some grouping but gives the idea of what tools you could use to make an AI agent. It has good intentions but mixes <strong>applications of AI</strong> with <strong>disciplines</strong> contributing to AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Krtc87OpczfnqxbB" /></figure><p>A different take on AI, from the point of view of the cognitive mode. I could not find here, though, anything related to decision making, logic, reasoning, or problem solving. Cognitive Computing seems to be a discipline branching out of AI, according to some authors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8xv9C0wq4JQovKMC.png" /></figure><p>In the next figure, I would just add reasoning, problem solving, and decision making. <em>Deep Learning</em> would go under <em>supervised learning</em>. Although not for long. There have been successes in some types of Generative Adversarial Networks or GANs -some of which use Deep Learning through <em>Convolutional Neural Networks</em>- where the GAN can discriminate and classify unlabeled data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cVr-vI48Prit-bkf" /></figure><p>This is another perspective. What are the disciplines that have made possible Artificial Intelligence. In this diagram we see few. I would add also: Economics, Neuroscience, Computer Engineering, Control Theory, Cybernetics, Linguistics, Statistics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/0*Zm3XqVv8GCuUyltn" /></figure><p>A different view on classification of contributing branches of AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PKR9ND_GV7diLBrV" /></figure><h3>References</h3><ul><li>“Artificial Intelligence: A Modern Approach” by Peter Norvig and Stuart Russell. 2010. <a href="http://aima.cs.berkeley.edu/">Link</a>.</li><li>“Computational Intelligence: A Logical Approach” by David Poole, Alan Mackworth and Randy Goebel. 1998. <a href="https://www.cs.ubc.ca/~poole/ci.html">Link</a>.</li><li>Nils J. Nilsson, 1998: “Artificial Intelligence: A New Synthesis”. <a href="https://www.google.com/books/edition/Artificial_Intelligence/LIXBRwkibdEC?hl=en">Link</a>.</li><li>“What are the branches of AI” by John McCarthy. <a href="http://jmc.stanford.edu/artificial-intelligence/what-is-ai/branches-of-ai.html">Link To Stanford University</a></li><li><a href="https://www.bbc.com/news/technology-15444222">Artificial intelligence community mourns John McCarthy</a>. BBC News, 25-Oct-2011.</li><li>Edward A. Bender; 1986: “Artificial Intelligence: Mathematical Methods”. <a href="https://www.amazon.com/gp/product/0818672005/ref=dbs_a_def_rwt_bibl_vppi_i4">Link</a>.</li><li><a href="http://jmc.stanford.edu/artificial-intelligence/what-is-ai/branches-of-ai.html">Branches of Artificial Intelligence by Prof. John McCarthy</a></li></ul><h3>Links</h3><ul><li><a href="https://www.linkedin.com/posts/alfonsorreyes_machinelearning-datascience-artificialintelligence-activity-6745455447035277312-iChS">Post in LinkedIn</a></li><li>Reproducible TikZ diagrams in GitHub: <a href="https://github.com/f0nzie/artificial_intelligence_diagrams">https://github.com/f0nzie/artificial_intelligence_diagrams</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61b174a3c9a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Granular Control of “R” Shiny apps with Docker containers and ShinyProxy]]></title>
            <link>https://oilgains.medium.com/granular-control-of-r-shiny-apps-with-docker-containers-and-shinyproxy-6f2df572b42b?source=rss-5c397fc2ba37------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f2df572b42b</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[shiny]]></category>
            <category><![CDATA[rstats]]></category>
            <dc:creator><![CDATA[AlfonsoRReyes]]></dc:creator>
            <pubDate>Mon, 20 Jul 2020 03:55:01 GMT</pubDate>
            <atom:updated>2020-07-20T16:36:18.267Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-iHn6o2dG2qUxsvijR2mkA.png" /></figure><p>One of the most impressive and rewarding tools for data science and machine learning is <a href="https://shiny.rstudio.com/">RStudio Shiny</a>. Shiny allows you to prototype the graphical user interface (GUI) of an R application in matter of minutes, and scale it up relatively quick in hours to days. At the end of the day, it will all depend of the ambition and size of the project, as well as how far you are willing to hand-holding your application before you release it to software engineering.</p><p>In this particular case, I am talking from the vantage point of a <strong>Domain Expert </strong>— specifically, Petroleum Engineering. So, some <strong>philosophical debate</strong> may arise on some <strong>sensible spots</strong>: if the domain expert should be coding in the first place; if the domain expert should let the project go to software experts for production; or rather, why not sub-contract the whole enterprise to an outside vendor. In fact, there is no universal truth written in stone on the subject. <strong>Flexibility</strong> seems to be name of the game nowadays.</p><h3>Some background</h3><p>One of the last applications I coded using a traditional GUI was a <strong>multi well scanning tool </strong>written in Python with <a href="https://www.learnpyqt.com/examples/">PyQt5</a>. If you are an engineer like me, and also code, then you know there is a <strong>delicate balance</strong> between the <strong>degree of involvement</strong> with the intricacies of low-level software development: the generation of executables, taking care of libraries and dependencies for the operating system, the deployment cycle, unit testing, debugging, version control, and so on. How much and how long we keep <strong>extending our reach</strong> of knowledge, start resonating stronger with tones of guilt as we further leave away our boundaries. One of those areas is the development of graphical user interfaces or GUIs. In some way, we have to demonstrate the <strong>navigational features</strong> and friendliness of our app while solving an engineering problem, otherwise loses part of its appeal and harder to sell it to management. Don’t get me wrong here. I need a simple way of expressing a GUI -but not simpler; I want a professional look — not a Frankenstein-like mix of buttons, panels and windows. The problem with advanced GUI frameworks such as <a href="https://doc.qt.io/qt-5/qtwidgets-index.html"><strong>Qt5</strong></a> is that they do provide high quality visualization products but at the same time it takes good portion of your time learning it. As a domain expert, should I? And I am talking about at <strong>prototyping level</strong> - not production class. There is this <strong>equilibrium phase</strong> of domain expertise, data science and software engineering.</p><p>And then, the <strong>eternal question</strong> still lingers: would there be a way that using the <strong>same programming language</strong> that I use to code algorithms and scientific functions, could be used to design the graphical user interface?</p><p>And then read something about <em>RStudio Shiny</em>, which contributed somewhat a bit for me switching from Python to R.</p><blockquote>R Shiny was that prototyping and scaling up tool I was looking for as a practical day-to-day problem solving engineer. And <a href="https://www.shinyproxy.io/">ShinyProxy</a> the easiest, flexible and affordable way of deploying R applications using <a href="https://www.docker.com/">Docker</a> containers in the best reproducible way.</blockquote><h3>Objective</h3><p>In this case, we are trying to demonstrate that a Shiny application packaged as a container can be started, controlled and stopped from <em>ShinyProxy</em> as is being run directly from the host. Although, <em>ShinyProxy</em> can be run also from a container, I will start explaining running it from the physical machine or host. Both work similarly. Maybe a couple of differences: the settings of <em>ShinyProxy</em> in the file application.yml, and the peace of mind of not having to install anything else than Docker in your machine.</p><h3>Requirements</h3><p>I assume that:</p><ul><li>You have some knowledge of <strong>Docker</strong> and have Docker installed in your machine either in Windows, Mac OS or Linux.</li><li>Although it helps to know <strong>R</strong>, to run this example you really don’t need to. The code is encapsulated in a container, and the application itself is very simple. In fact, this example may convince you to start coding also in R.</li><li>You know how install and run <strong>Java</strong> applications. <em>ShinyProxy</em> is jar file and runs with Java. But if it is containerized is one thing less to worry about.</li><li>You know how to clone a repository from <strong>GitHub</strong>.</li><li>You are familiar with <a href="https://code.visualstudio.com/"><strong>VS Code</strong></a>. Know how open project folders, how to start a terminal in a folder or sub-folder, and run basic commands from the terminal.</li><li>If you are a Windows user you already have a <strong>Unix</strong> terminal available via <a href="https://www.msys2.org/">msys2</a>, or <a href="https://www.cygwin.com/">cygwin</a>, or the terminal that comes with <a href="https://gitforwindows.org/">Git for Windows</a>, the program for version control.</li><li>Although Windows and Mac OS have their own virtualization kits, if you have trouble enabling them or installing them, I would recommend installing <a href="https://www.virtualbox.org/"><strong>VirtualBox</strong></a> as the virtualization tool. From the Docker point of view, it will be referred to as the docker-machine. I fell in love with Docker when I was a Windows 7 user.</li></ul><h3>Procedure</h3><p>To accompany this article, I have created a repository in GitHub with the necessary code, aids and files to run this in a reproducible way. You can find the <strong>links to the repositories</strong> at the end of this article.</p><p>I will be showing three examples: the first, with a simple Shiny app encapsulated in a container controlled by ShinyProxy installed in your machine. The second, two Shiny apps inside a <strong>R package</strong> plus ShinyProxy installed and working inside a container. And third, an example demonstrating how to add an additional standalone Shiny in a third container.</p><h3>Example 1</h3><p>We start with the repository shinyproxy-host-euler. which means “ShinyProxy will be installed and run from your physical machine to run the Shiny app Euler.</p><figure><img alt="Folder structure in the repository" src="https://cdn-images-1.medium.com/max/266/1*qmFCk0r5sJ1N0-OyOzLL4w.png" /></figure><p>In this repo folder there are two sub-folders:</p><ul><li>euler: contains a Dockerfile and everything required to build an image and run a container with a Shiny app;</li><li>shinyproxy: that contains the Java executable shinyproxy-2.3.1.jar along with a configuration file application.yml.</li></ul><p>These two folders serve two different purposes: build the container with the Shiny app euler as a fully running GUI application;shinyproxy will be a sort of container manager of the euler Shiny application in its own container. <em>ShinyProxy</em>, besides starting this Euler Shiny app and administering it, leaves the door open to control more Shiny applications running in containers as well.</p><h3>Steps</h3><h4>Step 1: Create the image with the Shiny app</h4><p>The first step is creating a Docker image that will provide the source for a running container hosting the Shiny app euler. To do this, we switch to the folder euler and build the image with:</p><pre>docker build -t euler-img .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*mUgHMJdGVxpE_55AWHJoXA.png" /></figure><p>We wait few minutes for the image to build. If you take a look at the file Dockerfile under the folder euler you will see several commands. What these commands do is creating a small <strong>operating system</strong> with all libraries, files, and dependencies required to run a Shiny app on its own. This is not related to your operating system; it is a totally independent application with its own operating system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*10eZLO897hmPkh7DXsE3ZA.png" /></figure><p>The other three files are the Shiny application server.R, ui.R, and a tiny settings file for handling the web port for Shiny to run in a browser Rprofile.site. These three files will be copied to the Docker image that contains that small operating system we mentioned before. if you are curious and want to know the size of that <strong>small OS</strong> with the Shiny app you will be glad to know it is approximately 900 MB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*HaaehCdvkKNJHs3ZFzJV1g.png" /><figcaption>Output of the creation of the Shiny app image</figcaption></figure><p>If you want to know the size of the image we just built, type this in your terminal:</p><pre>docker images | grep euler-img</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*KojNWkJ07w0i_SI29jSmJQ.png" /><figcaption>Size of the Shiny app euler-img</figcaption></figure><h4>Step 2: Run the container</h4><p>After the Docker image is built we can spin a container out of it and take a look at the GUI of the euler Shiny app. If we want to run a container from the euler-img image run the following command in your terminal:</p><pre>docker run -it                  `# interactive mode` \<br>    --rm                        `# remove container after use` \<br>    -p 3838:3838                `# assign web port for the app` \<br>    --name euler-k              `# name of the container` \<br>    euler-img                   `# name of the image`</pre><p>You get an output like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*_VatBvvfBUpsn1OrChjIWw.png" /><figcaption>A running container euler-k</figcaption></figure><p>Which is telling you Shiny is awaiting for a command in your browser at your local host at port 3838. Run this from your browser:</p><pre><a href="http://127.0.0.1:3838/">http://127.0.0.1:3838/</a></pre><p>Which will produce this output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*0WaT2GZsZpNaMtKedCgInQ.png" /><figcaption>Shiny app Euler running from container euler-k</figcaption></figure><p>This means that the Shiny app is running as intended via the Shiny port 3838.</p><p><strong>But this is not what we are looking for.</strong></p><p>Our aim is at controlling the Shiny application from shinyproxy. Why? Because it will give us more power and granular control over this and several other Shiny applications from a common interface, or as most call it these days, a <strong>dashboard</strong>.</p><p>For the moment, close the Shiny app running on port 3838 with Ctrl-C in the terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*ZwwPUNExTpN4lYqdttv2Lw.png" /><figcaption>Stop euler-k container with Ctrl-C</figcaption></figure><h3>Step 3: Run <em>ShinyProxy</em></h3><h4>Why ShinyProxy?</h4><p>So far we’ve got a Shiny app that is running from a container. Nothing surprising here.</p><p>As engineers and data scientists we may tempted to raise these questions:</p><ul><li>(i) what if we have more Shiny applications we want to run?</li><li>(ii) what if we have few users in our group we want to share our Shiny apps with for testing and experimentation of the application prototype?</li><li>(iii) what if we want data generated by the applications stored by the user that is using it?</li><li>(iv) what if we want a control panel or dashboard where we have a menu of the Shiny apps for the project?</li><li>(v) what if we want to run the Shiny apps independently -on demand, when needed- and avoid running a gigantic application?</li><li>(vi) what if we want to allocate one or several Shiny applications in one or different Docker containers because they have their own specific scientific libraries requirements?</li></ul><p><em>ShinyProxy</em> gives us those features.</p><h4>Java checks</h4><p>To run shinyproxy we need to run the Java application<br>shinyproxy-x.y.z.jar from the terminal. if you have cloned this repository, you will find <em>ShinyProxy</em> under the folder shinyproxy, along with a configuration file application.yml. The configuration file is very important because it tells <em>ShinyProxy</em> where the Shiny apps containers are residing, users and passwords, the port for the browser, the network, and the name of the application you want showing on the dashboard. Besides other advanced settings waiting for the moment you decide to scale up your R Shiny applications.</p><blockquote><strong>Note</strong>. You may need to indicate a common port for Docker and ShinyProxy to communicate. See the references at the end of this document.</blockquote><p>We assume you already have a running Java in your machine. Two ways of knowing you have a running Java:</p><pre>java -version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*RyeCmDq7LO7lDyB49fd2cw.png" /></figure><p>And for the <em>Java Development Kit</em> (JDK):</p><pre>javac -version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*jxuct3ApWsJKR_jb2qoaWA.png" /></figure><h4>ShinyProxy configuration file</h4><p>With Java installed in your host or physical machine then it would be now time to check the shinyproxy configuration file. It looks like this:</p><pre>proxy:<br>    port: 8080<br>    authentication: simple<br>    admin-groups: admins<br>    users:<br>    - name: jack<br>      password: password<br>      groups: admins<br>    - name: jeff<br>      password: password<br>    docker:<br>      internal-networking: false<br>    specs:<br>    - id: euler<br>      display-name: Euler&#39;s number<br>      container-cmd: [&quot;R&quot;, &quot;-e&quot;, &quot;shiny::runApp(&#39;/root/euler&#39;)&quot;]<br>      container-image: euler-img<br>​<br>logging:<br>  file:<br>    shinyproxy.log</pre><p>There are three key parts for this example to work. The port (8080), the network characteristics (internal-networking: false) and the image name (container-image: euler-img).</p><blockquote><strong><em>Note</em></strong><em>. Be warned: the tutorials and instructions on </em>ShinyProxy<em> in the web do not mention the setting </em><em>internal-networking: false. I struggled for few hours to make </em><em>shinyproxy communicate with the Shiny container. So, this networking setting is key.</em></blockquote><h4>Time to run ShinyProxy</h4><p>Now, let’s switch to the shinyproxyfolder. To run <em>ShinyProxy</em>, we will run the command:</p><pre>java -jar shinyproxy-2.3.1.jar</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*wHmUpeGkw5k8wSgXOetnng.png" /><figcaption>Under the shinyproxy folder</figcaption></figure><p>This is a very verbose output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aQl5h-wSZ-2myZb4ZxPhOA.png" /><figcaption>Output when running ShinyProxy</figcaption></figure><p>Now it is time head up to our browser and enter this address:</p><pre>127.0.0.1:8080</pre><p>This is the login page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*IMOA_h-J1S5AnQqCDqvknw.png" /><figcaption>Browsing ShinyProxy</figcaption></figure><p>Enter a user name and a password of those available in the file application.yml, that we listed above. I will use jack and password. And then we see the dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*0tospi_374GelbRaADl3_A.png" /><figcaption>ShinyProxy dashboard</figcaption></figure><p>In the dashboard we can only see one application, “Euler’s number” because that is all we have set up in application.yml configuration file.</p><p>Before I click on “Euler’s number” link, I want to show you how I know <em>ShinyProxy</em> is starting the container. I use a small monitoring tool called <a href="https://sysdig.com/blog/csysdig-explained-visually/">csysdig</a> for Docker containers.</p><p>It is empty because I haven’t pressed on the application link yet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O9Pf11DvQwfy2GIFecs0aA.png" /></figure><p>Now, I just clicked the link of the app. What ShinyProxy is doing is launching the Euler app container. It will take few seconds the first time, then it will remain in the cache memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*lXbF5z1rVZM-r_56QzTAxg.png" /><figcaption>ShinyProxy spinning the Euler container</figcaption></figure><p>After few seconds, the Shiny app appears:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*2SIfWBuQ_3mu4rckXJqLjw.png" /><figcaption>Euler app started</figcaption></figure><p>We move the slider a bit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*Nv-SuqiDnFy_Qr_QHrlQRw.png" /><figcaption>Slider moved to 100 precision bits</figcaption></figure><p>Let’s check csysdig. There is one active container derived from the image euler-img. The container ID is a random number under the column <strong>ID</strong> with an also random generated name under the column <strong>NAME</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Qsn2iGx55KTqP5O6UvW-Q.png" /></figure><p>The container will remain there while the Euler app is in use. Now, let’s see what happens when we stop <em>ShinyProxy</em>.</p><p>When I stop <em>ShinyProxy</em> with Ctrl-C, it will send stop signals to all the containers that are part of its network. This is a portion of the log when <em>ShinyProxy</em> is interrupted:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*1L88HfptedlUg_S92S9X3A.png" /></figure><p>And now we observe in csysdig the container for the Euler app has disappeared from the list of active containers. Which is expected and also awesome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*if2pj0TXsRvxHBgMhg-5-Q.png" /></figure><h3>Adding more Shiny applications as containers</h3><p>To see the real impact of using the <em>ShinyProxy</em> approach we have to add more Shiny applications as containers. In a separate repository I am including all the files necessary to reproduce this new case. Find the links to the three repos at the end of this article.</p><h3>Example 2</h3><h4>Run a R package with two Shiny apps as a container</h4><p>The build files will be located in the folder shinyapps-build. Originally this was a ShinyProxy demo but the names were little bit confusing so I renamed the package from shinyproxy to shinyapps, the Shiny applications, loaded different Shiny demos, used different names for the image and the R package itself. I hope this example is clear enough to understand running multiple containers from <em>ShinyProxy</em>.</p><p>Again, I will be using <em>VS-code</em> as my editor and terminal provider. There are three important folders:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*gsEdeEULh1pso9JQtLbjvA.png" /></figure><ul><li>euler: the previous containerized Shiny application</li><li>shinyapps-build: a R package with two Shiny apps that will be containerized</li><li>shinyproxy-container: ShinyProxy Java executable that will be containerized as well</li></ul><h4>Build the image for the R package shinyapps</h4><p>The R package shinyapps contains two small Shiny apps: run_07_widgets() and run_04_mpg(), both come with the <em>Shiny</em> package as examples. This is the R code for both apps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*uF2YUjoE2P0ERXkD46U3Xg.png" /></figure><p>The code is available in the repository. Pasting the image here looks better than pasting the code as text.</p><p>Right-click on the folder shiny-apps-build and open a terminal. See the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*xpa8XJGX2x2t4NVQN6zm9A.png" /></figure><p>The Dockerfile looks like this:</p><pre>FROM openanalytics/r-base</pre><pre>MAINTAINER Tobias Verbeke &quot;<a href="/service/mailto:tobias.verbeke@openanalytics.eu">tobias.verbeke@openanalytics.eu</a>&quot;</pre><pre>RUN apt-get update &amp;&amp; apt-get install -y \<br>    sudo \<br>    pandoc \<br>    pandoc-citeproc \<br>    libcurl4-gnutls-dev \<br>    libcairo2-dev \<br>    libxt-dev \<br>    libssl-dev \<br>    libssh2-1-dev \<br>    libssl1.0.0</pre><pre># packages needed for basic shiny functionality<br>RUN R -e &quot;install.packages(c(&#39;shiny&#39;, &#39;rmarkdown&#39;), repos=&#39;<a href="https://cloud.r-project.org&/#39;">https://cloud.r-project.org&#39;</a>)&quot;</pre><pre># install shinyapps package with demo shiny application<br>COPY shinyapps_0.0.2.tar.gz /root/<br>RUN R CMD INSTALL /root/shinyapps_0.0.2.tar.gz<br>RUN rm /root/shinyapps_0.0.2.tar.gz</pre><pre># set host and port<br>COPY Rprofile.site /usr/lib/R/etc/</pre><pre>EXPOSE 3838</pre><pre>CMD [&quot;R&quot;, &quot;-e&quot;, &quot;shinyapps::run_07_widgets()&quot;]</pre><p>Build the image with:</p><pre>docker build -t shinyapps-img .</pre><p>We can quick test the container with:</p><pre>docker run -it --rm -p 3838:3838 --name shinyapps-k shinyapps-img</pre><p>And watch one of the apps in the browser:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*K2oY_evYwXHERsiANerGRA.png" /></figure><p>Interrupt the container with Ctrl-C in the terminal. Although, we see one application here, this package contains two Shiny apps. We just can’t see them here until we launch <em>ShinyProxy</em>.</p><p>Next, we add these two Shiny apps to the <em>ShinyProxy</em> applications file application.yml.</p><h4>Run ShinyProxy as a container</h4><p>Now, let’s right-click on the folder shinyproxy-container in <em>vscode</em> and open a terminal. See the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*2a0MRuWVwRxloTdUiNTdZQ.png" /></figure><p>Open the file application.yml and see how we added the two Shiny apps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*3TWZHAXeRQSZuEMhVGJ0zw.png" /></figure><p>Also observe that there are two significant changes compared with the first example: we have internal-networking: true, and we added a network for the containers container-network: sp-example-net. This how Docker commands get passed between <em>ShinyProxy</em> and the containers.</p><p>Another thing that will be different is that we will be containerizing <em>ShinyProxy</em>; it will not run from the host or physical machine but from a container based on the following image Dockerfile:which is very simple: its function is provide a Java environment to be able to run the ShinyProxy jar file from within.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*UGuYofoQ25T_qxpc0YA9kA.png" /></figure><p>Let’s build the image with:</p><pre>docker build -t shinyproxy2-img .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*JBDi_G3OTTtLJERnMcUh-w.png" /></figure><p>Note. I am using the number 2 in shinyproxy2-img to mean that it will be handling 2 Shiny apps.</p><p>Now, it’s time to see all these containers working together. We run the ShinyProxy container with:</p><pre>docker run --rm \<br>    -v /var/run/docker.sock:/var/run/docker.sock \<br>    --net sp-example-net \<br>    -p 8080:8080 \<br>     shinyproxy2-img</pre><p><em>ShinyProxy</em> is up and running now in its own container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*tkmTS99iQX3UeHCqF6I_5Q.png" /></figure><p>We go see check csysdig and we found one container running:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vBFSdOB91VCudX9e61E0CQ.png" /></figure><p>Let’s open a browser with:</p><pre>127.0.0.1:8080</pre><p>We now see two applications in the dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*909IivMpaEwcXLP0QlCQKA.png" /></figure><p>We launch one of the apps <strong>MPG Application</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*UcFbblh1aS5jZxYhE7xFZQ.png" /></figure><p>From csysdig we now can see two active containers; one based on the image shinyapps-img, and the second from shinyproxy2-img.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HJcIXJ8VwpV5ZJtWNYiB-g.png" /></figure><p>Let’s stop everything by interrupting <em>ShinyProxy</em> with Ctrl-C. Back to csysdig, all the containers have been stopped.</p><h3>Example 3. Adding more Shiny apps as containers</h3><p>Our last example will add the Euler Shiny app to the set of applications administered by <em>ShinyProxy</em>. I will be copying this repo and creating a new one where I change the configuration for the 3 applications instead of 2.</p><p><strong>Add the new Shiny app to ShinyProxy configuration</strong></p><p>The changes I have to make are two: (1) add the new app to application.yml located in the folder shinyproxy-container; (2) and then build a new image that takes these changes. I will not reuse the same image names; I will be creating new ones so the example is very clear.</p><p>From the top (the first example), I copy+paste the section for the Euler app and inserted at the end of application.yml:</p><pre>- id: euler<br>      display-name: Euler&#39;s number<br>      container-cmd: [&quot;R&quot;, &quot;-e&quot;, &quot;shiny::runApp(&#39;/root/euler&#39;)&quot;]<br>      container-image: euler-img</pre><p>And save the modified application.yml:</p><pre>proxy:<br>  port: 8080<br>  authentication: simple<br>  admin-groups: admins<br>  users:<br>  - name: jack<br>    password: password<br>    groups: admins<br>  - name: jeff<br>    password: password<br>  docker:<br>      internal-networking: true<br>  specs:<br>  - id: 07_widgets<br>    display-name: Widgets Application<br>    description: Application which demonstrates the basics of a Shiny app<br>    container-cmd: [&quot;R&quot;, &quot;-e&quot;, &quot;shinyapps::run_07_widgets()&quot;]<br>    container-image: shinyapps-img<br>    container-network: sp-example-net<br>  - id: 04_mpg<br>    display-name: MPG Application<br>    container-cmd: [&quot;R&quot;, &quot;-e&quot;, &quot;shinyapps::run_04_mpg()&quot;]<br>    container-image: shinyapps-img<br>    container-network: sp-example-net<br>  - id: euler<br>    display-name: Euler&#39;s number<br>    container-cmd: [&quot;R&quot;, &quot;-e&quot;, &quot;shiny::runApp(&#39;/root/euler&#39;)&quot;]<br>    container-image: euler-img<br>    container-network: sp-example-net</pre><pre>logging:<br>  file:<br>    shinyproxy.log</pre><p><strong>Attach the new Shiny app to the shared network</strong></p><p>You see only a minor change: we added the container network sp-example-net for the Euler app. All the containers must belong to the same network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*ZqoOcOnjj2C0yb92dlThWQ.png" /></figure><p><strong>Rebuild the ShinyProxy Docker image</strong></p><p>Let’s build the new image for the three applications from the folder shinyproxy-container with:</p><pre>docker build -t shinyproxy3-img .</pre><p><strong>Run the new container</strong></p><p>And launch the new container:</p><pre>docker run --rm \<br>    -v /var/run/docker.sock:/var/run/docker.sock \<br>    --net sp-example-net \<br>    -p 8080:8080 \<br>    shinyproxy3-img</pre><blockquote>Note that we are now adding two new parameters: -v /var/run/docker.sock:/var/run/docker.sock to create a volume, and --net sp-example-net to connect to a shared network..</blockquote><p>Open the browser at 127.0.0.1:8080</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*uPTs9yCU-T9sfej72BCAvw.png" /></figure><p>There are three applications now. It will take few seconds to start the container and launch the <em>Shiny</em> app from the third container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*UgYf8H8ZdQkwKnHe0ffOcg.png" /></figure><p>Finally, our third application powered by the third container:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*dHKSsFcax7G5rRSCLbDEDA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-DsOfZYrwXJmcxUyLPlVYw.png" /></figure><p>And we launch the “<em>MPG App</em>”. In csysdig we see now the three containers that were spun off from the images euler-img, shinyapps-img, and shinyproxy3-img.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTjr6kPhJRdmjyWhvXx5oA.png" /></figure><h3>Epilogue</h3><p>If you have reached the end of the story, you may be thinking about all the possibilities. That you could combine Shiny apps inside packages, standalone Shiny apps, with the potential of splitting the applications in several containers depending of their requirements and complexity.</p><p>We haven’t explored other advanced features of <em>ShinyProxy</em> that involve sharing data among users, or make compartmentalized data stores. Or make applications available to certain user or groups. There is plenty more to learn.</p><p>One important aspect of data science, we didn’t talk much about in this article, is <strong>reproducibility</strong>. Docker containers contribute to that area enormously because they go to the point of replicating the exact condition of the operating system, packages, libraries, settings, software versions, of the machine where you originally run the experiments or developed an analysis. This makes the effort of learning containers worth it.</p><h3>Notes</h3><h4>Sharing the host network port with Docker</h4><p>To let ShinyProxy communicate with Docker we need to establish the port 2375. The details for setting this up are provided here: <a href="https://www.shinyproxy.io/getting-started/">https://www.shinyproxy.io/getting-started/</a>. This is only required if we run <em>ShinyProxy</em> directly from the host or physical machine. If you are using <em>ShinyProxy</em> from within a container, you don’t need to worry about this because the network between containers and the <em>ShinyProxy</em> container is created brand new and shared for this exclusive purpose.</p><h4>Running ShinyProxy from a container</h4><p>If you are running the <em>ShinyProxy</em> jar file from a container you have to make sure that internal-networking: true, and that each of the containers declared in application.yml contains a line sharing an exclusive network. Like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*K9-2QMYFsuosdqw4Vf9sIQ.png" /></figure><p>You can see this small difference from the configuration file we used for the Euler application: we are indicating a shared network with container-network: sp-example-net. This network sp-example-net is created in advance with <em>Docker</em> with only one line:</p><pre>docker network create sp-example-net</pre><h3>Links</h3><ul><li>Repository for the first example: <a href="https://github.com/og-analytics/shinyproxy-host-euler">https://github.com/og-analytics/shinyproxy-host-euler</a></li><li>Repository for the second example: <a href="https://github.com/og-analytics/shinyproxy-multicontainer-shiny">https://github.com/og-analytics/shinyproxy-multicontainer-shiny</a></li><li>Repository for the third example: <a href="https://github.com/og-analytics/shinyproxy-multicontainers-shiny-3">https://github.com/og-analytics/shinyproxy-multicontainers-shiny-3</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f2df572b42b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>