<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Graham Bryan on Medium]]></title>
        <description><![CDATA[Stories by Graham Bryan on Medium]]></description>
        <link>https://medium.com/@grahambryan?source=rss-b5e7afea8030------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*4exJIZYTXOfb7mBT</url>
            <title>Stories by Graham Bryan on Medium</title>
            <link>https://medium.com/@grahambryan?source=rss-b5e7afea8030------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:42:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@grahambryan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Randomized Optimization in Machine Learning]]></title>
            <link>https://medium.com/geekculture/randomized-optimization-in-machine-learning-928b22cf87fe?source=rss-b5e7afea8030------2</link>
            <guid isPermaLink="false">https://medium.com/p/928b22cf87fe</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mimic]]></category>
            <category><![CDATA[simulated-annealing]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[hillclimb]]></category>
            <dc:creator><![CDATA[Graham Bryan]]></dc:creator>
            <pubDate>Wed, 21 Apr 2021 16:37:43 GMT</pubDate>
            <atom:updated>2021-04-22T08:52:27.052Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this blog I look to go over some of the basics of Randomized Optimization Algorithms.</h4><h3>Bayesian Stuff</h3><p>Bayesian probability is used within the machine learning world and understanding Bayes’ rule at a high level is key.</p><blockquote>Bayes’ rule relates the conditional probability of one event to the actual condition.</blockquote><p>In Bayesian Learning, Bayes’ rule helps us to figure out the most probable hypothesis given an input space and some domain knowledge. The great thing is this can simply be expressed as, <em>argmax Pr[h|D] h∈H</em>, where D is the input space chosen from some probability distribution and <em>h </em>our hypothesis that is in the set of all possibile hypotheses, <em>H</em>. Applying Bayes’ rule will give an even friendlier expression that relates the likelihood our hypothesis outputs what we are looking for, prior domain knowledge of that output, and the probability distribution of the input space.</p><p>In Bayesian learning we simply apply bayes rule to the problem over all possible hypothoses and take the argmax. But finding some of the parameters in the friendly Bayes’ rule expression can be difficult. Lets try and answer a few. You can either uses the maximum a posteriori (<em>MAP</em>) or maximum likelihood (<em>ML</em>) hypotheses to help find prior domain knowledge of you hypotheses. ML seems to be recommended since its final expression is simply the sum of squared errors and minimizing it is the way to go to find a good hypothesis.</p><p>This all boils down into more in-depth Bayesian and probability theories such as Bayesian Networks, Inference, and Naïve Bayes that are the basis for many machine learning algorithms that I will not discuss here. There is a lot more here and I suggest looking over <a href="https://teapowered.dev/assets/ml-notes.pdf">Chapter 4</a> of the notes I have been reading since it is very helpful to see the equations and he gives great generalized examples.</p><h3>Unsupervised Learning: Randomized Optimization</h3><p>Optimization problems are given a input space, objective, and a goal, where the objective is a fitness function in order to find the best output (i.e., goal) within the input space. Optimization tends to target problems such as finding the best process, route, root, or parameters to an algorithm. In the real-world the input space can be huge and a complex function where it is hard to find the best thing to solve your problem and often you end up finding local optima (<em>maybe the next best thing</em>) instead of global. Thus, the need for randomized optimization that provides algorithms to search an input space that will “most likely” lead to the best answer.</p><h4>Hill Climbing</h4><ul><li>Randomly guesses a starting point and searches towards a single “best” input within the local neighborhood of the initial starting point.</li><li>Easy to get stuck in a local optima, but you can increase how many guesses are made to increase the chances of finding, at least, a really good local optima (and possible the global maximum) — this is the <em>random restart hill climbing algorithm.</em> If a local minimum is ever found we just try again starting a newly chosen random starting point.</li><li>Relies on <em>argmax</em> of the input space.</li></ul><h4>Simulated Annealing</h4><ul><li>Based off random hill climbing, the simulated annealing algorithm says that we don’t necessarily need to keep guessing new starting points and move up the “hill”, but instead improve on the guesses we already made by exploring them more.</li><li>Finding a balance between the number of times we start looking at a new neighborhood and exploring that neighborhood with more than <em>argmax</em> is the key. It gives us more of a chance of locating a global minima</li><li>The exploration of the neighborhood uses an acceptance probability function to determine how seriously to ingest bad points. This is driven by a <em>temperature</em> parameter — as the <em>temperature</em> goes to 0, this algorithm behaves exactly like hill-climbing (leading to the local maximum) and as <em>temperature</em> goes towards infinity, we accept more values regardless of how bad it is (behaving like a random walk through).</li><li>This all leads to a property that tells us that the probability of ending at a particular input is directly proportional to how good it is</li><li>Generally start with High temperatures and slowly decrease it (this is called the cooling factor)</li><li>Repeated annealing can be very slow depending on how complex your cost function is. Tuning this algorithm can be a delicate and tedious process.</li></ul><h4>Genetic Algorithms</h4><ul><li>Based off biology, this algorithm looks at populations (random sample input space) which mutate (local search) and have cross-overs, thus leading to different population groups combining to “maybe” produce something better over a certain amount of time (iterations of improvement).</li><li>The key take-away that makes this algorithm different than just being random hill-climbing in parallel, is cross-over. First we iteratively update the population by evaluating values with a fitness function (<em>truncation</em> or <em>roulette wheel strategy</em>) then cross-over occurs to generate a new population of more “fit” individuals based off pairs of individuals probabilistically as opposed to just taking the local maximum.</li><li>The algorithm is less likely to fall into local optima since it continuously replaces values with other values (<em>parents with offspring</em>) instead of a gradient descent that moves from one value to the next smoothly.</li><li>Representing an optimization problem as input to a genetic algorithm is difficult.</li></ul><h4>MIMIC</h4><ul><li>MIMIC wants to learn more about the complex function space rather than a single point within it — it just cares more. Broken down, MIMIC tries to model the underlying probability distribution and iteratively improve it over time. Thus, it “remembers” previous searches of the input space unlike any of the the algorithms mentioned previously.</li><li>The probability distribution considers uniform samples within the input space that are considered “good enough” on a fitness level. The choice of distribution depends on a threshold <em>theta, </em>that when looking at a minimum <em>theta</em> the distribution is uniform over the entire input space, and when a maximum it only looks at the local optima for the function.</li><li>MIMIC starts with a distribution over the entire input space and iteratively improves it till we reach optima. Each iteration takes a certain increasing percentage of the dataset and then takes “good enough” values in order to estimate the next probability distribution till we meet whatever percentage stopping criteria that the user chose.</li><li>MIMIC assumes a probability distribution can be estimated and that we actually chose “good enough” values to make that estimation. This estimation is done using dependency trees, a type of baysesian network. In short, we generate a minimization cost function (which tells us the similarities between any two distributions using <em>KL-divergence</em>)<em> </em>that then is maximized due to the introduction of a new term that maximizes the mutual information between features, which gives us the ability to create a connected graph (where nodes are features and edges are mutual information between features) and since we are trying to maximize mutual information, the edge weights are inverted so the graph can be spanned in order to find a new probability distribution…</li><li>MIMIC likes structure, so when the optimal value depends on the structure instead of exact values MIMIC will perform well as opposed to a fitness function that cannot be represented by a probability distribution. Due to the “storage” of relationships between features, MIMIC will usually converge in less iterations than the other algorithms but may take more time and will require more processing memory.</li></ul><blockquote>References Used:</blockquote><blockquote><a href="https://github.com/mohamedameen93/CS-7641-Machine-Learning-Notes/blob/master/UL01.%20Randomized%20Optimization.pdf">https://github.com/mohamedameen93/CS-7641-Machine-Learning-Notes/blob/master/UL01.%20Randomized%20Optimization.pdf</a></blockquote><blockquote><a href="https://teapowered.dev/assets/ml-notes.pdf">https://teapowered.dev/assets/ml-notes.pdf</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=928b22cf87fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/randomized-optimization-in-machine-learning-928b22cf87fe">Randomized Optimization in Machine Learning</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Networking with Docker-Compose: A Simple Ping Example]]></title>
            <link>https://medium.com/swlh/networking-with-docker-compose-a-simple-ping-example-7b6979dfc94b?source=rss-b5e7afea8030------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b6979dfc94b</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[ping]]></category>
            <category><![CDATA[networking]]></category>
            <dc:creator><![CDATA[Graham Bryan]]></dc:creator>
            <pubDate>Wed, 23 Sep 2020 21:54:06 GMT</pubDate>
            <atom:updated>2021-02-26T17:02:44.041Z</atom:updated>
            <content:encoded><![CDATA[<p>TL;DR The networking capabilities Docker-Compose abstracts for docker users has significantly helped my mental state and should help yours too! Stop using bash scripts and hardcoding IP addresses for your docker container network. It leads to hacky solutions and can be prone to errors.</p><p>Docker-Compose creates a single network for which all of its services can communicate over. This feature makes docker-compose an incredibly powerful tool. You will no longer have the self induced headache of remembering the IP address’ for all the containers you launched or creating your own docker network from scratch.</p><p>In this article, Ill show you a simple example of this awesome feature docker and docker-compose gives us.</p><p>For the example, I have put together a custom docker image based off the latest ubuntu with iputils-ping installed. The <em>Dockerfile</em> looks like this:</p><pre>FROM ubuntu<br>RUN apt-get update &amp;&amp; \<br>    apt-get install -y iputils-ping<br>CMD bash</pre><p>Below is a <em>docker-compose.yml</em> file I have put together in order to demonstrate the network. There are two services, <em>service-1</em> and <em>service-2</em>, that both spawn from the image, <em>my-image, </em>based off the <em>Dockerfile</em> above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5b7c0658fe311ec9f5fe39dca6ea2b4/href">https://medium.com/media/d5b7c0658fe311ec9f5fe39dca6ea2b4/href</a></iframe><p>The application of each service is to try and ping the other 3 times, ping &lt;service name&gt; -c 3.With docker-compose, it will automagically create a single network under the hood for you and map the containers IP directly to the service name so it can be resovled. Once you launch the services with docker-compose up the following is the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fax_V_-bwzfu38LvA5-1lA.png" /><figcaption>Results of running <strong>docker-compose up</strong></figcaption></figure><p>As you can see, the ping was successful! Both services were able to resolve and ping the other by the service name itself. Otherwise, we would need to know what the IP was for each container that was launched in order to do this simple execution.</p><p>To see more about the networking services docker-compose has see their <a href="https://docs.docker.com/compose/networking/">documentation</a>. This feature can easily be applied to more complex systems where messaging between services is essential. This makes the complicated network setup and tracking much easier.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b6979dfc94b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/networking-with-docker-compose-a-simple-ping-example-7b6979dfc94b">Networking with Docker-Compose: A Simple Ping Example</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embedding Grafana into your Web App]]></title>
            <link>https://medium.com/swlh/embedding-grafana-into-your-web-app-da6837545fe4?source=rss-b5e7afea8030------2</link>
            <guid isPermaLink="false">https://medium.com/p/da6837545fe4</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[embedded]]></category>
            <category><![CDATA[nextjs]]></category>
            <dc:creator><![CDATA[Graham Bryan]]></dc:creator>
            <pubDate>Sun, 13 Sep 2020 21:15:36 GMT</pubDate>
            <atom:updated>2020-09-13T23:19:20.300Z</atom:updated>
            <content:encoded><![CDATA[<h3>Embedding Grafana into a Web App</h3><p>My co-workers and I have built a full-stack web application with next.js, flask, postgres, and <a href="https://grafana.com/">Grafana</a>. Why do we need Grafana? We use it for specific visualizations that are hard to generalize in our main stack. It allows for customizable and version controllable dashboards that look great and can be embedded into web applications. This gives us the ability to add dynamic visualizations with ease.</p><p>In our <a href="https://nextjs.org/">next.js</a> application, a server side JavaScript and React framework, we use its native page system. In a main page we embed a Grafana dashboard using a simple <em>iframe</em> html tag and a few state defined variables in order to create a meaningful url. Below is a snippet of how we use the tag:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42bac46c1a4bbc22771a754f4ddf3ace/href">https://medium.com/media/42bac46c1a4bbc22771a754f4ddf3ace/href</a></iframe><p>The line,</p><pre>${darkMode ? &#39;dark&#39; : &#39;light&#39;}</pre><p>uses our <em>darkMode</em> state, taken from our React global state management context, which allows us to easily match the Grafana theme with ours!</p><p>This of course assumes a Grafana server is running. If it is not, feel free to take a look my previous article, <a href="https://medium.com/swlh/easy-grafana-and-docker-compose-setup-d0f6f9fcec13">Easy Grafana and Docker-Compose Setup</a>. We map the server url to the <em>GRAFANA_DASHBOARD_URL</em> variable so we can embed it in the iframe. This variable is easily passed into the system using a <a href="https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration">next.js runtime config file</a>, <em>next.config.js</em>. An example of the configuration file and how it is used within a .js file is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24a30bdc9ebfa50a5637ab723d9e68d5/href">https://medium.com/media/24a30bdc9ebfa50a5637ab723d9e68d5/href</a></iframe><pre>import getConfig from &#39;next/config&#39;;<br>const { publicRuntimeConfig } = getConfig();<br>const { GRAFANA_DASHBOARD_URL } = publicRuntimeConfig;</pre><p>We updated the configuration file in order to allow embedding so we do not get a deny error. This is easily done by updating the servers configuration and restarting it. The key setting below in order to do this is the <em>allow_embedding</em> config.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4e6f672ec0beb50c9f5a439e3512623/href">https://medium.com/media/b4e6f672ec0beb50c9f5a439e3512623/href</a></iframe><p>And that is it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da6837545fe4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/embedding-grafana-into-your-web-app-da6837545fe4">Embedding Grafana into your Web App</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Easy Grafana and Docker-Compose Setup]]></title>
            <link>https://medium.com/swlh/easy-grafana-and-docker-compose-setup-d0f6f9fcec13?source=rss-b5e7afea8030------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0f6f9fcec13</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Graham Bryan]]></dc:creator>
            <pubDate>Fri, 04 Sep 2020 20:02:46 GMT</pubDate>
            <atom:updated>2021-12-20T17:11:23.259Z</atom:updated>
            <content:encoded><![CDATA[<p>My team has been working on solutions in order to quickly maintain, scale, and visualize internal software suites used for work in modeling and simulation, networking, and more. Before stumbling on <a href="https://grafana.com/">Grafana</a>, an incredible open source project for dashboard generation, we were coding our own custom plots in Javascript. The customization was nice, but it took a lot of time and it forced us to edit the source code often. From never knowing how to use Grafana it took no time to setup and we basically had all the same plots as before!</p><p>Now, we are able to launch an entire visualization suite alongside our <a href="https://www.docker.com/">Docker</a> development environment. I wont get into the specifics of the environment, but our teams have generally switched to running all software development kits with docker and <a href="https://docs.docker.com/compose/">docker-compose</a>. We are able to launch the SDK, a database, and the grafana visualization in one go. This allows for a seamless start-up for our developers and awesome looking plots. All you need is docker and docker-compose installed!</p><p>This quick article will show a grafana user to provision, version control, and add grafana to a docker-compose setup so a user can launch a powerful development and analysis tool in a single go.</p><p>Here is a look at the compose file and the associated Dockerfile.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3723592bd58d2a9a399d68ff6ab997ba/href">https://medium.com/media/3723592bd58d2a9a399d68ff6ab997ba/href</a></iframe><p>Note, you could just launch the compose with image: grafana/grafana instead of using build: but this allows for more customization, easier user installation, and versioning.</p><p>If you wish anything to persist when you make changes to your dashboard or configurations, you must add a named volume to the setup. To create this volume,</p><pre>$ docker volume create grafana-data</pre><p>I am assuming a user already has or knows how to create a <a href="https://grafana.com/docs/grafana/latest/reference/dashboard/">json model</a> for the dashboard and <a href="https://grafana.com/docs/grafana/latest/features/datasources/add-a-data-source/">add a data source</a>. If you do not, you can still follow the setup and create your own dashboard later!</p><p>There are three things we add to the docker image: provisioning, grafana.ini, and dashboards. All of which are easily versioned controlled and can be released to update all your developers environments.</p><p>The <a href="https://grafana.com/docs/grafana/latest/administration/provisioning/">provisioning</a> is added so a user does not have to do any configuration to setup datasources or dashboards and are instead created automatically when launched. Here is what the structure looks like:</p><pre>|-- provisioning<br>        |-- dashboards<br>        |   `-- dashboard.yml<br>        `-- datasources<br>            `-- datasource.yml</pre><p>The .yml are configurations for dashboards and datasources. For example they can look like,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a65d3996ecf1cb371367f9bc05ad76a1/href">https://medium.com/media/a65d3996ecf1cb371367f9bc05ad76a1/href</a></iframe><p>The next configuration file, grafana.ini:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4e6f672ec0beb50c9f5a439e3512623/href">https://medium.com/media/b4e6f672ec0beb50c9f5a439e3512623/href</a></iframe><p>It is used to set multiple grafana <a href="https://grafana.com/docs/grafana/latest/administration/configuration/#configuration">configurations</a> for ease of use. There are tons of possible options here that will meet most users requirements.</p><p>The dashboards directory contains your json model(s) for your dashboard(s).</p><pre>|-- dashboards<br>    |   `-- my-dashboard.json</pre><p>And that is it. This was a brief look at a quick way to integrate grafana into your workflow. With these created you can easily version control the entire setup and push it out to users. Launch the setup with,</p><pre>$ docker-compose up</pre><p>The docker image will be built for you and the service will launch the grafana homepage at <a href="http://localhost:9000/">http://localhost:9000</a>.</p><p>As you continue and want to make this a more versatile setup you could add a database service and your own docker development environment. This would allow you to launch a powerful dev environment with a single command all connected via its own docker network.</p><p>[Edit: 12/2021]</p><p>Some users have had trouble with the code written here, so one has been nice enough to open source their setup in this library <a href="https://github.com/annea-ai/grafana-infrastructure">grafana-infrastructure</a> using a mariadb backend. There are many ways to do this, but providing more examples to readers in one place is always helpful!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0f6f9fcec13" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/easy-grafana-and-docker-compose-setup-d0f6f9fcec13">Easy Grafana and Docker-Compose Setup</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>