<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dave Thompson on Medium]]></title>
        <description><![CDATA[Stories by Dave Thompson on Medium]]></description>
        <link>https://medium.com/@limscoder?source=rss-1066a64cf0c3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*eg7s9957U1rbTmFaWsb0QQ.jpeg</url>
            <title>Stories by Dave Thompson on Medium</title>
            <link>https://medium.com/@limscoder?source=rss-1066a64cf0c3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:04:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@limscoder/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DataSidecar for Prometheus Time Series Analysis]]></title>
            <link>https://medium.com/freshtracks-io/datasidecar-for-prometheus-time-series-analysis-f8597f8405a1?source=rss-1066a64cf0c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8597f8405a1</guid>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Dave Thompson]]></dc:creator>
            <pubDate>Mon, 29 Oct 2018 16:42:42 GMT</pubDate>
            <atom:updated>2018-10-29T16:43:08.456Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-w4dvbztLUvjpfYMmIZQ_A.png" /></figure><p>I investigated using FaaS for scalable <a href="https://prometheus.io/">Prometheus</a> rule evaluation in <a href="https://blog.freshtracks.io/faas-for-scalable-prometheus-rule-evaluation-c10fc9082989">part 1</a> of this series. I found FaaS works great for rules driven by simple models, but the performance characteristics of FaaS added hard constraints for more computationally intensive multi-variate models as discussed in <a href="https://blog.freshtracks.io/tensorflow-model-execution-for-prometheus-time-series-data-b986188f0edd">part 2</a>. I will investigate using the <a href="https://github.com/open-fresh/data-sidecar">data-sidecar</a> service for executing multi-variate models against Prometheus data in this post.</p><h3>The Data</h3><p>Like the previous posts, this example uses a model attempting to predict future cryptocurrency prices. The models used here are multi-variate LSTM networks trained and executed with <a href="https://www.tensorflow.org/">TensorFlow</a>. This post focuses on how to continuously execute the models in a production environment rather than the models themselves. These models are very similar to the ones described in <a href="https://blog.freshtracks.io/tensorflow-model-execution-for-prometheus-time-series-data-b986188f0edd">part 2</a>, so refer back to that post to learn about how the models work. The prediction data is available live at <a href="http://www.predictatron.net/">http://www.predictatron.net/</a>. Select the “sidecar” option to view data from models described in this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jGpWAW-aLxYUduLfS1mDZQ.png" /><figcaption>Predicted btc/usd exchange rate. Black series is predicted value, green series is the actual value.</figcaption></figure><h3>data-sidecar</h3><p>The <a href="https://github.com/open-fresh/data-sidecar">data-sidecar</a> service is a <a href="https://golang.org/">GO</a> program developed at <a href="http://freshtracks.io/">FreshTracks.io</a> for the purpose of analysis of Prometheus time series data. The sidecar continuously and efficiently queries Prometheus time series data, performs data analysis at a specified interval, and then exposes the result as a Prometheus metrics endpoint so it can be scraped back into prometheus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*4SkSVsPxsFZF32T116MuJQ.png" /><figcaption>Prediction metrics exposed by the sidecar</figcaption></figure><p>FreshTracks created the data-sidecar for running proprietary models for the purposes of adaptive thresholds, forecasting, and anomaly detection of <a href="https://kubernetes.io/">Kubernetes</a> cluster metrics data. We can’t share these algorithms, so the open source code base contains only a couple of very simple adaptive threshold and anomaly detection methods.</p><h3>TensorFlow in Python and GO</h3><p>Our LSTM network is constructed with <a href="https://www.tensorflow.org/">TensorFlow</a>. The TensorFlow GO library is adequate for executing models, but is difficult to use for constructing and training models because it does not implement the full TensorFlow API. Use <a href="https://keras.io/">Keras</a> and <a href="https://www.python.org/">Python</a> to build, train, and persist the network instead, similar to the models used in <a href="https://blog.freshtracks.io/tensorflow-model-execution-for-prometheus-time-series-data-b986188f0edd">part 2</a>. The trained network is persisted to disk to be executed by the data-sidecar at runtime. The model building and training code is available in the <a href="https://github.com/limscoder/predictatron/blob/master/models/lstm/model.py">Predictatron repo</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*eCgSOj2o55_p2gtJpoIUuw.png" /><figcaption>Python code to build, train, and persist a TensorFlow graph in a format that data-sidecar can import.</figcaption></figure><p>The data-sidecar master branch does not contain the logic to execute TensorFlow models, so the examples in this post are based off <a href="https://github.com/limscoder/data-sidecar/tree/tensorflow">this TensorFlow enabled fork</a>. The data-sidecar <a href="https://github.com/limscoder/data-sidecar/blob/tensorflow/scoring/tf/tf.go">imports persisted TensorFlow models</a> on start-up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*TdyMyJ65l0UTVVUcKcslbw.png" /><figcaption>Go code for importing persisted TensorFlow graphs.</figcaption></figure><p>The data-sidecar continuously queries all series data tagged with the label ft_target=&quot;true&quot; from a Prometheus instance, and <a href="https://github.com/limscoder/data-sidecar/blob/tensorflow/scoring/tf/scorer.go">executes the TensorFlow model</a> against the series data whenever new datapoints are available. Model meta parameters such as the number of input and output steps are defined in the model’s params.json file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*Q3VHfLBkQT0jIJJsSZBIlw.png" /><figcaption>Go code that executes a model against series data and exposes the result to a Prometheus endpoint.</figcaption></figure><h3>Running the data-sidecar Service</h3><p>The data-sidecar must be carefully configured for everything to work properly. Significantly, the data-sidecar must be configured to use the same query steps and time duration as the model was trained with. In this example the models were trained on series containing datapoints at 1 minute intervals. Configure the following command line options to match our model.</p><ul><li>prom=<a href="http://prom.predictatron.net:9090/">http://prom.predictatron.net:9090</a> : Prometheus source</li><li>resolution=60 : time between datapoints (60 seconds)</li><li>lookback=60 : historic steps to include in initial query (60 minutes)</li><li>tfpath=/models/lstm/model-btc_usd-5m : location of persisted models</li></ul><p><a href="http://prom.predictatron.net/">Predictatron</a> runs data-sidecar within a Docker container running on <a href="https://cloud.google.com/compute/">Google Compute Engine</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T4RpzqiBt9TBQKZyDxF42g.png" /><figcaption><a href="https://github.com/limscoder/data-sidecar/blob/tensorflow/Dockerfile">Dockerfile</a> for Predictatron’s data-sidecar.</figcaption></figure><h3>Results and Scaling</h3><p>Running multi-variate models within <a href="https://github.com/open-fresh/data-sidecar">data-sidecar</a> is much more performant than the FaaS approach demonstrated in <a href="https://blog.freshtracks.io/tensorflow-model-execution-for-prometheus-time-series-data-b986188f0edd">part 2</a>. While I didn’t do any formal benchmarking, the data-sidecar can handle many input series and many models with reasonable resource constraints. data-sidecar scales horizontally in 2 dimensions. Both input series and models can be sharded among different data-sidecar instances. This is the most promising approach to scaling Prometheus time series data analysis, and the reason that <a href="http://freshtracks.io/">FreshTracks.io</a> developed the data-sidecar.</p><p>The prediction data from this post is live at <a href="http://www.predictatron.net/">http://www.predictatron.net/</a>. Select the “sidecar” option to view data from models described here.</p><h3>Running Custom Models</h3><p>The open-source version of data-sidecar described here does not yet have a plug-and-play model interface, but the <a href="https://github.com/limscoder/data-sidecar/blob/tensorflow">TensorFlow example code</a> used for this post should be a good starting point if you’re willing to write a little GO code. Create a <a href="https://github.com/limscoder/data-sidecar/blob/tensorflow/scoring/tf/scorer.go">Scorer struct</a> to implement your model’s algorithm, and wire it up in <a href="https://github.com/limscoder/data-sidecar/blob/tensorflow/main.go">main.go</a> to execute against incoming series data. Then configure your Prometheus instance with a starget at sidecar-url/metrics to scrape the resulting metrics back into Prometheus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*Dz10OQbst3wQZ8_3JZII7g.png" /><figcaption>Prometheus configuration for scraping data-sidecar.</figcaption></figure><p>Future improvements to data-sidecar may include a plug-and-play model interface to make it easier to execute custom models with minimal code changes, as well as supporting online models that continuously update in response to new data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8597f8405a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/freshtracks-io/datasidecar-for-prometheus-time-series-analysis-f8597f8405a1">DataSidecar for Prometheus Time Series Analysis</a> was originally published in <a href="https://medium.com/freshtracks-io">FreshTracks.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TensorFlow Model Execution for Prometheus Time Series Data]]></title>
            <link>https://medium.com/freshtracks-io/tensorflow-model-execution-for-prometheus-time-series-data-b986188f0edd?source=rss-1066a64cf0c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b986188f0edd</guid>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[cloud-functions]]></category>
            <dc:creator><![CDATA[Dave Thompson]]></dc:creator>
            <pubDate>Wed, 03 Oct 2018 18:48:36 GMT</pubDate>
            <atom:updated>2018-10-04T01:08:07.372Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*86ugmzIv-Fp0frjsxo25bQ.png" /></figure><h3>TensorFlow Model Execution for Prometheus Time Series Data (Part 2)</h3><p>I investigated using FaaS (function as a service) functions for scalable <a href="https://prometheus.io/">Prometheus</a> rule evaluation in <a href="https://blog.freshtracks.io/faas-for-scalable-prometheus-rule-evaluation-c10fc9082989">Part 1</a> of this series. The rules in Part 1 were simple univariate models that could be evaluated quickly with low computation overhead. This post expands on that by exploring methods to execute more complex multivariate models in the context of FaaS.</p><p>The models for this example attempt to predict (badly) the price of cryptocurrency at several points in the future. The model’s predictions can be explored at <a href="http://www.predictatron.net/">http://www.predictatron.net/</a> (select Bitcoin, LSTM, 15 minutes). All of the code used in this post is available at <a href="https://github.com/limscoder/predictatron">https://github.com/limscoder/predictatron</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9-Yh50d1xTWQcQyLReMO2g.png" /><figcaption>Black series is the predicted value, green series is the actual value.</figcaption></figure><h3>The Data</h3><p>The time series data used as model input is queried from the Prometheus server described in <a href="https://blog.freshtracks.io/faas-for-scalable-prometheus-rule-evaluation-c10fc9082989">Part 1</a> of this series. A <a href="https://github.com/limscoder/predictatron/blob/master/models/lstm/fetch.py">Python script</a> queries the data and saves it as CSV (comma-separated values) to use for model training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*0GznmSOeVddUC248k4wP4Q.png" /><figcaption>Python code to query time series data from Prometheus.</figcaption></figure><h3><strong>The Model</strong></h3><p>The <a href="https://github.com/limscoder/predictatron/blob/master/models/lstm/model.py">model</a> used in this example is a multivariate LSTM (long short-term memory) network that uses several different cryptocurrency-related time series as input, then outputs a cryptocurrency price prediction. The code builds an LSTM network using <a href="https://keras.io/">Keras</a> in Python. Keras is a modular open source neural network library that can run on top of <a href="https://www.tensorflow.org/">TensorFlow</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HobiJetUaR1CwakuLa_MxQ.png" /><figcaption>Creating an LSTM network with the Keras library.</figcaption></figure><p>The network is trained using the <a href="https://keras.io/backend/">TensorFlow backend</a>, and the resulting network weights are saved to a file for later use. As you might guess, the model does not converge well, but the point of this example is to explore methods for running and persisting TensorFlow models against Prometheus time series data rather than building an accurate model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*VwsStOcbHVlWI_aPguugcw.png" /><figcaption>The model sorta, kinda converges if you squint</figcaption></figure><h3>The Metrics</h3><p>The predictions generated from the pre-trained LSTM network will be exposed in the Prometheus exposition format following the techniques used in <a href="https://blog.freshtracks.io/faas-for-scalable-prometheus-rule-evaluation-c10fc9082989">Part 1</a>. The metrics endpoint must use a Node runtime, because TensorFlow does not yet support the Python 3.7 runtime used by <a href="https://cloud.google.com/functions/">Google Cloud Functions</a>.</p><p>The <a href="https://github.com/limscoder/predictatron/tree/master/metrics/lstm/models/model-btc_usd-15m">trained and persisted TensorFlow models</a> are deployed alongside the <a href="https://github.com/limscoder/predictatron/blob/master/metrics/lstm/index.js">endpoint code</a> as a Javascript/Node FaaS function that loads the model and executes predictions with the <a href="https://js.tensorflow.org/">tensorflowjs library</a>. Prometheus is configured to scrape the FaaS endpoint as a metric target. The function queries historic time series data, uses the data as input to the model, and exposes the resulting prediction as a Prometheus metric every time the endpoint is scraped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*G1mrCklFeuPQ41pElnnQVA.png" /><figcaption>JS/Node code to load the Keras/TensorFlow model containing trained network weights from disk, query historic data, and execute the model for prediction output.</figcaption></figure><p>The model loading and prediction steps take computation time. The Prometheus scrape timeout can easily be exceeded if we try to execute too many model predictions in a single scrape. A major advantage of the FaaS method described here is easy scaling. URL query parameters can be used to scale predictions across several Prometheus scrape targets. The params field is configured to set the query parameters and create a separate scrape target for each currency in the Prometheus configuration below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*Sa8tO0MKDEySbWFtY5u15w.png" /><figcaption>Prometheus scrape configuration uses separate target jobs to scale model execution.</figcaption></figure><p>Each of these endpoints returns a set of Prometheus metrics containing a predicted value for some point in the future, based on query parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*6tXICjXfa4rUcxzV3VNgMA.png" /><figcaption>Prometheus metric endpoint with Bitcoin price prediction.</figcaption></figure><h3>Model Updates</h3><p>The model is retrained offline whenever an update is desired. Training happens in whatever environment is most effective, the new network is persisted in a platform agnostic format, the updated model is pushed to the code repo and the FaaS function is re-deployed. The model update will be seamlessly transparent to the Prometheus scrape job.</p><h3>Limitations</h3><p>The FaaS method described here works great for the relatively small network used in this example, but unfortunately this method of executing Keras/TensorFlow models did not work out as well as I had hoped with larger networks due to several issues described below.</p><ol><li><strong>Function deployment size limitations <br></strong>Google Cloud Functions have a conservative limit on the size of code that can be deployed. This restricts the size and/or number of persisted models that can be deployed with a function. This could be corrected by deploying a separate function endpoint for each trained network. This could also be corrected by persisting the model in an external storage bucket instead, but that would exacerbate the next issue.</li><li><strong>Model loading is slow <br></strong>Loading the Keras/TensorFlow model with network weights from disk into memory is surprisingly slow. Since FaaS functions are stateless, a cold start can take significant time to load the model into memory and risk timing out the Prometheus scrape.</li><li><strong>Model execution is slow</strong> <br>Tensorflowjs has <a href="https://github.com/tensorflow/tfjs-node">C++ and GPU bindings</a> for improved performance with Node. Unfortunately, I could not get these bindings working properly within the Google Cloud Function runtime environment, so I had to fallback to the pure-Javascript version of TensorFlow that’s normally intended for lightweight models being run in a browser. This significantly slows down model execution. I’m guessing there’s a way for the FaaS function to trigger a model run in some more appropriate corner of GCE, but I haven’t had time to explore that option yet.</li><li><strong>Network updates are too slow <br></strong>It is not realistic to train and update an online model in response to new data due to the performance constraints outlined above.</li></ol><h3>Summary</h3><p>Executing TensorFlow models within FaaS functions is a simple way to get started with time series data analysis for Prometheus. While this method can work well for prototyping, there are performance constraints that limit its usefulness in production systems. In Part 3 of this blog series, we will investigate using the <a href="https://github.com/open-fresh/data-sidecar">data-sidecar project</a> to execute production-ready models for Prometheus series analysis.</p><p>The data-sidecar is an open-source stateful service designed by the <a href="https://freshtracks.io/">FreshTracks.io team</a> with the purpose of efficiently executing time series analysis models as new data points become available. The data-sidecar’s feature set resolves the limitations of the FaaS implementation. In Part 3, we will look at executing the TensorFlow LSTM models within the data-sidecar server.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b986188f0edd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/freshtracks-io/tensorflow-model-execution-for-prometheus-time-series-data-b986188f0edd">TensorFlow Model Execution for Prometheus Time Series Data</a> was originally published in <a href="https://medium.com/freshtracks-io">FreshTracks.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FaaS for Scalable Prometheus Rule Evaluation (Part 1)]]></title>
            <link>https://medium.com/freshtracks-io/faas-for-scalable-prometheus-rule-evaluation-c10fc9082989?source=rss-1066a64cf0c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c10fc9082989</guid>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[google-cloud-functions]]></category>
            <dc:creator><![CDATA[Dave Thompson]]></dc:creator>
            <pubDate>Mon, 24 Sep 2018 18:33:23 GMT</pubDate>
            <atom:updated>2018-10-04T01:07:37.317Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_BQRLUrkDVoihwxVhbLN4A.png" /></figure><p><a href="https://prometheus.io/">Prometheus.io</a> is a popular open source application used for persisting and querying high dimension time series data. Prometheus contains a limited set of built-in statistical functions, but more sophisticated forecasting and machine learning methods require external computation. The more computationally intensive analysis methods can be moved into FaaS (function as a service) functions for simpler deployment and easier scaling.</p><h3>App Concept</h3><p>This post walks through the creation of an application with the goal of forecasting cryptocurrency price changes. This is a difficult data set to forecast in real life, but the example application exposes simple concepts that can be applied to any time series data analysis. The focus of this post is not on the effectiveness of the models themselves, but rather on the techniques used to execute the models in a scalable fashion.</p><p>A working example of the application is available at <a href="https://www.predictatron.net/">www.predictatron.net</a> and the code is available at <a href="https://github.com/limscoder/predictatron">github.com/limscoder/predictatron</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*Q3oZ0do-3iarW0JWtxEmrA.png" /><figcaption>Select a currency, prediction method, and time range to visualize forecasts. The cyan series shows the actual value, while the black series shows the predicted value.</figcaption></figure><h3><strong>Running Prometheus</strong></h3><p>The first step of building the Predictatron app is to retrieve cryptocurrency price data and persist it in a format that’s convenient to query. Prometheus is the obvious choice for this task. The easiest way to run Prometheus is to run the <a href="https://hub.docker.com/r/prom/prometheus/">Docker container</a>. Predictatron.net runs a container within a <a href="https://cloud.google.com/compute/">Google Compute Engine</a> instance, which automatically runs a configured container on startup.</p><p>The Predictatron repo contains a customized <a href="https://github.com/limscoder/predictatron/tree/master/prometheus">Prometheus container</a> with scrape targets configured for the app, and the built container is published publicly at <a href="http://gcr.io/starry-tracker-215615/prom">gcr.io/starry-tracker-215615/prom</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*5uRYpNHSXyGC0JumbefgTw.png" /><figcaption>The Prometheus configuration for scraping cryptocurrency price data.</figcaption></figure><h3>Populating Source Data</h3><p>Once Prometheus is up and running, the next step is to expose data on an endpoint to scrape. The Predictatron Prometheus container is configured to scrape metrics data from an endpoint located at <a href="http://us-central1-starry-tracker-215615.cloudfunctions.net/coin">us-central1-starry-tracker-215615.cloudfunctions.net/coin</a>. This endpoint is running as a <a href="https://cloud.google.com/functions/">Google Cloud Function</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XGHcYf4X5WYBwXUHk29uTw.png" /><figcaption>Function deployment details.</figcaption></figure><p>Prometheus scrapes the endpoint at the configured interval (1 minute in this example). Every time the endpoint is scraped, it invokes a FaaS function that dynamically retrieves cryptocurrency price data from <a href="https://coinmarketcap.com/">coinmarketcap</a> and outputs the data formatted with the <a href="https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md">Prometheus exposition format</a>. The function’s code is written with Python and can be seen in the <a href="https://github.com/limscoder/predictatron/blob/master/metrics/coin/main.py">metrics/coin directory</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*LLOOEfCsw-3fAzO-qlXDFQ.png" /><figcaption>Cryptocurrency price data in Prometheus exposition format</figcaption></figure><h3>Series Inspection and Forecasting</h3><p>Cryptocurrency price data appears as a series once Prometheus is successfully scraping the coin endpoint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJdA1v95v5K5TKxa1OWH-A.png" /><figcaption>Time series data queried by metric name `btc_usd`.</figcaption></figure><p>Prometheus contains several basic built-in aggregation functions for analyzing data including: sum, avg, increase, and holt_winters. The predict_linear function uses linear regression to forecast future series values at arbitrary times, and that’s the forecasting method used by the linear option in Predictatron. The following <a href="http://prom.predictatron.net:9090/graph?g0.range_input=1h&amp;g0.expr=predict_linear(btc_usd%5B6h%5D%2C+900)&amp;g0.tab=1">Prometheus query</a> predicts the series value 15 minutes (900 seconds) in the future using 6 hours of historical data.</p><pre>predict_linear(btc_usd[6h], 900)</pre><p>Thepredict_linear computations are quick enough that they can be queried on-demand, but it can be helpful to record forecast results as their own series in order to evaluate the effectiveness of historical forecasting. The following <a href="https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/">recording rule configuration</a> will persist the forecast as a new series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*PX2PZZYRanhtH5co-ED4pg.png" /></figure><p>The derived series is named predict_linear:btc_usd and the predict_past and predict_future labels are added to identify the forecast’s parameters. The prediction series can now be queried with a time offset to compare against the source series to determine historic prediction error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BPucqsRfTxuOtFfVm8C-4w.png" /><figcaption>Prediction error: `predict_linear:btc_usd{predict_future=”900&quot;} — (btc_usd offset 15m)`</figcaption></figure><h3>FaaS for Rule Evaluation</h3><p>Prometheus recording rules are sufficient for simple operations like predict_linear, but implementers will run into a couple of problems with this method. The first is that more complex rule expressions take longer to execute and scaling becomes an issue as the number of recording rules grows. The second problem is that implementers may want to invoke more complex operations that can’t be expressed as a Prometheus query expression.</p><p>The Predictatron app contains an <a href="https://medium.com/@josemarcialportilla/using-python-and-auto-arima-to-forecast-seasonal-time-series-90877adff03c">ARIMA</a> series forecasting method that can’t be easily expressed as a Prometheus query expression. FaaS offers a straightforward way to implement and scale more complex rule evaluations. Predictatron uses <a href="https://github.com/limscoder/predictatron/blob/master/metrics/arima/main.py">a function</a> that defines an ARIMA model in Python.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*o0a2yO9B3pYJ7Qy_xSfX2g.png" /><figcaption>ARIMA model in Python using `statsmodels` package.</figcaption></figure><p>The function is deployed to a FaaS and exposes the forecast result in the Prometheus exposition format. Prometheus is configured to scrape the function endpoint and records the forecast as a new series. The FaaS function is auto-scaled, but different models and parameters can also easily be sharded across multiple function endpoints to ensure that each individual scrape completes in the desired duration.</p><p>FaaS functions are a useful strategy to deploy and scale any type of series rule evaluation and are not limited to forecasting analysis. The architecture of Predictatron is diagrammed below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*Z-tTQcjFlxEGh-sUVab79Q.png" /><figcaption>Predictatron app architecture</figcaption></figure><h3>Stateful Models</h3><p>The linear regression and ARIMA models used in this example were chosen because they are simple to understand and computationally inexpensive, so they can be executed on-demand. Both of these models are univariate models, and they look only at the historical values of the time series being forecast. Univariate models in general are inadequate for forecasting a series like cryptocurrency, since the events causing price movement are not present in the series itself.</p><p>Stay tuned for <a href="https://blog.freshtracks.io/tensorflow-model-execution-for-prometheus-time-series-data-b986188f0edd">Part 2</a> of this blog series where we will investigate more powerful and computationally intensive multivariate forecasting models. These models are too slow to train on-demand and require stateful execution, necessitating a different approach than FaaS rule evaluation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c10fc9082989" width="1" height="1" alt=""><hr><p><a href="https://medium.com/freshtracks-io/faas-for-scalable-prometheus-rule-evaluation-c10fc9082989">FaaS for Scalable Prometheus Rule Evaluation (Part 1)</a> was originally published in <a href="https://medium.com/freshtracks-io">FreshTracks.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Load Testing Prometheus Metric Ingestion]]></title>
            <link>https://medium.com/freshtracks-io/load-testing-prometheus-metric-ingestion-5b878711711c?source=rss-1066a64cf0c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b878711711c</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Dave Thompson]]></dc:creator>
            <pubDate>Thu, 30 Aug 2018 19:43:22 GMT</pubDate>
            <atom:updated>2018-11-05T19:40:42.408Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="http://prometheus.io/">Prometheus</a> is a popular open source project for time series ingestion, storage, and queries. The Prometheus ecosystem is rapidly expanding with alternative implementations such as <a href="https://github.com/cortexproject/cortex">Cortex</a> and <a href="https://github.com/improbable-eng/thanos">Thanos</a> to meet the diverse use-cases of adopting organizations. These alternative implementations share a common <a href="https://prometheus.io/docs/instrumenting/exposition_formats/">exposition format</a> that is in the process of being formalized as the <a href="https://openmetrics.io/">OpenMetrics</a> standard.</p><p>It is important for organizations to understand series throughput limits to properly scale their metrics ingestion system, but alternative implementations have different performance and scaling characteristics. <a href="https://github.com/open-fresh/avalanche">Avalanche</a> is a simple metrics generation tool that can be used to load test metric ingestion throughput. Avalanche serves a metrics endpoint with configurable series generation, seen in the example below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*iXk3lO95e270jK7tQSjNpA.png" /><figcaption>Example metrics generated by Avalanche.</figcaption></figure><h3>Running Avalanche</h3><p>The easiest way to run an Avalanche server is with the public <a href="https://www.docker.com/">Docker</a> image.</p><pre>docker run -p 9001:9001 quay.io/freshtracks.io/avalanche</pre><p>Avalanche supports several flags for configuring the generated series:</p><blockquote>metric-count</blockquote><p>Configure the number of metric names exposed in the endpoint. This option is useful for testing how the number of series present on the endpoint impacts ingestor performance.</p><blockquote>series-count</blockquote><p>Configure the number of series <em>per-metric. </em>The total number of series rendered to the endpoint is equal to (metric-count * series-count). This option is useful for testing how the (metric-count/series-count) ratio impacts ingestor performance.</p><blockquote>label-count</blockquote><p>Configure the number of labels present in each series.</p><blockquote>value-interval</blockquote><p>Update series values every {interval} seconds.</p><blockquote>series-interval</blockquote><p>Update the ‘series_id’ label every {interval} seconds, cycling series. This option is helpful for testing series creation and termination performance.</p><blockquote>metric-interval</blockquote><p>Update metric names every {interval} seconds, cycling series. This option is helpful for testing series creation and termination performance.</p><h3>Measuring Ingestion Performance</h3><p>Configure Prometheus to <a href="https://prometheus.io/docs/introduction/first_steps/#configuring-prometheus">scrape itself</a>, plus add the following scrape config for Avalanche:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*mi892_pY38XJgqTschlY-Q.png" /><figcaption>Avalanche scrape target</figcaption></figure><p>Some interesting Prometheus metrics to watch while scraping Avalanche include:</p><ul><li>scrape_duration_seconds</li><li>prometheus_target_sync_*</li><li>prometheus_target_scrape_*</li><li>prometheus_tsdb_*</li><li>prometheus_local_storage_*</li><li>prometheus_rule_*</li></ul><h3>Clustering</h3><p>Real-life Prometheus instances are likely to scrape multiple hosts and endpoints. This behavior can be easily replicated with a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Kubernetes deployment</a>. An example deployment configuration to deploy 5 instances of Avalanche looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*-IQf-2DrhclPAD55FEesMQ.png" /><figcaption>Avalanche deployment config</figcaption></figure><p>Next, configure Prometheus to run as a pod in your cluster and update the Prometheus scrape config to use <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E">Kubernetes service discovery</a> to find each exposed Avalanche endpoint with a scrape target config as shown in the example config below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*ntsnanECBEnVCdF1UP48XQ.png" /><figcaption>Avalanche pod scrape target config</figcaption></figure><p><a href="https://github.com/prometheus/node_exporter">Node Exporter</a> and <a href="https://github.com/google/cadvisor">cAdvisor</a> metrics can provide insights into performance and resource utilization of Prometheus once it is running in a pod and scraping Avalanche endpoints. Resource usage is generally correlated to total series/second ingested and “prometheus_target_interval_length_seconds” will exceed requested scrape intervals when Prometheus is under-provisioned.</p><h3>Alternative Implementations</h3><p><a href="https://github.com/open-fresh/avalanche">Avalanche</a> is useful for guiding parameter tuning, scaling, and development of other metrics collection systems that adopt the <a href="https://openmetrics.io/">OpenMetrics</a> standard. An example is <a href="https://github.com/cortexproject/cortex">Cortex</a>, which has much different performance characteristics than Prometheus, and exposes a host of metrics useful for performance investigations prefixed with cortex_.</p><p>FreshTracks uses Avalanche to test and scale our Cortex system, and we’ve found that operations such as rule evaluation, chunk flushing, and series termination are particularly useful to track because certain metric workload combinations can significantly degrade ingestion performance. This data has been useful to help us proactively avoid and tune for those combinations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b878711711c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/freshtracks-io/load-testing-prometheus-metric-ingestion-5b878711711c">Load Testing Prometheus Metric Ingestion</a> was originally published in <a href="https://medium.com/freshtracks-io">FreshTracks.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go lang first impressions]]></title>
            <link>https://medium.com/freshtracks-io/go-lang-first-impressions-731c5accb6d7?source=rss-1066a64cf0c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/731c5accb6d7</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Dave Thompson]]></dc:creator>
            <pubDate>Mon, 08 Jan 2018 17:10:31 GMT</pubDate>
            <atom:updated>2018-01-08T17:10:31.976Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnxBwMR1cPNLhqc37pgecw.png" /></figure><p>FreshTracks.io is using many great tools from the <a href="https://www.cncf.io/">Cloud Native Computing Foundation</a>, which has led to me starting to read and write <a href="https://golang.org/">go code</a>. This article describes my first impressions of the language and development environment.</p><p>I’ve written production code in a dozen languages from C to Javascript, so I have a good amount of comparison material, but I’m a relative golang newb so be warned that I maybe missing some detail. This covers the highlights and lowlights I’ve encountered so far.</p><h4>The nice parts</h4><p>Fat binaries are great. Want to run my go program? Download a single file and run it. No runtime download, env setup, or dependencies to grab, and no version issues to contend with.</p><p>Golang is fast. <a href="http://benchmarksgame.alioth.debian.org/u64q/compare.php?lang=go&amp;lang2=gpp">Benchmarks</a> show speed approaching C++ for most tasks, and much faster than higher level dynamic languages.</p><p>Golang is safer. You don’t need all the verbose and fragile bounds checking and validation code like C/C++, and different concurrency patterns can reduce multi-threading complexity (maybe).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3KpZy_5uvaRc9TColFWYaw.png" /></figure><p>Development tools are simple and work out of the box. Unit testing is built-in and tests use standard code instead of an assert library. The compiler is super fast, which you will under-appreciate until you go back to a platform with a slower build process.</p><p>Code formatting, linting, and debugging in <a href="https://code.visualstudio.com/">vscode</a> is painless. I haven’t tried other ides because vscode works great and can be configured by installing a single plugin.</p><p>Golang supports high level features like closures and slice syntax. These features are second nature to Javascript and Python developers, it’s nice to have them here.</p><p>Objects are simple. Instead of supporting classes or inheritance, sources of pain in many languages, golang only allows functions to be bound to structs. It’s the best of both worlds, simple and semantic.</p><p>There are a bunch of helpful little nuggets such as <a href="https://golang.org/ref/spec#Selectors">automatic dereference in selector expressions</a>, <a href="https://tour.golang.org/basics/6">multiple return values</a>, and <a href="https://golang.org/doc/effective_go.html#embedding">struct embedding</a> that makes your code less verbose.</p><h4>The type system</h4><p>The golang type system’s simplicity is refreshing. It works well and exactly as you would expect. The lack of generics makes it difficult to code custom data structures, but golang’s built-in data types are sufficient for most tasks.</p><p>I do miss higher order functions like map and reduce, which can’t be easily defined without generics, but I think the development team made the right choice by choosing simplicity and safety as the main goals of the type system.</p><h4>Memory management</h4><p>Memory management in golang is extremely flexible. Developers can create slices and maps and dynamically add and remove elements and golang will figure it out for you like a higher level language, but developers also have the choice to use fixed sized arrays and structs to more carefully manage memory when needed.</p><p>I’ve read about some memory details that can be tricky to debug when you get into memory optimization, such as <a href="https://dougrichardson.org/2016/01/23/go-memory-allocations.html">escape analysis causing variables to be unexpectedly allocated on the heap</a>, but it seems like memory management is sufficient for most use cases. Extra care might be needed if you’re expecting C level control for a memory bottlenecked app, but golang offers a <em>ton</em> of control compared to higher level languages like Java.</p><h4>Features I miss</h4><p>I definitely miss persistent immutable data structures like those found in Clojure and Immutable.js. Golang code tends to be mutation heavy with many impure functions, and this makes it difficult to debug and track down where mutations are happening. Unfortunately the lack of generic types makes it difficult to write a 3rd party immutable lib.</p><p>Golang lacks decent dependency management. I still haven’t completely figured out the mechanics, but when you go get a dependency I think it downloads master and sticks it in GOPATH, so all of your go projects use the same, not obviously specified, version of that dependency. There are some 3rd party solutions available, but the community hasn’t made any strong decisions on what’s best yet.</p><p>3rd party library selection is not as large as other platforms. The amount of libs you’ll find is much smaller than Java, Python, or Javascript. For example, Google doesn’t support a full-featured <a href="https://www.tensorflow.org/">TensorFlow</a> library yet, and I had trouble finding a well supported and actively maintained <a href="http://graphql.org/">GraphQL</a> implementation.</p><p>I wish golang used file-level variable scoping. Golang uses “package” variable scoping instead, which may span several files. You see variables being accessed in one file, but their declaration and/or instantiation is hidden off in some other file within the same package. IDEs can help here, but I still find it difficult to read go code that uses this pattern, especially when the variable is <em>mutated</em> in multiple files, and <em>double-extra-especially</em> when variable access needs to be synchronized among go routines across multiple files.</p><h4>What I’m still learning</h4><p>Golang error handling is confusing. The docs recommend C style error code returns using the helpful error type, but the language also supports panic, which is kinda like an exception, but somehow different. I’m not sure when I should use one over the other, but I think standard exceptions would be easier for most use cases.</p><p>Golang supports multiple ways to synchronize data between concurrent go routines. Golang docs claim channels are the <a href="https://tour.golang.org/concurrency/2">greatest thing since sliced bread</a>, but <a href="https://www.jtolds.com/2016/03/go-channels-are-bad-and-you-should-feel-bad/">some people disagree</a>. Real-life go code uses a mix of channels, shared memory + mutexes, and closure callbacks to synchronize data between go routines. I haven’t done too much concurrent coding with go yet, but it appears you need to thoroughly understand all three paradigms, their performance implications, and their tradeoffs before embarking on the journey, especially if you are using third party libs that mix and match these styles.</p><p>I haven’t done any localization with go yet, but I have noticed that strings are plain byte arrays similar to strings in Python 2x, rather than encoded character sequences like Python 3x, Java, or Javascript. This may make localization more difficult to implement, and I need to dig into this issue a bit more.</p><h4>Conclusion</h4><p>Golang has tradeoffs like all coding platforms. The pieces I like tend to be the “simpler than other platform” parts, and the confusing parts tend to be related to pieces that seem like they were baked into the platform before being fully thought out.</p><p>Overall, I’d recommend golang when you need a safer, friendlier C/C++ or a simpler and faster high level alternative, but consider investigating other options if you want a large 3rd party library selection or require data structures unavailable within go’s standard library. I hope the upcoming <a href="https://blog.golang.org/toward-go2">golang v2</a> focuses on making improvements in core areas mentioned above instead of simply adding new features, because I’d love to see the platform become more capable and useable while retaining its simplicity and performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=731c5accb6d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/freshtracks-io/go-lang-first-impressions-731c5accb6d7">Go lang first impressions</a> was originally published in <a href="https://medium.com/freshtracks-io">FreshTracks.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[D3 with React (Part 2)]]></title>
            <link>https://medium.com/freshtracks-io/d3-with-react-part-2-9364ef2c0a5a?source=rss-1066a64cf0c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9364ef2c0a5a</guid>
            <category><![CDATA[d3js]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Dave Thompson]]></dc:creator>
            <pubDate>Mon, 04 Dec 2017 16:48:16 GMT</pubDate>
            <atom:updated>2017-12-04T18:50:52.591Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fb4MoWwE7N0lR2CoWvvJ9A.png" /></figure><p>In <a href="https://blog.freshtracks.io/an-introduction-to-d3-data-binding-part-1-1fe1418b958a">part one</a> I introduced D3’s data binding API. In part two I’ll introduce techniques to wrap D3 within a <a href="https://reactjs.org/">React</a> component to facilitate state management for the <a href="https://limscoder.github.io/hexaplex/">interactive app pictured below</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*5A8uxa_uFQBsh9gm570vJw.png" /><figcaption>Click on a hexagon to flip it’s color and the color of surrounding hexagons.</figcaption></figure><p>D3 is great for building complex visualizations, but it lacks an easy method for state management. React provides state management tools in the form of state and props. The repo below contains an example of integrating D3 into a React application:</p><p><a href="https://github.com/limscoder/hexaplex">limscoder/hexaplex</a></p><h3>D3 + React</h3><p>Several libraries exist to represent D3 operations as React components such as <a href="http://www.reactd3.org/">reactd3</a> and <a href="http://recharts.org/">recharts</a>. These libraries are great for constructing charts and other simple visualizations, but the D3-&gt;<a href="https://facebook.github.io/jsx/">JSX</a> mapping loses much of D3’s power and expressiveness. D3’s main advantages over simpler solutions are it’s powerful data binding API and flexibility to draw anything without being tied to a handful of pre-conceived data visualization types. For these reasons, I recommend sticking with D3’s native API and manually wrapping the functionality within a React component.</p><p><strong>Rendering a grid of hexagons with D3</strong></p><p>Stop here and read <a href="https://blog.freshtracks.io/an-introduction-to-d3-data-binding-part-1-1fe1418b958a">part one</a> before continuing if you are not familiar with D3’s data binding API. The code below generates an array of datums representing a grid of hexagons for D3 to render using the concepts from the <a href="https://blog.freshtracks.io/an-introduction-to-d3-data-binding-part-1-1fe1418b958a">part one</a> post. Hexagons are colored magenta when active === true and gray when active === false.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9175f8f25b697aadd3cb9e0a05dae142/href">https://medium.com/media/9175f8f25b697aadd3cb9e0a05dae142/href</a></iframe><p><strong>D3 rendering within a React component</strong></p><p>How can this code be integrated with a React component? The approach below uses a React component to render the SVG element, and then the grid of hexagons is rendered with D3 in the componentDidMount lifecycle method, using React’s ref property to store a reference to the SVG element. The data from the hexData function is stored in state.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/36ac42a2a30222a7c48b93938167c6da/href">https://medium.com/media/36ac42a2a30222a7c48b93938167c6da/href</a></iframe><p><strong>Respond to user interaction</strong></p><p>The component draws a grid of hexagons, but it needs to respond to user interaction to turn it into a functioning app. The D3 event handling system will execute a callback when a hexagon is clicked, and the callback will call the component’s setState method with an updated array of data. The component will also need a componentDidUpdate method to update the visualization when the component’s state changes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c447bf84932cf032a3b97da4bbf933fc/href">https://medium.com/media/c447bf84932cf032a3b97da4bbf933fc/href</a></iframe><p>There’s a lot going on here that needs to be unpacked. The component lifecycle operates as follows:</p><ol><li>Component’s constructor method is called and state is initialized with hexData representing the initial state of the hexagon grid.</li><li>React renders SVG element to DOM and ref is stored via the ref callback.</li><li>React calls the componentDidMount method after rendering.</li><li>componentDidMount uses D3 data enter selection to render a grid of hexagons matching the data in state.hexData. Each rendered hexagon has a click event handler.</li><li>User clicks a hexagon.</li><li>click event handler toggles each datum’s active property based on it’s location relative to the clicked hexagon, and then calls the component’s setState method with the updated data. It’s good practice to treat state as an immutable object, so the updated datums are always new objects in this example.</li><li>React renders to virtual dom in response to setState, and calls componentDidUpdate after the update is complete.</li><li>The componentDidUpdate method animates all hexagons that transitioned from active to inactive or vice-versa using a D3 data update selection.</li><li>5–8 is repeated every time a hexagon is clicked.</li></ol><p>This example shows different transition behavior in componentDidMount (animates hexagon grid scaling and centering) and componentDidUpdate (animates hexagon color). This type of flexibility is difficult to achieve using “d3-as-react-component” libraries, but easy using D3’s data selection API and React’s lifecycle methods.</p><p>The number of hexagons is static in this example, but the componentDidUpdate method could animate hexagons being added and/or removed to the grid by using the enter and exit selections explained in the <a href="https://blog.freshtracks.io/an-introduction-to-d3-data-binding-part-1-1fe1418b958a">part one</a> post. D3’s selection diffing allows these updates to be executed in a performant manner.</p><p><strong>Integrating with a state management framework</strong></p><p>The React component above uses state for tracking the hexagon grid’s state. Moving the grid state out of state and into props allows the same techniques to be used with any popular state management framework such as <a href="https://github.com/apollographql">apollo</a>, <a href="https://redux.js.org/">redux</a>, or <a href="https://github.com/mobxjs/mobx">mobx</a>. The event handler callback will also need to be moved into props to make the component reusable in different contexts. The example below uses props instead of state.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fd21089c9208cbcbe3f68259d034a41/href">https://medium.com/media/8fd21089c9208cbcbe3f68259d034a41/href</a></iframe><p>React + D3 is a powerful (and fun!) way to develop maintainable interactive visualizations. React handles the state management in a predictable manner, while D3 handles the visualizations gracefully.</p><p>Check out the <a href="https://github.com/limscoder/hexaplex">repo</a> to see all the pieces put together in a working app with only ~100 lines of code. The goal of the app is turn all of the hexes the same color. I have no idea if it’s actually solvable ;)</p><p><strong><em>FreshTracks.io is a monitoring tool that layers machine learning on top of Kubernetes and Prometheus. To learn more, </em></strong><a href="https://www.freshtracks.io/beta"><strong><em>join our beta</em></strong></a><strong><em> today!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9364ef2c0a5a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/freshtracks-io/d3-with-react-part-2-9364ef2c0a5a">D3 with React (Part 2)</a> was originally published in <a href="https://medium.com/freshtracks-io">FreshTracks.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An introduction to D3 data binding (Part 1)]]></title>
            <link>https://medium.com/freshtracks-io/an-introduction-to-d3-data-binding-part-1-1fe1418b958a?source=rss-1066a64cf0c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fe1418b958a</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Dave Thompson]]></dc:creator>
            <pubDate>Mon, 27 Nov 2017 14:51:00 GMT</pubDate>
            <atom:updated>2017-12-04T17:29:32.849Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F0eMzIcMX8_eMgjG4UfI7w.png" /></figure><p>This is part one of a two part post. I’ll introduce D3’s data binding API in part 1 with code examples and add state management with React to build a working app in <a href="https://blog.freshtracks.io/d3-with-react-part-2-9364ef2c0a5a">part 2</a>.</p><p>D3 is most commonly used as a tool for drawing vector graphics within Javascript apps, but its true power lies in its flexible data binding API. D3’s data binding functionality works with <em>all</em> DOM elements, and is not confined to SVG.</p><p><a href="https://d3js.org/">D3</a></p><h3>D3 DOM selections</h3><p>DOM selections are objects that represent a set of elements matching a <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors">css selector</a>. Attribute and style changes can be applied to all elements in the set, which should be familiar to <a href="https://jquery.com/">jQuery</a> users.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ddb6d158b7092efdb845d1d210242d98/href">https://medium.com/media/ddb6d158b7092efdb845d1d210242d98/href</a></iframe><h3>D3 data selections</h3><p>Data selections are the core of D3’s data binding API. A data selection is created by diffing an array of data against a DOM selection. This example selects .app-user DOM elements and matches them to an array of user data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/41856e931e461fa22dbaadae8b70d349/href">https://medium.com/media/41856e931e461fa22dbaadae8b70d349/href</a></iframe><p>The resulting selection can be thought of as an object representing the difference between DOM state and data state, and each selection contains 3 sets:</p><ul><li>update selection: existing DOM elements that match a datum</li><li>enter selection: non-existing DOM elements that match a datum</li><li>exit selection: existing DOM elements that don’t match a datum</li></ul><p><strong>Update Selection</strong></p><p>The base selection is called an “update selection” because the bound datums can be used to <em>update</em> the properties of the DOM elements they are bound to. This example updates the text of each element to match the datum’s name and changes the color of each element to match the datum’s role.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71d7e7e06800fdebbad1495b17abdb99/href">https://medium.com/media/71d7e7e06800fdebbad1495b17abdb99/href</a></iframe><p>The style function creates an inline style, which is very convenient for encapsulating styles and avoiding CSS style bleed between elements. You can also use any style library that generates class names, such as <a href="https://github.com/threepointone/glamor">glamor</a>. I recommend starting with inline styles.</p><p><strong>Enter Selection</strong></p><p>Each element of the enter selection represents a datum that was not matched to an existing DOM element. Use the append function of an enter selection to <em>add</em> new elements into the document.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2da86e88823830f810dafbc0d2f39b60/href">https://medium.com/media/2da86e88823830f810dafbc0d2f39b60/href</a></iframe><p><strong>Exit Selection</strong></p><p>Hopefully you’ve already guessed what the exit selection is. Each element of the exit selection represents an existing DOM element that does not match any datums. The exit selection can be used to <em>remove</em> elements from the document.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/985b2151455385a6b09d046288602e99/href">https://medium.com/media/985b2151455385a6b09d046288602e99/href</a></iframe><p><strong>Data matching</strong></p><p>The data function uses index position for matching. In the example above the first element of users will match the first DOM element matching the CSS selector li.app-user, the second element of users will match the second DOM element and so on.</p><p>data accepts a second argument, a key function that is used to uniquely identify a datum. Using a key function can dramatically reduce the number of DOM modifications when the ordering of the data changes frequently. The key function returns a value that uniquely identifies a datum.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf774cee153e8a161a186a7c79dc6b8f/href">https://medium.com/media/bf774cee153e8a161a186a7c79dc6b8f/href</a></iframe><h3>Transitions</h3><p>By now you maybe thinking: “Ok, so D3 does DOM vs data diffing for me, but why do I have to describe the state of these 3 selection sets (update, enter, exit)? It seems more complicated than React where I only have to describe a single, final state and React manages all of the additions and removals.”</p><p>D3’s selection system is more complicated than React, but it also gives you a more expressive way to describe your UI. This is most evident when you want to apply specific behavior to elements transitioning between states, such as animating updated, added, or removed elements. Animating these transitions can be tricky in React, but is trivially easy with D3 selections. The example below animates the color of updated elements and the opacity of added and removed elements.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50a23120d5ddd8964f6ebac500915787/href">https://medium.com/media/50a23120d5ddd8964f6ebac500915787/href</a></iframe><p>The 3 set selection paradigm may seem foreign at first, but complex transitions become incredibly easy to express once you are familiar with how the selection system works.</p><h3>Learn more</h3><p>D3 data selections have many more features that don’t fit in this post. Check out the D3 project documentation.</p><ul><li>Data re-ordering is gracefully handled by default, but there is an API for customizations which maybe useful especially if you want to animate data ordering changes: <a href="https://github.com/d3/d3-selection/blob/master/README.md#selection_order">ordering docs</a>.</li><li>D3 has an event handling system for DOM interactions: <a href="https://github.com/d3/d3-selection/blob/master/README.md#handling-events">event docs</a>.</li><li><a href="https://github.com/d3/d3-selection/blob/master/README.md#joining-data">In-depth details of D3’s data join system</a>.</li><li><a href="https://bost.ocks.org/mike/nest/">Old, but good deep-dive into nested selections.</a></li></ul><p>Stay tuned for <a href="https://blog.freshtracks.io/d3-with-react-part-2-9364ef2c0a5a">part 2</a> when I’ll investigate wrapping D3 code in a React component to add state management and build a functioning app.</p><p><strong><em>FreshTracks.io is a monitoring tool that layers machine learning on top of Kubernetes and Prometheus. To learn more, </em></strong><a href="https://www.freshtracks.io/beta"><strong><em>join our beta</em></strong></a><strong><em> today!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fe1418b958a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/freshtracks-io/an-introduction-to-d3-data-binding-part-1-1fe1418b958a">An introduction to D3 data binding (Part 1)</a> was originally published in <a href="https://medium.com/freshtracks-io">FreshTracks.io</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>