<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jon Mease on Medium]]></title>
        <description><![CDATA[Stories by Jon Mease on Medium]]></description>
        <link>https://medium.com/@jonmmease?source=rss-91ccb2ee730e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kJmtHrajmd2fFrsKlSUyOg.jpeg</url>
            <title>Stories by Jon Mease on Medium</title>
            <link>https://medium.com/@jonmmease?source=rss-91ccb2ee730e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:55:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jonmmease/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introducing VlConvert]]></title>
            <link>https://medium.com/@jonmmease/introducing-vlconvert-c763f0076e89?source=rss-91ccb2ee730e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c763f0076e89</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[vega-lite]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jon Mease]]></dc:creator>
            <pubDate>Mon, 10 Oct 2022 17:05:42 GMT</pubDate>
            <atom:updated>2022-10-17T19:14:51.239Z</atom:updated>
            <content:encoded><![CDATA[<h4>Embedding JavaScript in Python with Rust</h4><figure><img alt="Left-to-right arrow labeled VlConvert pointing from the Vega-Lite logo to the Vega logo" src="https://cdn-images-1.medium.com/max/960/1*T9dAZmigENflJeC5OPWGLw.png" /></figure><h3>Background</h3><p>As I’ve written about <a href="https://medium.com/@jonmmease/announcing-vegafusion-570f62207ba7">previously</a>, <a href="https://vegafusion.io/">VegaFusion</a> is a project that provides server-side scaling for data visualizations specified using the <a href="https://vega.github.io/vega/">Vega</a> visualization grammar. The core VegaFusion runtime is written is Rust and embedded as a native Python library.</p><p>Vega-Lite is a higher-level visualization grammar that compiles into Vega. The popular <a href="https://altair-viz.github.io/">Altair</a> library provides a Pythonic API for creating Vega-Lite visualizations, where the actual compilation of Vega-Lite to Vega is implemented by the <a href="https://www.npmjs.com/package/vega-lite">Vega-Lite Typescript library</a>.</p><p>Currently, VegaFusion’s Python support is tightly coupled to a <a href="https://ipywidgets.readthedocs.io/en/stable/">Jupyter Widget</a> which performs this Vega-Lite to Vega compilation in the web-browser. I would like to break this tight coupling by moving the Vega-Lite to Vega compilation to Python, making it possible to use VegaFusion in Python environments that support Vega but not Jupyter Widgets.</p><h3>The Challenge</h3><p>This raises the question, how can a Typescript library be embedded in Python? Vega and Vega-Lite support <a href="https://vega.github.io/vega/usage/#node">running in Node.js</a>, so one option is to have the Python library call out to Node.js as a subprocess. This works, but it’s not very satisfying as it requires the end user to install Node.js, which is not available on PyPI and so cannot be installed automatically as a dependency when installing VegaFusion. It also requires somehow distributing a JavaScript bundle that includes Vega-Lite and all of its dependencies.</p><p>What I really want is a way to embed the Vega-Lite bundle and the Node.js runtime (or something like it) directly into a self-contained native Python library.</p><h3>Potential Embedding Solutions</h3><h4>Node.js in C++</h4><p>Based on the documentation, it is possible to <a href="https://nodejs.org/api/embedding.html">embed Node.js</a> into a C++ application, and so it should be possible to wrap the Node.js runtime in Python by adding a C-shim around a custom C++ library. Or perhaps <a href="https://www.boost.org/doc/libs/1_80_0/libs/python/doc/html/index.html">Boost Python</a> could be used to interface with Node.js directly. Regardless, I don’t enjoy working in C++ (or dealing with C++ build systems), so I didn’t seriously consider this option.</p><h4>Deno in Rust</h4><p>Deno is an exciting project created by <a href="https://en.wikipedia.org/wiki/Ryan_Dahl">Ryan Dahl</a> (The original author of Node.js). As described on the <a href="https://deno.land/">Deno website</a>:</p><blockquote>Deno is a simple, modern and secure runtime for JavaScript, TypeScript, and WebAssembly that uses V8 and is built in Rust.</blockquote><p>If you’re interested in learning more about Deno, I encourage you to check out the project website and this excellent <a href="https://blog.logrocket.com/what-is-deno/">blog post</a> by L<a href="https://logrocket.com/">ogRocket</a>.</p><p>For the problem at hand, the key feature of Deno is that it’s developed as a collection of Rust crates that can be <a href="https://github.com/denoland/deno/tree/main/core/examples">easily embedded</a> in a Rust project. Such a project can then be exposed to Python using the excellent <a href="https://pyo3.rs/">PyO3</a> crate.</p><p>This approach, wrapping a core Rust library for use in Python with PyO3, is exactly how VegaFusion works, and so it was an appealing starting point for this project.</p><h3>VlConvert</h3><p>I’ve created a new project called VlConvert (<a href="https://github.com/jonmmease/vl-convert">GitHub</a>, <a href="https://pypi.org/project/vl-convert-python/">PyPI</a>) that performs Vega-Lite to Vega compilation using the <em>Deno in Rust</em> approach described above.</p><p>This particular library is pretty niche, but I wanted to share some details on how it works in case they are useful to others who might want to accomplish something similar. All in all, I found this to be a pretty elegant approach to making a JavaScript library available in Python.</p><h4>Vendoring Vega-Lite</h4><p>The <a href="https://github.com/jonmmease/vl-convert/tree/v0.1.0/vl-convert-vendor">vl-convert-vendor</a> crate in the VlConvert project is responsible for downloading the Vega-Lite source code (and all of its dependencies) using <a href="https://deno.land/manual@v1.26.1/tools/vendor">deno vendor</a> and creating the <a href="https://github.com/jonmmease/vl-convert/blob/v0.1.0/vl-convert-rs/src/module_loader/import_map.rs">import_map.rs</a> source file that embeds the contents of all of the vendored JavaScript files as Rust strings using the <a href="https://doc.rust-lang.org/std/macro.include_str.html">include_str!</a> macro.</p><p>One thing to note, Vega-Lite and it’s dependencies are pulled from the <a href="https://www.skypack.dev/">Skypack CDN</a>, which is designed to work well with Deno. See the <a href="https://deno.land/manual@v1.14.2/npm_nodejs/cdns"><em>Packages from CDNs</em></a> section of the Deno documentation for more info on what that means.</p><h4>Rust library</h4><p>The vl-convert-rs crate (<a href="https://github.com/jonmmease/vl-convert/tree/v0.1.0/vl-convert-rs">GitHub</a>, <a href="https://crates.io/crates/vl-convert-rs">crates.io</a>) is a pure Rust library that interfaces with Deno. It implements a custom module loader (as an alternative to Deno’s built-in FsModuleLoader used in the deno_core <a href="https://github.com/denoland/deno/blob/main/core/examples/fs_module_loader.rs">examples</a>) that loads module source code from import_map.rs. This is how VlConvert is able to embed the Vega-Lite source code directly in the library binary.</p><p>After initializing Vega-Lite, conversion requests are handled by executing JavaScript snippets that return the converted Vega specifications as strings.</p><h4>CLI application</h4><p>The vl-convert crate (<a href="https://github.com/jonmmease/vl-convert/tree/v0.1.0/vl-convert">GitHub</a>, <a href="https://crates.io/crates/vl-convert">crates.io</a>) uses <a href="https://docs.rs/clap/latest/clap/">clap</a> to wrap the vl-convert-rs library as a simple CLI application. The vl-convert CLI can be installed using <a href="https://doc.rust-lang.org/cargo/">cargo</a> with:</p><pre>$ cargo install vl-convert<br>$ vl-convert --help</pre><pre>CLI application for converting Vega-Lite visualization specifications to Vega specifications<br><br>Usage: vl-convert [OPTIONS] --input-vegalite-file &lt;INPUT_VEGALITE_FILE&gt; --output-vega-file &lt;OUTPUT_VEGA_FILE&gt;<br><br>Options:<br>  -i, --input-vegalite-file &lt;INPUT_VEGALITE_FILE&gt;<br>          Path to input Vega-Lite file<br>  -o, --output-vega-file &lt;OUTPUT_VEGA_FILE&gt;<br>          Path to output Vega file to be created<br>  -v, --vl-version &lt;VL_VERSION&gt;<br>          Vega-Lite Version. One of 4.17, 5.0, 5.1, 5.2, 5.3, 5.4, 5.5 [default: 5.5]<br>  -p, --pretty<br>          Pretty-print JSON in output file<br>  -h, --help<br>          Print help information<br>  -V, --version<br>          Print version information</pre><p>If there is demand, this CLI application could also be published as a self-contained binary for each operating system.</p><h4>Python library</h4><p>The vl-convert-python crate (<a href="https://github.com/jonmmease/vl-convert/tree/v0.1.0/vl-convert-python">GitHub</a>, <a href="https://pypi.org/project/vl-convert-python/">PyPI</a>) uses <a href="https://pyo3.rs/v0.17.2/">PyO3</a> to wrap vl-convert-rs as a Python library. Using the <a href="https://maturin.rs/tutorial.html">maturin</a> build tool, this crate is compiled to native Python wheels for Linux, MacOS, and Windows.</p><p>A result that I’m very happy with is that the compiled wheels are only ~14MB in size. While not apples-to-apples, this compares quite favorably to the ~30MB size of the compressed Node.js binary for MacOS.</p><h3>Future work: Image export</h3><p>A natural extension of VlConvert is to support static image export. Vega already supports static image export in Node.js, so this <em>almost</em> already works. The difficulty is that Vega, when running in Node.js, relies on the <a href="https://github.com/Automattic/node-canvas">node canvas</a> library for text measurements and raster image generation. Unfortunately, the node canvas library does not work in Deno.</p><p>An approach I would like to explore in the future is to update Vega to support custom text measurement calculations, and then try using <a href="https://github.com/RazrFalcon/resvg">resvg</a> for text measurement and svg to png rasterization.</p><h3>Update — 10/17/2022</h3><p>The image export approach overviewed above worked out well! It turned out to be straightforward to override Vega’s text measurement functionality, so no change was required in Vega itself to achieve accurate text placement using resvg instead of node canvas. Version 0.3.0 of VlConvert (both the Python library and CLI application) now supports both SVG and PNG static image export.</p><p>Try it out now on <a href="https://mybinder.org/v2/gh/jonmmease/vl-convert/main?labpath=vl-convert-python%2Fnotebooks%2Fconvert_vegalite.ipynb">Binder</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c763f0076e89" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Announcing VegaFusion]]></title>
            <link>https://medium.com/@jonmmease/announcing-vegafusion-570f62207ba7?source=rss-91ccb2ee730e------2</link>
            <guid isPermaLink="false">https://medium.com/p/570f62207ba7</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[vega-lite]]></category>
            <category><![CDATA[altair]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[vegas]]></category>
            <dc:creator><![CDATA[Jon Mease]]></dc:creator>
            <pubDate>Thu, 27 Jan 2022 14:17:32 GMT</pubDate>
            <atom:updated>2023-08-23T11:06:48.619Z</atom:updated>
            <content:encoded><![CDATA[<h4>Server-side acceleration for the Vega visualization grammar with Rust and WebAssembly</h4><p>Today, I’m happy to announce a new open source project I’ve been working on since stepping down as Plotly’s Chief Scientist this past fall. It’s called VegaFusion.</p><p>VegaFusion provides server-side acceleration for the <a href="https://vega.github.io/">Vega</a> visualization grammar. While not limited to Python, the initial application of VegaFusion is the acceleration of the <a href="https://altair-viz.github.io/">Altair</a> Python interface to <a href="https://vega.github.io/vega-lite/">Vega-Lite</a>. Its design was motivated by years of experience working on visualization libraries (<a href="https://plotly.com/python/">Plotly</a>, <a href="https://holoviews.org/">HoloViews</a>, and <a href="https://datashader.org/">Datashader</a>) and dashboard technologies (<a href="https://dash.plotly.com/">Dash</a>, <a href="https://ipywidgets.readthedocs.io/en/latest/index.html">Jupyter Widgets</a>, and <a href="https://panel.holoviz.org/">Panel</a>).</p><h3>Quickstart: Accelerate Altair in Jupyter</h3><p>VegaFusion can be used to provide server-side acceleration for Altair visualizations when displayed in Jupyter contexts (Classic notebook, JupyterLab, and Voila). It also eliminates the Altair MaxRowsError when generating a chart with over 5000 rows.</p><p>First, install the vegafusion-jupyter package, along with vega-datasets for the example below.</p><pre>$ pip install &quot;vegafusion-jupyter[embed]&quot; vega-datasets</pre><p>Then, open a Jupyter notebook (either the classic notebook, or a notebook inside JupyterLab), and run these two lines to import and enable the VegaFusion Altair renderer.</p><pre><strong>import vegafusion as vf</strong><br>vf.jupyter.enable()</pre><p>VegaFusion will now be used to accelerate any Altair chart. For example, here’s the <a href="https://altair-viz.github.io/gallery/selection_layer_bar_month.html">interactive average</a> Altair gallery example.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fxe8KQZ7hM1k%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dxe8KQZ7hM1k&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fxe8KQZ7hM1k%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e6f941f998384d5c8ef2cb5d975724f2/href">https://medium.com/media/e6f941f998384d5c8ef2cb5d975724f2/href</a></iframe><p>Histogram binning, aggregation, selection filtering, and average calculations will now be evaluated in the Python kernel process with efficient parallelization, rather than in the single-threaded browser context.</p><p>You can see that VegaFusion acceleration is working by noticing that the Python <a href="https://experienceleague.adobe.com/docs/experience-platform/data-science-workspace/jupyterlab/overview.html?lang=en#kernel-sessions">kernel is running</a> as the selection region is created or moved. You can also notice the VegaFusion logo in the dropdown menu button.</p><h3>Vega Ecosystem Background</h3><p>VegaFusion is designed to complement the Vega visualization ecosystem. In particular, the <a href="https://vega.github.io/">Vega</a>, <a href="https://vega.github.io/vega-lite/">Vega-Lite</a>, and <a href="https://altair-viz.github.io/">Altair</a> projects. If you’re not familiar with these projects, it will be helpful to take a few minutes to browse their documentation as background for understanding what VegaFusion adds.</p><h4>Transforms and Signals</h4><p>One powerful feature of the Vega visualization grammar is that it includes a rich collection of data manipulation functions called <a href="https://vega.github.io/vega/docs/transforms/">transforms</a>. Transforms have functionality that is similar to that provided by SQL queries or Pandas DataFrame operations, but they are specifically designed to cover data preprocessing tasks that are useful in constructing data visualizations.</p><p>For additional flexibility, Vega also provides the concept of <a href="https://vega.github.io/vega/docs/signals/">signals</a>. These are scalar variables that can be constructed using the <a href="https://vega.github.io/vega/docs/expressions/">Vega expression language</a>, which is a subset of JavaScript. Transforms can accept and produce signals.</p><p>There are at least two significant advantages to having data transformations and signals included in a visualization specification.</p><ul><li>First, it makes it possible for a visualization to accept raw data files as input and then perform its own data cleaning and manipulation. This often removes the need to generate temporary intermediary data files.</li><li>Second, it enables higher-level libraries like Vega-Lite to automate the creation of rich interactive visualizations with features like <a href="https://vega.github.io/vega-lite/examples/interactive_layered_crossfilter.html">cross filtering</a> and <a href="https://altair-viz.github.io/gallery/select_detail.html">drill down</a>.</li></ul><h4>Motivation for VegaFusion</h4><p>Vega makes it possible to create declarative JSON specifications of rich interactive visualizations that are fully self-contained. They can run entirely in a web browser without requiring access to an external database or a Python library like Pandas.</p><p>For datasets of a few tens of thousands rows or fewer, this architecture often results in extremely smooth and responsive interactivity. However, this architecture doesn’t scale very well to datasets of hundreds of thousands of rows or more. This is the gap the VegaFusion aims to fill.</p><h3>How VegaFusion Works</h3><p>VegaFusion has two primary components: The Planner and the Runtime.</p><h4>Planner</h4><p>The Planner starts with an arbitrary Vega specification (typically generated by Vega-Lite, but this is not a requirement). The Planner’s job is to partition the specification into two valid Vega specifications, one that will execute in the browser with the Vega JavaScript library, and one that will execute on the server with the VegaFusion Runtime.</p><p>VegaFusion does not (yet) provide full coverage of all of Vega’s transforms and all of the features of the Vega expression language. The planner uses information about which transforms and expression functions are supported to make decisions about which parts of the original Vega specification can be included in the resulting server specification. The advantage of this approach is that VegaFusion can accept any Vega specification, and as more support is added over time, more of the input specification will be eligible for inclusion in the server specification.</p><p>Along with the client and server specifications, the planner also creates a communication plan. The communication plan is a specification of the datasets and signals that must be passed from server to client, and from client to server in order for the interactive behavior of the original specification to be preserved.</p><h4>Runtime</h4><p>After planning, the server specification is compiled into a VegaFusion task graph. The job of the Runtime is to calculate the value of requested nodes within a task graph.</p><p>A task graph includes the values of the root nodes (those with no parents), but it does not include the values of any of the interior nodes (those with parents). Each node in the task graph is a pure function of the values of its parents. This enables the Runtime to calculate the value of any node in the Task graph from the specification, while keeping the overall task graph size small enough to be efficiently transferred between the client and the server. The Runtime uses fingerprinting and precise caching to avoid repeated calculations of the same nodes. The cache is “precise” in the sense that cached values can be shared across visualizations that share a common substructure, even if the full specifications are not identical.</p><h3>Technology</h3><h4>VegaFusion Technology Stack</h4><p>VegaFusion uses a fairly diverse technology stack. The Planner and Runtime are both implemented in Rust.</p><p>In the context of vegafusion-jupyter, the Planner is compiled to WebAssembly using <a href="https://github.com/rustwasm/wasm-pack">wasm-pack</a> and wrapped in a TypeScript API using <a href="https://github.com/rustwasm/wasm-bindgen">wasm-bindgen</a>. This TypeScript API is used to integrate the WebAssembly library into the VegaFusion Jupyter Widget.</p><p>The Runtime is wrapped in a Python API using <a href="https://github.com/PyO3/pyo3">PyO3</a>, resulting in the vegafusion-python package. The vegafusion-jupyter package is used to integrate vegafusion-python with Altair and the Python portion of VegaFusion Jupyter Widget.</p><p>The Task Graph specifications are defined as protocol buffer messages. The <a href="https://github.com/tokio-rs/prost">prost</a> library is used to generate Rust data structures from these protocol buffer messages. When Arrow tables appear as task graph root values, they are serialized inside the protocol buffer specification using the <a href="https://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc">Apache Arrow IPC format</a>. The binary representation of the task graph protocol buffer message is what is transferred across the Jupyter Comms protocol.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GNJ-kxHFMGkX9eTM.png" /><figcaption>VegaFusion Architecture</figcaption></figure><h4>DataFusion integration</h4><p><a href="https://github.com/apache/arrow-datafusion">Apache Arrow DataFusion</a> is an SQL compatible query engine that integrates with the Rust implementation of Apache Arrow. VegaFusion uses DataFusion to implement many of the Vega transforms, and it compiles the Vega expression language directly into the DataFusion expression language. In addition to being really fast, a particularly powerful characteristic of DataFusion is that it provides many interfaces that can be extended with custom Rust logic. For example, VegaFusion defines many custom UDFs that are designed to implement the precise semantics of the Vega expression language and the Vega expression functions.</p><h3>Examples</h3><p>VegaFusion is compatible with all of the examples in the <a href="https://altair-viz.github.io/gallery/index.html">Altair gallery</a>. Here are a few additional examples that demonstrate the benefits of server-side acceleration.</p><h4>Large Altair Gallery Example: Interactive Crossfilter</h4><p>The following example is a modifications of the <a href="https://altair-viz.github.io/gallery/interactive_layered_crossfilter.html">Interactive Crossfilter</a> Altair Gallery example that increases the dataset size from 2,000 rows to 200,000 rows. You can try it out on <a href="https://gesis.mybinder.org/binder/v2/gh/vegafusion/demos/93a9a68100de0d6282b114f8cf01cbb30c672267?urlpath=lab%2Ftree%2Fnotebooks%2Flarge_altair_examples%2Fflights_crossfilter.ipynb">Binder</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F90-s5VH33L4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D90-s5VH33L4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F90-s5VH33L4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5530860c17d150cad319de37fc7d35f1/href">https://medium.com/media/5530860c17d150cad319de37fc7d35f1/href</a></iframe><h4>10 Million Taxi Rides</h4><p>This example creates a dashboard to explore 10 million taxi rides in New York City from January 2015. It supports zoom-based rasterization of the pickup locations (similar to <a href="https://datashader.org/">Datashader</a>), and combines that with cross filtering across a histogram of trip distances and a heatmap of tip rate by hour and day of the week.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FymuxCAg49GI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DymuxCAg49GI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FymuxCAg49GI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5991bbde19f6b16903c99d66adbd909/href">https://medium.com/media/c5991bbde19f6b16903c99d66adbd909/href</a></iframe><h3>About the Name</h3><p>There are two meanings behind the name “VegaFusion”</p><ul><li>It’s a reference to the <a href="https://github.com/apache/arrow-datafusion">Apache Arrow DataFusion</a> library described above</li><li>Vega and Altair are named after stars, and stars are powered by nuclear fusion</li></ul><h3>Licensing and Funding</h3><p>At least until a sustainable funding model is established, VegaFusion will be developed under the <a href="https://www.gnu.org/licenses/agpl-3.0.en.html">AGPLv3 license</a>. This is a copy-left license in the GPL family of licenses. As with all <a href="https://opensource.org/licenses/alphabetical">OSI approved licenses</a>, there are no restrictions on what code licensed under the AGPL can be used for. However, the requirements for what must be shared publicly are greater than for licenses that are more commonly used in the Python ecosystem like <a href="https://opensource.org/licenses/Apache-2.0">Apache-2</a>, <a href="https://opensource.org/licenses/MIT">MIT</a>, and <a href="https://opensource.org/licenses/BSD-3-Clause">BSD-3</a>.</p><p><strong>Update:</strong> VegaFusion is now sponsored by <a href="https://hex.tech/">Hex</a> and version 1.0+ of VegaFusion are released under the same license as Vega, Vega-Lite, and Altair: BSD-3</p><h3>Roadmap</h3><p>This is only the beginning for VegaFusion. There are a lot of possible future directions, and many of them are detailed in the <a href="https://vegafusion.io/roadmap.html">roadmap</a>.</p><h3>Learn more</h3><p>Here are some ways that you can get in touch to learn more.</p><ul><li>Read the <a href="https://vegafusion.io/">documentation</a></li><li>Check out the code on <a href="https://github.com/vegafusion/vegafusion/">GitHub</a></li><li>Start a discussion on <a href="https://github.com/vegafusion/vegafusion/discussions">GitHub Discussions</a></li><li>And of course, if something doesn’t work please <a href="https://github.com/vegafusion/vegafusion/issues">open an issue</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hEH0FQNFfI4zv5Sn-InXfQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=570f62207ba7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Dash HoloViews]]></title>
            <link>https://medium.com/plotly/introducing-dash-holoviews-6a05c088ebe5?source=rss-91ccb2ee730e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a05c088ebe5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[holoviews]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[dash]]></category>
            <dc:creator><![CDATA[Jon Mease]]></dc:creator>
            <pubDate>Thu, 03 Dec 2020 16:20:38 GMT</pubDate>
            <atom:updated>2022-07-22T21:12:13.827Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re happy to announce the release of Dash HoloViews. This collaboration between the HoloViews and Dash projects makes it possible to build certain classes of interactive Dash applications without the need to manually define any callbacks.</p><p>Two particularly powerful use cases are the ability to automatically link selections across multiple plots (also known as crossfiltering) and to display large datasets using Datashader. Both of these use cases can be implemented on top of pandas, Dask, or GPU accelerated cuDF DataFrames.</p><blockquote>For a demo and Q&amp;A session for Dash HoloViews with Plotly’s Chief Scientist, watch our <a href="https://go.plotly.com/dash-holoviews">recorded webinar</a>!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MQKm_PZODiOsYCDIaFzzLA.gif" /><figcaption>Datashader and Linked Selections with Dash HoloViews</figcaption></figure><p>Dash HoloViews was just released as part of HoloViews version 1.14. You can install it today with pip:</p><pre>$ pip install holoviews==1.14</pre><h3>HoloViews Overview</h3><p><a href="https://holoviews.org/">HoloViews</a> is an ambitious project that aims to provide a flexible grammar of visualization types and plot interactions. HoloViews specifications can be displayed using a variety of technologies, including <a href="https://plotly.com/javascript/">Plotly</a>’s Graphing Library and <a href="https://plotly.com/dash">Dash</a>.</p><p>While HoloViews can be used to create a large variety of visualizations, for Dash users it is particularly helpful for two use cases: Automatically linking selections across multiple plots and displaying large data sets using Datashader.</p><p>HoloViews also provides a uniform interface to a variety of data structures, making it easy to start out by visualizing small pandas DataFrames and then scale up to GPU accelerated RAPIDS cudf DataFrames, or larger than memory Dask DataFrames.</p><p>For more background information, see the main HoloViews documentation at <a href="https://holoviews.org/">https://holoviews.org/</a></p><h3>Dash Overview</h3><p><a href="https://plotly.com/dash">Dash</a> is the easiest way to build highly-scalable analytic web applications using pure Python, R, or Julia. It is downloaded over 350,000 times per month and is relied upon by hobbyists, researchers, and businesses to share data analysis and visualization results, and to operationalize machine learning and data science models.</p><p>For HoloViews users, Dash provides a new deployment technology with excellent scalability characteristics. In contrast to other technologies that can be used to deploy HoloViews dashboards (Bokeh, Panel, and Voila), Dash is unique in that it stores all user-level session data exclusively in the user’s web browser. This means that server memory requirements depend only on the app itself, and they do not increase linearly with the number of simultaneous users.</p><p>Not only does Dash’s architecture make it possible for a single server to support many simultaneous users, it also makes it easy to horizontally scale an application across multiple servers using a load balancer like <a href="https://www.nginx.com/">NGINX</a> or <a href="https://plotly.com/dash/kubernetes/">Dash Enterprise Kubernetes</a>.</p><h3>Linking Selections</h3><p>HoloViews provides a really powerful link_selections transformation that automates the process of setting up linked selections across plots. This is a fairly sophisticated function that understands the semantics of how each plot type was generated from a source data set, and how selections in the plot’s local coordinates represent symbolic expressions that can be used to filter the original dataset. For more information, see the <a href="https://holoviews.org/user_guide/Linked_Brushing.html%5E">Linked Brushing</a> section in the HoloViews documentation.</p><p>Thanks to Dash HoloViews, the full power of the link_selections transformation can be used in Dash. Here is an example of linking a scatter plot to a histogram to visualize the classic iris data set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*F6SQ1c9NJoQnra-2Fab9YA.gif" /><figcaption>Automatic Linked Selections</figcaption></figure><p>See the <a href="https://dash.plotly.com/holoviews">Dash HoloViews documentation</a> for the full source code of this example.</p><h3>Visualizing Large Data sets with Datashader</h3><p>Another HoloViews feature that is particularly convenient for Dash users is the integration with <a href="https://datashader.org/">Datashader.</a></p><p>Datashader is a Python library for quickly creating a variety of principled visualizations of large datasets.</p><p>While the Plotly.js <a href="https://plotly.com/python/webgl-vs-svg/">WebGL accelerated</a> scatter trace can handle hundreds of thousands of points, Datashader can handle tens to hundreds of millions. The difference is that rather than passing the entire data set from the Python server to the browser for rendering, Datashader rasterizes the data set to a heatmap or image, and only transfers this heatmap or image to the browser for rendering.</p><p>To effectively use Datashader in an interactive context, it’s necessary to re-render the data set each time the figure viewport changes. This can already be accomplished in Dash by installing a callback function that listens for changes to the relayoutData prop, but coordinating the full update process is not straightforward.</p><p>With HoloViews, the datashade transformation can simply be applied to a scatter element without the need to write any callbacks defining how to re-evaluate the datashading operation in response to viewport changes.</p><p>This example loads the same iris data set, but then duplicates it many times with added noise to generate a DataFrame with 1.5 million rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*YCUWIQ-t5UH01UYru5OIDw.gif" /></figure><p>See the <a href="https://dash.plotly.com/holoviews">Dash HoloViews documentation</a> for the full source code of this example.</p><h3>Combining Linked Selections and Datashader</h3><p>It’s even possible to combine the link_selections and datashade transformations to create linked visualizations across large datasets. This example shows how the two previous examples can be combined to support linking selections across a histogram and a datashaded scatter plot of 1.5 million points.</p><p>Notice how the box selection tool can be used to select regions of space in each figure and how the selection of the corresponding points is displayed in both figures. Also, notice how the datashaded scatter plot dynamically resamples after zoom and pan operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*BKgT27Cs5PfpkW-mfXrvJA.gif" /></figure><p>See the <a href="https://dash.plotly.com/holoviews">Dash HoloViews documentation</a> for the full source code of this example.</p><h3>Mapping support</h3><p>HoloViews allows most 2-dimensional visualization types to be overlaid on top of a map. Here is a more refined example of displaying the 10 million row NYC Taxi data set that is provided by the PyViz project (<a href="https://examples.pyviz.org/nyc_taxi/nyc_taxi.html">https://examples.pyviz.org/nyc_taxi/nyc_taxi.html</a>).</p><p>The drop-off locations are rendered using Datashader on top of a Mapbox map, and is accompanied by a histogram of fare amounts. The datashaded scatter plot automatically resamples in response to zoom and pan events, and selections on both plots are linked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MQKm_PZODiOsYCDIaFzzLA.gif" /></figure><p>This full app is under 100 lines of code with zero callbacks. For the full source code, see the GitHub repository at <a href="https://github.com/plotly/dash-holoviews-taxi">https://github.com/plotly/dash-holoviews-taxi</a>.</p><h3>GPU Acceleration with RAPIDS</h3><p>Many HoloViews operations, including datashade and link_selections, can be accelerated on modern NVIDIA GPUs using technologies from the <a href="https://developer.nvidia.com/rapids)">RAPIDS</a> ecosystem. All of the previous examples can be GPU-accelerated simply by replacing the pandas DataFrame with a <a href="https://github.com/rapidsai/cudf">cuDF</a> DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RluM-tvkD3O5qEg-" /></figure><p>The ability to develop exploratory data analysis workflows using pandas, and then later GPU accelerate them with almost no code changes is one of the most exciting recent developments in the PyData ecosystem.</p><h3>Acknowledgments</h3><p>The Dash HoloViews integration was funded by the NVIDIA RAPIDS project as part of the ongoing <a href="https://medium.com/plotly/plotly-and-nvidia-partner-to-integrate-dash-and-rapids-8a8c53cd7daf">partnership</a> between Plotly and NVIDIA.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a05c088ebe5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plotly/introducing-dash-holoviews-6a05c088ebe5">Introducing Dash HoloViews</a> was originally published in <a href="https://medium.com/plotly">Plotly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Kaleido:]]></title>
            <link>https://medium.com/plotly/introducing-kaleido-b03c4b7b1d81?source=rss-91ccb2ee730e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b03c4b7b1d81</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[plotly]]></category>
            <dc:creator><![CDATA[Jon Mease]]></dc:creator>
            <pubDate>Thu, 16 Jul 2020 13:33:51 GMT</pubDate>
            <atom:updated>2020-07-16T13:33:51.723Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introducing Kaleido ✨</h3><h4>Static image export for web-based visualization libraries with zero dependencies</h4><h3>Background</h3><p>As simple as it sounds on the surface, programmatically generating static images (e.g. raster images like PNGs or vector images like SVGs) from web-based visualization libraries (e.g. <a href="https://plotly.com/javascript/">Plotly.js</a>) is a complex problem. It’s a problem that library developers have struggled with for years, and it has delayed the adoption of these libraries among scientific communities that rely on print-based publications for sharing their research. Today we introduce <a href="https://github.com/plotly/Kaleido">Kaleido</a>: an easy to install Chromium-based library for static image export for web-based visualization libraries.</p><p>The core difficulty is that web-based visualization libraries don’t actually render plots on their own. Instead, they delegate this work to web technologies like <a href="https://developer.mozilla.org/en-US/docs/Web/SVG">SVG</a>, <a href="https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API">Canvas</a>, <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API">WebGL</a>, etc. Similar to how matplotlib relies on various <a href="https://matplotlib.org/3.2.2/tutorials/introductory/usage.html#backends">backends</a> to display figures, web-based visualization libraries rely on a web browser rendering engine to display figures.</p><p>When a figure is displayed in a browser window, it’s relatively straight-forward for a visualization library to provide an export-image button because it has full access to the browser for rendering. The difficulty arises when trying to export an image programmatically (e.g. from Python) without displaying it in a browser and without user interaction. To accomplish this, the Python portion of the visualization library needs programmatic access to a web browser’s rendering engine.</p><p>There are three main approaches that are currently in use among Python web-based visualization libraries (e.g. <a href="https://plotly.com/python/">Plotly</a>, <a href="https://docs.bokeh.org/en/latest/index.html">Bokeh</a>, <a href="https://altair-viz.github.io/">Altair</a>, <a href="https://ipyvolume.readthedocs.io/en/latest/">ipyvolume</a>, etc.):</p><ol><li>The <a href="https://selenium-python.readthedocs.io/">Selenium</a> or <a href="https://github.com/pyppeteer/pyppeteer">pyppeteer</a> Python libraries can be used to control a full system web browser such as Firefox, Chrome, or Chromium to perform image rendering.</li><li>A custom headless <a href="https://www.electronjs.org/">Electron</a> application can be used to perform image rendering using the Chromium browser engine built in to Electron. This is the approach taken by Plotly’s current <a href="https://github.com/plotly/orca">Orca</a> image export library.</li><li>When operating in the Jupyter notebook or JupyterLab, a Python library can use the <a href="https://jupyter-notebook.readthedocs.io/en/stable/comms.html">Jupyter Comms</a> protocol to communicate with a custom Jupyter extension running in the browser. This extension can perform the image export and then communicate the results back to the Python process using the Comms protocol.</li></ol><p>While approaches 1 and 2 can both be installed using conda, they still rely on all of the system dependencies of a complete web browser, even the parts that aren’t actually necessary for rendering a visualization. For example, on Linux both require the installation of system libraries related to audio (libasound.so), video (libffmpeg.so), GUI toolkit (libgtk-3.so), screensaver (libXss.so), and X11 (libX11-xcb.so) support. Many of these are not typically included in headless Linux installations like you find in JupyterHub, Binder, Colab, Azure Notebooks, SageMaker, etc. Also, conda is still not as universally available as the pip package manager and neither approach is installable using pip packages.</p><p>Additionally, both 1 and 2 communicate between the Python process and the web browser process over a local network port. While not typically a problem, certain firewall and container configurations can interfere with this local network connection.</p><p>The advantage of options 3 is that it introduces no additional system dependencies. The disadvantage is that it relies on running within a Jupyter notebook, so it can’t be used in standalone Python scripts.</p><p>The end result is that all of these visualization libraries have in-depth documentation pages on how to get image export working, and how to troubleshoot the inevitable failures and edge cases that people run into. While this is a great improvement over the state of affairs just a couple of years ago, and a lot of excellent work has gone into making these approaches work as seamlessly as possible, the fundamental limitations detailed above still result in sub-optimal user experiences. This is especially true when comparing web-based plotting libraries to traditional plotting libraries like matplotlib and ggplot2 where there’s never a question of whether image export will work in a particular context.</p><p>The goal of the Kaleido project is to make static image export of web-based visualization libraries as universally available and reliable as it is in matplotlib and ggplot2.</p><h3>The Kaleido Approach</h3><p>To accomplish this goal, <a href="https://github.com/plotly/Kaleido">Kaleido</a> introduces a new approach. The core of Kaleido is a standalone C++ application that embeds the open-source Chromium browser as a library. This architecture allows Kaleido to communicate with the Chromium browser engine using the C++ API rather than requiring a local network connection. A thin Python wrapper runs the Kaleido C++ application as a subprocess and communicates with it by writing image export requests to standard-in and retrieving results by reading from standard-out.</p><p>By compiling Chromium as a library, we have a degree of control over what is included in the Chromium build. In particular, on Linux we can build Chromium in <a href="https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md#usage-as-a-c_library">headless</a> mode which eliminates a large number of runtime dependencies, including the audio, video, GUI toolkit, screensaver, and X11 dependencies mentioned above. The remaining dependencies can then be bundled with the library, making it possible to run Kaleido in minimal Linux environments with no additional dependencies required. In this way, Kaleido can be distributed as a self-contained library that plays a similar role to a matplotlib backend.</p><h3>Improvements</h3><p>Compared to Orca, Kaleido brings a wide range of improvements to plotly.py users.</p><h4>pip installation support</h4><p>Pre-compiled wheels for 64-bit Linux, MacOS, and Windows are available on <a href="https://pypi.org/project/kaleido/">PyPI</a> and can be installed using pip. As with Orca, Kaleido can also be installed using conda.</p><h4>Improved startup time and resource usage</h4><p>Kaleido starts up about twice as fast as Orca, and uses about half as much system memory.</p><h4>Docker compatibility</h4><p>Kaleido can operate inside docker containers based on Ubuntu 14.04+ or Centos 7+ (or most any other Linux distro released after ~2014) without the need to install additional dependencies using apt or yum, and without relying on Xvfb as a headless X11 Server.</p><h4>Hosted notebook service compatibility</h4><p>Kaleido can be used in just about any online notebook service that permits the use of pip to install the kaleido package. These include Colab, Sagemaker, Azure Notebooks, Databricks, Kaggle, etc. In addition, Kaleido is compatible with the default Docker image used by <a href="https://mybinder.org/">Binder</a>.</p><h4>Security policy / Firewall compatibility</h4><p>There were occasionally situations where strict security policies and/or firewall services would block Orca’s ability to bind to a local port. Kaleido does not have this limitation since it does not use ports for communication.</p><h3>Try it out</h3><p>Kaleido can be installed using pip…</p><pre>$ pip install kaleido</pre><p>or conda.</p><pre>$ conda install -c plotly python-kaleido</pre><p>Out of the box, Kaleido supports converting Plotly figures to PNG, JPG, WebP, SVG, and PDF output formats. Support for the EPS format is available when the poppler library is installed. This can be done either using <a href="https://anaconda.org/conda-forge/poppler">conda</a>, or a system package manager.</p><p>When Kaleido is installed, plotly.py 4.9.0+ will automatically use it for image export operations, falling back to Orca if Kaleido is not available. For example…</p><pre>import plotly.express as px</pre><pre>df = px.data.iris()</pre><pre>fig = px.scatter(<br>        df, x=&quot;sepal_width&quot;, y=&quot;sepal_length&quot;, color=&quot;species&quot;<br>)</pre><pre>fig.write_image(&quot;fig.png&quot;)</pre><p>This will produce a file named fig.png in the current working directory containing this image</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*AszcwPdTHKBRg6SV" /><figcaption>PNG image produced by Kaleido</figcaption></figure><h3>Beyond plotly.py: Kaleido Scopes</h3><p>While the development of Kaleido has been motivated by the needs of the plotly.py community, we know we’re not unique in facing these challenges. We’ve designed the C++ and Python portions of Kaleido using a basic plugin architecture (Kaleido plugins are called <em>Scopes</em>) with the goal of making it possible to support image export for other web-based visualization libraries. If you’re interested in adding support to Kaleido for another web-based visualization library, check out the <a href="https://github.com/plotly/Kaleido/wiki/Scope-(Plugin)-Architecture">Scope (Plugin) Architecture</a> wiki page and let us know how we can help!</p><p>Additionally, since the core of Kaleido is a standalone C++ application that receives export requests on standard-in and writes responses to standard-out, it is relatively straightforward to build wrappers for other languages besides Python. In fact, Kaleido support will be coming soon to the <a href="https://github.com/igiagkiozis/plotly">Plotly for Rust</a> library. If you’re interested in writing a Kaleido wrapper for another language, check out the <a href="https://github.com/plotly/Kaleido/wiki/Language-Wrapper-Architecture">Language Wrapper Architecture</a> wiki page and, again, let us know how we can help.</p><h3>Learn more</h3><ul><li>For more on static image export in plotly.py 4.9+ using Kaleido, see <a href="https://plotly.com/python/static-image-export/">https://plotly.com/python/static-image-export/</a></li><li>Kaleido is open source (MIT Licensed) and available on GitHub at <a href="https://github.com/plotly/Kaleido">https://github.com/plotly/Kaleido</a></li><li>For more information on using Chromium as a C++ library, see <a href="https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md#usage-as-a-c_library">https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md#usage-as-a-c_library</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b03c4b7b1d81" width="1" height="1" alt=""><hr><p><a href="https://medium.com/plotly/introducing-kaleido-b03c4b7b1d81">Introducing Kaleido:</a> was originally published in <a href="https://medium.com/plotly">Plotly</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bringing GPU support to Datashader]]></title>
            <link>https://medium.com/rapids-ai/bringing-gpu-support-to-datashader-62a693f2c554?source=rss-91ccb2ee730e------2</link>
            <guid isPermaLink="false">https://medium.com/p/62a693f2c554</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[open-source-software]]></category>
            <dc:creator><![CDATA[Jon Mease]]></dc:creator>
            <pubDate>Mon, 10 Feb 2020 19:00:19 GMT</pubDate>
            <atom:updated>2020-02-10T19:00:19.412Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bringing GPU support to Datashader: A RAPIDS case study</h3><p><a href="https://datashader.org/">Datashader</a> is a Python library for rapidly creating principled visual representations of large datasets. It is written in pure Python and relies heavily on <a href="https://numpy.org/">numpy</a>, <a href="https://pandas.pydata.org/">pandas</a>, <a href="https://dask.org/">Dask</a>, <a href="http://xarray.pydata.org/en/stable/">xarray</a>, and the <a href="http://numba.pydata.org/">Numba</a> Just-in-Time compiler. See the Datashader <a href="https://datashader.org/">documentation</a> for examples of the kinds of visualizations it is designed to create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*C583ZEDkYnzVQmgc9jRdhw.jpeg" /><figcaption>Visualization of 300 million data points (one per person in the USA) from the 2010 census</figcaption></figure><p>The <a href="https://rapids.ai/index.html">RAPIDS</a> project was <a href="https://blogs.nvidia.com/blog/2018/10/10/gtc-europe-keynote-rapids-volvo-clara-ai/">announced</a> in late 2018, and it has quickly grown to encompass a rich ecosystem of technologies for taking advantage of the full computational power of modern NVIDIA GPUs from Python.</p><p>This post describes our experience incorporating several RAPIDS technologies to bring GPU-acceleration support to Datashader.</p><h4>Prior challenges to GPU-acceleration</h4><p>It has long been an ambition of the Datashader project to provide optional support for GPU acceleration, but prior to the RAPIDS ecosystem this had been impractical for two primary reasons:</p><ol><li>Every call to Datashader would have required transferring large data arrays to and from GPU memory, substantially reducing the potential performance gains.</li><li>Adding GPU support would have required writing and maintaining separate CPU and GPU implementations of many of Datashader’s core algorithms, significantly increasing the complexity of the codebase.</li></ol><h4>Avoiding data transfer with cuDF and CuPy</h4><p>The two primary inputs to Datashader’s rasterization methods are pandas <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html">DataFrames</a> and xarray <a href="http://xarray.pydata.org/en/stable/data-structures.html#dataset">Datasets</a>. DataFrames are used for <a href="https://examples.pyviz.org/nyc_taxi/nyc_taxi.html">point</a>, <a href="https://datashader.org/user_guide/Timeseries.html">line</a>, and <a href="https://datashader.org/user_guide/Timeseries.html#Area-plots">area</a> rasterization while Datasets are used for <a href="https://datashader.org/user_guide/Grids.html">raster</a> and <a href="https://datashader.org/user_guide/Grids.html#Quadmesh-Rasterization">quadmesh</a> rasterization.</p><p>The <a href="https://rapidsai.github.io/projects/cudf/en/0.12.0/index.html">cuDF</a> library provides a pandas-like DataFrame that is backed by GPU memory, with DataFrame operations implemented as CUDA <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#kernels">kernels</a>. Similarly, <a href="https://cupy.chainer.org/">CuPy</a> provides a numpy-like ndarray that is backed by GPU memory which can be used as a drop-in replacement for the numpy ndarrays that are wrapped by xarray Datasets.</p><p>As of Datashader 0.9.0, cuDF DataFrames can be used as replacements for pandas DataFrames in point, line, and area rasterization. And as of Datashader 0.10.0, CuPy-backed xarray Datasets can be used as replacements for numpy-backed Datasets in quadmesh rasterization. In both cases, the output produced by the rasterization method is a CuPy backed xarray Dataset.</p><p>The RAPIDS ecosystem (cuDF and CuPy in this case) solves the data transfer problem because it makes it possible to execute entire data-processing and analysis pipelines on the GPU. With these updates, Datashader can now act as a data visualization operation within a RAPIDS pipeline which, as we’ll discuss below, can provide significant performance benefits.</p><h4>Code reuse</h4><p>Many libraries in the current RAPIDS ecosystem are designed as largely compatible drop-in replacements for libraries in the traditional PyData stack (cuDF for pandas, CuPy for numpy, cuML for scikit-learn, cuGraph for NetworkX, etc.), and they <em>require</em> a compatible GPU to operate.</p><p>In contrast, the goal for Datashader has always been to provide <em>optional</em> GPU support from within the existing code base, reusing the existing APIs. Accomplishing this without introducing significant complexity to the code base requires effective strategies for maximizing the amount of logic that can be shared between CPU and GPU code paths.</p><h4>Code reuse: Data structures</h4><p>Some of Datashader’s internal algorithms are implemented using the pandas DataFrame API and the numpy ndarray API. Thanks to the degree to which cuDF and CuPy are faithful to the pandas and numpy APIs, it was fairly straightforward to update these functions to support cuDF and CuPy.</p><h4>Code reuse: Custom algorithms</h4><p>As mentioned above, many of Datashader’s internal algorithms are implemented as custom Python functions that are compiled at run-time using Numba.</p><p>Numba has long supported CUDA as a <a href="https://numba.pydata.org/numba-doc/dev/cuda/overview.html">compilation target</a>, and with the growing interoperability of the RAPIDS ecosystem, this has become even more powerful. It is now possible to use Numba to write CUDA kernels that can operate on the contents of cuDF DataFrames and CuPy ndarrays without copying and without transferring data to and from GPU memory.</p><p>Top-level CUDA kernel functions (those wrapped with the <a href="https://numba.pydata.org/numba-doc/dev/cuda/kernels.html#kernel-declaration">numba.cuda.jit</a> decorator) require a small amount of CUDA-specific execution logic and can only be invoked on a machine with a compatible GPU. However, these kernel functions can call standard CPU-compatible numba functions (those wrapped with the <a href="https://numba.pydata.org/numba-doc/dev/user/jit.html">numba.jit</a> decorator). This capability has made it possible to reuse significant portions of Datashader’s custom algorithms between the CPU and GPU code paths.</p><p>For a complete minimal example of this approach to CPU/GPU code reuse, see the 1D histogram example at <a href="https://anaconda.org/jonmmease/datashader-rapids-histogram-example">https://anaconda.org/jonmmease/datashader-rapids-histogram-example</a>.</p><h4>Performance results</h4><p>With the adoption of RAPIDS technologies, many Datashader use-cases can now be accomplished 1 to 2 orders of magnitude faster than the corresponding single-threaded CPU implementations*. The following example notebooks were executed on a consumer-grade <a href="https://www.nvidia.com/en-us/geforce/graphics-cards/rtx-2080-ti/">GeForce RTX 2080 Ti</a> with 11GB of GPU memory:</p><ul><li>10–15x speedup for generating a visualization of 300 million points: <a href="https://anaconda.org/jonmmease/datashader-rapids-census">https://anaconda.org/jonmmease/datashader-rapids-census</a>.</li><li>120x speedup for generating a visualization of a 67 million element quadmesh: <a href="https://anaconda.org/jonmmease/datashader-rapids-quadmesh">https://anaconda.org/jonmmease/datashader-rapids-quadmesh</a>.</li></ul><blockquote>* For many operations, Datashader supports multi-core, parallel, and out-of-core execution using Dask and Numba. This can be used to parallelize workloads across multiple CPUs or GPUs, but is out-of-scope for this post. For more information, see the <a href="https://datashader.org/user_guide/Performance.html">Performance</a> documentation section.</blockquote><h4>Learn more</h4><p>If you’d like to learn more about Datashader, please check out the <a href="https://datashader.org/index.html">documentation</a> and the examples at <a href="https://examples.pyviz.org/index.html">PyViz Topics</a>. And, if you’d like to learn more about the RAPIDS ecosystem, check out the <a href="https://rapids.ai/">rapids.ai</a> homepage.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62a693f2c554" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rapids-ai/bringing-gpu-support-to-datashader-62a693f2c554">Bringing GPU support to Datashader</a> was originally published in <a href="https://medium.com/rapids-ai">RAPIDS AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>