For data analysts and business leaders focused on the retail industry, being able to use data to effectively identify and forecast trends is crucial to the success of their business and to getting consumers what they need, when they need it.

With supply chains still reeling and costs rising, it has never been more important for retailers to manage their inventory in the most effective way possible… and that involves exploring and understanding a tremendous amount of complex data.

And for those of us who are not retail data analysts, this data is interesting and relevant because it is often a leading economic indicator. A positive or negative monthly retail sales report can, quite literally, move our economic markets.

In addition, improperly managed inventory could directly impact everyone on a daily basis by the scarcity of things like toilet paper, infant formula and many others. This recent article from Katie Teague at CNET outlines a list of items that may be hard to find at your local store.

In an effort to better educate ourselves at Observable, and to offer a resource to anyone following the retail industry, we created an Observable Notebook that uses visualization techniques to shed light on the monthly retail sales numbers shared by the US Census bureau. If you’re not an Observable user yet, you can create your free account here.

We encourage our community to fork this notebook and explore the data alongside our team. We’ve converted it from Excel spreadsheets to a code-and-analysis-friendly format. And if you notice something interesting, be sure to connect with us via Twitter and share what you find!

For example, do you think dataviz could have driven timely insights that could have helped us avoid the toilet paper, infant formula scarcity we’ve faced?

This is a transcript of the talk I gave at OpenVisConf 2018:

Data visualization is about exposing patterns to the eye.

We are always seeking ways to tap into deeper patterns.

Patterns that feel distinctly human

Patterns we humans can recognize but can’t articulate to a computer

And patterns we didn’t even think to look for

When exploring a new dataset we have various tools in our analysis and visualization toolkit. These tools include things like averages and summary statistics, line charts and histograms, as well as an ever expanding catalog of custom visualizations.

Now I want to direct your attention to a relatively new set of tools that can change the way we explore large datasets.

These tools use Machine Learning to pull out patterns for us and give us new ways to navigate our data.

I’d like to demonstrate these techniques on my favorite dataset, Quick, Draw!

If you haven’t had a chance to play the game, the rules of Quick, Draw! are pretty simple. The game asks you to draw a word, and you try to get an AI to guess the word from your drawing.

When the Google Creative Lab built Quick, Draw!, they had the foresight to save anonymized copies of the drawings, altering the course of my life forever. At this point millions of people across the globe have played and Google has open sourced 50 million of the drawings they created. This means we have on average more than 100,000 drawings for each of the 300 words in the game to explore.

The Data

Let’s take a close look at the data, what it is and isn’t.

This dataset makes for a great demonstration, because its so fun but it is also representative of so many serious datasets. It has categorical data, such as which word is being drawn and which country the drawing originated from. We also have a few time related dimensions, such as how long the drawing took to draw (duration), a timestamp of when the drawing was made.

We also have the sequence of points that make up the drawing. It is this sequence of pen strokes that carries most of the meaning in this dataset, they capture the way we as humans represent abstract concepts across the globe.

They are also the most difficult to dissect with traditional data visualization techniques.

Data Visualization

Just because something is difficult, doesn’t mean it can’t be done. Since the dataset was released several amazing projects applied various techniques to surface interesting patterns in the data.

How Long Does it Take to (Quick) Draw a Dog? by Jim Vallandingham

This project explores questions around complexity and quality by utilizing stroke counts and drawing durations. Some quite interesting observations can be surfaced by interactively browsing the summary statistics of these attributes.

Note that the dimensions visualized here are the count of strokes and the duration of the drawing, both reduce the sequence of strokes to a single number. These numbers give us a manageable way to peek into the data, but can’t capture all of the features of the drawings by themselves.

How Do You Draw a Circle? by Nikhil Sonnad

This article takes a deep dive into simple shapes, they determine whether each circle is drawn clockwise or counter clockwise, allowing them to count this feature in the dataset.

They can then visualize this feature to communicate some understanding about cultural differences across the world.

“There are countless ways that we subtly, unconsciously carry our cultures with us: the way we draw, count on our fingers, and imitate real-world sounds, to name a few. That’s the delight at the heart of this massive dataset.”

Forma Fluens by Visual AI Lab @ IBM Research

This project takes a number of interesting approaches to visualize the data. One in particular is the use of visual averages to highlight cultural patterns.

Visual averages work by drawing thousands of faint transparent drawings on top of each other, surfacing the dominant patterns. This works especially well when we filter drawings by country where cultural patterns can emerge.

Visual Averages by Country by Kyle McDonald

Kyle McDonald took the concept of visual averages to an extreme in this epic tweetstorm.

He makes excellent use of small multiples to compare patterns across several categories.

These certainly give us some interesting points to reflect on, but it’s hard to dig deeper since all of the nuanced patterns get washed out by the averaging.

So what if we had a way to capture the nuance lost by averages, to automatically find the interesting features in the strokes and dissect the data by more than one dimension at a time?

Machine Learning

Enter Deep Neural Networks. These aren’t magic, but they do have some amazing capabilities and as it turns out, we have just such a network trained on the Quick, Draw! dataset. It’s called sketch-rnn.

While it’s super fun to play with the network and come up with creative applications for a drawing machine, what’s exciting for us as data visualizers is the patterns its had to encode in order to generate its drawings.

So how do we get at these patterns?

One way to do it is to ask the network how probable it thinks a given input drawing is, as Colin Morris did in his Bad Flamingos article.

On top we see flamingos the network thinks are highly likely and on the bottom are flamingos it thinks are very unlikely. This gives us an interesting lens to look at the data through, but it still reduces all of the data to a single dimension. That’s a problem because some of the most interesting depictions of flamingos are mixed in with words that are clearly not a flamingo.

We would like a broader view of the data, and we can get one once we understand a little bit more about how the network operates. Sketch-rnn belongs to a family of neural networks called auto-encoders which find ways to “compress” input data into a smaller representation that can then be used to generate new output.

The network is composed of two parts, an encoder network that tries to find a way to represent the data in much fewer dimensions than the input, and a decoder network that tries to accurately reconstruct the original data using only the encoded data.

We call the encoded data a latent vector, and it’s the key to unlocking our technique.

We can extract the latent vector for each drawing from the network, which gives us a way to compare the drawings numerically.

When we compare them we see that similar latent vectors mean similar drawings. In our network, a latent vector is 128 numbers, which is still kind of a lot to deal with. So we need a way to compare a lot of high-dimensional data points with each other.

Luckily there is a wonderful algorithm called t-SNE which is very helpful for visualizing similarities in high-dimensional data. It isn’t a silver bullet, but it offers us a very interesting way to explore our data. Here each drawing is represented by a small translucent yellow dot, the algorithm places similar drawings close to each other to create this two dimensional map.

We can zoom in on a small piece of this map, and see a group of similar drawings.

As humans using our eyes, the patterns here are fairly clear, namely the eyes and smile.

Let’s look at an entirely different cluster.

We can see that this cluster highlights a pretty different pattern, which is kind of sad.

Let’s get back to the idea of studying the complexity of drawings. Instead of using proxies for complexity like number of strokes or duration of drawing we can examine complexity directly.

Here is one way to draw a simple cat, all of these have a single stroke:

Here is another way to capture the essence of catness with simplicity, though this time with 3 strokes. In both cases you could put these in front of a young child and they would say that’s a cat!

Now we depart from metrics and get into humanities:

Here are some approximately equally complex cats, but clearly we are looking at whiskers and not smiles:

We don’t need to stop there, we can have it all!

So now we are navigating through a space that is much richer than a single dimension.

Let’s revisit the problem of averaging a concept like yoga.

The problem with averages is that they assume a normal distribution with a single mode. What we will see is that there are several modes when representing yoga, starting with the different poses.

The way people draw even a single pose:

And the way people give up:

At this point I want to pause and step back from our particular dataset of drawings and make sure we’re clear on the two things going on here.

The first is that t-SNE is a general technique for visualizing high-dimensional data.

The second thing is that Neural Networks can work on all kinds of data. In this figure, taken from Chris Olah’s amazing article Deep Learning for Human Beings, paragraph vectors are visualized with t-SNE to surface topics in Wikipedia articles.

So using Neural Networks to find patterns and t-SNE to visualize them is a good idea in general.

A more mathematical term for the high-dimensional landscape created by our neural network’s internal representation is the “latent space”. We can think of t-SNE as helping us draw a map of this space.

Much like a 2D map can never truly represent our 3D globe, a 2D t-SNE map won’t be able to show us everything thats happening in higher dimension.

But it can still be a very helpful way to navigate as we explore.

Here we’ve sampled a single drawing from each grid cell and opacity indicates the number of drawings in that cell.

Let’s briefly bring back the idea of averaging by country codes.

With this view we can instead filter down by country code. We can take a quick look at Japanese power outlets:

If we zoom in we see the predominant representation of a “Type A” outlet with two vertical holes. Unlike in the averages, we also get to see some fun outliers which seemed to have understood power lifting instead of power outlet

Let’s look at another word, octopus and revisit the idea of complexity.

We can filter our map to only octopi with 1 stroke, and sample from those areas. It’s probably easy to imagine drawing an octopus in 1 go like these.

If we instead filter to all of the complicated octopi, defined by having more than 14 strokes:

We find a really fun cluster

Conclusion

The different ways in which people draw are like different notes, the harmonics of a word and the clusters we’ve explored are the result of thousands of strangers harmonizing together.

Thanks

Martin Burch 🌳 on Twitter

The trouble with D3 is to build a visualization you must also have a deep understanding of SVG, DOM, JavaScript, geometry, color spaces, data structures, the standard model, and quantum physics

Frank Elavsky ᴰᵃᵗᵃ ᵂᶦᶻᵃʳᵈ on Twitter

⬇️ Thread of thoughts on this topic ⬇️ I've taught #D3js to nearly 300 faculty, staff, and students at primarily the grad/PhD level across various workshops and bootcamps. Some key reasons behind why D3 is hard, I'll outline below (in case anyone is learning or curious) https://t.co/JzmSCy3NXD

The misconception at the heart of these threads is that d3 and data visualization are the same thing. That to do data visualization one must learn and use all of d3, and to use d3 one must learn all of data visualization. Instead, I like to emphasize that d3 is a toolkit for communicating complex data-driven concepts on the web.

What I want to get across here is how we can get a more holistic view of d3’s role in web-based data visualization. Let’s use a metaphor inspired by Miles McCrocklin where data visualization is likened to building furniture. All kinds of people might get into building furniture, for all kinds of reasons, especially when they see the beautiful things other people are making:

People see the impressive output and naturally desire the ability to make it themselves, they ask how it is done and often hear “it was made with d3.” This is the start of the problem, because when someone hears that it was made with d3, they think “oh, I should go learn d3”. They go over to the documentation and see something like this:

Many of these tools seem baffling, they require knowledge about woodworking and processes we’ve never thought about before, or even knew we might need to think about. We feel overwhelmed and discouraged, it seems the path to something that seemed within reach is long and treacherous.

This is where I believe we can change things for the better, rather than changing the toolset we can guide people based on their goals along more suitable paths for them. Let’s examine a few common situations where people find themselves wanting to do interactive data visualization and how we might plot a better course for each.

The designer

Our designer is already comfortable communicating ideas visually, they know how to break down complex problems and map them to relatable concepts. They have a suite of tools that enhance their ability to express whats in their mind. They often are not very familiar with programming, perhaps they have some experience with basic HTML and CSS for putting together static web pages. They’ve seen what people can make with d3 and are driven to be able to do the same. When they try to understand what looks like a very small amount of code in a bl.ock they get very confused.

What part of this is JavaScript? What part is specific to d3? What is an asynchronous request? What is this DOM I keep hearing about?

For these folks, d3 offers great power and flexibility, but first they must learn some foundational technical skills to operate in this environment. I often recommend Scott Murray’s excellent d3 tutorial (and book) which covers basic HTML, CSS and JavaScript concepts. I also recommend experimenting with exporting SVG from design tools like Illustrator and Sketch and imbuing them with interactivity and data magic in the browser.

When starting out, I often encourage designers not to focus on the enter/update/exit pattern, reusability or performance concerns. It’s much more helpful to focus on getting the desired output, once you have something close there are lots of friendly folks that can help you make it more performant or robust.

The analyst

Our analyst is already comfortable working with data, writing queries and calling powerful functions with complex APIs. They have a workflow in a powerful environment like R Studio or Jupyter Notebooks. Most likely they come to d3 because they want to publish their analysis in some way. While the analyst is typically more comfortable programming than the designer, they are likely not familiar with the idiosyncrasies of programming in a web browser environment.

What is the difference between SVG and Canvas? What is the JavaScript equivalent to Pandas/Tidy? Why can’t I draw a line chart with an SVG line? What is this “d” attribute on a path?

For these folks I also recommend a primer in web development to familiarize themselves with concepts like the DOM. Again, my favorite starting point is Scott Murray’s d3 tutorial (and book). I would also recommend a crash course in JavaScript and JSON, exporting data from their normal environment as JSON for visualizing in the browser.

When starting out, I often encourage analysts to ignore a lot of d3’s utility functions, as they are probably more familiar with the powerful functions in their own environments. Instead, I think its best to focus on exporting the data into an easy-to-consume JSON or CSV format that matches existing examples.

The software engineer

Our software engineer is an interesting case, because although they have a lot of the foundational skills and knowledge around web development, some of d3’s tools require a foreign way of thinking. In our metaphor, the engineer doesn’t just care about making furniture, they are working on the entire building. There are frameworks and infrastructure that the furniture has to fit inside.

What is this enter/update/exit business? Why are you messing with my DOM? Transitions… How do I unit test those?

Many developers will already be intimately familiar with the DOM and JavaScript, so my advice is to actually try and ignore the parts of d3 which focus on the DOM. Instead, become familiar with some key utilities for data visualizations like d3-scale. D3 is broken up into many smaller modules so it’s pretty easy to cherry-pick the functionality you want to use.

I also emphasize separating the layout of data from the visualization, so using a module like d3-hierarchy you can generate a data structure with d3 and then render it into the DOM using your framework of choice.

Silver bullets

These situations are loose archetypes, many people will fall somewhere between them and that’s perfectly fine too. The idea is to separate out the goals and constraints so that we can better guide the diverse folks entering our community.

I personally think of web standards as the lowest common denominator for global communication. The graphics APIs are not ideal but if you want to instantly distribute your data-driven experience to billions of people I think it’s reasonable to pay the price of a relatively steep learning curve. The underlying concepts of 2D graphics, visual design, user experience design, information architecture and programming all transfer directly to many other endeavors besides data visualization.

But sometimes, a chair is just a place to sit, we don’t have time or money to care that much and IKEA will do just fine! In those cases there are plenty of charting libraries that only need a little bit of configuration to get going.

Elijah Meeks has made a great map of the d3 API that breaks down the toolbox into useful categories in his recent article.

I’ve also attempted to map out the d3 learning landscape in my article The Hitchhiker’s Guide to D3, which gives some links and starting points for what I believe are some of the more essential concepts.

A while back I interviewed a handful of data visualizers who learned d3 in the process of expressing themselves and the datasets they cared about. The common theme was that they had started with goals. They learned what they needed from d3 along the way to achieving those goals.

So grab a map and plot your own course through the vast world of Data Visualization. You can find some trails others have blazed with Blockbuilder search, try out JavaScripts very own Notebook environment Observable, and join over 3,000 like-minded chair makers, I mean data visualizers on the d3 slack channel.

Good luck, I look forward to seeing your visualizations!

I’d like to thank Erik Hazzard, Kerry Rodden, Zan Armstrong, Yannick Assogba, Adam Pearce and Nadieh Bremer for their feedback on this article.

The Trouble with D3 was originally published in DailyJS on Medium, where people are continuing the conversation by highlighting and responding to this story.

The landscape for learning d3 is rich, vast and sometimes perilous. You may be intimidated by the long list of functions in d3’s API documentation or paralyzed by choice reviewing the dozens of tutorials on the home page. There are over 20,000+ d3 examples you could learn from, but you never know how approachable any given one will be.

If all you need is a quick bar or line chart, maybe this article isn’t for you, there are plenty of charting libraries out there for that. If you’re into books, check out Interactive Data Visualization for the Web by Scott Murray as a great place to start. D3.js in Action by Elijah Meeks is a comprehensive way to go much deeper into some regions of the API.

This guide is meant to prepare you mentally as well as give you some fruitful directions to pursue. There is a lot to learn besides the d3.js API, both technical knowledge around web standards like HTML, SVG, CSS and JavaScript as well as communication concepts and data visualization principles. Chances are you know something about some of those things, so this guide will attempt to give you good starting points for the things you want to learn more about.

Communicate Complex Concepts

Before we dive into data visualization principles and technical skills, lets take a second to be aspirational. There are some amazing examples of what’s possible in this medium, whether its a New York Times article, r2d3, distill.pub, datasketch|es, polygraph, or ncase. Be sure to leave a comment with any inspirations I didn’t list here!

Don’t just look to others though, one of the most important things you can do is set an aspiration for yourself. I found out from interviewing some of the top data visualization practitioners using d3.js that one of the best ways to learn is to set your sights something you really want to build and figure out what you need to build it.

Visual Representations

D3 does not introduce a new visual representation. Unlike Processing, Raphaël, or Protovis, D3’s vocabulary of graphical marks comes directly from web standards: HTML, SVG, and CSS
 — http://d3js.org

Charts are just rectangles with shapes inside of them. There are a handful of configurations of those shapes we recognize as common charts or graphs. What d3 provides is a way to define your own visual representations by manipulating graphical marks or creating your own shapes. It makes it easy to add interactions to visuals and declare how your visuals behave. You are here to learn how to express things that aren’t possible to express in any other medium.

If you want to learn about some of the principles behind the different kinds of marks and graphical representations people use, you can’t go wrong exploring the Grammar of Graphics.

Don’t worry though, there are a ton of creative things you can do with just circles, rectangles and some careful positioning. Start simple, always try to get something to show up on the screen and build up from there.

On the web

One of the reasons you want to use d3.js is so you can share your work instantly with anyone who uses a web browser (at least half of the people on Earth!). That means you need to have at least a passing understanding of the medium you’ll be working in, which is HTML5. So before you even start calling d3 API functions you’ll need some basics in SVG, HMTL and CSS. You’ll probably also want to learn some Canvas if you want to be serious about rendering a lot of data (don’t worry, its actually kind of easier to learn than SVG). I recommend this is a great intermediate tutorial on using d3 with Canvas after you are comfortable with the basics of both.

For SVG I recommend starting with this short but sweet SVG primer by Scott Murray. Play around with manually creating SVG elements and seeing how they work. Use a tool like BlockBuilder to quickly get started without setting up any kind of development environment. You may want to refer to the MDN reference site for SVG. Once you’ve mastered the basics, check out SVG beyond mere shapes by Nadieh Bremer.

You don’t have to use SVG to make your visualization, it’s relatively common to manipulate HTML elements like <div> tags with d3. You will need to be familiar with CSS positioning to make it work well. You can even mix HTML, SVG and Canvas all at once!

It can be a bit overwhelming to figure out which rendering system you should use, and even how to use any one of them. I’m going to reiterate the importance of knowing basic HTML, CSS, SVG (and a little Canvas) before getting started with d3.js!

Getting started with d3.js

How do you build up your visualization from first principles? Piece by piece and a whole lot of utility functions. As you’ve probably seen, d3’s API is massive, so lets call out a few of the utilities that will be particularly helpful early on.

d3-scale

Some of the most fundamental tools in the d3 tool belt are scales. Start with the Introducing d3-scale post by Mike Bostock to get an overview of what they are and how to use them. No matter what kind of visualization you do, you will likely use at least one kind of scale to make it happen.

d3-shape

SVG paths are pretty intense (see this thorough overview), so d3-shape includes functions that will make them easier to create and manipulate for some use cases. Read Mike’s Introducing d3-shape article to get an overview of what it offers and how to get started using them. d3-shape can also help you render things like lines, areas and arbitrary paths to Canvas with just one extra line of code!

d3-selection

One of the hardest parts of d3 to learn is the selection system, also referred to as the General Update Pattern. It took me months of banging my head against my desk to internalize it, but don’t let that scare you! You don’t actually need to be a master of selections to get a ton of cool stuff done. When you are ready to give it a shot, you can start with the d3-selection README and be sure to click on the links like Thinking in Joins.

d3-collection

Manipulating data is an incredibly important part of visualizing it. This can often be the hardest part to do depending on how good your data is and how well you understand it. Having more tools that make working with the data by reshaping it, slicing and dicing or aggregating it can really help. To this end I recommend getting familiar with d3-collection, particularly the nest function.

d3-hierarchy

Continuing the theme of working with data, the key part to many visualizations is laying out your visual representations based on your data’s structure. Some common functions for helping you do this are found in d3-hierarchy for making things like trees, treemaps or circle packs.

d3-zoom

A common interaction behavior you may want to add to your visualization is zooming. Mike has a series of examples that show various ways of adding zoom to your visualization with d3-zoom. Check out zoom’s close cousin d3-drag for some additional interaction help!

d3-force

One of d3’s capabilities that inspires a lot of people is the force layout. It’s pretty easy to use but definitely difficult to master. Just don’t let it pull you to the dark side! See d3-force for more details.

Search!

One final tip you can use for any API function is to use BlockBuilder’s search capability. You can also limit your searches to examples that use d3 v4!

Community

Connect with like-minded folks! We have a vibrant online community in the d3.js slack channel. Find a meetup near you, the SF Bay Area has one of the biggest groups, but you can probably find some groups near you. If you’re really dedicated, come to the annual d3.unconf held in SF every fall!

Many thanks to Erik Hazzard for editing and helping shape this post. Thanks to Kai Chang for his suggestions in this post but more so for helping shape the d3 community. Thanks to #teaching-d3 in the d3js Slack, especially Sebastian and John for their feedback. Of course Infinite thanks to Mike Bostock for creating the playground we all get to work in!

I should note that at Google, the UXE role has two lenses. Everything in this post concerns the “design lens” which is focused on prototyping as opposed to the “engineering lens” which is more focused on production code. It is possible to move fluidly between the two, and it is not uncommon for UXE’s to lead design work as well.

First we should establish some context around the role. If we break up the product development process into three major categories we’d get Research, Design and Engineering. Each of these categories can have dedicated professionals (like Google does) or skilled generalists who can wear multiple hats (most startups).

The focus of each of these domains is to answer a different fundamental question about the product. The aim of Research is to answer why. Why do people have the problems we are trying to solve? Why would they or wouldn’t they use our solutions? Design is all about figuring out what we should build. It’s also about what we shouldn’t include, what the solution looks like and what the product does. Finally, Engineering is concerned with how we build the product.

You may be tempted to string these disciplines together with a linear narrative, that we first research to understand why, then design to figure out what and then let the engineers figure out how to do it.

In reality there are feedback loops that connect each discipline and discussions that dig deeper into each of these questions. The answers to each question we ask about why brings up new questions about what to build. While trying to figure out how we can build our product we come up with new questions for what it should be and sometimes this makes us rethink why we are pursuing a particular solution in the first place.

Trying to map out all the possible ways we can tie together Research, Design and Engineering would wear us out worse than that traveling salesman. Instead, lets look at the two main opportunities I see for a UX Engineer: connecting why to what and connecting what to how.

Why to What

One of my favorite parts of the product development process is helping to get from why to what, it usually means creative applications of all my technical skills to quickly produce something that resembles a solution just enough so that we can get new information. It’s a delicate balance between achieving quality that isn’t distracting and avoiding waste on ideas that are dead on arrival. Before I joined Google I drew this diagram to illustrate where I like to spend my time:

What I’ve since realized is how much of a two-way street the why-to-what process really is. We don’t just ask why once and get the answer, most of the time we have to try several ideas, refine how we ask the questions to get better answers. Often we get to something we think is what we need to build and need to confirm that it actually answers the user’s why.

The output of this process is typically a combination of static mocks and interactive prototypes built from any variety of tools. Personally I’m doing a lot of raw HMTL/JS/CSS with some d3.js and throwing in the occasional Material polymer component. That being said, the result of this process is a better understanding communicated between any number of people involved in this phase of the product development.

What to How

Another key opportunity for those of us that straddle design and engineering is aiding in the communication between what and how. In many places Designers and Engineers are separated along this boundary, with designers describing what they want and engineers figuring out how to build that. I find this a limiting perspective because it can place people on opposite sides of an adversarial relationship. The ideal situation would see everyone trying to reach a deep and wholistic understanding of all the questions.

This is where a UXE can help the most, by increasing the bandwidth of communication between the two groups you reduce the amount of miscommunication and friction that occurs. Examples of this include building out high-fidelity interaction and motion prototypes that express a designers intentions more truthfully. It can also mean getting access to real data to surface edge cases and realistic scenarios before a design gets fully baked. My personal focus has been on working with data to facilitate the why-what-how loop around data visualization problems, which deserves a post of its own!

Gifts that keep on giving

From what I’ve seen, the above activities can be incredibly effective in speeding up and raising the quality of the product development process. There is another area I believe UX Engineers can focus on that can have a disproportionate impact and it is a side-effect of our process. I’m talking about building tools for those around you. I framed this as a perspective to adopt rather than a specific initiative or responsibility because from what I can tell it works best when you spot an opportunity. These opportunities arise in discussions with various coworkers who experience painful processes that could be alleviated via automation.

We build our prototypes with a ruthless focus on efficiency and effectiveness, balancing between fast turnaround and successful communication. If we take the same approach to building internal tools it can be surprising how helpful we can be. It just requires to look at internal development processes we encounter with the same mindset we approach our prototypes.

Let me know what you think about my perspective of the process and the role, it’s a relatively new one in the space. I’m also excited to continue my journey into the design side of product development, coming from a math and software engineering background. I certainly feel fortunate to be learning from extremely skilled practitioners in all 3 disciplines on top of being valued for doing inter-disciplinary work.

User Experience Engineering was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

On October 16th one hundred d3.js practitioners gathered for two days of intensive sharing, learning and building at the 2016 d3.unconf. We had bigger ambitions than our previous two years, and we pushed ourselves and the community in new ways.

We took our tradition of animated badges (thanks Gifpop!) to another level this year by building a self-serve app called Badgetron to let people make their own animated designs!

Don McCurdy on Twitter

Day 1 was wonderful. See you tomorrow, #d3unconf! https://t.co/zuUZJ5GjiP

The badges were sponsored by Metis, who runs a 6 week d3.js class in both SF and NYC.

One thing we want to do better next year is give everyone even more time to design their badges. Almost 40% of attendees designed their own badge, with enough time next year we should get much closer to 100%!

Day 1

After everyone got their badges and their breakfast in Galvanize’s stunning SF location, we settled in to hear the only talk of the day: Rachel Binx’s fascinating journey with d3.

Keynote: Rachel Binx

Rachel got her start with d3 at Stamen and has since used it design data-driven jewelry and help coordinate Mars missions at NASA JPL. You should really watch her talk as she does a better job summarizing exciting applications of data visualization than we possibly could do in one paragraph:

Rachel’s talk set the tone of curiosity and discovery, which was perfect because the rest of the day wasn’t about listening to talks, it was about participating in discussions. We broke out into 3 sessions for about an hour at a time. Covering a variety of attendee-chosen topics ranging from GIFs to Machine Learning (detailed with links to notes at the end of this post).

The energy at day one was amazing, many people who only knew of each others twitter handles, github usernames or slack handles got to meet in person for the first time. We had people fly in from as far as France to join the fray and share ideas. Both beginners and experts learned from each other, shared tips and resources and traded inspiration.

The organizers know that the real magic of the event comes from the attendees, but we also know that the environment in which attendees gather can have a large impact on their experience. We put a lot of thought into they day-of experience, from the breakfast and lunch to the timing and location of sessions and chill spaces for catching a breather between constant socialization. We hired an event planner to ensure smooth logistics and more structured planning, freeing up the organizers to focus on the content and the experience as well as fundraising through sponsorships to keep our ticket prices low.

If there is one big lesson we learned this year it is to start planning even earlier. Even with 2 prior years of experience we would have loved to have more time for every aspect of the process. My biggest mistake was to block further planning on key pieces like locking down a location. In 2017 we look forward to an even better multi-day event by getting all the balls rolling in parallel!

Diversity Scholarships

Another tradition we are proud to continue is the diversity scholarships we offer to members of the community who might otherwise not be willing to attend. Whether its for financial reasons or because underrepresentation discourages some people from attending homogenous tech events we wanted to provide an avenue for encouraging everyone to attend. This year we were able to provide 17 out of 100 people with free tickets thanks to illumio’s sponsorship!

Day 2

For the first time the d3.unconf had a second day! We gathered in Google’s San Francisco office for a day of hacking and learning. We got a taste of Google’s infamous catering thanks to a sponsorship from Google Cloud.

Keynote: Mariko Kosaka

Mariko gave us an inspiring keynote with plenty of encouraging ideas. Her work with d3 is both exemplary and challenging, bringing data well outside the usual confines of a two-dimensional chart.

Don’t take my word for it, watch her talk and find out:

After the keynote the room was buzzing with energy, new friends wanted to continue conversations and many people were excited to try their hands at new ideas. This was our first time trying to facilitate such a large event where people would focus on coding rather than talking. We had many questions about how this might work, how much structure should we try to impose? How do we encourage a balance between absorbing information and putting it to practice?

We ended up with some answers, though as usual nothing is as simple as it may seem. We heard excited praise as well as disappointment about the exact same elements of the day. One key part was a series of workshops led by members of the community. Many thanks to Noah Veltman, Fil and Kristin Henry for leading these. A lot of attendees enjoyed these workshops, which were only 45 minutes long and fairly unstructured, delighting in the spontaneity of discovering new concepts and trying new ideas. At the same time others had expectations of more structure and hopes of matching them at their skill levels. We anticipated this to an extent and had various community members lead intro-to-d3 sessions throughout the day. These were appreciated but could have benefited from more preparation on the organizers behalf.

In any case, much was learned, as can be seen in the communal notes from Noah’s workshop.

Fil’s Voronoi workshop included a fun hands-on activity drawing voronoi diagrams on paper before diving into any code. This is something we will certainly encourage future workshop leaders to do as it was a fun and engaging way to learn about this powerful technique.

As far as I know, we didn’t learn any lessons about lunch. Good food with good company in a beautiful setting was a great way to fuel up for a full day of hacking and learning!

#d3-meta

A cornerstone of the d3.js community is learning by example and with over 18,000 examples we have an embarrassment of riches. Being a community of data visualizers, it is only natural that we would want to visualize data about our community.

Micah Stubbs put together a curated dataset of blocks and their meta-data for attendees to visualize. We had a room reserved for anyone to come and try their hands at visualizing the data, with guidance from Micah and myself. It was great to have an accessible dataset that was relevant to most people in the room, it removed a lot of friction for people who wanted to test new visualization skills and techniques on data they could relate to. You can see some of the outcomes of the sessions in the show-and-tell section below!

Hacking

Another important component of day 2 was simply creating space for people to hack. We had two rooms where people could find a seat at a table, plug in their laptops and start coding. Many people did just that and enjoyed a few hours of coding around like-minded individuals. We heard a lot of keyboards but also a fair amount of laughs along with several oohs and aahs.

Topics

We wanted to do a better job of capturing all the knowledge that gets shared at the unconf, so we prepared a series of Google Docs for people to take notes in. All of the sessions linked here have a wealth of examples and resources to explore!

Generative Art: Notes & Links

d3.v4: Notes & Links

VR/AR: Notes & Links

Maps: Notes & Links

Modular and Reusable: Notes & Links

GIFs: Notes & Links

WebGL: Notes & Links

Machine Learning: Notes & Links

Clustering: Notes & Links

You can see all of the Shared notes if you want to dig deeper!

Show-and-tell

We had an amazing show-and-tell session, a tradition we borrowed from our d3 meetups. Over 20 people shared what they have been working on lately or what they built during the hack sessions earlier in day 2.

Micah Stubbs on Twitter

d3 blocks on a-boxes a #d3js + #aframevr experiment 🙌 @donrmccurdy @aframevr #d3unconf https://t.co/ThTOb88bH7 https://t.co/pNG1rf8qMB

I wish I had been able to capture all of the show-and-tell examples with links and photos, but a good part of the excitement is being there to see whats happening as people share for the first time! We even had some demos that we were asked not to capture with exclusive behind-the-scenes looks at data visualiztions used in products we know and love.

See you in 2017

With all the things we learned at our third annual event, we look forward to an even better 4th annual d3.unconf!

I can’t wait to see who we can bring together and what we can accomplish. It is such a joy to be a part of the d3.js community and I hope to see you at our next event!

❤

What we set out to do

We all set aside 4 hours of a Sunday afternoon to get a hands-on introduction to the leading tools for web based mapping visualization. With a repo full of workshop materials, we dove into GeoJSON, Leaflet, MapboxGL and d3.js. From the start it was clear we wouldn’t be able to cover all of the exciting examples we had collected, so our main focus was giving a solid foundation for exploring the rich world of online web-mapping and data visualization.

What went right

well organized material in a relaxed setting

There are two positive things that stood out in the feedback we heard from the students, and they go hand in hand. The first was that we connected the dots for them in terms of how all of these tools and techniques fit together. With so many d3.js examples out there in the wild, it can be overwhelming trying to find a starting point

The second thing people liked was learning by doing. It’s one thing to be introduced to new and confusing concepts like map projections and GeoJSON its another thing to get your hands dirty and start feeling for yourself how they work.

What could have been better

Something we heard from several students is that they wanted more time. We take that as a compliment, if people are asking for more instead of running away screaming we must be doing something right! This may be related to something else that was brought up, we tried to cover too much. We love this subject matter and want to share as much as we can, but we need to make the best of the limited time we have together to set people up for success. Our goal is to get the content just right so you have enough to work with when you get home (or back to work) and continue building your knowledge.

We heard several people loud and clear, you want more hands on. It’s a tough balance introducing new concepts and making time to play with them. We tried to prepare many examples that could be explored after class beyond the few we went through together. Next time we will make sure to make more space for hands on activities.

Designing with Spatial Data

What we set out to do

Stamen has been making the best damn maps and data visualizations for years. The results coming out of the studio might seem like magic but we wanted to shed some light on what is actually a very interdisciplinary exploration. Eric kicked off the day with an overview of Stamen’s design process, giving a behind the scenes look at how they think about data and design.

There is a lot to think about when it comes to making maps, from colors and accessibility to geopolitical boundaries it can be intimidating to start designing from scratch. One of our main goals was providing context for the tools and resources available to design with. Alan took us through a thorough survey of mapmaking design principles and technologies, with interactive examples along the way.

What went right

Very informative, a good primer to the complicated world of data vis on maps

It turned out people were really interested in Stamen’s design process. it can be mystifying how such beautiful output starts from raw numbers, where does one begin? how does one find their way? how do you start such a process inside an existing organization? We spent a good amount of time discussing these questions and more at the end of the workshop.

People were also very happy with the context we provided, they let us know we helped put various puzzle pieces into place for them.

What could have been better

We heard again that people wanted even more hands-on material. This is something we will definitely focus on in our next set of workshops.

Something we didn’t expect but learned could be better was the order of the 2 workshops. Next time we will put this class first. We originally thought we’d give people the technical building blocks first and then show them how we reason about them. It seems like it might be more effective to give people more context & overview before we dive in to solving problems with tools. People seemed to enjoy both classes but we hope to make them even more effective by doing things in the right order.

Lets do it again!

Stamen is hosting the second round of classes November! We are really looking forward to trying out some improvements and doubling down on what went well.

Sign up now!

Working & Designing with Spatial Data was originally published in Hi.Stamen on Medium, where people are continuing the conversation by highlighting and responding to this story.

For the third year in a row we are gathering together 100 d3.js practitioners to discuss, share and collaborate at the d3.unconf. The past two years we brought together designers, developers and data analysts for a single day of intense discussions.

This year we have 2 days: October 16 & 17, 2016

We learned a lot from the past two years, and the biggest takeaway was that one day just isn’t enough. You get to meet people you’ve only seen on twitter, you get to see inspiring works-in-progress and share your ideas all day long. You leave the event with a full notebook, your brain worn out from constant stimulation but filled with anticipation because you want to try out at least three things you heard about and you’re finally ready to go for that little idea you’ve been toying with.

But then you wake up the next day and you have to get back to life and work and your ideas go to the back burner. Not this year, this year you wake up the next day and eat breakfast with the same folks that inspired you. You find a place to sit with your new friends or a quiet spot in the hacking room. You spend the afternoon coding to your hearts content or designing with people who fly to the other side of the world tomorrow. At the end of the day you can show everyone what you tried, and you’ll see what they were inspired to do.

The d3.unconf is all about making time and space for our community to come together and do what we love. It’s an environment where everyone is learning, not by sitting down and shutting up, but by participating actively. We can’t promise what topics will be covered this year, because you determine that, but we can promise you it will be a great place to bring up whatever d3 & data visualization topics are on your mind.

day 1

We continue our unconference tradition of attendee-driven discussions throughout the day. We start with breakfast and give people time to show up. We have whiteboards with time slots that you can add topics to as well as vote for topics others have added. Before we get into the sessions Rachel Binx will inspire us with a talk drawn from her extensive experience with d3. The rest of the day will be 45 minute sessions to discuss your favorite topics. We always keep a space free to just chill, hack or have casual conversations. We will close the day out with a short presentation and an unofficial happy hour nearby.

day 2

This is where things get interesting, similar to day 1 we start with breakfast and an inspiring talk. This time by Mariko Kosaka, a great speaker and creative wielder of JavaScript. What follows is again totally up to you. You can choose to attend (or lead) one of a series of 45 minute guided tutorial sessions or you could carve out a couple hours to code away on your favorite idea or project. You can form a team and try to visualize some of our prepared datasets or you can continue a deep conversation from day 1. Our goal is to give you the time and the space to make the most of the energy you get from being surrounded by 99 other d3 & data visualization enthusiasts!

Mariko Kosaka on Twitter

Yey quick sort scarf is done! I always wanted to mix 2 of my favorites - datavis &amp; knit! Image generated by d3.js 💕

Who will be there

While we can’t just list off everyone who’s signed up already, we can promise that the organizers will be there. If thats not enough, we do know there will be plenty of teachers & tutorial makers in attendance, including Kai & Shirley who are teaching the upcoming Metis d3.js course.

We also know engineers and designers who are using d3 at companies like Apple, Twitter, Netflix, Uber and Google will be there. Of course we have plenty of creative students, design studios and startup hackers in attendance as well!

So if you’ve already signed up, we hope you are even more excited now! If you haven’t signed up…

Sign up!

P.S. If you didn’t attend in 2014 or 2015 you may not know how sweet everyone’s badges were:

This year we are going to take it a step further and let you make your own animated badge design. We will show you how once you get your ticket!

When it comes to making maps online there are many tools available, but they all have one thing in common: Geographic coordinates go in and a two-dimensional image comes out. Converting from geographic coordinates (longitude,latitude) to pixel coordinates is accomplished via a map projection. Most of the tools made for online map making focus on facilitating the navigation of a map with layers of geospatial data. Leaflet and Mapbox GL are two such tools and they do their job well. There is one tool that focuses on the projections and lets you decide how to present the geospatial data: d3.js projections.

What happens when you bring geospatial data into the rich and expressive d3.js data visualization ecosystem? The possibilities are endless, and this post attempts to explore and categorize some of them.

custom style

One of the first things that stands out is the potential for custom styling. Once you have used a projection to create SVG elements you can style them to your hearts content. The following example utilizes some of SVG’s features like <defs> and <use>, and the textures.js library to render custom patterns on a map.

It’s not just SVG either, here is an example of generating a sketchy style using an algorithm that renders to canvas.

The key idea here is that we get to use all of the tools we are familiar with in HTML5, SVG and CSS to express our geospatial data. We have complete control over the rendering, which leads to some powerful possibilities.

custom data

The most common use of d3.js projections is customizing the representation of data that has a geospatial component. The following examples show a small sample of the variety of ways that people have visualized data in a geographic context.

From these examples we can see that the size, shape and even number of maps can all be changed to suit the visualizations purpose. The wide variety can be seen in this search of d3 examples.

custom transitions

A powerful concept in d3 is transitions. Animating a visualization between two states when the underlying data changes is a great way to provide consistency to the viewer. This is no different when using geospatial data as d3 can animate a transition between two projections. You can also transition the parameters of the projection, such as the center, scale and rotation.

Projection transitions are used here to help understand the differences between projections.

This example transitions between two different representations of the data, a standard choropleth map and a dorling cartogram.

This example allows the user to choose between a globe projection and the traditional Mercator projection.

Transitions can also be abused to induce sea sickness.

As they say, “change is the only constant” so being able to incorporate change into your visualization opens the door to many interesting posibilities.

custom interaction

One of the more exciting aspects of web based visualization is the ability to include interaction. Because projections allow you to create DOM elements, you can leverage the existing browser APIs (made easier with d3’s selections and event handlers) to provide custom interactions with your geospatial data. These examples show some simple but powerful interactions with the mouse cursor.

In the following example a zoomed-in version of the map is rendered on the right hand side, centered on wherever the cursor is pointing in the left map.

This example uses fish-eye distortion around the cursor to allow the user to inspect densely plotted points inside the same map.

custom geometry

It may be desirable to create geometry from data in order to approximate it, or group it together visually. With d3 projections you can run any algorithm that works on two dimensional coordinates. A fairly common algorithm is the voronoi diagram seen in these two examples.

A lesser known but useful technique is to generate a concave hull to group data together.

The key takeaway is that d3 projections bring you into the well-studied realm of 2D graphics, your tool set can be expanded from mapping-specific to anything that has been invented for manipulating x,y coordinates.

custom composition

It is not just the rendering of the geographic shapes that can be customized, but what you do with them once they are rendered as well. Many geographic features are instantly recognizable (at least to relevant audiences) and can be used as icons. d3’s projections make it convenient to render single features independently, for arrangements outside of a traditional map.

Layering two locations in the same projection make for an educational example.

Once your geospatial data becomes a set of DOM elements, you can lay it out with the same flexibility and specifity as any other HTML and CSS project.

extending projections

d3 includes an extensive set of projections, and that set can be extended when people implement other projection with d3’s API.

Some countries have territories that are not geographically close, so this library makes composite d3 projections that plot them together.

All projections introduce some sort of distortion when they attempt to flatten the 3D globe on to a 2D plane. This example allows you visually inspect the distortion of several d3 projections.

Interrupted projections allow an aspiring cartographer to decide how the earth should be cut in the process of flattening it out, BECAUSE THE EARTH IS NOT FLAT.

You can actually print out the map and cut along the “interruption.”

Glue the result to a tennis ball and you’ve created your own custom globe!

custom conclusion

We’ve covered custom styles, data layouts, transitions, geometries, compositions and projections. The real conclusion is that we are just getting started with the potential for geospatial visualization. Use the links in all of the above examples as starting points, and don’t forget to search for more.

If you are in the San Francisco Bay Area at the end of June or beginning of July

Sign up

for our two d3.js & mapping workshops to get a solid foundation in coding and designing with d3.js projections! Read more about our philosophy and methodology behind the workshops.

An ode to d3.js projections was originally published in Hi.Stamen on Medium, where people are continuing the conversation by highlighting and responding to this story.

learning and teaching d3 together

Learning d3.js and data visualization is a long and sometimes frustrating journey. First, there is the overwhelming amount of choice. From the tens of thousands of examples, to the hundreds of API functions and millions of ways you can combine them with SVG, Canvas or WebGL. Even if you can chart a course across this vast ocean you may find many icebergs lurking just below the surface. d3 on it’s own is just a collection of commonly useful functions, but it touches many technologies that have their own ecosystems, APIs and paradigms to learn. Just ask anyone who wanted to connect some data points with a simple line about the SVG <path> element and the d attribute.

So how do we help people navigate this overwhelming sea of possibilities? I have many ideas but I want to focus on a particular strategy: 1-day intensive workshops. There are tons of online resources, but nothing beats a focused session with an expert to help put everything in context. I’m convinced that a few tweaks to the traditional workshop format can create a powerful learning experience for those trying to make this data-driven journey.

what we’ve learned about learning d3

Four years of running the Bay Area d3 Meetup have given me the privilege to meet all kinds of people climbing d3’s steep learning curve. From graphic designers to backend engineers, from college students to public administration officials we’ve had all kinds of people with all kinds of backgrounds come together and try to learn this one JavaScript library. We convene to watch expert talks, to share our techniques, to practice our craft on public datasets and to learn from each other. We’ve hosted over 100 events, from small gatherings at coffee shops to 100+ people filling up auditoriums. A few patterns have stood out to me watching all these people learn.

crossroads

One of the most interesting aspects about d3 is that it is the intersection between design and code, math and art, data and story. It brings together people with wildly different skill-sets and perspectives who have no choice but to collaborate and learn from each other. There is just too much that goes into data visualization to have one individual master it all. While that may sound discouraging, it is actually quite exciting. Learning how to communicate with and bring together these diverse skills and talents is definitely within reach. You don’t need to become a master of everything if you know how to ask for help and where to find it.

peers over everything

We’ve hosted many inspirational speakers who show amazing projects, but the most energizing aspect of our talks tend to be the Show & Tell session at the end. We give anyone who wants to demo 2 minutes, which usually results in 5 or 6 rapid fire demos of projects at various stages of completion. These demos tend to kick off all kinds of excited discussion, perhaps it is their imperfection that makes them more accessible, or the fact that the person is stuck on something just like you are (or were) that opens up the connection. Knowing that you are not alone in on this journey is a comforting and exciting feeling, and surrounding yourself with like-minded individuals can really improve the experience.

creating an environment

Something I think about a lot when organizing d3 meetups and events is the environment we are creating. The basic formula is very simple: pick a time and place, maybe line up some refreshments and then announce the event.

The “traditional” format of one or two speakers creates a lecture environment, where the attendees consume a presentation. This creates a rather passive environment, though the socialization between and after the talks is often stimulating.

We have been iterating on another format over the years, which we now call a data dive. We still pick a time and a place and try to provide refreshments, but there are no main speakers. The environment we try to create here is a productive one, where attendees participate by spending the night coding and collaborating. We do provide some basics, like clean public datasets that can serve as starting points, or experts who can answer questions and give short tutorials. We have been pleasantly surprised time and time again by the projects that come out of these, or the progress that people make when they have a couple focused hours in a productive environment.

designing a learning experience

Recognizing these patterns has inspired us to design a new type of learning experience for learning at the crossroads by creating an environment for learning with peers. I’m imagining a series of 1-day workshop events that can be taken individually or together, each one an intense session of learning and doing.

We know that there is no shortage of content online, I’ve organized some of it for several workshops in the past. These events are not about downloading content into your brain, they are about navigating the crossroads. Each event will be a sign-post, a foothold for a specific topic or set of techniques. We will make sure everyone has context for the techniques and technologies we cover, how any particular function or concept fits in with the rest of the ecosystem.

The format for each event will be a mixture of hands-on and guided experimentation. The necessities will be provided: coffee, well-organized examples, clear reference material and more exercises than we can fit in 4 hours. We will build confidence with first-hand experience and excitement with examples crafted to produce aha moments. We will provide the environment, and the students just bring their laptops and their goals.

Perhaps the biggest departure from traditional workshops is that students are asked to apply to participate. We want to ensure that everyone in the room is a peer, motivated to learn and bringing their skills and perspective to the table. Asking people to apply adds some friction for both the students and the organizers, but we believe this is essential to create the environment we want to see. Maybe we won’t get a ton of students, but the ones we do get will define the experience.

let’s do it

We are excited to pilot the first series of workshops with Stamen Design, a long-standing pillar of the data visualization community. The first workshop will focus on working with geospatial data, while the second workshop will focus on designing with geospatial data. Mapping is one of the oldest forms of data visualization, and creating maps has gotten a lot more interesting on the web thanks to d3.js and other tools we will cover. With Stamen’s deep experience in designing and making maps we think this is the perfect crossroad to start with.

Sign up

Be a part of the best damn learning environment we can think of!

d3.js workshops was originally published in Hi.Stamen on Medium, where people are continuing the conversation by highlighting and responding to this story.