Welcome back!

Each edition we try to bring you a set of interesting finds in interactive technology. If you missed the previous edition, there’s another collection of great links waiting for you when you’re done here.

This issue we’re covering:

• Speaker Diarization
• Descriptive Scene Creation
• Automatic Interaction Systems

Follow me on Github or get in touch with me on LinkedIn!

Speaker Diarization

Although we’ve recently seen vast improvements in speech recognition systems such as Amazon’s Alexa and Google’s Assistant, those improvements have for the most part come in systems that deal with a single speaker. They’re great at figuring out what’s being said — just not who’s saying it.

Speaker diarization is the process of segmenting out distinct speakers from a single audio stream. A quick way to think about it is: who spoke when.

Diarization is a necessary piece of any voice-based system that would create rich text transcripts from courtroom proceedings, meetings, podcasts, and panel discussions.

It’s important that such a system recognize new speakers it has never heard before, across a variety of languages and voices and independent of background noise. Transcripts can contain additional information such as speaker positions or neural network generated descriptions of visual elements.

I’m interested in using those transcripts in combination with the audio itself as the data source for generative content. A few examples:

• We can take data from a company like Bridgewater, who records employee conversations, and turn every conversation into a repeatable learning environment. We can put you in the scene, pause it before a decision, and ask: What would you do?
• We can take the entire history of Radio Drama and create virtual movie sets where we can shoot a million movies from any angle with any lens we want. Reimagine a favorite television show without having to recast it.
• We can take audio of a podcast or panel discussion and feed it into an environment where the audio drives an avatar’s facial or skeletal animations. Sound comes from the position of the currently speaking character, since we can map speaker classifications to entities. You’ve got a ticket to an endless virtual concert hall — the room could be replicated infinitely or expanded to match available hardware capabilities.

IBM is into speaker diarization. It is built into Watson now.

Who's speaking? : Speaker Diarization with Watson Speech-to-Text API - IBM Cloud Blog

Amazon’s not quite there yet with their Transcribe service, but it’s coming soon.

Speech To Text - Amazon Transcribe - AWS

Microsoft is into speaker diarization as well.

Notice a trend? You’ll find Google represented in the “Speaker Diarization with LSTM” paper below.

Here are some cool Python libraries to get you started with your own experiments.

Pyannote has absorbed some other libraries, Tristounet. It’s being actively developed, but be wary: “The API is unfortunately not documented yet.” :/

GitHub - pyannote/pyannote-audio: Neural building blocks for speaker diarization: speech activity detection, speaker change detection, overlapped speech detection, speaker embedding

Around here, we LOVE a paper with code. Or is it code with a paper? Here’s some excellent work on diarization in broadcast television using deep learning. Thanks to the LIMSI team in France!

yinruiqing/change_detection

Regarding datasets: if you’re already paying for transcripts of calls or meetings from another service, you’ve got a great resource.

If you don’t, or if your dataset is small and needs augmentation, here are a few different speech corpuses: meeting recordings in English, Finnish Parliament proceedings, and French Radio and TV.

• AMI Corpus
• aalto-speech/finnish-parliament-scripts
• ISLRN

I-vectors were all the rage in diarization for a while. “ I-vectors convey the speaker characteristic among other information such as transmission channel, acoustic environment or phonetic content of the speech segment.”

Front-End Factor Analysis for Speaker Verification - IEEE Journals & Magazine

Then along came deep learning. Here’s are a newer paper that starts to move away from the idea of i-vectors.

Descriptive Scene Building

One unavoidable dilemma when building virtual worlds is the void.

You can build absolutely anything you want, but you’re faced with doing so in an endless expanse that goes on as far as you can see in either direction. It’s pretty intimidating to get started, to say nothing of the effort required to fill such a vast space with attractive content.

To generate an interesting object, you’ve got to be a specialist at creating sculpts by 3D modeling, lighting, and painting textures — all while using a manual Human Interface Device like a mouse, keyboard, 3D mouse or set of 6DOF controllers.

Or you have to become an expert at scouring the internet to find objects someone else has created that fit your budget — often by compromising on the initial concept. While this practice allows creators to avoid losing momentum and occasionally leads to delightful outcomes, the end result may not be the ideal form.

After all that, you still need to arrange all of these pieces of content in the space by using a combination of rotation and movement handles attached to individual objects, one object at a time.

And if you want the objects to be anything more than decoration, you have to begin to script interactions that specify any and all possible uses of an item beyond basic physics — e.g. a piece of paper that you can write on, crumple up and throw in a trash can, that will burn when its put in a toaster, etc. (More on that below in the section on Automatic Interaction Systems)

It should be easier.

If we can find the words for it, we can make it so.

Hence, descriptive scene building as a way forward.

We’ll be able to create and change the spaces that we inhabit online in real-time. And we’ll do so by describing with natural language what we want to see and where we want to be. Intelligent systems will get to know, riff off of and even preempt our desires as they adjust the world around us to suit our current mood and intent. The atoms that we build with in mixed reality can be the smart nano-dust that engineers dream of. As we describe our desires they will manifest around us.

And if we’re having trouble articulating our desires, the ambient intelligence will help us discover them .

Pragmatic ambiguity will be a feature — if there is a possibility space wherein a number of arrangements would match your specification, you may be asked to curate your preferred embodiment. Or the system might show an arrangement that oscillates over time. Imagine a room that’s never the same but always recognizable.

This 2006 paper seems to have kicked things off. Can you imagine trying to do this with the natural language processing tools available at the time? I bet they’d love another go.

Things advanced in 2009–2010, along with NLP.

And in the past few years we’ve started to approach systems that can handle situations like: “There is a room with a chair and a computer”. It gives me goosebumps to think about all the possibilities.

I highly recommend checking out Angel Chang’s body of work, since her research continues to push the boundaries of what’s possible in these areas.

Angel Xuan Chang | Angel Xuan Chang

One interesting problem in descriptive scene building is dealing with a phrase like “A window over there”. Over where? Understanding spatial intent is important, and Bayesian techniques can be used to filter out unlikely intents and aid decision-making.

Meet Matterport’s newest employee: Thomas Bayes

Automatic Interaction Systems

As I mentioned above, even if you can generate a scene or environment you’ve still got to wire it up with code or animations to make it come alive.

How do we do that at scale, in a way that is interesting and feels authentic?

This first paper takes recordings from depth sensors of people performing common interactions like “watching TV” and then allows for procedural generation of similar activity.

A few people doing the same activity will quickly open up a wide range of possibilities — as each node is attached to the graph, a character can explore a larger action space between each node for activities to perform. That is, after observing us both the character would be able to “watch TV” mostly the way that I “watch TV”, but also sort of the way that you “watch TV.”

I’d love to take it a bit further and see what happens in with a generative adversarial network — what would it look like if the network running the character just barely continued to classify the movement that it creates as “watching tv” while also making the movement as much “write on a whiteboard”as possible? This works with images with different styles of painters, and often produces interesting results. Could it work with motion too? Will it dream up new gestures or spit out nonsense?

PiGraphs: Learning Interaction Snapshots from Observations

You may need to watch the video to fully grok the next paper, but what’s happening here is that nobody actually animates the pirate’s feet — the character is able to move over and transition between a variety of terrain without specialized programming to account for each situation. Props to these folks for having a video, a paper, and code :)

sreyafrancis/PFNN

Curious Resources: Volume 3 was originally published in Curious Resources on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome back!

Each edition we try to bring you a set of interesting finds in interactive technology. If you missed the previous edition, there’s another collection of great links waiting for you when you’re done here.

This week we’re covering:

• E-Commerce Inside of Games
• Synthetic Biology and Simulation
• Deep Learning for Weapon Detection

Follow me on Github or get in touch with me on LinkedIn!

E-Commerce Inside of Games: Buy the Story

If you can sell your branded products to players at the correct time in the story, you’ll provide delight on demand and build a meaningful bridge between the physical and digital worlds. You’ll make people happy and make a ton of money! But the integrations should be more than trying to sell all of your swag at the end of every level. They need to mean something.

I’d love to do an integration that unlocks a T-shirt for purchase only after you’ve beaten a particularly difficult boss— if you saw someone else wearing the T-shirt in public, you’d know they were legit. Or in a cooking game, order a package of ingredients to match a meal you just made in the game kitchen!

Shopify has made it easy to connect with Unity using their new SDK.

• Shopify/unity-buy-sdk
• Shopify SDK for Unity

If there’s anything missing from the new unity-buy-sdk, you might be able to patch it with functionality from this more mature C# library.

nozzlegear/ShopifySharp

Or setup a lambda function on AWS and connect to Shopify in any language you want by invoking the lambda function inside of Unity with a network request.

• awsdocs/aws-unity-developer-guide
• Shopify/shopify_python_api
• getconversio/go-shopify

Synthetic Biology & Simulations: It’s Alive!

The application of technology to the creation of life is today’s computing revoluion. It comes down to cost and time-to-iteration — and both are dropping precipitously. If there is no penalty to experimentation, great strides forward will occur with regularity. When it comes to synthetic biology, we’re almost there. In the meantime, simulations can help save time and money.

If you know Verilog from programming FPGAs or embedded systems — well, you’re already awesome — but did you know how awesome? You can make living circuits!

GitHub - CIDARLAB/cello: Genetic circuit design automation

Or for those with web development background, write standards-compliant creatures with JavaScript

• SynBioDex/sboljs
• the Synthetic Biology Open Language (SBOL) | a language for genetic design

Model some bacteria populations to learn what you can

CellSimulationLabs/bsim

Or simulate a whole worm inside of your computer with OpenWorm

openworm/OpenWorm

This is the engine behind the body physics in OpenWorm

openworm/sibernetic

Once you’ve reached the limits of simulation, order parts from an online database of plasmids and do it “for real”

Addgene: Homepage

Deep Learning for Weapon Detection

While technological change is no substitute for actual policy, computer vision systems can augment policy and provide more options to policymakers. Computerized threat or weapon detection is clearly in the public interest, but I couldn’t actually find very much code out there.

That’s probably why Homeland Security recently teamed up with data-science contest host Kaggle on a $1.5 million challenge for passenger screening at airports.

Passenger Screening Algorithm Challenge

Since there isn’t much code, I’ve found a few recent papers instead:

• An Automatic Hybrid Approach to Detect Concealed Weapons Using Deep Learning
• Developing a Real-Time Gun Detection Classifier
• Automatic Handgun Detection Alarm in Videos Using Deep Learning

The first of these papers addresses detecting concealed weapons. Here, detection of weapons in high-security areas might be improved with deep learning through image fusion and hyperspectral imaging. Varied types of input data can be put together to form the tensors to train the network on, which should make the network more robust

However, this benefit may be limited to expensive installations or specialized equipment. The hope would be to find some kind of ubiquitous field or sensor (wifi?) where the presence of a concealed weapon might be detected in a broader scope. Weapons made with ceramics and 3D-printed materials will continue to make this task difficult, even for specialized imaging and algorithms.

The latter papers deal with detecting weapons in plain sight, a task that sounds easier than it actually is, considering the millions of hours of video uploaded to the internet or fed through security systems every day. A classifier can recognize weapons in security footage and social networks alike that might have been overlooked by human eyes. The appropriate civil liberties need to be exercised when deciding how to act upon this information, but these kinds of safeguards might constitute an early-warning system that we can build upon for a more peaceful future.

Okay, so I did manage to find a few image fusion and hyperspectral imaging tools. The basic idea is to collect as much data from as many sensors as you can so that you can combine them into a single data-rich information vector.

• rjw57/dtcwtfusion
• HPSCIL/cuSTARFM
• hyperspy/hyperspy
• spectralpython/spectral

A lot of these types of projects are trained on images of weapons from the Internet Movie Firearms Database. Which is a really cool resource that I’m sure provides some interesting training data. Real life, however, is often more subtle than a movie, so I’m not sure I’d trust these projects in the wild without some retraining. Hasta la vista!

Internet Movie Firearms Database - Guns in Movies, TV and Video Games

Curious Resources: Volume 2 was originally published in Curious Resources on Medium, where people are continuing the conversation by highlighting and responding to this story.

This week we’re covering:
- Spatial Audio (Ambisonics) for 360/ VR / AR / XR
- GraphQL For Faster Front-End Iterations
- Live Streaming 360 Video to YouTube From Inside of Unity
- Assorted Deep Learning Finds: Poetry, Audio, Hi-Res Images, Unity Agents

Hi! I’m starting a series of posts that will collect some of the interesting repositories and articles that I’ve come across lately. I’ll also dig up references from my ever growing toolbelt of Github stars. I love research, exploring patterns, making connections, and creating new possibilities.

I own a company that specializes in experiences — get in touch with me on LinkedIn!

Spatial Audio (Ambisonics) for 360 / VR / AR / XR

I’ve been thinking of applying to Google’s “Jump Start” program to get my hands on their new YI HALO 360 camera rigs for a series of creative shoots involving magical creatures in abandoned places. One of the questions on the application asks about spatial audio aka ambisonics, which remains somewhat unexplored compared to reactive visual systems. I went back through some of my finds:

Google has a spatial audio SDK for ambisonics called Resonance
https://developers.google.com/resonance-audio/

The Unity package is on the Releases page here
https://github.com/resonance-audio/resonance-audio-unity-sdk

Web devs can use this library
https://github.com/resonance-audio/resonance-audio-web-sdk

The engine behind the Web SDK
https://github.com/GoogleChrome/omnitone

For advanced users, you can edit in your favorite DAW as a VST
https://github.com/resonance-audio/resonance-audio-daw-tools

Google’s encoding specs for audio and video
https://github.com/google/spatial-media

Facebook has some slick software, but its source is nowhere to be found
https://facebook360.fb.com/spatial-workstation/

An Ableton-specific ambisonics solution
https://github.com/EnvelopSound/EnvelopForLive

GraphQL For Faster Front-End Iterations

Front-end developers building client applications can sometimes get blocked waiting for back-end API changes to keep up with design.

What if you could just go ahead change that API from the front end? What if you didn’t have to worry about whether a particular endpoint exists — if the data is there, you can construct one yourself. Even better if you could keep that app trim by only getting the data you want when you make a request, instead of a ton of bloat.

To that end, I’ve been considering whether to recommend a GraphQL API instead of a REST API to an e-commerce client. It’s important that they be able to move quickly. While doing my research, I came across some interesting resources:

Starting from this excellent article
https://time2hack.com/2018/02/introduction-quick-guide-to-graphql-for-backend-frontend/

GraphQL query visualizer
https://github.com/graphql/graphiql

GraphQL server for Express
https://github.com/apollographql/apollo-server

Interesting how GraphQL is just an interface on top of whatever database you want to use. For example, you can use it with MongoDB https://www.compose.com/articles/using-graphql-with-mongodb/

Or PostgreSQL
https://github.com/postgraphql/postgraphql

The spec itself, although I’d start with articles and tutorials instead
https://github.com/facebook/graphql

Some front end tools for interacting with GraphQL
https://github.com/apollographql/apollo-client

Specifically for React
https://github.com/apollographql/react-apollo

A whole framework from Facebook that couples React with GraphQL
https://github.com/facebook/relay

Modifying control flow of queries and results
https://github.com/apollographql/apollo-link

Shorthand for writing queries
https://github.com/apollographql/graphql-tag

Live Streaming 360 Video to YouTube From Inside of Unity

I’ve been trying to sort out a pipeline to live stream equirectangular video captured from a camera inside of a Unity scene to YouTube 360. I want viewers to be able to see inside of live game worlds as they’re happening. I’ve found a couple of useful leads:

Here’s one to output using FFMPEG (which could then stream to YouTube)
via the amazing Keijiro
https://github.com/keijiro/FFmpegOut

But at what speed? The new scriptable render pipeline in Unity 2018 might enable a faster solution
https://github.com/keijiro/AsyncCaptureTest

I also evaluated a paid option
https://github.com/SystemFriend/GPUVideoEncoder-Trial

The paid option makes use of this very interesting concurrency library
https://assetstore.unity.com/packages/tools/spicy-pixel-concurrency-kit-3586

Assorted Deep Learning Finds

Generate poetry from images using CNNS and RNNs
https://github.com/rossgoodwin/neuralsnap

Get audio into a Pytorch tensor — more audio Deep Learning projects, please!
https://github.com/pytorch/audio

Speech at super-resolution could lead to clearer communications.
https://github.com/jhetherly/EnglishSpeechUpsampler

Make music alongside a Google AI — a new generation of instruments and live performances may be collaborations with a computer
https://github.com/tensorflow/magenta

Future Alert: with the right kind of sensors, deep learning should enable subvocal speech recognition — the ability to communicate without making sound, which would come in handy in smart gear for divers or firefighters
https://github.com/Mohamed-Elbially/ThesisCodes

Many state-of the art networks have been trained on extremely small images (300px by 300px) due to GPU memory size constraints— however, they have been very accurate despite this limitation. I’m excited about anything trained on high resolution images, since it will help us understand whether smaller training images are sufficient or only scratching the surface of what’s possible. More pixels!
https://github.com/NVIDIA/pix2pixHD

Unity is a great environment to train deep learning agents — especially multi-agent environments.
https://github.com/Unity-Technologies/ml-agents

Here’s a hide/escape example
https://github.com/HappySlice/hide-escape

Here’s one with pirates!
https://github.com/HugoCMU/pirateAI

Curious Resources: Volume 1 was originally published in Curious Resources on Medium, where people are continuing the conversation by highlighting and responding to this story.

https://www.prettymuni.com

A web app that uses D3.js and vanilla JavaScript to display and update vehicles from the San Francisco Muni system in real-time.

New colors each refresh!

Showing public transit vehicles in real-time

D3.js is a great open-source data visualization library from Mike Bostock, formerly of the New York Times.

I won’t go too deeply into code in this article, you can check that out at the Pretty Muni project GitHub. What I will do here is provide a line of reasoning for many of the choices that I have made, both visually and in code. My goal is to show how these considerations influence and guide each other during the process.

Additionally, I try to remain conscious of the costs of deploying and accessing the application. Serve the most people at the lowest price! This extends to the personal cost of user time, where I try to make continuous incremental improvements.

I had fun building something interesting this week, but I’m sure there’s a lot I can improve on. Please send me your suggestions and pull requests!

Enjoy the read!

Drawing the base maps — geoJSON layering, colors, time to first interaction, map sizes

With SVG, the order of the elements on the page is the order that the elements are added to the element. If I tried to fetch all of the maps in parallel, the draw order was haphazard depending which was retrieved first. It’s pretty straightforward to reorder the actual SVG DOM nodes containing the maps using built-in methods.

The map files are in geoJSON format,but I dipped quickly into the world of topoJSON, which promises smaller file sizes and more features. However, my early experiments actually ended up with larger file sizes in this particular case so I decided not to pursue it further at this time.

In regards to vehicle colors, Muni does provide colors for each route, but in my personal experience the colors aren’t really used for finding a vehicle — it’s much more route-name based. However, I could be wrong about that and it is hard to dell without more data. If this were a functionally-oriented application (i.e. for the actual Muni organization) I might have gone with the API provided colors.

I assigned random colors to each of the layers, which ends up being sometimes ugly but often fascinating. It’s also a great way to get people to revisit and refresh the page. It ends up looking sort of like a Fauvist version of Mini Metro.

At first, I explored libraries for procedurally generating sets of related colors. The best library I found only does up to 16 colors. I enjoyed the random results enough to hold off on locking the colors down until a later version. I might stick with the randomness but tune it a little bit with an algorithm to distribute colors as far as possible from the previous colors in the set across all 80+ Muni routes. There is a section later about choices regarding route paths and colors.

Of course, with such an interesting map, users are going to want to zoom and scale. I decided to use native SVG scaling and transformations to accomplish this, instead of re-projecting each data point at every zoom level. The result is a smoother, high resolution zoom. An upgrade to passive event listeners could help, especially on mobile.

The file size of the maps is a problem. Almost 9mb for the streets.json file alone. To my great surprise, the streets geoJSON file comes over the network at only 1.2mb. Gzip for the win! Still pretty big, but enough savings to make the app usable at 1.9mb total for all maps.

Later, I realized that waiting for the streets base map to load shouldn’t block the user from starting with the other base maps. I made some changes so that the streets file loads separately from the other base maps. Once it has arrived, I insert its DOM node into the SVG at the correct level. The result is a much reduced time to first interaction, a better user-experience and a big win in terms of how it felt to use the app.

Getting the vehicle data — XML, JSON, Promises

http://www.nextbus.com/xmlFeedDocs/NextBusXMLFeed.pdf

One utility function I didn’t hesitate to use was an XML to JSON parser. I haven’t worked with XML enough in my career to know all of its foibles, and didn’t want to run into them one at a time writing my own parser. Therefore, I leaned on the crowd and on one clever implementation in particular. I did have to update it to supress some Chrome warnings, but it was only changing one or two lines. This allowed meto parse and process the Nextbus API responses in familiar JSON format.

It didn’t take long to start to see a tangle of callbacks beginning to develop in this part of the app. So in the name of readability, I dropped in an ES6-Promises polyfill and wrapped the Nexbus API requests. Now I can easily get and use the vehicles and route configurations in my other functions without getting too verbose.

Drawing the vehicles —adding circles, headings, colors,text, teardrops & transitions

Vehicle groups consist of these elements: circle, text, heading drop, and heading dot. Routes contain many vehicle groups. When I update data for a route, the vehicle groups are transitioned to their updated locations.

I first visualized the headings as dots within the outer ring of the vehicle circle. While this was aesthetically pleasing, it didn’t translate quickly or at great distances. It was too subtle.

I didn’t want to switch to using a triangle or another hard-edged primitive to represent the bus. The circles were nice and clean. My answer was to put a teardrop shape underneath the circle — the heading drop. I moved the heading dot out further and gave it the same color as the route, which helps distinguish between overlapped vehicles.

Route Paths — Too much color?

I was conflicted about whether to add route paths — everything looked really busy already with the streets, colors and vehicles. Maybe it would be too much. I had to see for myself, so I coded up a couple of methods do add route paths to the map when a user chooses a route in the selector.

At first glance, my suspicions seemed confirmed. There is a lot going on. But the more I used the app with paths and showed it to friends, the more I realized that they do communicate a lot about what is going on. Taking a little bit of opacity off of the streets map helped but destroyed the frame rate and made everything sluggish. Perhaps there is a compromise — I like the look of the route paths with the streets turned off. A monochrome palette for the entire set of base maps could calm things down too. I tried black, white, and grey streets, which all worked depending on the colors of the neighborhoods. Yet there is a certain delight in some of the weird combinations that the random selection uncovers. And refreshing is easy.

I’m torn. For now, color roulette, until I get more feedback :)

Creating the route selector — first, see what’s out there. But sometimes you’ve got to roll your own

The native HTML5 <select multiple> element is a mess. On the desktop you need to press control or command and click to make your selections. The Rolodex-style spinner given to users on mobile by iOS Safari is better, but not by much. Either way, the styles involved can be difficult to write or completely out of your control.

To sidestep these issues and avoid maintaining separate codepaths, I decided that this might be a good place to use a library. I’m a fan of Material Design and have successfully used the Materialize library on previous projects. So I decided to give it a try here. While adding a dependency on jQuery just for this made me cringe, the stylistic advantage of a <select multiple> that renders as a dropdown with checkboxes was surely worth a few extra kilobytes.

This looked better. Everything was hunky dory until I tried it on mobile. Then I ran into a nasty Materialize bug — multiple selects cannot be closed on mobile after they are opened. This is a non-starter. I toyed with the idea of using the browser-default on mobile and the Materialize version on desktop, but that would mean feature detection and maintaining different code paths. With all of that, for little gain, I decided it was time to scrap Materialize and the <select multiple> element altogether.

What I ended up creating is a much more flexible, faster route selector that requires less scrolling and looks the same everywhere. It also provided an opportunity to use the route colors to make a connection between what the users chooses and what the user sees. Larger screens have room for the full route names. The controls are fast, even on mobile.

Bonus: I was able to remove jQuery and Materialize from my project.

Interfaces — GUI glue, responsive CSS styles, loaders, saving, sharing

I wrote some standard JavaScript code to handle creating the user interface elements and binding input events to them. This app is straightforward enough that using an MVC framework like Angular or React really seemed like overkill. Not to mention the likelihood of integration issues with the chosen framework and D3. The native DOM methods are actually pretty useful, if verbose, and you usually know what you’re going to get. I even made the close button sticky without too much fuss :)

Since the maps loaded are fairly large, here’s inevitably a bit of initial user wait time. I chose to use a simple, CSS-based loader that is hidden once the maps are drawn. I’d like to upgrade this stylistically to be in line with the rest of the app. Speaking of styles, I enjoyed writing the CSS for the app. A couple of simple media queries really helped alleviate the mobile vs. desktop differences I could surface in my somewhat limited testing.

Saving an SVG to an image is a bit of a kludge. I did some research and found a function that mostly works, although in the end it is missing some styles that are applied to the entire page and not at the SVG level. I’m going to work on that feature a little bit more before I deploy it.

For sharing, I defined Facebook and Twitter compatible metatags, as well as some Slack-style pull-through tags for really fancy sharers. I flirted with the idea of buttons, but didn’t want to disrupt the visual flow, and there are already a lot of built in ways to share links.

Get real —Cross-browser compatibility, performance

Cross-browser and cross-device testing is an essential but sometimes painful process. It quickly becomes race to apply a patch here, a shim there. What’s a dev to do?

Try to put your time developing for where users are likely to come from. This requires knowing your starting user base, as well as paying close attention to your analytics. Spend some time looking at different reports of browser statistics and compare them to the reality at your organization.

If you have to guess, here are some general observations: Currently on desktop, Chrome is crushing it, with a large spread separating it from Firefox and Internet Explorer. The top 3 browsers on mobile at time of writing have a closer spread — Chrome, Safari, and the Alibaba’s UC_Browser.

These decisions come down to resources and reach. The more you have, the can do. Apportion wisely. As far as what I was able to test with Pretty Muni:

On desktop Chrome, things work the best.

On desktop Firefox, things work after a quick to update my XML parser to use .value instead of .nodeValue, but my heading drops are ending up rotated in strange places.

On desktop Microsoft Edge, zoom scrolling results in wild scaling of the base maps. Rendering bugs about and performance suffers. I can’t imagine that older Internet Explorer versions must be doing any better :/

Some mobile testing in Chrome led me to the look into the new Passive Event listener API, which I tried to shim into the current version of D3.js, but ran into some errors that forced me to back off for the moment. The feature might be worth spending some time getting working correctly in order to avoid hiccups during the processing of initial geoJSON maps and achieve really great performance.

The largest performance increase would probably be reducing the number of SVG elements that are being drawn and observed. I’ve already filtered out vehicles that haven’t moved and headings that haven’t changed from their respective animations.

Deployment— https, GitHub pages, mixed-content, proxying apis, node.js heroku, DNS w/ cloudflare

My first approach was to serve the app completely from the client side using a service like GitHub pages or Amazon S3. But when I pushed to GitHub pages, I was greeted with a blank screen and a console full of mixed-content errors. How come?

You can’t use http:// assets on a site served over https://

Simple as that. Production ready code should be served securely. Therefore, the requests to the Nextbus service over http:// were conflicting with the browser’s internal safety barometer.

Nextbus isn’t likely to change their API before I need to deploy (although if they do add https:// the whole internet might do a cartwheel). Therefore, I’ve got to figure out a way to secure my request.

Fortunately, writing a proxy in node.js is a matter of just a couple of lines. The server takes requests to Nextbus and passes them along. To be fair, this proxy server is a bottleneck. It ads an additional server step to an app that is otherwise completely client-side. It also makes it look to Nextbus like there are an awful lot of requests coming from the IP address that belongs to the app. So far they don’t seem to be throttlers, but we’ll see. After pointing the Nextbus urls in my app at the proxy, the page proceeded to load as expected.

I tried deploying the app on my own static server on Heroku using Express, but worried about co-locating the proxy and the app — too much traffic and the whole thing would go down. Better to keep them separate. I don’t have an AWS account setup right now, otherwise I might have just put everything in an S3 bucket. Besides, why pay for all that bandwidth

I’m an avid GitHub user and eventually I came back around to GitHub pages. I had to name my distribution folder ‘docs’, but I was happy to deal with a little awkwardness in exchange for free hosting. (I’ll try to fix that at some point with a deploy task, likely written in gulp.). Now the app and the proxy server for requests are separate, and I’m not paying for hosting the client-side assets. Woohoo!

I used Google Domains to purchase the www.prettymuni.com domain name, but decided to try CloudFlare as my DNS provider because they offer more routing options and features. I set the A records at Cloudflare to point at GitHub’s Pages servers. The final step in securing the site was adding page rewrite rules at Cloudflare to turn all http:// requests into https:// requests.

Why does any of this matter? In addition to actually wanting the content to appear everywhere without mixed content warnings, security now weighs into the SEO rankings of many search engines.

Some quick delivery statistics vis-à-vis bandwidth:

• 107kb JavaScript files
• 2kb CSS
• 11kb Fonts
• 1.9mb JSON to fetch all maps
• route fetches are 1.1kb per route from NextBus
• ~83kb to get all routes at once

Conclusion

A lot can be improved in terms of performance and I have plenty of ideas for additional features — generative sound, anyone? Really though, I’d be happy to just figure out those pesky heading drop rotations.

But that’s all for now!