As a creative technologist I get a lot of joy out of building visual experiences for physical spaces. Many of Bluecadet’s projects run in highly trafficked locations for years, and if you’ve ever worked on a long-term media installation, you’re probably aware of the challenges involved in making them bulletproof. As exhibits scale in size and complexity, the process of installing can become a delicate but repetitive chore. To streamline and standardize this process, we developed Launchpad: an open-source tool that facilitates the deployment and management of exhibit apps and content.

Why Build Our Own Tool?

Many of our projects are content-heavy, native apps with long runtimes distributed across many PCs. Our experiences need to be reliable, consistent across exhibits and support many types of frameworks like Unity, Cinder or Web. If things go wrong — especially on a large canvas — they need to fail gracefully and recover quickly. We also often pull content from a wide variety of sources and need to cache that content offline for quick and reliable access.

Typically we need to tick these boxes during our installs:

• Configure Windows OS for exhibit environments
• Run a watchdog to restart processes on exit
• Prevent escape from our apps into the OS
• Regularly sync remote content locally

There are plenty of great tools for some of these tasks, like Helios Interactive’s Phoenix, Stimulant’s AM/PM, Local Projects’ ofxApp, Windows Task Manager, Ansible and PM2. But none of them provide all the specific settings for PC bootstrapping, monitoring capabilities, content pipelines or extensibility that we needed.

Standardizing our Process

Before Launchpad, we had long install checklists for each PC that required manual configuration and testing. However, this was prone to errors and time-consuming to manage. During a project for the Gerald R Ford Presidential Library, we found ourselves spending three days in an AV closet installing dozens of programs one-by-one, manually.

For a long time, our watchdog of choice was Phoenix, which is a robust tool but hard to manage and sequence multiple concurrent apps with. For content, we either wrote custom Python scripts or used rysnc and ssh, which was either very specific and time consuming or limited our choice of backends and infrastructure.

echo "Downloading media..." 
rsync -rlvz --size-only --ipv4 --progress -e 'ssh -p 2222' --delete --exclude='*.csv' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/wp-content/uploads/*/*/ ./media

echo "Downloading data..."
rsync -rlvz --size-only --ipv4 --progress -e 'ssh -p 2222' --delete --include='*.csv' --exclude='*' $ENV.$SITE@appserver.$ENV.$SITE.drush.in:code/wp-content/uploads/*/*/ ./data

Automating at Scale

Our next big project was for The Henry Ford Museum — coincidentally another Michigan-based Ford institution — with one of the more complex installations running on 20 PCs.

Tired of spending days poring over punch lists, we knew we needed a better solution. That’s why we set out to build a suite of scripts that would automated every step of the process—an early version of Launchpad.

This initial version was built as a plugin for AM/PM. While it worked wonders for our first project at The Henry Ford, we quickly realized that it lacked the necessary extension options and control over launch sequences and app states. As we geared up for our next challenge with over 20 individual programs at the MIT Museum, we knew we had to up our game.

Standalone Package

The next version of Launchpad was built from the ground up as a standalone process or a Node dependency. It combined the best parts of our previous AM/PM plugin and was designed to be easily extended and modular. We swapped in PM2 as our watchdog of choice, which allowed us to run it as a subprocess to manage multiple apps at the same time, get better state readings and add add features like window management.

But that’s not all — we revised our content pipeline to support a range of CMS platforms, improve caching and gracefully rollback content in emergencies. We also added text and image processing capabilities, making it easy to resize, filter and crop images or convert CMS-specific syntax to HTML or any other format. Additionally, Launchpad now has streamlined log management and provides hooks to run commands at various stages of your app lifecycle.

Why You Should Use Launchpad

Launchpad has significantly simplified our process of installing complex interactive programs across museums, institutions and other large venues. It takes the guesswork out of configuration and reduces days of work down to minutes for us. After years of testing it on permanent exhibit installs with across many types of backends and frameworks, it’s become a crucial part of our exhibit operations and we’re excited to open source it.

What Launchpad Can Do

• Run multiple apps in parallel, with control over launch sequence and window order
• Fail gracefully and restart apps when they exit
• Automate PC configuration for exhibit environments, e.g. disable consumer features like edge swipes, notifications, updates and prevent PCs from going to sleep
• Sync content from APIs like Sanity, Strapi, Contentful, Airtable, and generic JSON (Wordpress, Drupal, REST APIs)
• Cache content locally in case of network outage, with graceful rollback error handling
• Support for text transformations like markdown to HTML, image processing, resizing, filtering, and cropping

Getting Launchpad

Launchpad is available on NPM and GitHub:

npm i @bluecadet/launchpad

Check out our documentation to get started and learn more.

We’ve deployed many projects with Launchpad and are actively maintaining it. If you have questions or suggestions, submit an issue and we’ll look into it.

Streamlining Exhibit Installs with Launchpad was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Bluecadet, we have been using Pantheon’s Build tools and Circle CI to deploy projects for a few years now. We have also customized the CircleCi jobs for our own agency needs. It has made the development process more robust by giving us tools to manage different environments as well as run tasks like accessibility checks and visual regressions. But a CI Process also posed a maintenance burden on our clients, some of whom do not have internal DevOps or development staff. With an eye toward making the handoff to our clients easier and more accessible, we started investigating Github Actions as a potential continuous integration alternative.

First, I want to give a shout out to Tyler Fahey and his article from 2020. This got me started and I think there are some significant updates I wanted to address, hence this article.

Why do we use CI? We have two main needs for CI. First, running Composer so we do not have to commit module and library dependencies to our project repository. Second, we use Node (and Gulp) to compile all our custom front end CSS and JS, and checking in these assets can cause merge conflicts. After that, depending on the project we run some tests (mainly visual regression — but that is a different article). Using Pantheon’s Build Tools to setup a new site, Composer actions are built into the setup and adding in the Node processing was easy to add into the CircleCi config.

Another thing that is worth pointing out, as an organization, we like to have our canonical repo live on our GitHub account, to take advantage of Github Issues and ultimately as a backup for the codebase. When we first started using Pantheon, we would actually push our changes to both our GitHub repo and the Pantheon repo. We typically have small teams so this was not a huge burden, but it still required developers to remember to push up to both remotes. Having a CI process that triggered on a push to Github and then pushed to Pantheon on its own made our internal dev/deploy process more consistent.

In recent years though, Pantheon has been adding some cool new features that we wanted to learn about and take advantage of.

1. Integrated Composer (basically, able to run composer install on Pantheon platform). Very cool! We now do not need CI for this process. But we still need it for our Node and Gulp processing.
2. Upstreams, which really is not new, but allows for one-click updates in the Pantheon dashboard. As a developer, one-click updates scare me, but after handing off a project to a client, it is pretty handy.
3. Autopilot, which seems like a great feature, runs automatic updates, and visual regression! To be honest, I do not know much about this feature, but want to continue to look into it. AGAIN, With an eye toward making the handoff to our clients easier, this seems like a great feature when we hand off a project to our clients.

With the existing Build Tools setup, we do not have access to those features (as far as I understand it, maybe there are some workarounds?). We also need to update our CI process. We did a lot of customization, and the result is hard to maintain. I had too much fun learning and playing with all the possibilities CI provided. At this point in time, it is allowing us to take a good look at GitHub Actions and so far seems to be a great solution that fits our needs.

Step 1: Starting

The first step to get started now would be to read Tyler Fahey’s article. He really does a great job of explaining the process in the article, and anything here would just be a copy/paste of that article anyways.

Step 2: Adding in Integrated Composer

OK, now that you have read that, the next step for me would be how to add in Integrated Composer. Most of this happens on the Pantheon side when we create the project. Through Pantheon’s UI, create a new Drupal 9 site with Integrated Composer. You can do WordPress too, there are just a few more steps. Read the Docs on Integrated Composer. Once you push code, Pantheon will do the rest.

So here is my Github Workflow (This should look extremely familiar if you read Tyler’s article):

Step 2B: Deploy to any branch

We can update this slightly so we can push to any branch. Just be careful how you name your branches due to Pantheon’s environment naming conventions.

We changed:

• the ‘on’ trigger to build on any branch
• the git push command to push to the correct branch

Step 3: Adding in Node and Gulp

Next, we want to add in Node and the relevant node commands. Our typical process uses Gulp, so you will see commands related to that, but you should be able to substitute it with whatever you need.

Let us walk through what we added. First, I borrowed a lot from Pantheon’s build tools to help me understand what needed to happen and how to do it. They have some nice helpful functions and I imagined I could accomplish everything we needed to without it, but it seemed easier just to add it in here for the moment. Lines 16–23 we are installing Terminus and the build tools plugin. What I ended up really needing was this function to cut the .gitignore file. In our “local” repo, we ignore the production assets that Gulp builds (css, js, etc). But we do not want to ignore those files on Pantheon, because otherwise styles and javascript will be missing. Lines 25–27 are running the command to modify the .gitignore file.

If you are familiar with Pantheon Build Tools process, you will know that there is always this one “Build Assets” commit that is on Pantheon’s repo which is usually huge and contains everything built from Composer as well as Gulp. We basically are replicating that, but now we only need the production assets from our Gulp build.

Next (lines 29–46), we are adding in all of our Node stuff. We are going to install Node first. Restore any cache for node packages to help speed up the action. Then, finally run the needed commands to build out our Front End assets.

Now that we have all these new files, we need to get them to Pantheon. Previously, we just pushed to Pantheon, but now we need to create that “Build Assets” commit and then push. The deploy step is updated to handle this.

Congrats!! We are now using Pantheon’s upstream, and Integrated Composer, GitHub Actions AND Node!

Read on if you are interested in another use case we ran into…

BONUS CONTENT: Ignoring files only on the Pantheon Repo

An issue we ran into (and a number of users from what I am told from Pantheon) is if Composer changes files that are not really supposed to be modified from a Composer change, it causes the process to fail on Pantheon. So no CMS! Not good. Also, this was a hard error to understand, but if you think everything is working but your changes are not updating on your build, check the “View log” for that push in your Pantheon dashboard. At the end there will be some errors.

Our use case is that we are using a handy little plugin, joachim-n/composer-manifest, which creates a small yaml file, composer-manifest.yml, indicating exact versions of each composer package installed. This keeps a nice log in your git history for when a package updates, but you did not explicitly change your composer.json file. You could read the lock file, but that is just insane!

In order to bypass this problem I basically need exactly the opposite of what the Build Tools does in cutting the .gitignore file. And after a little trial and error, I also realized we need to remove those files from being tracked in the repo! So I created a new ignore file, .gitignore-panth. As a standalone file, this has no effect on anything as far as I can tell. If it messes with git in any way, we can rename it. We can then use this file to define files or directories we want to ignore on Pantheon, just like any other .gitignore file. Here is the deploy step now with the 2 new lines we need.

- name: deployer
      env:
        pantheon_repo: '${{ secrets.PANTHEON_REPO }}'
      run: |
        BASE_BRANCH=${GITHUB_REF##*/}
        git remote add pantheon $pantheon_repo
        git config user.mail "bc-bot@bluecadet.com"
        git config user.name "Bluecadet Bot"
        git rm `cat .gitignore-panth | sed 's/^\///g'`
        cat .gitignore-panth >> .gitignore
        git add .
        git commit -q -m 'Build assets for dev.'
        git push --force pantheon HEAD:refs/heads/$BASE_BRANCH

First, we delete the files with the git rm command, then we simply add the contents of the .gitignore-panth to the real .gitignore file with the cat command.

Pantheon and GitHub Actions was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Every year at Bluecadet we like to make a festive technology project to celebrate the holidays and ring in the new year, and 2021 was no different! We’re always researching and experimenting with new technologies to stay at the top of our game, and recently AR has been on our minds a lot.

Due to COVID, museums and brands have been eager to find new ways to engage their audiences using contactless, or even remote, interactions. Many are wondering whether AR is a viable option, and we think it definitely has a lot of creative potential to explore.

However, at the moment the fastest, most stable, and most robust AR engines (Apple’s ARKit and Android’s ARCore) are only available for native apps, which means creators have to navigate the world of app store publishing and visitors have to install another app, which may only work on newer, expensive phones.

Fortunately, a few platforms and technologies (8th Wall, Zappar, AR.js, and others) have been popping up recently that are using web technologies to provide at least some of the same features as their native counterparts. Now we can build fun AR experiences that nearly anyone can try out without any special apps or fancy hardware!

Go ahead and scan the QR code below to see for yourself!

In case the app isn’t working or you don’t have a phone nearby, here is a recording of what you would see:

How it works

After evaluating a few different platforms and code packages, we ended up landing on 8th Wall. Their Cloud Editor, live app previews, multiple 3D framework integrations, and built-in project hosting and deployment all make it easy to set up, build, and launch small to mid-sized projects like ours in no time!

World tracking

At the core of our AR experience is 8th Wall’s magical world tracking functionality, which uses a custom SLAM engine built in C++ and compiled for the web with WebAssembly. A JavaScript package developed by 8th Wall interacts with the engine and exposes an API with interfaces for ThreeJS, A-Frame, and other modern 3D web frameworks.

We used 8th Wall’s Cloud Editor with ThreeJS for this experience, which was pretty easy to get up and running with thanks to 8th Wall’s sample projects, which can used as starter templates for new projects.

8th Wall’s JavaScript package provides a scene object that automatically stays aligned with the floor of the environment around you in real-time, adjusting both its orientation and scale to fit as best it can. For me the “a-ha” moment happened when I realized that the XZ plane of the scene was constantly being realigned to the horizontal surface found by the SLAM engine — as you add meshes to the local coordinate system of the scene, they ALL get transformed as 8th Wall aligns the scene to the environment!

Assets

The 3D models and animations that you see in the experience were all created by our wonderful Motion Designer Siji Chen using Blender, then exported to compressed GLB files. On the web, asset size makes a huge difference in UX because the assets have to be downloaded by the browser on page load. Our models were ~7–10MB normally, but only ~2MB compressed!

In ThreeJS the assets were loaded with GLTFLoader and DracoLoader, then animated by just loading up and playing all the baked-in animations created by Siji using ThreeJS’s animation system. Having all the animations, materials, and geometry all in a single file really saved us time!

The lighting and environment map setup was borrowed from Don McCurdy’s glTF Viewer app, whose source code helped in a pinch.

UI

The experience also includes some flat 2D UI components (built with standard HTML, CSS, and vanilla JavaScript) that sit on top of the canvas, providing some additional features like:

• A “Help” dialog with usage tips, including tips for screen reader users.
• A “reset” button that allows you to force the world tracking engine to reset, in case the 3D models get “stuck” somewhere you don’t want them.
• A manual control interface allowing you to explore the objects in the scene using crosshairs without having to move your phone.

Accessibility

We always want to make sure that the projects we create can be enjoyed equally by the widest possible audience, which means creating content and interfaces that are accessible to people with a wide range of abilities.

We work hard to incorporate guidance from the Web Content Accessibility Guidelines (WCAG 2.1 level AA) into our work, and for normal web apps they are a tremendous resource. For AR experiences, though, there are some challenges that are not so easy to solve, so we have to get creative.

There is still a lot of research to be done on the topic of AR accessibility, so we recommend looking into communities like XR Access and A11yVR to see what people are thinking about. The XR Accessibility User Requirements Working Group Note published by the W3C’s APA working group is worth a read too!

UI accessibility

All of the flat 2D UI components were built using simple, semantic HTML elements and minimal ARIA so that screen reader users can easily find, understand, and operate all the controls on the screen.

Using <button>s for buttons, text alternatives for icons, and standard patterns for common components (like modal dialogs) are all easy ways to make a huge difference in accessibility!

Manual control interface

In the bottom left corner of the screen is a small gamepad controller icon button that toggles a set of crosshairs and some directional control buttons.

As the crosshairs pass over objects in the scene, human-friendly labels are shown identifying what those objects are. These labels also get read out automatically by screen readers so that blind and visually-impaired visitors can explore the scene too.

For blind and visually-impaired visitors, it’s important that content that is being displayed visually is also available as text somehow (see WCAG 1.1.1). This is especially tricky in AR because part of what a visitor sees on screen is the actual room around them, which is hard (maybe impossible) to appropriately describe in an automated way.

For the digital content though, we tried to include information that seemed relevant like general descriptions of the scene, announcements of state changes (like opening the present), and contextual instructions explaining how to interact with the scene.

Here is a recording of what it’s like to use the manual control interface with TalkBack on an Android phone. It’s not perfect, but it’s a start!

Object labelling

In future AR apps, we’re not sure whether a manual control interface will be the way to go, but we are confident that it’ll be important for digital content to be labelled with metadata somehow to convey what it represents for assistive technology users, kind of like alt text for 2D images.

In theory we could maintain a separate tree of label data whose structure matches the structure of the models in the glTF file, but keeping them synchronized as designs are worked on would be difficult.

In our models I noticed that the designer had labelled various “collections” of meshes with human-readable names just to help organize them in Blender, which actually get included in the glTF data. With a little post-processing (like removing the numbers) these could work as labels!

As a visitor moves the crosshairs around in the manual control interface, a ray is cast from the camera to the scene (through the crosshairs’ intersection point). When an object is found, its parent model (“collection” in Blender) is identified and its human-readable name displayed as a visible text label and announced through an ARIA live region.

For the most part this works great, until multiple models get grouped together under an object named “_center” at the last minute! Maybe in future projects this can be mitigated by the designer, but even better would be a file format that decouples object names from their nesting structures.

Future work

Accessibility in AR is still an emerging field, with lots of work ahead to come up with inclusive standards and proven patterns that professionals can follow without having to become researchers themselves.

This presents a real challenge for institutions that want to be at the cutting edge, delivering inclusive experiences that attract and excite visitors. Many of these institutions want their work to be as inclusive as possible, and are legally obligated to do so, but simply don’t know how when it comes to AR.

Over time, standards and best practices will inevitably emerge as the industry continues to build and share solutions, and we’re excited to see what that brings! Until then, we’ll continue to do our best to experiment with AR, follow the on-going research, and listen to feedback to help create the future of accessible immersive experiences!

AR Winter Wonderland: Celebrating the Holidays With WebAR was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.

• vvvv (gamma and beta)
• QT Quick
• Max
• Vuo
• OPENRNDR
• TouchDesigner
• KodeLife
• Processing
• Unity
• Unreal Engine
• GoDot

This first article in the series will focus on vvvv. But before we dive in, let’s look at why we’re exploring these tools and what our goals are for this exploration.

Potential for Prototyping

Prototyping is an integral part of Bluecadet projects as well as our R&D. When we’re exploring potential concepts in strategy, testing and proving a concept, or investigating technical feasibility, we always try to integrate rapid prototyping. We have a few tools that as a studio we’re quite handy with, but it’s always helpful to keep adding to that toolkit.

R&D at Bluecadet often involves producing, low-fidelity experiments to evaluate if an idea is worth pursuing. It’s often this interesting experimental work that clients see and are inspired by for their own projects with us. Exploring a wider set of tools here allows us to keep pushing the boundaries of our work and experiments.

Pushing the Boundaries of our Work

Bluecadet has created some really exciting digital and physical experiences throughout the years. We’ve expanded from websites to digital touchscreens to large-scale environmental experiences. We always want to continue to push that work to be bigger and better experiences. This has led us into spaces like VR, AR, physical LED installations, and digital experiences involving real world objects. As we explore these new tools, we’ll be looking for the potential doorways that open up as a result of this experimenting.

With this in mind, let’s take a closer look at vvvv.

vvvv

vvvv (often pronounced four vee or vee four) is a hybrid visual/textual live-programming environment for easy prototyping and development. Coming personally from a traditional development background, and more recently a bit of work with TouchDesigner, vvvv’s hybrid environment felt like a great place to start my explorations.

When Should I Use vvvv?

If you look at the vvvv showcase video, you can see it being used for projects like large-scale projection mapping and light shows, virtual reality and augmented reality, digital art experiences, interactive installations, and physical computing. The possibilities are immense for vvvv. I think the versatility of vvvv and the range of mediums you can work with is one of its biggest strengths.

vvvv also has a lot of potential as a tool for rapid prototyping. The visual programming component of the vvvv environment can make iterations on projects fairly quickly. Additionally, the ability to create projects without any traditional code might make vvvv a good option for designers who want to be closely involved in the prototyping process.

vvvv comes with some core libraries (including 2D/3D collision detection, 2D drawing with SVG and PDF export, async File IO, among other core features), 3D rendering (based on Stride 3D Engine), VR output, as well as low-level shader based workflow. There is also support for devices like the Kinect, Leap Motion controller, and TheEyeTribe Eyetracker, to name a few. Additionally, outside of the standard vvvv library, there are add ons called packs. These offer additional functionality for your vvvv projects.

vvvv Beta

Currently, there are two versions of vvvv: vvvv gamma and vvvv beta. Both are similar but have their own unique strengths and weaknesses.

vvvv beta is the older of the two versions. While this may seem like a drawback, at the time of writing I have found it to be better documented than vvvv gamma. The sample projects and the online community posting their own vvvv Beta work is much more extensive.

vvvv Gamma

vvvv Gamma, which was released in April 2020, is newer. In general I found vvvv Gamma a little bit more intuitive than vvvv Beta on the surface. However, the lack of samples and documentation (at least compared to vvvv Beta) did make it a little bit challenging to pick up. As I ran through some older tutorials on vvvv Beta, I was able to find similar features in vvvv Gamma once I figured where they were and how to use them. vvvv Gamma feels like the better choice going forward with the caveat to reference vvvv Beta for a more extensive body of work.

Personal Experiences

If you’re coming from a more traditional programming background, you may find vvvv’s potential exciting, but a little frustrating to pick up. The learning curve can feel a bit steep at times. In my experience, there were quite a few things that seemed very easy to do right out of the box, while I found myself struggling to do things I would consider simple in a more traditional programming environment. The tutorials that come with vvvv are good to get started, but if you want to achieve the larger potential of vvvv, it’s important to understand some of the fundamental concepts and workings of the tool. I found that a lot of these foundations of vvvv are similar to more traditional programming tools, but structured just differently enough that it’s not totally obvious and takes some time to understand.

Findings & Takeaways

Generally vvvv feels like a very powerful tool for those who are able to take the time to learn how to use it. During the time I was experimenting with it, I never felt totally comfortable with it as a tool, but its potential feels very in line with a lot of the work Bluecadet does. I think with more experience and learning, vvvv will be a good tool for us, especially for rapid prototyping.

vvvv Samples & Resources

I wanted to share some samples and resources I came across in my research.

RGB Hunt - A small room physical interactive where users interact with projected objects on the walls of the space.

EleBook Prototype - A physical book projection interactive where each page offers a different set of interactions.

Manchester Digital Laboratory / Omniversity of Manchester Press Tutorial -I found this tutorial for vvvv Beta really helpful.

2021 Prototyping Tool Exploration: Introduction & vvvv was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the past year, we’ve had a renewed interest in building collaborative applications and virtual spaces where people can come together. While full-featured multiplayer networked games can be complex, we’ve found a simple socket.io server on Glitch to be an effective tool for rapid multi-user prototyping.

Where multiplayer gets complicated

Multiplayer games get complicated because of latency.

If I send an input to move to the left, my local client immediately renders that movement so the game feels snappy. However, it may take 100 milliseconds for that input to get to the game server, run the game simulation to determine what happens with a given input, and then another 100 milliseconds to send the resulting game state to all players. Now, there’s a 200 millisecond difference between where I think I am and where the other players do.

Competitive multiplayer games get complicated because they must create the illusion that this 200 millisecond difference does not exist. If you’re interested in how this is done, check out this fantastic resource by Gabriel Gambetta.

(Oh yeah, also networked physics.)

What’s not so complicated

Luckily, we can do a lot without solving this latency issue. For many applications, whether it’s a simulated world or a collaborative tool, we can simply run our virtual environment on each client, respond to updates from the server, and not worry about compensating for the milliseconds between us.

A Demo Multiplayer App with Glitch and Socket.io

The following demo app will get you started with the concepts involved in adding simple multiplayer presence to an interface. The frontend is built with React. We will not be covering basic React concepts, so some level of familiarity is assumed. The server is built with Node.js/Socket.io and is hosted on Glitch. Glitch has been a fun development server environment for our multiplayer prototypes. It’s instantly deployed and allows for quick collaborative iteration.

Get the full code repos here:

• Frontend React App on Github
• Backend Node/Socket.io server on Glitch

The Frontend

https://github.com/ChrisArasin/simple-multiplayer-client

Our demo app will render a user as a colored circle and move them on click. We’ll use react-spring to interpolate between user position updates. While not suitable for every use case, getting fewer state updates and handling more of the movement in a client-side interpolation is a nice simplification and can still communicate presence effectively.

The Player

The player component renders a dot with a position and color. React-spring’s useSpring hook will interpolate between position updates with some configured settings (which are quite fun to play with).

The Game Board

Our Game board renders all other players at the latest updated position from the server, renders the current player in their latest input position, and captures user input. We separately render the current player so the user does not feel a delay between their input and the application’s response to a server update.

Client-Side Network Class

The client-side network’s init method is called on mount of the application. This establishes the socket connection, sets at listener for the stateUpdate events the client will receive from the server, and sends an initialize event to the server which will register the current player with its randomly chosen color.

When the user clicks to move, we both locally update the user’s position and send this position update to the server.

The Node Socket.io Server

Glitch: The friendly community where everyone builds the web

Our server will accomplish a few tasks:

• Keep track of the current state of all players
• Handle new users joining
• Send updates to each client of the current player state
• Remove a user when they leave.

Player State

Our state data will be an object holding player information, keyed by their IDs. The createPlayer function will be called when a user joins to build a new player object.

Handling Player Events

We set up a handler for when the client side application calls io.connect. In this handler, we set up listeners for the three events the client can trigger.

When the client application sends the initialize event, we create a new player with the corresponding id and color, and add that new player data to the game state. We also begin emitting updates to all players if this is the first player joining (discussed in the next section.)

When the client sends a positionUpdate event, we update that player’s position in game state.

When a player leaves, the disconnect event is sent, and we remove that player from state.

Broadcasting State Updates

Once users have joined and we are updating their position in the server-side game state, we must broadcast these updates to all players.

The core of this function is io.emit('stateUpdate', players) call. This sends a snapshot of the current player data to all connected users. On the client side, this state object is used to interpolate each player to their updated position.

We have a stateChanged flag which prevents an update from being sent if there has not been an event that changed the application state since its last update.

We then confirm that players are still connected, and schedule another stateUpdate message at a given interval.

Extending the Pattern

This pattern can be summarized as:

1. Client-side: Emit a user action as a message to the server
2. Server-side: Update shared application state
3. Server-side: In a game loop, broadcast that shared state to connected users
4. Client-side: Interpolate between state updates

In our Holiday Oasis, we used this flexible pattern to model all user actions, including movement on different surfaces, eating holiday treats, and triggering dance floor fireworks. It should serve as a solid foundation for bringing multiplayer features to a variety of contexts, whether it’s a real-time collaborative application or a simulated 3D environment.

Just Enough Multiplayer, A Simple Start for Prototyping Realtime Multi-User Spaces with Socket.io was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.

As with any display technology, it helps to look carefully at what its capabilities are in order to make sure it’s successful. Our goal was to test three different displays and come up with some best-use-case scenarios and a few design principles for visualizing the most impactful content. The four displays discussed below are Looking Glass Factory’s (LGF) 8K and Portrait offerings, the Magnetic 3D 50" Wildfire Pro and the Sony Spatial Reality Display, referred to as “8K”, “Portrait,” and “SRD” respectively.

Whenever possible, videos are presented below in full resolution and frame rates so the displays can be fully appreciated and evaluated. Obviously, the full 3D effect can’t be fully appreciated with a single camera lens, but these should give a rough sense of parallax, visual artifacts, etc.

Special thanks to all the vendors listed here. Sony and LGF generously sent us test units (some of which were in early prototyping phase), and Magnetic 3D organized a safe in-person review and walkthrough.

Tech Overview

The displays below accomplish a 3D effect through 2 different methods, each of which have specific design constraints and caveats for use.

Light Field Displays

The LGF and Magnetic 3D displays make use of a sophisticated lenticular film on the screen surface that channels not only slightly different images into each eye (i.e. a “stereo-pair”), but also a different set of images based on where you are located with respect to the screen.

In order to drive these screens, content must be sent to them with all possible views of the 3D subject stitched together into a “quilt.” The internal processor of the screen receives the quilt image and arranges the thin slices of pixels to work with the micro lens to achieve the 3D effect. As seen below, LGF displays require a 45 image quilt to achieve a smooth, high-fidelity parallax effect. By contrast, Magnetic 3D displays are capable of running a single 2D image (for use as a regular display), a double image stereo pair, and 9 image quilt.

It is important to note that this technology does not allow for vertical parallax, meaning the stereoscopic effect is only created along the horizontal axis, from left to right. And while the 3D effect is certainly perceivable while standing still, it is much more effective when the user can move and truly appreciate the parallax.

Due to the nature of the technology, some subtle visual artifacts will always be present and are important to keep in mind:

• Moving between stereo-pairs (i.e. the images of the left and right eye get momentarily switched) a subtle “cross-eyed” effect may be noticeable. This is less apparent on displays that have more images in their quilt as there are more image pairs available with less horizontal distance between them.
• As the maximum viewing angle allowed by the micro lenses is reached, the image will “fail,” giving conflicting visual information.
• Within the 3D scene being visualized, content positioned too close or too far from the viewer may cause image ghosting. This results in a “sweet spot” for content somewhere in the middle that gives the best parallax effect while reducing artifacts.

The greatest advantage of this technology is that it provides a multi-user, “glasses-free” approach, allowing multiple people to enjoy the content with no additional equipment (i.e. barrier for entry).

Eye/Facial Tracking Displays

Facial tracking displays like Sony’s SRD take advantage of the micro lenses mentioned above in addition to a facial tracking camera embedded within the display’s housing. This allows the display to track the position of the user in space and show a view of the 3D subject that would correspond to where they are with respect to the screen. This allows for both horizontal and vertical parallax, giving users more freedom to examine the subject from various angles.

The biggest limitation of this methodology is that it is truly a single user display. Any other viewers in the space aside from the primary, facially tracked user will see a warped perspective designed solely for the first user. A similar problem exists if there are no users present: the display doesn’t know what perspective to show. In this case, it is important to leverage the facial tracking data (or lack thereof) and have the screen fall back to a 2D image to attract users as opposed to a visually distorted image. Sony’s SRD SDK does this automatically.

Naturally, these types of displays will also suffer from the shortcomings of facial tracking technology in general. Masks and other types of facial coverings, whether pandemic or culturally related, will limit tracking ability. Darker skin tones are also a known failure point for many facial tracking toolkits so it is important to thoroughly test lighting conditions, device positions, and other factors to maximize accessibility for all users.

Designing For 3D Displays

With any of the displays below, one thing is exceedingly clear: they are NOT regular screens. The content intended to be shown MUST be carefully curated and designed to maximize the 3D effect. Below are some principles and suggestions we noted to make 3D visuals more successful.

• Solitary “hero” objects work best as they provide the largest contrast between the foreground object and the empty space they exist in. Seeing a scene that extends across the whole display and gets cut off at the boundary slightly ruins the illusion. Note the difference between the LGF 8K Squid video versus the LGF rabbit scene below.
• Construct the overall scene to give the viewer as many visual hints for depth as possible. Things like lighting, movement, and background selection give the brain additional cues that cause us to perceive depth. A high contrast object on a flat background, artificially scaling an object up as it moves toward the user in 3D space, and backgrounds that provide perspective or a horizon line will amplify perceived depth.

• Tailor the content to the spatial setting and use case. Insetting the display within cabinetry or setting it just beyond reach will prevent users from getting too close and ruining the illusion. Using large hero objects/3D text with dramatic depth transitions call attention from a distance while using subtle parallax effects to augment closer viewing.
• Be careful with text use. Each display’s own SDK provides ways of rendering text for maximum readability but image artifacts can cause text to be difficult to read or even disorienting if not used carefully.
• All the displays reviewed have some type of lenticular coating making them highly susceptible to damage from touching, scratching, and finger oils. Protective glass is necessary in publicly viewable use cases.
• In light field displays, be wary of visualizations requiring lots of small objects like particle systems or point clouds. The small dots partnered with visual artifacts from the lenticular displays can create unflattering moire patterns and visual confusion.

Looking Glass Factory Displays

Looking Glass 8K

Upon first view, the 8K display is a substantial piece of equipment. It is very solidly constructed with a thick pane of optical glass and a solid aluminum chassis. Make sure you get a spotter when moving it around and have proper support for any mounting scenario! In addition to substantial hardware, there are a substantial number of pixels. 8K worth of pixels is a lot to drive in addition to the overhead of rendering 3D content in 45 different views required by LGF. Make sure your computer is equipped with a capable graphics card with 2 display port connections along with a USB 3.0 connection to control the display. While LGF recommends a Nvidia RTX 2080, we tested the 8K successfully on a GTX 1080 Ti.

Looking Glass Portrait

The portrait display is quite small and lightweight, lacking the thick glass of its larger siblings. At the time of review, the portrait was still in development and included pre-baked content which did not require a PC to run, only a USB cable to power the internal Raspberry Pi 4.

(Apologies for the portrait orientation videos for those on desktop!)

Notes

• Optimum viewing distance was roughly 6-12 feet, but viewing up close was not bad. From far away, many visual artifacts disappear.
• The image on the 8K is surprisingly crisp considering that the many pixels are split up by the 45 image quilt (equivalent to a roughly 1145 x 644 pixel image).
• 45 discrete views within the quilt makes visual artifacts between stereo-pairs very subtle and is overall a pretty seamless experience while moving around.
• The measured horizontal viewing angle is about 50 degrees total, meaning about 25 degrees off-center until the image “fails.” At the maximum viewing angle the 8K will give a conflicting stereo-pair causing a nonsensical image, while the Portrait contains a privacy filter that causes the image to darken at extreme angles before the “fail” point.
• The thick glass on the 8K display is beautiful and eye catching, however it will produce a bit of glare so the location of the display with respect to ambient lighting should be planned carefully.
• The Portrait lacks the glass and because of this it looks like you’re peering into a portal.
• LGF requires an OS level service called HoloPlay (available for Windows and OSX) and has plugins for Unity, Unreal, Maya, Blender, DepthKit and more.
• Visualizing text or other flat imagery needs to be done thoughtfully and positioned carefully within the 3D scene on the convergence plane (see “LGF Text” video below).
• For debugging/development, the 8K screen must be attached as a secondary, extended display but micro lens prevents the desktop from being readable so it is not quite usable as a monitor during debugging/setup. We did not test using the LGF as a stand-alone display.
• LGF supports display tiling relatively easily within the SDK in order to build a video wall. However, all displays have a fairly large bezel that should be factored into the video wall design. For the Portrait display, the privacy filter should be taken into account when tiling as the image may go dark when viewing several side by side.

Magnetic 3D Display

Magnetic 3D has a wide array of offerings, with screens of all sizes and are available to work out custom solutions from small displays to large 100" screens and tiled video walls. We were able to view the 50" Wildfire Pro display and sample content with a Magnetic 3D representative, but were not able to drive the display with our own content.

Notes

• The 50" Wildfire is a 4K display that utilizes a 9 tile quilt, meaning each image is roughly 1,280 x 720 pixels.
• Fewer images in quilt mean more discrete “steps” as you move between stereo-pairs from left to right and slightly more noticeable “cross-eyed” artifacts when between stereo-pairs.
• Screen is very flexible and has settings via remote control to switch between a single image, double image, and 9 tile quilt. This is very helpful when switching back and forth between computer desktop and 3D content or when lower fidelity is needed for performance considerations.
• M3D officially supports the display with a Unity SDK, but the display could potentially work with other platforms provided they output the same 9 perspective views.
• Since custom displays are not uncommon for the company, lower bezel displays can be leveraged to make video walls or embed displays into exhibit/installation cabinetry.
• Perceived depth of display feels a bit deeper than LGF displays by a small margin, about 12-18," with the sweet spot of the visualization being 8-10".

Sony Spatial Reality Display

In a different category from the previous displays, the Sony SRD is a facial tracking display and contains a camera embedded on top of a 15.6" 4K display. The screen and camera are set within a prism shaped metal housing and includes two triangular pieces that attach via magnets that help frame the content and enhance the illusion of depth. The display requires a PC with a simple HDMI port and USB connection for the camera, and as expected requires a strong GPU capable of real time 3D rendering of 4K content.

Notes

• The 3D effect is superb, allowing users to feel a very real parallax as they move around the object both horizontally and vertically.
• Unity SDK is available and contains examples for displaying content and getting facial tracking data.
• Facial tracking feels fast and responsive with very little lag or visual artifacts as you move around the display.
• The SRD is set to “fail” gracefully back to a 2D image if no users are found. If multiple users are visible, it will persistently track the first user to step into frame.
• Consistently takes less than 1 second to find face and adjust the perspective when going from obscured to unobscured face.
• Facial tracking is possible on faces wearing masks but not great. It depends on lighting, face angle, etc. The author has medium-to-light brown skin and found the tracking to be on par or better than most open source facial tracking toolkits. However, the testing the display in a dark room definitely deteriorated the tracking quality so be sure to test in different lighting conditions.
• Screen is shiny and glare prone, but the facial tracking is highly dependent on ambient lighting to find users.
• Provides the most “real” hologram experience of all displays seen. Items truly feel touchable.
• Viewing angle of the display is very wide, however facial tracking may fail at very wide viewing angles. It has the widest viewing angle of any 3D display seen.
• Biggest pitfall: It is truly a single user experience. Seeing someone else use the screen feels very strange. If a camera happens to lose its primary user, it may quickly latch on to another user standing far away, which feels very alienating for the first user.

Wrap Up

All in all, we’re excited to find opportunities to use these displays in the right setting. Together with the right designs, they can be highly eye-catching and impactful without feeling tacky or gimmicky.

Lastly, I’d like to give a special thanks Looking Glass Factory, Magnetic 3D and Sony for giving us the opportunity to access and evaluate their displays as they graciously sent us test units, arranged (safe) in-person reviews, and talked us through using their products.

Disclaimer: Bluecadet was not compensated in any way for this review by the aforementioned partners but we do appreciate their openness and the time they spent informing us of their offerings.

3D Displays: Opportunities & Design Principles was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Here at Bluecadet, most of our multilingual sites we have developed in WordPress, but when we get into custom integrations, we move into the Drupal realm. I’m constantly programmatically creating and updating nodes with data from external sources and I’ve been curious about how this would work within a multilingual Drupal site. I did not think it would be a big deal, just another programming pattern to learn how to code in Drupal.

The quick test

To begin, I just made my local Drupal 9 test site multilingual, with English and Spanish enabled. There are a lot of helpful articles out there to get you started if you are curious, but it is not crazy. Moving around the admin side of the site though, you have to be careful. I just need to slow down and be purposeful about what I am clicking.

After it was all set up, I simply played around in Drupal Devel’s PHP editor, using the default “Article” Content Type that is present when you install a site. The process was just a simple step to get the node translation. Editing and updating the node and its translation was as straightforward as expected. Below gets into some of the details.

Time to get into the weeds

If you regularly work with your nodes in code, this should look familiar.

1. Started out creating a node as usual:

2. Now I create the Spanish translation and save it:

3. And now to edit the node, we load and edit the English version:

4. And load the Spanish translation and edit:

Unlike most things with Drupal, this came together pretty quickly with little fuss. I’m sure as I dive in further there will be a lot of nuances to discover, but I can’t wait to work on a multilingual integration for a production site.

For those interested, here is the full Github gist:

Drupal Programmatic Translations R&D was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.

When we created the Connections AI Table with The Henry Ford Museum of American Innovation (THF for short), our goal was to demonstrate the interconnectedness of the museum’s collection. THF curators did the heavy work of identifying meaningful connections between many of the objects; we wanted to extend that experience to as much of the collection as possible. To fully provide endless connections to our users, we leveraged the computational power of artificial intelligence (AI). Through our research and engineering we found that while the current-stage AI is no substitute for human curators in systematic thinking, it can be adopted as an effective tool to suggest visual connections between seemingly unrelated objects. The result was AI-curated connections that offer never-ending interactive exploration that is both insightful and full of surprises.

In this post, we share our early work that formed the foundation of the AI aspects of the project. We started prototyping with a full export of THF’s digital collection—containing tens of thousands of object images and related information—and came up with two approaches to create visually meaningful paths between images in the dataset: visual similarity paths and dominant color paths.

Visual Similarity Paths

To start, we used a pre-trained deep learning model (VGG16) to analyze every image to get the model’s internal representation of this image (technically called latent representation, in the form of an array of numbers). With this information we could find the location of every object in the “THF universe” (technically, latent space), as indicated in the following graph:

For any two objects in this universe, we can create a visual similarity path by simply selecting a series of objects between them. But more specifically, how do we select these objects?

Naïve Approach: Linear Interpolation

Our first approach was simple. In the latent space, draw a straight line between the two objects and evenly divide the line into a few steps. For each step select an image that is closest to the dividing point from the latent space. This is almost like taking the 2D visualization above, tracing a line through objects, and presenting those back as a path of visually similar objects—but we’ll do this in a higher-dimensional space.

The original 4096-dimensional latent space is too sparse for the number of images we have: If we simply draw a line between two images in that space, most parts of the line will be in empty space, so the closest images to each step could be drawn from pretty far away. Even if we reduce the latent space to 10 dimensions using the PCA (primary component analysis) algorithm, this situation still happens and is reflected by duplicated images in the following example path.

After reducing the dimension of the latent space to 3, we mostly get non-repeating paths, as the following example shows.

We also have control of the number of steps we sample along the path. The more steps a path has the smoother the transition is supposed to be. However, the paths formed by linear interpolation only make sense to some degree. After all, when drastically reducing the dimension, we lost a lot of information from the latent representation. Our next approach, shortest path, is able to find more convincing visual connections because it can take advantage of the information in the original high-dimensional data.

Sophisticated Approach: Shortest Path

First, build an interconnected network (technically a graph) of object images using the latent representation data. For each image, find the closest k images in the latent space, and set up a direct connection between them. For all other image pairs, the only possible way to connect them is through other images. Then, for two randomly selected images, run the shortest path algorithm to find a path between them.

With this approach we also have a few parameters under control. Similar to linear interpolation, we can reduce the dimension of the latent space. But with shortest path, the higher dimension we keep the more information we make use of, and therefore the better the result. The benefit of dimension reduction is time saving when building the graph. With digital collections there is hardly ever a case where we would need to update this “live” for the user, so we were able to run our algorithm overnight as a daily routine. It has all the time it needs.

Different from linear interpolation, we can’t explicitly choose how many steps we want the path to be. Instead, we choose how many direct connections to assign to each image when building the graph. The more direct connections the denser the graph is and the shorter an average path will be, and vice versa. However, we don’t want to assign too few direct connections to each image: that might result in image pairs for which a path cannot be found, since the graph would not be fully connected.

Example Paths

In the original 4096-dimensional latent space:

We can also slightly reduce the dimensionality of the data for some moderate speed gain. In the reduced 300-dimensional latent space, the paths are nearly as good with occasionally more confusing connections. We ended up not doing this in production, as the speed gain wasn’t worth the weirdness of some of the connections.

Note that although we can’t explicitly ask for a fixed-step path, we can always extend a path by forming a new one starting from the endpoint of the existing one. The following examples (with each new path starting from the end of the last path) are again from the 4096-dimensional latent space:

Or we can form multiple paths from the same beginning object:

We can play with longer—and sometimes weirder—connections by selecting a random “middle” point and building a compound path from A to Z to B (Z = the random middle):

Dominant Color Paths

Besides connecting objects with visual similarity, what’s also conceptually interesting (and visually appealing) is connecting them with objects of gradually changing colors, forming a gradient.

To start, we used the k-Means algorithm to get the dominant color of every image. A color is represented by a 3-dimensional vector. Therefore, we can run the same linear interpolation and shortest path approach described above with the color data. The catch is that not all images are vivid enough to pick up their main colors—a few pale images in the path and the color transition won’t make much sense to the eye. After some experimentation, we figured out the following steps to ensure beautiful color paths:

First, we cropped each image to its central 50% part to reduce the influence of background colors—especially since THF’s collection objects are not uniformly documented against the same background. Then, we ran the k-Means algorithm with k=3 (to cluster all pixel colors into 3 clusters and select the centroid color of each cluster). We sorted the selected colors of an object in descending order according to chroma (chroma is computed by first converting the RGB color into LAB color space and calculating sqrt(A² + B²)). Essentially chroma gives us a mathematical representation of the image’s vibrancy or color intensity. We then selected the first color that has a chroma no smaller than 2000 (colorful enough) and has a cluster size bigger than 25% of the whole cropped image as the dominant color of the image. If none of the colors meet the criteria, we discarded the image because it will be confusing to the eye in a color path.

We use the same linear interpolation and shortest path approach to find paths in the color data. Because the color space is 3-dimensional (low-dimensional) by default, the images seem to be distributed rather evenly. Therefore, both the “naive” linear interpolation approach and the “sophisticated” shortest path approach work pretty well. Actually, linear interpolation might work better because the color path feels more “linear” whereas the shortest path can “zig-zag” in the color space.

Example Paths

Below are paths among object images, formed by linear interpolation. Each image has its dominant color below it. Note that when randomly selecting the beginning and ending images we made sure they are distant enough in the color space to allow interesting transitions between them.

Paths among object images, formed by shortest path algorithm:

Where to from here?

The learnings from this phase went directly into the design and development of the Connections AI Table, with integrated AI-generated connections becoming a key component of the experience. From here, we dove into production:

• Designed a playful, explorative, and meaningful user experience around the technology
• Optimized the performance for the interactive experience
• Integrated the Python/Tensorflow-based AI backend with the C++/Cinder-based interactive frontend
• Streamlined the data processing routine to allow continuous updating of the museum collection data

What is fascinating in looking back on this work is that in a way we had to become curators ourselves. As you can see above, we had to hone the algorithm and its parameters to find connections that were meaningful for our users. All of the examples are technically “correct” to the AI—but it took actual human input and critical thinking to build something that we felt made sense.

To learn more about the full creative process for this project, read Playful Prototyping with Machine Learning.

Endless connections: Using AI with digital collections was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.

We’ve seen a lot of awesome stuff with the new(ish) iPhone 12 Pro’s LiDAR scanner, like this great post at Sketchfab. Naturally, we had to give the process a try ourselves — with an eye towards using this data in creative tech applications.

TLDR: use an off-the-shelf app to get started faster, and familiarize yourself with MeshLab if you’d like to render these very detailed captures in code.

Dipping Toes: Sketching in Unity

We do a lot of our AR prototyping in Unity, so that’s where I started. It gives you a lot of features for free, including making meshes of real-world objects and allowing those meshes to occlude objects.

There’s more to be shared there… stay tuned.

A big downside of Unity’s implementation is that you don’t get access to the raw LiDAR data, which includes points, distances, and colors. To play more with capturing the real world, you need to either a) roll your own native point cloud code or b) use an off-the-shelf app.

I chose the latter, following Sketchfab’s lead and downloading 3D Scanner App and SiteScape.

Off-the-Shelf Apps: Capturing & Cleanup

These apps are great, mostly! Most importantly for me, SiteScape natively exports point clouds PLY files, which are supported by both modeling software as well as THREE.JS, our WebGL framework of choice. 3D Scanner is more interested in making 3D meshes; cool, but not my ideal case here.

Anyone who has experimented with depth camera data knows: there’s pretty much always some processing you need to do before showing off your shiny model in code. SiteScape is mostly meant for architectural use so it a) captures a ton of points and b) keeps the origin of the model based on where you started scanning. I needed to edit the density and position of my point cloud to make it workable on the web.

Our go-to quick modeler/exporter/cleaner-up-er is usually Blender. It remains great, but sadly does not support PLY point clouds. It can view ‘em, and cannot export them. On to another tool!

MeshLab is another open-source tool, focused specifically on simplifying 3D scans for other uses (e.g. 3D printing).

I followed three steps here:

1. Reset origin

• Using the Transform tool found in Filters → Normals, Curvature and Orientation, I reset the origin to “Center on Scene Box”

• I checked my work by turning on the axes via the lil’ button at the top of the screen. Do that, so you don’t use your mind.

2. Translate

• Using that same tool, I moved my y-axis down to place the origin — and our future user—on the ground
• Make sure you select “Preview” before you apply!

3. Reduce

• I used the catchy “Simplification: Clustering Decimation ” tool under Filters → Remeshing, Simplification, and Reconstruction to simplify my mesh

• I played with a few percentage-based settings here — settling on reducing by ~2%.
• Note: many guides will recommend using Quadric Edge Collapse Decimation to simplify. That is slightly easier, but we have no faces since we’re a point cloud. ;(

Bringing it into Code: PointClouds in Three.js

Full disclosure: this mesh could and should be simplified further! But I was keen to try it out in its (near) full glory.

Loading the mesh

Loading and displaying a PLY file in Three is fairly simple, once you’ve setup your file and added the PLYLoader external file.

var mesh, material, geo; // these become important later

// setup your loader + load your PLY
var loader = new THREE.PLYLoader();

loader.load( '../3d/arches.ply', function ( geometry ) {
   geo = geometry;

   material = new THREE.PointsMaterial( { 
      vertexColors: THREE.VertexColors, 
      size: 0.05, 
      sizeAttenuation: true 
   });

   mesh = new THREE.Points( geo, material );

   // my meshes were still oriented wrong for THREE
   // and needed to be scaled; you may not need these

   mesh.rotation.z = - Math.PI / 2;
   mesh.scale.multiplyScalar( .5 );

   scene.add( mesh );

});

The options in the material proved crucial

• vertexColors : load color data from your vertices
• size: rendered size of points; I played with this a lot for effect… it’s very small bc my mesh is in meters
• sizeAttenuation: change point size based on distance from camera

Making it Move with Some Noise

I used the very simple SimplexNoise.js to create a “chaos” state for my points I could tween to.

1. Setup Points: this code lives in my “loaded” function

... 

// cached original points
pointsGood = geo.attributes.position.array;

// noise points: array based on # of points in mesh
pointsNoise = new Float32Array(pointsGood.length);

// points rendered at any given frame
pointsLive = new Float32Array(pointsGood.length);

var index = 0;

for (var i=0; i<pointsLive.length; i++){
   var x = pointsGood[index];
   var y = pointsGood[index+1];
   var z = pointsGood[index+2];

   // my use of x,y,z here is arbitrary!
   pointsNoise[index++] = this.simplex.noise3D(x,y,z);  //x
   pointsNoise[index++] = this.simplex.noise3D(z,x,y);; //y
   pointsNoise[index++] = this.simplex.noise3D(y,z,x);; //z
}

2. Tween between the two, once we’re loaded. In your animate() function…

// keep a running increment from 0 - 1
// my "pos" var started at 1 in setup()
pos = pos * .9 + .1;

// loop through and set "live" points
var index = 0;

for (var i=0; i<pointsLive.length; i++){
   var x = pointsGood[index];
   var y = pointsGood[index+1];
   var z = pointsGood[index+2];

   // lerp between "good" and "noise" points

   pointsLive[index++] = 
      THREE.Math.lerp( pointsNoise[index], x, pos ); 
   pointsLive[index++] = 
      THREE.Math.lerp( pointsNoise[index], y, pos );
   pointsLive[index++] = 
      THREE.Math.lerp( pointsNoise[index], z, pos ); //z
}

// tell Three your points are updated
geo.addAttribute("position", 
   new THREE.BufferAttribute( pointsLive, 3 ) );

geo.attributes.position.needsUpdate = true;

And voila! A fast-ish demo of a big LiDAR mesh in Three.JS.

Check it out here. (Yes, it takes a long time to load, still!), and if you’d like, look at my sketchy code here.

What’s Next?

Now that we have our pipeline down, there’s a lot left to try:

• New types of scans: I’d love to scan some smaller objects, and play with simplification settings to find the balance between “good” and “portable”.
• More efficient code: The above is a prototype, so it’s lazy. Moving manipulation of the points into a shader would make it infinitely faster, and open up possibilities of tweeting between different scans, having the points drift every frame, etc.
• Rolling our own scanner? SiteScape is awesome because it’s free and super full-featured. But, it smartly culls a lot of data for you, which led to some underwhelming scans when I tried to capture statues in Grand Army Plaza. Since we might be more focused on museum-style use cases, we’ll eventually need to try out our own, special toolset.

Until next time, ✌️

Hello LIDAR World was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.

Moving to Google Workspaces

Here at Bluecadet we recently upgraded to using Google Workspaces, which included the ability to use Google Drive to store all our project files. Previously, these files were scattered across an internal server, Dropbox, and individual Google Drive accounts. Our team created a flexible file structure template for our projects with directories for supporting our most common workflows and file shortcuts to key documents and files. Unfortunately, testing revealed a problem: a simple copy paste of the parent directory did not preserve those shortcuts (they referenced the original directory in the template project, rather than the new one in the copied project).

Shortcuts

Let’s take a step back and look at what exactly a shortcut is and why it’s a problem. A shortcut is a link that references another file or directory. In a complex file structure it is sometimes handy to have a quick shortcut to another directory multiple levels or directories away. For example, if you have a directory containing notes for developers, it might be nice to have a shortcut to that same folder under your client discovery directory.

However, when you copy the file structure, the shortcut remains pointing to the directory in the template structure and not the new directory in your new project.

Our Google rep suggested that we use the API to help maintain the directories and shortcuts when copying them to a new project folder. And this is where I entered, tasked to make it all work! As a web developer typically deep in PHP code, I was excited to dive into something a little different, the Node.js world.

Breaking down the problem

Here is our little project scope: We had a specific shared drive. In that shared drive we had a directory that contained multiple templates, depending on project type, with each containing about 80 directories and shortcuts. So, I had to be able to copy any of these templates. The templates were a work in progress so it was important to be able to just copy and pate the template structure in Google Drive, instead of hard-coding the templates in a config file. Oh and, of course, it had to be done ASAP so we could get the team using the new structure.

The first step was just getting connected to the Drive API. In the past I’ve always had problems dealing with authentication and Google APIs, however, their basic node.js example got me up and running pretty quickly and to be honest that was not as terrible as I thought. Pleasant surprise!

Directory wrangling

Second step was simply getting a basic understanding of how Google treats files and directories. I played a lot. After I had about 20+ tabs open with pages from different parts of the Drive API documentation, I was getting pretty familiar with it and started to dive into my actual task, copying an entire directory structure. And now we move from the nice little contained Google examples into the fun of a real project. It never works as easy as their examples. I created a test directory structure, all 5 directories, 2 levels deep.

The first hurdle was efficiently getting the directory structure. There is no way to call the API and say give me all the children of a directory. I can query for EVERYTHING on a shared drive and extract what I need, or recursively work through the hierarchy. In order to limit my total API calls I chose the first option.

After that hurdle, it was a pretty straight forward process. Find a directory, copy it. And I was sort of done! Or so I thought…

A long way to shortcuts

The next step was to create the shortcuts. This is also pretty simple, except for the fact that the directories need to already be created. Now I had to manage ids; ids from the original template structure and the ids of the new structure and their relationships. Small hurdle, but added complication. But note that Google does provide a handy method to generate IDs so I can build this relationship structure before actually calling all the API endpoints to create directories and shortcuts.

At this point we are talking about a few hours of work, nothing crazy, and I am feeling really good about it and I take a step back. All of a sudden it hits me — since this is a Node script, how are project managers, the ones creating new projects, going to do this? I don’t want to be bugged every time we start a new project. And I definitely don’t want to have to try and train project managers on installing and running a NPM package. I can’t even imagine. Next hurdle.

Creating a tool for the team

There are a few options out there that I could have done, however I’ve been really interested in Electron.js apps and its potential. So not getting any advice or discussing it with anyone, and thinking it would only take an hour or two, I went right in.

Needless to say, I went from dealing with hurdles to hitting a number of walls… but it was so much fun. I do like a challenge. And, if you are asking, of course I underestimated the 1 or 2 hours to convert it.

First major wall was getting Google authenticated within the Electron App. Electron is a browser but also, it’s not. And it doesn’t like playing with things outside of its little world. You know, security and all. I’m sure there was a better way to do it, but I eventually got it working. I do think this is what took me the most amount of time transferring everything to Electron.

The next wall for me was fully understanding how Electron’s main.js file interacts with the window you are actually seeing. Initially it was so perplexing. It also could have been that it was midnight and I was on hour 6 of what I was sure should have taken me an hour. Probably not though. Needless to say, when I took a break, let my brain recoup and actually read the documentation, it all started to become clear.

The next steps were to build out the UI a little bit, and how a user would choose a template and enter the new project title. Nothing Earth shattering here, but it was really fulfilling to see this “thing” come to life. Command line tools are really cool and powerful, but all of it is so hidden. It is very underwhelming at times. I did all this coding, I run the process and all I get is a new blank command prompt. Yeah! It wasn’t much but adding the UI felt good.

Now, of course, I jumped the gun and told the higher ups it was done and working. They did want it ASAP. Also, do you remember I said my test was 5 directories? And the actual template structure was 80+? Well, 10 minutes before I go to show it off, I switch it over to the real template directory structure. And it stalls… nothing. No output (I threw in a lot of debug messages as it is creating the directories) after the first 10 directories.

Well, after my “oh crap” moment, and rescheduling some meetings, I went back in. So, of course, I tried creating way too many new projects to debug what was going on: Will this work_001, Will this work_002, How about this one_003, please God let this one work_004, WHY WONT THIS WORK_005, did that actually work or was I not paying attention_006. Since this was in the shared public drive, I did get a few people asking me about my mental health. I was fine, just tired of naming things Test ###.

Error handling

Unfortunately, I didn’t take as close a look at the error messages as I should have. Another note, set proper reject methods and messages for your js Promises. It could save you time when debugging. Apparently copying 5 directories you do not hit any of Google API’s rate limits. When copying 80+ you do hit those limits. That fixed, we were good to go, but just running slower in order to not hit the limit. Also to note, you have to make one call to the API for each directory creation, instead of sending all the data at once. Kind of annoying.

Long story short I had a lot of fun “playing” and building up my js skills and after a solid week’s worth of work, we had a little desktop (Mac only) app to copy directories in Google Drive. We plan on building it out a bit more in the future, and I look forward to the new challenges.

Google Drive, Node.js, and a custom Electron app was originally published in Bluecadet on Medium, where people are continuing the conversation by highlighting and responding to this story.