Those of us who have been on professional film productions anytime in the last two decades or so know the utter importance of having a qualified and talented DIT on set. DIT stands for “Digital Imaging Technician” and involves skills far more than simply copying and verifying media from digital cameras to hard drives. A really good DIT is able to work closely with the cinematographer and director to help inform creative decisions and help communicate to important stakeholders.

My expectations for a DIT have always been high. Having a strong post production background and invented many of the workflows I use on all of my 360 and 180 immersive video projects, I have built a skillset that proves extremely valuable to myself, but also to other creatives when we collaborate. I want to share my techniques and tools here, in hopes that it inspires others to jump in, learn, and improve upon what I’ve started.

A VR DIT needs to be able to accomplish the following tasks:

• Organization of Media from many cards. This means having super fast storage and enough bandwidth to import data and back it up. Having several USB 3.0 interfaces along with several card readers is a good idea. Most traditional DITs roll in with one or two readers, but 360 cameras often capture media on six or eight. The Yi Halo uses a whopping 17 micro SD cards. In order to do the job on set and deliver a fast turnaround, you need a good system and you need to understand data rates, bandwidth of the peripheral bus, hard drive speeds, etc.
• Rough stitching. Knowledge of PTGui and stitching software such as Mistika VR is essential. You should be able to take the images from the card and generate a rough stitch for review by the creative team. And you should be able to do this as quick as possible. Nobody wants to wait around for hours while you struggle with the technology.
• Color management. This is less important with some cameras, but with many high end productions now filming with professional level cameras, understanding how LUTs work and how to interpret a raw image is very important. Especially for any stakeholders, they might need to see a “rough grade” so they feel confident in the image that is being captured.
• Live preview. Many 360 cameras do not offer a great way to see what is actually being captured. While I understand this is traditionally the job of a VTR technician, on the small crews of a a 360 production, my expectation is that the DIT also understands how to help the team review what is being captured. The DIT should be able to unwrap the fisheye images from the cameras into equirectangular space and rectilinear space for review at various FOV sizes, including headset preview if needed.

The Light Sail VR DIT Cart

Here is what I have built and why I chose each component.

Filmtools Vertical Cart
I like this cart because it’s compact and tall. I’ve added a few shelves onto mine as well as a triple baby pin adapter to hang my monitors.

Atomos Sumo 19" Monitor
This monitor acts as a confidence monitor, but also has the ability to record. Recording the unwrapped image is a really great way to offer quick playback on set for stakeholders to confirm they are ready to move onto the next setup.

APC 1500x UPS Battery Backup
Having the whole system running on a battery backup is smart because it allows me to unplug my cart and push it to the next setup or location without having to shut everything down and boot back up again. Mine gives me about 12 minutes of run time.

TP-Link AC750 WiFi Adapter
This tiny portable wi-fi hotspot allows me to create my own ac-hoc 5Ghz network so that I can connect my Oculus Quest 2 in AirLink mode wirelessly.

Custom Built PC
You can use whatever you like here, but I’d recommend an NVIDIA graphics card that’s good enough to stitch at the razer size you require, as well as a good CPU and plenty of RAM. You’ll also need a Blackmagic Decklink 8K Pro card to handle video input/output from professional camera rigs. Personally, I roll with a RTX Titan, AMD 1950x Threadripper, 96GB RAM, and an extra USB 3.0 PCI card. The case I use also has handles, making it pretty easy to pick up and move.

Asus ProArt 24" Monitor
This is just a standard HD computer monitor, but with pretty good color. It’s calibrated with my iDisplayPro. I am looking to potentially swap this out with a touchscreen monitor.

Oculus Quest 2
The Oculus Quest 2 is my favorite headset for comfort, price, and quality. I have it linked up via AirLink so that I can have a truly wireless experience. By utilizing a custom app I built to push NDI signals to the headset, I am able to view stuff on set, in headset. This is incredibly valuable to be able to make creative decisions regarding camera height, proximity, and stereo convergence.

Needed Software

There are a few pieces of software I highly recommend adding as part of your kit if you decide to become a VR DIT.

Hedge
Hedge is fantastic for backing up and transferring footage from cards. It’s fast, very customizable, and you can even check the status of transfers from your phone. Very cool.

PTGui
The classic stitching software. This is very useful for creating custom stitch templates for Mistika VR (to deal with custom camera geometry or difficult scenes). I use an older version (version 10) which is more compatible with some of the tools Andrew Hazelden have created.

Mistika VR
It’s important to have a tool that’s able to handle any camera input and quickly stitch together an image for viewing. Mistika VR is fast and flexible. Being able to also spot potential stitching issues on set can also help inform the team of any plates they may want to get as well as help get ahead of post production headaches before leaving the set.

DaVinci Resolve Studio
Since we’ve built our setup with a Blackmagic 12K camera, DaVinci Resolve Studio (and Fusion, which comes bundled with) are pretty good tools to have. Especially when paired with Andrew Hazelden’s KartaVR, which is a free set of tools designed to help folks work in spherical formats inside Fusion. KartaVR has an amazing tool that will generate STMaps from PTGui stitch files so you can use them in applications such as Touch Designer for realtime stitching.

Touch Designer
This is the application that creates all the real-time magic that allows us to unwrap the image and broadcast it to other sources. I’ve only scratched the surface of this amazing tool, but it’s worth it.

Have questions? Building your own cart? Want to work as a DIT in VR? Get in touch and let’s talk. matt@lightsailvr.com

Since the pandemic began, the team at Light Sail VR has been working from our homes. For the most part, it hasn’t been too challenging. We check in via Google Hangouts every morning and at the end of the day. We hop into VR around lunch time to touch base and play a short game. We upgraded our home computers and heavily utilized Google Drive to synchronize project files and media. We managed to deliver several high profile projects this way for our clients. This all seemed to be working well, until it wasn’t.

The problem of working this way is that it doesn’t scale to technically challenging and large projects. Especially given the poor upload speeds offered on most residential internet plans, there were several pain points that came to light in the middle of our last project for adidas.

So, after a year of working from home, and even with the thought of returning to an office soon, we decided to take the company completely remote for all of our post production work.

Why Go Remote?

Before I get into the how, let me answer why we decided to do this.

First, I’ve become accustomed to being around my family and it’s been great to be closer to my two young daughters. I feel like I’ve been more present in their lives than ever before. Spending long days and late nights in an office when crushing under a deadline was hard. Now, I feel like I can set up a render, check in with the family, etc.

Second, some of the folks we love working with have relocated outside of Los Angeles and having a dedicated remote work system means that we can now collaborate with them (and anybody else) no matter where they decide to live.

Our goals were clear. We needed a system that allows us to:

• Give editors and artists the ability to utilize our hardware that’s built to handle large format immersive workflows, including lots of fast shared storage and powerful graphics cards.
• Take advantage of our in-house render farm.
• Provide remote viewing support (including in headset) so creatives and stake holders and weigh in with notes in real time.
• Meets strict security standards.

I’m happy to say that we’ve accomplished all of those.

Our System

We elected to go with Teradici for our remote desktop solution. Teradici is the only company with PCoIP that provides a secure, encrypted tunnel to each machine and is capable of rendering the screens at high resolution and perfect color accuracy. This is critical for the kind of work we do. We’re in good company, as Pixar, Disney, Dreamworks, etc. have all used this system in the past year to deliver amazing content.

Setting it up required a bit of googling around Windows Servers, Ubuntu, VMs and Active Directory, but now we have the ability to create user accounts and assign them specific workstations on which to complete their work. These machines are securely located in our office, hooked up to our storage, and contain all the licenses for the software we use. It is a lot less work than setting up a remote editor with the tools and workflows (and sometimes hardware) to get the job done.

Furthermore, it means that we can send large files to be processed by our Render Farm. Before this solution, the Render Farm laid mostly dormant as it wasn’t practical to send large files across the internet. It also wasn’t practical to move the Render Farm to my house, or the editor’s house. For me, this is one of the best reasons to invest in a remote system like this.

Remotely accessing the computers works great. It honestly feels like I am sitting directly in front of the machine from the comfort of my home. However, confidence monitoring is an area that presented a challenge. How could we offer the experience of sitting in an edit bay with the editor and giving creative feedback quickly.

For this, we turned to Medialooks’ Video Transport along with the free NDI Tools available for Adobe Premiere Pro. Premiere Pro’s playback engine outputs an NDI stream of either the source or record monitor. Video Transport’s publisher takes that and pushes it to the web, where the editor can send a web link to stakeholders to review.

Reviewing immersive content presented another challenge as typically you have to have a headset hardwired to a PC. To solve this, we built a custom Unity app that I’ve published the source code for on my GitLab to take the feed received by Video Transport and display it in the headset. Currently it supports monoscopic and stereoscopic 360 videos. By connecting my Oculus Quest 2 via AirLink to my PC and launching this app, I can now review footage being edited on the timeline in real-time from the comfort of my home. The latency is extremely small and clarity quite good.

The Cost / Benefit

All of this technology doesn’t come cheap. Teradici is about $1250/year. Video Transport for up to 5 point to point connections is $1250/year as well. However, we’ve already invested a considerable amount of capital into the technology in our office and when you take a look at the time savings, security benefits, and vast expansion of the talent pool, I believe that these solutions will more than pay for themselves while providing an even better experience for clients that are increasingly more global.

For me personally, getting to create while still being available to my family has led to a better work/life balance and that’s worth everything to me.

I hope you’ve found this article helpful. Have you taken your company remote? What prevents you from doing so?

This is Part 2 of a two part series that chronicles how I built a workflow with this camera to capture and produce extremely high quality monoscopic 360 video. Here we’ll talk about the post production workflow I’ve been battle testing over the past few weeks, while part 1 talks in depth about the production and how the camera is to use on set. If you are considering purchasing or renting this system to use on a project, I strongly recommend reading both and understanding exactly what this camera is and is not capable of before diving in.

To follow along, check out this sample content. It’s quite large. You’re welcome:

Front: https://lightsail-public.s3-us-west-2.amazonaws.com/A001_01211618_C005.braw

Back:https://lightsail-public.s3-us-west-2.amazonaws.com/A001_01211621_C007.braw

This was shot at full 12K resolution at 60fps using Q0. The lens was set to f4.0. There was no IR filter installed, so I expect the colors to be quite red and difficult to nail. I’m looking forward to seeing how you decide to color it.

I cut my teeth in the industry designing post production workflows and learning how to manage jobs on a budget and deadline with a team of people. I am always looking for the fastest and most scalable way to produce content and that was my goal when battle testing these workflows.

Mistika Boutique is still my preferred post production tool for creating 360 media. Yes, it is expensive, but it saves me so much time that it easily pays for itself IMO. However, as of this writing, it can’t work with the native BRAW files from this camera, forcing a transcode into something in order to use it. That may not be as much of a deal breaker as originally thought, since utilizing BRAW in Resolve with the Fusion tab also seems to bring my workstation to its knees.

The DaVinci Resolve Post Production Workflow

Filming a project on a 360 camera without understanding the post production workflow is a recipe for disaster. I can’t tell you how many projects I’ve come across since beginning to work in this industry that have either failed to finish or weren’t able to finish in the way the creatives envisioned because of this fact. So, I knew that figuring out the post workflow for the camera would be a first priority before filming a commercial project with it.

TL:DR You need a serious machine to work with the best footage this camera can capture.

• At least an NVIDIA RTX Titan. You need the 24GB of VRAM. Anything less won’t do. You might be able to use two cards. The VRAM is the most important part.
• 128GB RAM is HIGHLY recommended. Without it, I was getting all sorts of errors while attempting to stitch in Resolve.
• FAST storage and LOTS of it. These files can be HUGE and also you want fast read/write times to be able to access the files quickly. Faster, the better.
• Ideally, access to a FAST shared storage server such as a Lumaforge Jellyfish and several other computers to help you render. More on this later in this article.

DaVinci Resolve Studio is what I used to work with the media we captured. It ships free with the purchase of the camera, but even without, it is only $295 for a perpetual license. That comes with a license to use Fusion Studio standalone with it’s unlimited render nodes. Having been used to the ridiculous pricing that is Nuke, that’s a screaming good deal any way you look at it. Resolve Studio supports the 12K BRAW codec, and its intuitive editing interface means you can complete your entire project in a single app.

Here are the tools you’ll need to complete this workflow:

• PTGui 10 (not the latest version)
• DaVinci Resolve Studio 17 — I am using the beta (7)
• KartaVR (available via Reactor)
• STMapper (also available via Reactor)

Interestingly, Fusion Studio does not support BRAW. But you can copy/paste the nodes between the two programs, which is pretty cool. For now, I suppose you could transcode to an image sequence format and send to a render farm, if that’s your thing. My experience tells me you might want to explore that, because while the following workflow is good, it is MIGHTY slow.

Ok, time to get to work.

Installing KartaVR

KartaVR is a massive set of FREE plugins and scripts for Fusion by Andrew Hazelden. Seriously, we all owe him a lot for making this workflow even usable. He’s a genius and an incredibly nice guy. Join the KartaVR Facebook group. Buy him a coffee.

You get KartaVR by first installing Reactor, available here. It couldn’t be easier. Drag and drop the .lua file you download into an empty fusion comp and let the magic happen. Once the download manager pops up, make sure you install KartaVR and the KartaVR 3rd Party Libraries.

You are also going to want to grab STMapper as it’s a vastly faster UV map positioning tool that take advantage of your GPU. You do have an RTX Titan, right? It’s available under Tools -> Warp

After you’ve installed, restart DaVinci Resolve Studio and let’s continue.

The Offline/Online Workflow

Working with this much footage at full resolution is a pain. After trying all sorts of transcodes and proxy workflows, I honestly think it might be best (depending on your project of course) to do a very rough edit with simply the fisheye lens. The reason I say this is that Blackmagic’s voodoo of working with the 12K BRAW footage natively is just so good that it allows you to start editing/syncing right away. This way you can roughly edit the usable footage and not waste time on frames you don’t need. Remember, at 60 frames/second even the 2 seconds it takes for the 2nd AC to get out of the shot with the slate can eat up a lot of processing power! Let the cut be a little heavy, you can always trim it later.

To do this, I would create a sequence for each take and sync my front and back cameras. Then I would edit my video using the take sequences as the media. Once I am happy with my rough assembly, I can simply “unpack” the compound clips to get all of my footage with my front camera on V1 and my back camera on V2 in a single sequence. Duplicate this sequence. It’s now time to stitch.

Transcoding

I really wanted to find a workflow that allowed me to work with the BRAW files natively, but to speed up rendering with this workflow, converting to frame sequence is really the way to go… provided you have enough fast disk space, which can get rather expensive.

When stitching in Resolve from the BRAW files, I would average .5 to 1fps. That’s pretty slow. After converting to frame sequences, that number was much higher. DPX files in particular are super fast to playback because the CPU doesn’t have to do much work. It’s all dependent on disk speed. But having that much disk space with high enough speeds can be hard. My best guess is that this workflow is best suited to work on a few shots at a time, which just means taking extra time to make sure the stitch and VFX are all final before moving on.

Before exporting out DPX files, make sure the white balance and exposure of your clips is optimal. Those can usually be changed in the raw image settings, so you’ll want to review (ideally with a preview LUT) before exporting to make sure you don’t need to adjust anything.

Also, be sure when exporting to check your timeline color space and gamma. If you are exporting a Rec709 timeline, you are going to be throwing away a ton of color information. If sticking with Resolve, I suggest setting up your timeline to use the DaVinci WG/Intermediate color space anyway.

PTGui

In order to use PTGui, we first have to generate still images from our footage. I’ve found that the fastest way to do this is the following:

1. Starting in DaVinci Resolve, make sure the sequence size is set to the full vertical resolution but a 2:1 aspect ratio. In my case, this would be 12960x6480.
2. Layout all my clips in a single sequence with the back camera angles on V1 and the front camera angles on V2 in a checkerboard fashion.
3. Head over to the Color page. Right click and select “Grab All Stills -> From Middle”. All of your stills should now appear in the gallery. By default they are named in the following format: [track].[shot number on track].[version — which is usually 1]. You can customize this format if you like in the preferences.
4. Select all the stills in the Gallery and select export. You now have a folder with still images from each setup in your edit.

The next step is to stitch using PTGui. I’m not going to get into the nitty gritty of how to do that here, but I will give a few pointers that seemed to work for me.

• The HAL250 lens on this camera has a focal length of 4.3mm, a 250 degree FOV and is a circular fisheye. The URSA Mini Pro sensor is the equivalent of a 1.4x crop.
• Make sure your crop circles are really accurate. It should be the same for each image, since it was on the same camera!
• After the first stitch, manually delete any control points on moving objects such as trees, sun shadows, etc. You may need to add some if you find the results not quite good enough.
• When optimizing, uncheck Optimize lens FOC. I’ve found that changing minimize lens distortion to heavy + lens shift improved my results.

When you are done in PTGui, your preview should look perfectly stitched. Save the file.

NOTE: I think it’s perfectly possible to do this workflow on a single shot to generate the UVMaps, but it might not be perfect for every shot if you have changing geometry (room size). I would do a single shot, use the following UVMap workflow on all the shots and then return to PTGui to hand stitch any shots that don’t look good enough.

Stitching in the Fusion Tab

Back in Resolve Studio, I return to my assembly sequence and duplicate it again. I then start converting the pairs of clips into Fusion Clips. It’s time to head into Fusion and stitch!

Before I do so, be sure to change the project resolution to your final output size such as 8192x4096. You can try working at a lower resolution too, but you’ll need to add some transform nodes in the graph and a control node to be able to easily switch resolutions since the UVMaps are a set size and seem to not be resized by Fusion for some reason.

Above is my node graph and I’m going to walk you through it. The key script I have to run before I can create this is Andrew’s “Generate UV Maps” script. This will take your PTGui project and create STMaps to unwrap the fisheye footage in Fusion using the STMapper node. You’ll want to use the following settings:

This will generate in PTGui two files that we’ll bring in to Resolve as our FrontUV and BackUV.

Plug the UVs into the yellow triangle on the STMapper node. Plug your video into the green triangle. Clicking on the merge node (I renamed mine to ‘Stitched’ using F2) and hitting the ‘2’ key loads it into the 2nd screen. Most likely one of the images is greatly overlapping the other, so I use a AlphaMaskErode node to smooth it out. That works for quick stitches, but for final stitches it’s better to manually paint your masks and plug them into the mask channel. Andrew Hazelden has a cool workflow that uses some sort of image analysis to draw the perfect seam line, but I haven’t quite gotten there. If anybody has tips, I’m all ears.

At this point it’s probably looking pretty good, but it’s hard to be sure in the distorted state. Let’s take a look at a preview!

You can preview in magic window style by clicking on the three dots -> 360 View -> Latlog. Use Shift + Right click to move the camera around. Or, if you have a VR headset hooked up to your PC (such as my Quest 2 running Oculus Link), hit the 3 key and it’ll boot up a preview on the headset. VERY cool. This is a feature that Mistika Boutique doesn’t have and I love it.

You should check the stitch to make sure it’s perfect. If it isn’t, you can add a GridWarp node after the UV, before the STMapper node and nudge stuff into place.

Now it’s time to paint the tripod out. I use the LatLongPatch tool and then Fusion’s excellent paint node. Honestly, it feels very quick and easy. I then merge that back into my footage.

Another approach is to render out a still frame, paint in Photoshop or Affinity Photo, and composite back in the still image. Your call here.

Overall, I found the process pretty straight forward and it was extremely powerful to accomplish all of this in a single app.

The “Gotcha”

Speed. Simple as that. This is really slow with render speeds anywhere from 1 frame/second to 6 seconds per frame. Yikes. If working inside DaVinci Resolve, there isn’t really any way to speed that up either.

If you convert to DPX frames as suggested above, you can load your media onto super fast NVME drives for a huge increase in performance. Each DPX frame is about 400MB (that’s megaBYTES) which means ideally you need a PCIe 4.0 M.2 drive to achieve close to realtime speeds. And even then, 8TB will only net you about 6 minutes of footage and cost upwards of $1700 to put together. Perhaps compressed EXR formats would fare better, but I haven’t had the time to test.

I also always got better stitching results using Mistika Boutique and I’m not sure why. There are a couple of workflow speed improvements in Mistika that make stitching happen in quite a few less clicks. Rendering speeds were still on the slower side (.5fps) but more consistent. However, I much prefer painting and color grading in Resolve. Plus the addition of headset preview is really nice.

Conclusion

This is a workflow I’m going to continue to explore. I’m sad Blackmagic doesn’t make it easier or provide as many quick tools to work with immersive media vs. Adobe Premiere because the underlying technology of Resolve is pretty impressive. While I complain that it’s slow in Resolve, this project wouldn’t even be usable in Premiere Pro. I know the number of us doing immersive 360 projects is still small, but fingers crossed it eventually makes it onto Blackmagic Design’s dev team list.

What would you like to see? How could you improve upon my workflow? Download the sample footage and drop me a line: matt@lightsailvr.com or follow me on Instagram @immersive_matt

Matthew Celia is the Co-founder and Creative Director at Light Sail VR, a commercial production company specializing in immersive media. You can follow him on Instagram @immersive_matt

I’ve been working in immersive media for the past five years having directed and produced over a hundred individual pieces of content for top agencies, brands, and clients. I’m a firm believer in having the right camera for the job and I know that one camera won’t always work for everything, but this year I decided to take the plunge and invest in a camera of my own to continue to tinker and build new workflows with.

Why I Chose This Camera

In October 2020, I directed a series of videos that we shot with a Sony Venice and Entaniya HAL 250 lens. Throughout the post production process I really enjoyed the speed and ease at which we were working, as well as the tremendous quality of the image and colors captured by that camera. After spending a few years working with stereoscopic footage, I found that I didn’t really miss the 3D depth, which was always difficult to get right anyway and often sacrificed a lot of sharpness. Furthermore, being able to render such high dynamic range and beautiful colors left me with an image that felt so cinematic and beautiful to look at.

When Blackmagic Design announced their 12K camera, I felt like this would be a good match for this style of filming. Paired with the HAL 250 4.3mm, it should offer the full 250 degrees FOV horizontally and about 205 degrees vertically. This means I should be able to get a 11.5K 360 stitch at 60 frames per second. All of this at a fairly reasonable price point and a camera body with the flexibility of doing dome work and virtual production to boot. That was exciting to me.

I set about building out a rig in addition to the camera and lens. Here are the components and why I chose them.

When filming using a nodal setup, I needed a way to slide the camera back far enough so it would rotate around the nodal point. I also need this to be easy to adjust in small increments because even 1mm off can mean a bad stitch. It’s also a good idea to try and raise the camera up so the patch job of the tripod is easier. Because the camera rig weighed in at about 12lbs and had to hang off the back, I also needed something sturdy that wouldn’t flex too much. I quickly found out that I could spend more on a tripod than I did the camera body! After some research, I found the Arca Swiss plates are a good budget interchangable system that’s compatible with a lot of stuff. I picked up two long Arca Swiss clamps and used some standoff risers I had in my kit to attach them to each other and give it some height. With an Arca Swiss plate on the camera, I could attach it to the Nodal Ninja 6 I already had in my kit. The Nodal Ninja 6 is fantastic. It’s well built and has a bubble level and a nub that can be used to lock in the nodal point.

Sliding the camera back puts a lot of pressure on the mount, so it’s smart to have the least amount of weight as possible. I already owned a few V Mount batteries, so I bought some clamps to hang them on the tripod legs. I also mount my Atomos Shogun 7" monitor. Having an external monitor is really important for focus as well as a focus chart.

The one drawback with my Shogun I can see is that Atomos seems unwilling to add Blackmagic profiles to its monitors and the color accuracy of the monitor to the LCD panel on the camera is quite different. This is something I’m looking into further.

Lastly, I decided instead of CFast cards, I would record directly to SSDs. I think this makes the post workflow incredibly fast as I can simply plug in the drive and get to work on the road. SMALLRIG makes a pretty solid SSD mount and so far this system has been great. Use whatever you like, just be sure to double check Blackmagic’s qualified media list to be sure the media is fast enough to record at the settings you need.

Production

We went out to film some test footage at Vasquez Rocks here in Southern California. Being a nodal rig, this camera is not good for every kind of filming. If I wanted to film crowds and large amounts of moving objects in an uncontrolled environment, it’s just going to be difficult. It also is a fairly large rig with a lot of components. A single operator is not going to be able to run and gun this rig around and get great results. It demands precision and expertise. That’s ok! The results that can be achieved in my opinion are far worth the effort. Usually I’d use this rig in studio shoots, so filming outdoors was definitely a challenge.

Some key tips to keep in mind when filming with this rig:

• Define where your stitch lines are going to be. It’s less about seeing a stitch line, because by filming nodal there is no parallax to deal with, but rather more about what the action of the scene is and what is moving along those stitch lines. Be aware of anything moving!
• Always check critical focus.
• I think the sweet spot of sharpness and depth of field with this lens is f5.6.
• Make sure the camera is level when flipping around. Otherwise you could be introducing another axis of potential parallax.

Some “Gotchas”

After I had a chance to review the footage in post, a few downsides became obvious.

First, in order to get the lens to mount, I had to remove the built in IR filter. This is easy to do, but I noticed a lot of IR pollution when filming outdoors in the sunlight. To solve this, I found this set of filters that I was able to attach to the rear element of the lens. I’m not sure it’s 100% effective, but it is quite a bit better.

Second, flares can be a huge issue. With such an enormous lens, unless you desire flares, I think it’s a smart idea to flag the flare off the lens and shoot some plates that will composite together in post.

Third, the camera mount I was using was fine, but felt a bit wobbly. I’ve ordered some new parts and will update this post once I find a more rock solid solution. Either way, I’m not paying for this.

Lastly, the post production workflow took some serious time to nail due to the fact that only Resolve Studio was able to work with the BRAW files natively. It also requires a hefty computer. But more on this in the next article…

Conclusion

I’m really happy. I know as I continue to work with it, I’ll be able to really dial in. There are a ton of small things that continue to impress me. I love having SDI connections. I love that you can trigger and set all the camera settings via Bluetooth. I love that I can jam timecode and make syncing sound a lot easier. I also love that I can swap out lenses and use it for 2D work. But most of all, I love the quality of the image and how much more cinematic our content shot with this setup will look.

And now, since I know everybody was asking, here are two sample clips in BRAW of a short scene we shot.

Front: https://lightsail-public.s3-us-west-2.amazonaws.com/A001_01211618_C005.braw

Back:https://lightsail-public.s3-us-west-2.amazonaws.com/A001_01211621_C007.braw

This was shot at full 12K resolution at 60fps using Q0. The lens was set to f4.0. There was no IR filter installed, so I expect the colors to be quite red and difficult to nail. I’d love to know your thoughts and how coloring it goes. Here is a sneak peek of what I accomplished by pushing the grade pretty far.

Matthew Celia is the Co-founder and Creative Director at Light Sail VR, a commercial production company specializing in immersive media. You can follow him on Instagram @immersive_matt

Last December, Light Sail VR shot a music video for the incredibly talented Jacob Collier and the first single of his new album Djesse Vol. 2. It was an incredibly rewarding experience and we’re very proud of the finished product. Jacob’s creativity and collaboration is so vibrant throughout the piece and he hid so many fun references for his fans to explore.

While initially this was supposed to only be released as a 360 video, there were concerns about how sometimes people watching 360 videos tend to feel like there is too much to take in and how the story can get lost. We were asked to make a 2D version. The request made a lot of sense. After all, headset adoption is not incredibly high and while engagement with 360 videos is, the view count is often lower than the averages for a 2D film. Personally, I find myself less apt to watch any 360 video unless it is in a headset. The whole moving the phone around concept never appealed to me.

We knew we could do a simple reprojection of the 360 video and call it a day, but we decided since this was going to be the version that probably garnered the biggest audience, we needed to create something more creative. Below, you can watch both the 360 and 2D versions of the music video. These were both created from the same 8K finished lat long master file.

I think what excited me most about this project was the fact that it was an excellent example of shooting a 360 video with a single high end 2D camera and then finishing both the 360 and a reprojected 2D version. It’s something we want more clients and collaborators to embrace. The audience reaction has been very positive.

So without further ado, read on to find out a bit of the behind the scenes magic.

TECHNICAL CONSIDERATIONS

When Jacob and his team came to us wanting to make a music video, one of the first requests they had was that they wanted it to feel cinematic. They had tested the YiHalo and weren’t happy with the results. We love the YiHalo, but know exactly how they felt. There is no comparison between a collection of small cheap action cameras shooting 8-bit 4:2:0 highly compressed H264 and a RED Helium that shoots in raw with 16.5+ stops of dynamic range. It’s night and day in terms of cinematic quality.

Since the creative concept of the music video already had us duplicating several copies of Jacob playing instruments and we were filming in a controlled environment, we knew we could utilize a technique called “nodal capture” to create our 360 image from a single standard 2D camera.

This allows us to stitch each camera angle together with minimum parallax between each quadrant. It’s a clever technique that when done right can still deliver the feeling of having the entire 360 sphere captured at once. The added benefit is that we have a very high quality image that looks cinematic.

Jacob was going to move around the space quite a bit and we had large flats to contend with so we paired our RED Helium camera with the Entinaya 250 degree fisheye lens. We then still filmed in four directions (north, east, south, west) and that gave us a ton of overlap for compositing.

Shooting nodal comes with the trade off of needing to have a very detailed plan of attack. For that, we turned to Unity where we were able to mock up and pre-viz all the blocking. It was incredibly important for us to figure out the blocking ahead of time so we knew that it would all stitch together seamlessly when we merged the camera angles in the edit.

The shoot consisted of over 35 individual shots that we stitched and merged together to create the seamless 8K final video.

Creatively we wanted to embrace the sense of childlike wonder that I remember from my days in grade school theater. Michel Gondry’s The Science of Sleep was a touchstone reference for us as well as Harold and the Purple Crayon and Where the Wild Things Are. We very much wanted the feeling to be magical and nostalgic.

More importantly, we really wanted mostly everything to be done practically. All the sets were moved by hand to create the transitions between scenes and a “hero” Jacob singing the main vocals would journey around the 360 space, helping guide us through the story.

POST PRODUCTION INTO 2D

With all the compositing, stitching, and rotoscope work of the 360 video completed, we ended up with an 8K file. Now, this is very rough math, but seeing as the FOV of 360 video players is somewhere in the 70–90 degree range, I estimated that an 8K file would give me the ability to generate a 1080p HD 2D version.

We utilized Re:VisionFX Re:Lens software in Adobe Premiere Pro in order to convert the video from lat long projection to either rectilinear or tiny planet. It’s actually really fun to use because it feels like you are directing a 2D camera with a zoom lens that’s shot from the center of the 360 world. I found that I could zoom in, pan, tilt, and animate all the keyframes to feel smooth and follow the story.

The hardest part was figuring out what we wanted to feature in the 2D cut. Jacob and I sat together as we cherry picked our favorite moments that we felt told the best story and worked on framing for timing and flow.

THAT’S A WRAP

For me, 360 video proves again to be an endlessly creative medium of expression to try new things both on a technical and story level. Taking this piece into 2D offered us a chance to experiment in cool ways we haven’t thought of before. What do you think? Success? Failure? Either way, let’s keep pushing this new storytelling medium forward.

- Matt Celia | Light Sail VR

VR180 is an interesting new format in the world of immersive media, finding itself placed snugly between the traditional 2D video format we are all used to, and the 360º degree video format that can be more daunting. With that in mind, we set out to discover how exactly VR180 technology works, how it can be used to tell strong narratives in different POVs, how a desired effect can be achieved, and how to work around problems that may exist in this format.

We discovered that VR180 can work well in 1st-person; and that traditional 2D filming techniques to move the field of view, such as pans, tilts, and dollys, can work well when planned out. However, this technology requires creative thinking and may require multiple attempts to get a desired solution. The following is an in-depth look at the reasoning behind our decisions and some conclusions we have drawn.

MOVING POV IN 180

One of the first things we should cover is why POV in 180º makes sense. 180º is as close as we can currently get to emulating the average human viewpoint (the average human has a 210-degree wide field of view), but this field of view is also extremely close to the angle range we see in traditional 2D formats. This allows a creator to make more direct narrative decisions relating to perspective, opening up a much wider range of opportunities when filming. However, one must be careful as VR180 also isn’t a perfect emulation of actual human POV, and this can create a feeling of surrealness. The challenge of VR180 POV is that people don’t live a sedentary lifestyle: We are constantly moving, turning our heads, looking around, etc. Our perception of the things around us is not limited by a creative medium, so translating our own perspective to VR requires some thought.

After our tests, as you’ll see below, there are a few things we have learned that can drastically improve the audience experience and avoid nausea and discomfort.

1. Keep your camera pans less than 90º
2. Keep subject central in your 180º view
3. Add in dip to black fades for transitions: coined as “blinks” by Matthew Celia. Keep transitions anywhere from 6 to 10 frames depending on the speed of motion.
4. Only show motion on one plane at a time, i.e., isolate pans and directional movements
5. Use body mounts and post stabilization if movement is essential to story
6. Be careful of adjusting the camera height mid-shot, as it disorients perspective

By focusing on the 1st-person POV, we wanted to better understand new techniques that may arise from filming in this medium. We also wanted to experiment with as many different types of shots as possible to get a full range of scenarios to work with in post.

The first video was shot by Light Sail VR’s Interns, Iggy Cossman and Nick Reed. They shot this from the first-person POV of Iggy, while Nick showed him around the office. The big problems with this video were that the camera was filmed completely handheld without any stabilization added in post, and was limited to one continuous take from start to finish.

The second video, shot by Cody Pomeroy and Nick Reed, was from a similar first-person POV but filmed on top of a tripod. While filming on a tripod does take away from the immersion of continuous motion, this is rectified by the fact that less motion makes for a smaller risk of motion-sickness, and that movement can be simulated by using dip to black fades for a few frames (6–10 in our testing), which makes the viewer feel as though they are blinking while viewing the video.

To show the comparison, here are some side-by-side GIFs from both videos that demonstrate what video elements we found worked well for the viewer, as well as ones that resulted in an uncomfortable viewing experience. For these comparisons, we will be focusing on the biggest issues that we faced in production, as well as the types of shots that we believe will be the most commonly used in this format. For the most part, Video Two was made in an attempt to fix the problems that we had in Video One, as you will see in the following GIFs.

Comparison 1: Horizontal Motion (Forward)

Notice the difference between Video One and Two. While Video One does provide continuous motion, this is unnecessary. Video Two gets the viewer where they need to be faster while also conveying movement and alleviating any uneasiness that comes from unstabilized motion. Unless the creator needs a narrative reason to dolly from one spot to another, the second option is by far the best choice. Unlike traditional filming, static shots are almost always the way to go in terms of visual appeal for the viewer in VR180. We found that similar to 2D video, one can go straight to the next significant action without losing continuity or immersion in the story. It is best to execute this in a way that is not disruptive for VR180, such as the quick fade method used in Video Two, that is consistent and keeps your audience grounded in the narrative.

Comparison 2: Camera Pans

In this comparison, Video One and Two both require camera pans to get a better look at the kitchen area. Video One doesn’t work because the pan will likely not match up with the direction the viewer is currently looking, resulting in immediate discomfort and breaking the immersion. Video Two still has a slight motion (less than 90º), but by dipping to black and creating a sense of “blinking”, we are helping to cue the viewer that we are moving while not causing any discomfort to them. This is a great way to still get motion in this video format without sacrificing how your audience may feel or react to the shot.

Comparison 3: Extreme Angle Changes

Videos One and Two here present a major problem that someone might face in creating 180º videos: How do you effectively move in extreme angles in a format that isn’t traditional 2D (thereby using smart cuts to fix the issue) and that also isn’t 360º (thereby completely foregoing that issue)? Video One would require extreme planning to pull off effectively, whereas Video Two circumvents the rule by shifting from one location to another in a speedy process that mimics traditional 2D camera work. Notice also how we stay consistent with the 180º rule in this format; the actor exits and enters from left-of-frame.

Comparison 4: Vertical Tilt

Here we are presented with the challenge of vertical motion. While this likely isn’t a shot that will happen often, it still exists. Video One isn’t smooth, because the camera is attached to an actual actor. It creates a motion that the viewer would rarely mimic inside a viewing headset. Surprisingly, despite this, we did find that a vertical tilt — while seated in a static position — works well in a first person POV. This may be attributed to the fact that the average viewer usually only looks left and right in a headset as opposed to up and down. In Video Two, the motion still exists without discomfort, but can be a confusing shift in both X and Y-coordinates for the viewer. While both options have pros and cons, it appears that a middle ground may yet exist that takes the best of both options.

Take a look for yourself by clicking the links to both of our videos below. Make sure to view the experiences in stereoscopic 3D in your own headsets and please share your thoughts in the comment section to start a larger conversation about this exciting technology!

https://www.youtube.com/watch?v=jWUQtthW5SE

https://www.youtube.com/watch?v=3YSnZlBFfvM\

Conclusion

There are always unseen challenges that arise when working with new technology, and VR180 technology is no exception to this rule. Within just a simple two-minute video, we encountered multiple problems and had to devise creative solutions to fix them. We quickly figured out that a body rig on an actor would simply not work in most scenarios and that this issue could be fixed by using a tripod and selectively shooting movement as necessary. There are still plenty of other scenarios that we will face in future endeavors: How do wide shots work? Inserts? The 180º Rule? The list goes on and on as we work with a new format and style of filmmaking. VR180 certainly has plenty of uses and tools in its arsenal, and it will be interesting to see how other creators will use this technology in their own way.

We’ve recently launched a new show called “Now Your Turn” on Geek & Sundry that was filmed using Google’s new VR180 format.

When VR180 was announced, it was met with a fair amount of skepticism from the 360 video community. This article won’t debate the merits of VR180 except to say that I don’t think the existence of VR180 takes anything away from 360 video. For reasons I will get into, I believe they are different mediums with different creative uses and audiences. So let’s move on.

This was the first time Light Sail VR had worked with the VR180 cameras and workflow, so naturally there was a lot to learn. Here are my tips and tricks, hoping it will help you on your VR180 production.

THE IDEA OF “Now Your Turn”

“Now Your Turn” was born out of some testing we did over a year ago at Geek & Sundry. My business partner Robert and I thought it would be exceptionally cool to not only learn how to play a board game, but to actually feel like you are sitting at the table playing with the funny, amazing talent on their channel. For me personally, I always enjoy learning a game by sitting with my friends and watching them play.

When we heard of Google’s VR180 format, we thought it would be a perfect fit for three reasons:

1. Sitting at a table gives you a motivated reason to look forward and never behind you.
2. The financial costs of working with VR180 are less due to no longer needing to do any time consuming compositing work such as removing the tripod and lights, etc.
3. The improved resolution and excellent 3D of VR180 would make the experience more immersive for our audience.

Unlike 360 video, this wasn’t storytelling through place. This was all about the people and the board game in front of you. It made perfect sense for this new medium so we brought the idea to Google. Google was eager to work with us and was really excited about the project.

THAT DAMN CROP

One of the most interesting (read: challenging/frustrating) aspects of the VR180 format is the fact that outside of VR mode, YouTube basically crops the center part of the image and you are stuck with it. There is no reframing in post and worst of all (as we discovered) no guides.

Robert and I did our first test with the camera and were shocked to find out how underwhelming the experience is on a browser in 2D. Even though I wished VR headsets were available to everyone, I know that the majority of people will not be experiencing my content in a VR headset, despite the fact that it’s MUCH better when viewed in one.

We needed a solution.

I took my equirectangular output and brought it into Photoshop. I first used Photoshop’s spherical view to unwrap the image and the FOV to approximate the same view as I was seeing in the YouTube player. Using the exact science of “eyeballing”, I drew a square box that covered what I could see in the crop on YouTube. This told me what would be visible. I ended up inverting it and creating a transparent PNG that I could overlay on top of the ZCam Controller app on my PC. I used Custom Desktop Logo to accomplish this.

Having the title safe crop was incredibly important to the show. We needed to make sure game elements didn’t fall outside the view. We also wanted a way to make sure our talent was framed correctly.

You are free to make your own, but in case you want to use ours, here it is: https://drive.google.com/file/d/1aTYoHMe6-0JURTlK5by0pKAkJT4Stfzv/view?usp=sharing

I understand the reasoning behind the crop. A lot of people watching 360 videos (myself included) on the phone are annoyed that the image moves around, or maybe it starts accidentally when we’re looking at our feet and we see only black. The static crop ensures that the center view is always visible.

What’s difficult is that post-manipulating what’s presented inside the crop is currently not possible and we don’t have easy-to-use guide zones on set to help us when capturing content. Also, creative decisions that work great in VR180 don’t always translate well to the center crop and this means certain story elements can be left out. We had to work extra hard on “Now Your Turn” to ensure that graphic elements made up for the fact that the audience couldn’t see their own hands outside of VR.

Creatively, it is very restricting to have to design one video file for two very different ways of consuming the content. I hope other solutions present themselves going forward.

Otherwise, working in VR180 was very much the same as working in 360. I decided we would always work with the final resolution media and utilize proxies since that was the workflow we were already used to with cameras like the Odyssey and the Halo.

The advantage of working with 180 footage is that since we didn’t have to worry about a seam line, we could utilize more of our traditional effects like those from Boris FX.

IT’S BETTER IN 180

I’m really proud of Light Sail VR’s first VR180 show. It was a creative challenge and ultimately I find the episodes fun to watch and the format feels organic to VR. While they stand on their own viewed on a desktop, they really do shine in a cardboard or Daydream View headset. Looking down at crabs hands always makes me laugh!

Some people say that VR180 is a cop out. That we shouldn’t embrace it only because it’s easier. I agree with that last point. Don’t embrace VR180 because it is easier. Embrace it because you find something creatively rewarding about it. Working with VR180 footage was technically easier, which left us more time to be creative rather than fighting the tech. I think this made for better content in the short time frame in which we had to deliver. I think “Now Your Turn” is a show built for VR180 and makes good use of the medium’s strengths.

Go check out the episodes at Geek & Sundry and let us know what you think in the comments! New episodes will be dropping every Friday between now and International Tabletop Day on April 28th!

If you are just catching up with us, I highly recommend checking out the first two parts about the planning and filming of Speak of the Devil.

• Part 1: Designing a Mesh Narrative
• Part 2: It Takes A Village

And then, be sure to watch our trailer!

We had an incredibly ambitious goal of having something out by Halloween. Let me be the first to admit that the timeline in our heads was ridiculous and naive. We frankly had no idea how much work we had ahead of us. Obviously we missed that deadline by several months, the reasons for why I will cover in a future article.

This part gets a little technical, because creating VR content IS really technical. The tools are very new or non-existent. Workflows are barely established and we don’t have a 100 years of history helping guide decisions. In short, asking the internet a question rarely turns up the correct answer because there are few to none attempting any kind of project on this scale. To my knowledge, Speak of the Devil is the first truly large scale mesh narrative in VR. If you know of others, please share!

We had to change a lot of our workflow and constantly adapt to changing deadlines and technical hurdles.

A WHOLE LOT OF DATA

For Speak of the Devil, we were lucky enough to be in the Google Jump Start program which gave us access to the camera, but also Google’s Jump Cloud stitching. There are some very smart people at Google who have designed one of the best automatic stitchers out there.

We had shot approximately 6TB of footage that needed to be uploaded to the cloud for stitching. That’s a ton of data. Thankfully, we were able to head to YouTube Space LA and utilize their gigabit upload pipe (thanks Libor!). It took about 24 hours to send everything up to the cloud. Amazingly, we started receiving notifications that the stitches were completed and ready for download between 12–36 hours after those shots were uploaded. Every single shot from the over 7.5 hours of material was ready in two days. That’s incredible and a huge reason why using the Google Jump camera is so popular. Of course, it took us another 24 hours to download!

From the cloud, you receive an H.264 over-under file with dimensions of 5760x5760 running at 59.94fps and a 4:2:2 color space. Adobe Premiere Pro does not like these files… at all.

We needed to transcode all of our footage right away into something that Premiere Pro would have a much easier time working with. We elected to create our own proxy files as DNxHR LQ 8bit files at a resolution of 2048x2048. I think upon reflection, I would have generated higher quality files since we weren’t able to see the small details in the forest, which meant that we didn’t catch some errant crew members until way late in the workflow. Talking with the Jump engineering team, we also figured out later that you could have Jump create the proxies and use Adobe Premiere’s proxy feature to link up to them.

Aside from the editorial proxies, we also needed to have all used footage transcoded into a high quality frame sequences for work in Nuke where we would do our top and bottom stereo compositing. We had planned on using DPX files since our Mistika monoscopic VR pipeline uses them and we find it to be rock solid, however a quick calculation of our running media, the number of versions and intermediates we would need, and some headroom for mistakes in our workflow (which we knew we would at some point) led us to believe we were going to need over 100TB of storage. Holy hell!

Enter Lumaforge to save the day. We already use their 30TB Jellyfish Mobile as our main shared storage in our office. We needed them to help us with something much bigger. What I love about Lumaforge is that it is a company founded by indie filmmakers just like us. They understand the grit and hard work it takes to create something that pushes the envelope. They also know how to make awesome tools that made our lives easier.

We hooked up all three of our workstations to a 125TB tower they brought over. This meant we could be editing on one machine, organizing on another, and exporting on a third. Or at the end of the day, we could queue up renders on all three machines and have a chance of being finished. The time savings were enormous as well as being able to work from drives running upwards of 800mb/s read and write.

However, we quickly discovered that not only would we have needed a tremendous amount of space, we would have needed an even faster pipe than 10GbE could offer. We decided after a few scenes that DPX just wasn’t going to offer any advantages and we switched to Zip16 OpenEXR. There were so many formats to consider, but this is a good standard offering much smaller file sizes and rock solid workflows with Nuke, which would be a major part of our pipeline. Even with the switch to a more compressed format, our final project ended up taking about 110TB of space.

With our house in order, it was time to get down to tackling all these scenes.

EDITING

We brought in Chris Willett as our lead editor for this project. We’ve collaborated a bunch before and not only is he extremely organized, he’s also always game for creating something different.

We started by simply creating all the empty spots in the forest, choosing the best 15 or so seconds to create an endless loop. It became obvious very quickly just how many locations and shots there were going to be and we needed a way to track progress.

I wrote a Medium article about using Kanbanchi, which is a service very similar to Trello, but with a deeper Google Drive integration. Go read that here. The TL:DR is that Kanbanchi proved invaluable to tracking shots through our whole pipeline (and notes) as well as collaborating with VFX and sound.

With so many shots to track, Chris and I settled on a pipeline to make sure everything went as smooth as possible. Every location had a folder that contained a sequence with every component (equirectangular, plate shots, etc) broken out and synchronized. Most of the plate elements were turned off and the sequence was then nested into an edit sequence where we could add dissolves and color effects for the deaths. We had to make sure that our cutscenes transitioned seamlessly into the loop scenes.

What became extremely obvious was that it was very hard to tell whether the scenes were working for two reasons:

1. Final audio was going to play an enourmous role in successfuly creating the mood
2. We were unable to really experience the narrative until we put these scenes into a game engine and worked out those bugs.

This meant that the quality control process we were so used to was turned upside down and we were going to have to integrate the game engine programming side of things as quickly as possible in order to confirm that everything fit together and played how we envisioned it when writing. Having never done a project like this, it was hard to understand.

12 WAYS TO DIE

The death scenes were the most fun and also some of the more challenging creative choices we had to make. With 12 unique deaths, creating enough variation to stay interesting was tough.

Some of the deaths we captured on set were falling flat in the edit bay. Frankly, having the demon yell at you, wasn’t really enough to incite fear, so we needed to amp it up. Chris and I played around with timing and sound. We even had a good opportunity to crack open our copy of Mettle’s Mantra VR.

My favorite deaths are the two scenes where Lindsey and Brian vomit flies into your face. For this effect, we turned to Josiah Reeves from Occlusion VFX to help model the flies in 3D and composite them into the scene.

It was by pure coincidence that our 12 deaths plus one way out equalled 13. We thought it a fitting number for our project.

HORROR FILMS ARE ABOUT WHAT YOU HEAR

In a horror film, sound design is one of the most important elements in helping build tension. If you ever feel like you are getting too scared, don’t close your eyes, turn off the sound!

For Speak of the Devil, we reunited with Eric Wegener who worked with us on our first 360 horror project, Paranormal Activity: The Ghost Dimension. I’ll let him explain his creative process:

I needed to constantly balance between the creative, design side and the literal, technical side of the soundscape. The amount of material needed was staggering, with 50+ unique locations in the forest, each one involving multiple ambient looping backgrounds and cutscenes. So the big challenge in that regard was to make each spot unique and interesting, but to make the whole forest feel real, cohesive and connected.

One way we did this was by enabling the player to hear distant sounds from adjacent locations, which not only makes for a creepy and realistic spatial effect (i.e. unseen characters yelling for help nearby), but draws the player toward certain locations in order to advance the story.

Once we established the realistic, literal sounds of the forest, we engineered methods to add more spooky, abstract layers on top, to try and mess with the player’s head and freak them out. These sounds are mostly made up of creepy tonal elements, whispers, breaths, laughing, crying, satanic chants, etc. They are triggered either as “looping beds”, once the player experiences certain events in the story, or as “random one-shots”, which are totally unpredictable and can happen at any time.

This combination proved extremely effective at steadily ramping up the feeling of isolation and dread as you progress, while the added randomization really keeps you on edge from start to finish, and across multiple playthroughs.

We loved working with Eric throughout this process and think that the immersive sound is one of my favorite parts of the whole narrative.

FINISHING THE WORKFLOW

We love shooting on the GoPro Odyssey, but one thing we don’t like is that it doesn’t capture the top and bottom of the sphere. Robert and I were going for maximum immersion and looking down or up and seeing a blank area just wasn’t going to cut it. That meant we needed to take our plate shots of the zenith and nadir and composite them back into 3D space. This is not an easy task. Especially for the number of shots we had.

We knew from our experience working with the Google Odyssey on Immerse that we’d have to utilize Nuke to get the highest quality results.

We worked with Jeremy Vanneman from Treehouse Studios as our stereoscopic supervisor. We wanted to get started as soon as possible but since we were still in the process of editorial it wasn’t feasible to send over a giant drive with all the footage. Especially given how large of a drive we would need to send.

Instead what we did was create cards in Kanbanchi and attached a still frame of the equirectangular along with still of the top and bottom plates. Jeremy would composite those together using Nuke and then send us back the Nuke script. We would then reconnect to the high resolution frame sequences and render. I think our shortest render was 8 hours and our longest render something like 75 hours.

Halfway through the finishing of the VFX, the Google Jump team released a new feature called ‘Sweet Spot’ that allowed us to send scenes back to the cloud to be restitched if they had weird optical flow artifacts. Since we shot this film in a forest, we had tons of artifacts in our footage. In fact, if you take a look at the scene in our trailer where the Cultist character is offering you the bloody deer skull and that same scene in the final narrative, you’ll find that the version in the narrative is way improved due to us reprocessing it with Sweet Spot. We couldn’t get to all the scenes, but we were able to use this great new feature on several to help improve the quality dramatically.

Our render created a separate EXR frame sequence for the left and right eye ready for our color grade. When we began this project, we had planned on completing our color grade in Davinci Resolve since I already knew the software so well. Speak of the Devil left us with an interesting problem.

You see, to maintain the accuracy of the geography in the narrative, we shot every single setup with camera 9 (the front camera of the Odyssey) facing true north as told by a cheap magnetic compass we strapped to the top of the rig. This meant that there is a lot of action that takes place due south (or directly on the seam line).

Davinci Resolve doesn’t understand 360 video. That means if I were to do a power window across the seam, the risk of it leaving a seam line would be very high and since I couldn’t even check the grade in a headset, I wouldn’t know until another multiple hour render. That was time we just didn’t have. Again, I was going to have to modify our workflow.

Fortunately, the folks at Assimilate have created a tremendously powerful post production tool that’s fully ready for 360 video with SCRATCH VR. Not only does Scratch VR give us the ability to use windows and masks over the seam line, but it also has full headset support for preview.

I colored Speak of the Devil using a Windows Mixed Reality headset. The ability to flip up, adjust settings, then flip down and check what I had done immediately was a huge timesaver. It did mean learning an entire new program and I made a lot of mistakes that required me to redo and rerender a ton of my work. Still, I’m really happy with the look of the final.

ARE WE DONE?

We were in post production from the end of September until the middle of January. To us, who usually work in the branded content space on short five minute commercials, it felt like an eternity, but several people told us that it was actually really fast for the size and scope of project we were attempting to pull off. We had to modify our techniques and workflow in the post production process several times, which was frustrating, but forced us to learn a lot and develop a plan to be able to replicate this style of media across future stories.

Still, we weren’t finished yet. As we were completing all the final VFX, sound, and color, we were also hard at work with Wemersive and The Oberon Group figuring out how to program an entire logic engine and stream it all to your mobile phone.

Pick up the penultimate chapter next week in PART 4: It’s not a movie and it’s not a game, detailing how we used the Unity game engine to create the interactivity and logic engine of the narrative.

Download Speak of the Devil NOW for Google Daydream and Gear VR

Coming soon to Oculus Rift, and HTC Vive.

A Light Sail VR original. Speak of the Devil was filmed on the GoPro Odyssey as partners in the Google Jump Start Program. Media storage was sponsored by Lumaforge. Hosting and Unity Development in collaboration with Wemersive and The Oberon Group.

Production is my favorite part of creating 360 live action VR. There is such a great energy being surrounded by so many creative people, and although the days can be grueling, seeing so many people come together to execute a vision is one of the best feelings in the world.

Below I’m going to talk in depth about a few topics I feel are especially relevant to the 360/VR community. I’ll talk about using art direction to help create a world, how we worked with talented actors to achieve a believable performance, and shooting on the GoPro Odyssey thanks to our participation in Google’s Jump Start program. Film production in VR is much the same as it is with normal movies/TV shows/commercials. It takes a dedicated team of people to create it. A village, so to speak.

If you aren’t familiar with Speak of the Devil, watch our trailer below.

CREATING THE WORLD

In my last article, I briefly touched upon our scout and the 360 photos that we took of each location. Each location was given a scene number. These numbers became the bedrock of organization for the entire process and a great way to communicate with members of our creative team.

In creating 360 content, we’ve learned that art direction is a hugely important piece of building the world, perhaps even more so than 2D entertainment. After all, we’re communicating story not only with dialog, but with location and space. For Speak Of The Devil, we knew we were going to heavily lean on our art department to help make our forest look creepy and plant several Easter eggs for a dedicated audience member to find. Not to mention, we were trying to create a scary horror project that was shot in the middle of the day! I struggled to find references for horror films that took place in broad daylight and felt that a talented production designer would be able to help add to the creepy “lost in the woods” vibe I wanted.

Enter Kendra Bradanini. An accomplished Production Designer with an impressive reel and talented team consisting of Propmaster Lawrence Humphreys and Art Director Hailey Duvall. I talked with Kendra about the challenge of augmenting our creepy wood over 56 unique locations and some ideas we had such as a dead animal carcass, skulls, runes, and rock formations. I shared with her the website detailing all 56 locations so she could get some ideas of where to hang items, etc. She responded back with a look book that explained her creative vision of the project. I loved it. She and her team were hired.

It was really important that every location had at least one defining feature we added, so that people in the narrative could have something to recognize that location by. This worked really well for certain locations, and not as well for others, mostly due to the low resolution of the final playback experience.

One of the biggest thing we needed her help with was designing and constructing the altar that Lindsey uses to find and summon the demon with. It needed to feel natural, but also serve the needs of the story.

Because we had limited resources and a significant amount of forest to cover, I had designed our shooting schedule to stay localized to certain parts of the forest, so that we could be filming while Kendra and Hailey would be setting up the next locations. Lawrence would stay with camera team to make small adjustments as we were filming. This way we could also repurpose high cost items (such as our deer carcass) and custom built items such as our hanging effigy.

In hindsight, certain items worked great, and others were really challenging for the 3D stitch. We were in a forest, which is hard enough alone for the optical flow algorithms, but adding any sort of thin moving objects like some of our hanging stick effigies just didn’t translate well.

Our art team did a tremendous amount of work creating this world and, most importantly, putting it all back the way we found it. We were very careful to respect the earth, working with the forest service to leave as little impact as possible and protect the natural plants and wildlife we encountered.

THE WORLD BECOMES THE STAGE

To bring our story to life I knew we would have to have solid performances from our actors. I’ve sat through my fair share of horrible performances in VR. I’m not sure why acting in VR is so hard. Maybe it is because we can’t cut a performance together and we’re used to having that option. If you haven’t worked in theater or had an extensive amount of time to rehearse (which is very hard on the shoestring budgets in VR) it becomes obvious very quickly that a three to five page scene that you have to nail the timing, performance, and technical considerations of in a single take is challenging.

I had a meeting a few months back with a A-list commercial director whose work I admire. He watched my work and although he had a lot of kind words, he remarked that he felt the acting took him out of the story. I was a bit hurt, but upon discussing it further with him, I think his note was that he wanted a deeper authenticity to performances in VR.

Speak of the Devil has its roots in a campy horror narrative, but I was determined to cast a believable couple that audiences could connect to.

I had met Taylor Murphy-Sinclair a few years back and always wanted to work with her. Her natural personality seemed to fit Lindsey’s character. I still had three more roles to cast.

Longtime friend and collaborator Jackie Sollitto came aboard as a casting director on this project. She has an amazing eye for talent and suggested Michael Terry Grant for the role of Brian. He was an instant yes. Our initial phone conversation revealed that he loves classic horror films (as do I) and was interested in working in VR.

Through our makeup artist, KC Mussman, we found Mick Ignis. Mick is known for his creature work and KC suggested that we work with someone who was going to be comfortable sitting in a makeup chair for hours. Because the Windego was a suit, he could be double cast as both the Cultist and Windego creature. Poor Mick was in makeup for the Cultist for around five hours. This meant some creative scheduling (more on that later), but having an actor who knows how to embody this kind of creature work was stunning. It was my first time working with such extensive prosthetic makeup and I learned a lot about how long this stuff takes to put on, and take off! Word to the wise, a makeup test is always recommended before you start production so that the artists can work with the actor’s skin and understand the particulars of how each piece will fit together.

The special effects makeup on our cast was crucial in keeping the experience feeling organic and helping our actors feel connected. I think our team did an amazing job.

While we’re on the point of a great makeup team, we also brought in a whole pressurized blood system so that we could get seriously bloody. Or Mussman and Nelson Cooper helped create some of the most fun we had on set. My only regret is that I wish the blood played better in VR. I should have blocked it closer because the resolution and lighting still isn’t quite there yet. When you think about it, most movie scenes cut to a close up. We don’t have that luxury in VR, so I should have blocked the action closer to the camera.

Working with actors in VR is amazing. It’s very creative and freeing. The actors love being able to improvise and try new ideas and I encourage them to do so. Since we aren’t shooting coverage, my approach has always been to treat performances more like a stage play, with the camera as our audience, and encourage a lot of eye contact with the viewer to draw us in.

We would run the scene a handful of times and talk about the motivations and discuss blocking particulars that would work best for camera. Rehearsals are really important because the actors have to nail the entire scene without cutting. For a five page scene, this can be tough. Taylor, Mike and Mick were great with suggestions and they really brought a lot to the project.

I think it is really important for cinematic VR to focus on achieving a higher of level acting in VR and that’s only going to happen as we figure out what techniques work well and hopefully convince talented people to collaborate. As we push the envelope with longer, more ambitious storytelling project, we can’t forget that audiences are going to connect with the actors over cool technology every time.

USING THE GOPRO ODYSSEY

Using the GoPro Odyssey was both a blessing and a bit of a curse. It was amazing because it by far produces my favorite 3D 360 footage at an extremely high quality and comes with Google’s automated stitching. It’s a bit of a curse because it’s quite heavy to lug around.

I had constructed our build to include a few extra features. First, I mounted two additional GoPro cameras on the bottom of the rig to catch any feet that would go into the dreaded “blank” area. I wanted our actors to be able to get CLOSE without chopping off their limbs. I also mounted a GoPro to the top so that I could capture above as well. To power these GoPro’s all day without needing to change batteries, I used a USB power brick that I mounted.

We were using LAV microphones for most of our audio, but I wanted to run our Hear360 8Ball and Zoom F8 recorder to capture as much forest ambience as possible. I had mounted a monopod to the center column of the tripod so that I could place the 8Ball dead center and mount our Zoom F8 onto the bottom. All in, it was a pretty good setup that although it was on the heavy side, we were able to pick up and move fairly quickly without needing to reset up in each new location. Our production audio mixer, Helena McGill of Noctvrnal, monitored the LAV levels in her headphones and used an iPhone paired via Bluetooth to the F8 to monitor levels from afar and input shot information. It was a pretty good setup!

Honestly, the worst part was carrying the battery pack around! If only we had been able to use the new YiHalo, that would have made things a TON easier.

Google’s stitching has a great feature that allows you to send two shots up to the cloud and tell it that you will be using them together as a composite. This meant that it was fairly easy for us to stay in frame and watch the action, knowing that we could shoot a “clean plate” where we would move to the other side of the camera. Sending both shots to the Google cloud, they would come back stitched with the same lens geometry and easily able to be stacked in premiere and garbage matted. This worked really well despite the fact that we were in the forest outdoors. I’ll cover the specifics of this in the next article I write in this series talking about our post production process.

The last VFX plate we always grabbed was a nadir shot with another GoPro camera. This is what we mostly used to mask out the tripod.

All in, it was a lot of shots to remember to take and keep track of. Our assistant camera Jake Block was again a hero by keeping really detailed notes on an Android tablet.

STAYING ORGANIZED

With so many scenes and variations, staying organized was a must. 56 sounds like a lot of scenes. It felt like many more. It’s so easy to miss one. Many of the 56 locations also had two or three different variations to correspond with the mesh narrative.

Scheduling all these shots was a massive undertaking. Not only did I have to work around actor availability, makeup times, and art set dressing time, but I also had to account for the changing light of the sun. By no means did we succeed in maintaining a consistent time of day. We would have had to shoot over the course of 3–4 weeks to make that happen. But we tried to be smart in the order in which we shot.

For example, we’d tend to shoot a lot of the empty scenes and 1st act stuff towards the beginning of the day. This was good for two reasons. First, it allowed our actors time to rehearse and get into makeup. Second, our story begins with us waking up in the morning, so the morning light tends to be consistent with scenes you would hit early in the story. Later scenes, after the demon has been summoned and you are on the verge of impending death, looked creepier with the long afternoon shadows.

THAT’S A WRAP

We were in Big Bear Lake for a week at the end of September. Monday was our scout prep day and we filmed Tuesday, Wednesday, and Thursday. Friday was spent capturing empty locations and pickups for anything we had missed.

Cast and crew all stayed in a huge (very oddly laid out) cabin we had rented for the week. My wife, Katie, home cooked all our food and snacks. My barely one year old baby also joined. It was a fantastic guerrilla indie film vibe. We wanted our first interactive live action project to come to life, and so the blood, sweat and tears were worth it.

Pick up the story next week in PART 3: Post Production Workflow Madness, detailing the production of this project and how exciting it is to see a vision come to life.

Download Speak of the Devil NOW for Google Daydream

Coming soon to GearVR, Rift, and HTC Vive.

A Light Sail VR original. Speak of the Devil was filmed on the GoPro Odyssey as partners in the Google Jump Start Program. Media storage was sponsored by Lumaforge. Hosting and Unity Development in collaboration with Wemersive and The Oberon Group.

If you aren’t familiar with Speak of the Devil, watch our trailer below.

PART 1: DESIGNING A “MESH NARRATIVE”

When I was 10 year old, I played MYST for the first time. It was a transformative experience (much in the same way that putting on my first VR headset was a few years back). I decided after I finished MYST that I wanted to make my own version someday. I remember using my dad’s clunky digital camera to capture a “walk-through” of the woods behind my house. I used HTML hotspots to create that feeling of walking through the forest. 20 years later, now I want to do it in VR.

I had just seen Sleep No More in New York City and while some parts of the narrative structure went over my head, I was really intrigued by the idea of trying to something similar to immersive theater in VR.

My business partner, Robert Watts, and I started to create a narrative story with the location interactivity of MYST. We didn’t want it to be a game. We didn’t want it to just be a movie. We wanted to explore what is in between the two with live action video. It was a very ambitious goal, but one that we had set out to do over two years ago when we first launched Light Sail VR.

BUILDING THE STRUCTURE

Robert and I began to sketch stuff out. First we thought that we had to make the map big enough so that it would feel like you could explore anywhere. We settled on a 7x7 grid and placed our “spawn” point right at the center.

We had built some excellent relationships with horror projects we’ve done (Paranormal Activity, The Parksville Murders, The Off Season) so we decided to lean into the horror genre and also have the opportunity to work again with talented people we love. Our story was going to be about a couple lost in the woods who accidentally summon a demon. At this point, the story is very loose. We begin by pointing at locations on the grid and saying “Let’s hear scary whispers in this square!”, or “This is where she summons the demon,” etc.

We quickly realized that we had to design the narrative in a way that would transform into act breaks and then close up the grid so someone watching doesn’t accidentally ruin the building of the tension by getting lost and hopelessly wandering in circles. So on our whiteboard we duplicated our grid anytime a major event happened. This forced us to think in “Act Breaks”, which helped define the narrative story structure we were after. For example, the “inciting incident” of Brian getting his arm hurt (an indication that this was not going to end well for our characters), the summoning of the demon, and then the death of one or both of our characters were all moments that move the horror narrative forward and that we wanted the audience to experience on some level. Making sure that certain scenes wouldn’t happen until those milestones were met was important to keep the story coherent.

Eventually you’d either be killed by the demon or cultist character, or you’d find your way out of the forest. We planned our grid so that you would be boxed in by deaths. Coincidently, we ended up with 12 deaths and one way out, for a total of 13 unique endings. Very fitting for a horror piece. The structure morphed a few times as we played with subplots, additional characters, and how exactly we were going to communicate all this information to our audience.

By the time we took a look at all our scenes and how they were fitting together, we ended up with a map that looked like a mesh net of interwoven story beats. Our audience could engage with the story in a number of ways, coming from many different locations on our map. This wasn’t a branched narrative, in which the audience is given a binary choice that creates a tree. This was a new kind of storytelling much closer to the immersive theater productions I had been inspired by. We needed another term. This is how we coined the term “mesh narrative”.

At this stage there was certainly a lot of plot holes and work still to be done. There were a few things I had in mind that I think were extraordinarily helpful in the design of this interactive narrative.

1. Characters needed to enter and exit each scene with motivated reasons so that the loops could feel endless and organic. This was really important to nail down, but made tougher by the fact that the forest at times was pretty open.
2. Characters should engage with the audience, but not ask questions of them. At all times I wanted to audience to feel like an observer, trying to mind their own business. It was also important that the audience member could be any age, any gender, any race and still feel a part of the story.

In hindsight I think I would have added more characters and subplots and a few less locations. But I’ll talk about all of our lessons in the final post of this series.

FLESHING OUT THE STORY

We also were trying to create this on a budget, so we were very flexible on creative decisions as long as they still served the narrative intention of the scene: that is to say that each scene needed to move forward the story.

For example, one of our biggest challenges was figuring out what exactly this monster would look like. We pulled some reference photos and took them to KC Mussman, our makeup artist from Nocturnal Designz. She went over several books with us that were filled with different looks and we chatted about the story and characters. She mentioned that she had just put together a Windego for a trade show and had all the molds and parts still in her workshop. It was perfect. I want to stress how important it is to work collaboratively with talented professionals who can bring a lot of their art to a vision if you let them.

KC’s Windego gave us some narrative backstory about why THAT particular demon had been summoned. We worked all that’s into the script and the character arcs. It is why Lindsey seems so selfish. She puts her work and that quest for success above all else. KC’s suggestion of also covering up our cultist character in burns was equally compelling and helped us write his backstory too. We imagined that he had been worshiping this demon for some time and the proximity of being this close has given him radiation burns. All these small details went into building this world we were trying to create.

With the larger details in place, two things were key to moving forward. First, we needed to hone in on our location. Second, I needed to figure out how to write a script.

After some research, Robert and I settled on Big Bear Lake as being the perfect location to film. It took us three separate scouting trips to lock our location and translate our scene grid from the 7x7 sketch on the whiteboard, to what the scenes would actually be during production. We ended up purchasing a Garmin GPS unit so that we could be accurate on each waypoint that we scouted. We also took photos with our Gear360 camera and created an interactive tour that allowed us to quickly reference each of the 56 locations we had scouted. Using Pano2VR made this process incredibly easy and we only needed it as reference. This site (which we downloaded onto our tablets) would come in handy when in production when talking to our art department about what I needed in each location.

Click here to visit our “scouting” website.

With our locations in place, it was time to revisit the whiteboard and start plotting which scenes would go in which spots and then churning out script pages so we could go out to our cast.

I ended up writing the script like I would for any screenplay. I first started by outlining the beats I wanted to capture, except this time I would write that same narrative beat a few different ways for different play throughs. The script took on a rough sort of order in order for me to focus on whether I was building a narrative arc enough. We ended up at about 20 pages or so.

The problem was that there wasn’t really a way to play through this in the interactive sense. We had some ideas about printing note cards or putting the script into an interactive story tool like Twine, but none of that happened because we were suddenly part of Google’s Jump Start program and we needed to get into production, FAST.

Again, in hindsight, I think it is crucial to put your script into some sort of interactive tool in order to test it out and we would have saved ourselves a lot of problems in post had we done so. But this was all part of learning as we were creating. We now have those proper tools in place.

WRAPPING UP

I don’t think we fully grasped what a monumental challenge this was going to be and we would hit several logical roadblocks later. At this stage, however, we were excited and ready.

Our scripts went out to cast. Robert finished the pre-production logistics. Permits were in place, an amazing crew was signed on. The train had left the station and we were going to produce our first original project.

Pick up the story next week in PART 2: It Takes A Village, detailing the production of this project and how exciting it is to see a vision come to life.

Download Speak of the Devil NOW for Google Daydream

Coming soon to GearVR, Rift, and HTC Vive.

A Light Sail VR original. Speak of the Devil was filmed on the GoPro Odyssey as partners in the Google Jump Start Program. Media storage was sponsored by Lumaforge. Hosting and Unity Development in collaboration with Wemersive and The Oberon Group.