In a recent project I was facing the challenge of creating a machine learning powered photo booth solution. The software allows its users to take pictures of themselves in a car, have the background automagically removed without the usage of a greenscreen, and replaced with another more pleasing background scenery. Among other things, performance was key here. The segmentation needed to be finished by the time it takes the user to exit the car and walk up to the nearby pickup station. That gives us around five to ten seconds.

In this story you’ll learn how you can use Flask, a micro web framework written in Python, to drastically reduce the time it takes for the pre-trained Mask R-CNN model to generate bounding boxes and segmentation masks for the captured photograph.

Mask R-CNN

The Mask R-CNN for object detection and segmentation framework is a Python implementation based on the work published by Facebook Research and allows to get bounding boxes and classifications for a variety of object types, such as cars and people, for any input image provided. Of course its performance is heavily bound to the computational power of the host machine.

That said, we built a powerful rig equipped with a Nvidia Titan RTX GPU with CUDA support and a high end CPU for any CPU bound work that needed to be done, which is mostly OpenCV based image processing and error corrections. With this setup we achieved a total computation time of around 15 seconds per photograph for an input / output resolution of 1920x1920 pixels.

Here’s one example of the processing steps from input to output with me and my colleague:

So how does Flask help here?

Once you have started playing around with Mask R-CNN you will notice that the model and its weights need to be loaded and initialized whenever you want to run segmentation on an image. Say you had a Python script called my_segmentation_script.py which takes the path to the input image as an argument, you’d execute it like this:

Now this runs fine and will perform the required work, assuming you have setup Mask R-CNN properly. The issue here is that once the script is done it will clean up any allocated resources and release the model and its weights again. So for any subsequent image they have to be loaded again, which costs us around up to 10 seconds of time! So let’s try and get rid of this overhead using Flask.

Follow the instructions over here to install and setup your Flask web service. Once it’s running you’ll be able to create Views which you can think of as API endpoints. Following example code creates a GET endpoint for the api/v1/ route of your flask app that returns "Hello World" when called in a browser:

All we need to do now is create a view / endpoint for our segmentation python code to execute and pass it an input and output path as parameters so it can read in the image and output the segmentation at the desired path. The important differentiation is that we will only load the model and the weights once when the Flask app starts and then re-use it throughout the session / lifetime of the service. Here’s the full code and then we’ll take a look at it in more detail:

Most if this is just boilerplate code for importing the required Python modules and creating the Flask view. Pay attention to lines 11 through 23 as we are initializing the keras session here and then loading in the model weights. At line 23 we are then finally assigning the created session to be the active session.

Next we are simply defining a POST endpoint that takes the input and output path in its request body. Note lines 30, 31 and 32 here as this is where we are making sure that we are using the globally defined session and graph variables we just created.

Using main.doSegmentation(rcnn, input_file_path, output_file_path) we are then passing everything to our segmentation code, which makes use of the pre-loaded weights.

Summary

By loading the model weights only once and then using them in a shared session throughout the lifetime of the Flask service we were able to save another ~10 seconds of computation time for a captured photograph to run through background segmentation. Combined with decent hardware this fulfills our requirements in performance for the photo booth to work.

Please share in the comments if you run in to any issues or if you have questions. I hope you find this little trick useful and can boost your performance just as I did!

Using Flask to optimize performance with Mask R-CNN segmentation was originally published in Medialesson on Medium, where people are continuing the conversation by highlighting and responding to this story.

Very often you’ll find awesome coding tutorials on Unity which will include a combination of these “pitfalls” for simplicity:

• Only public fields in scripts
• Make everything a Singleton
• Non performance-optimized code
• Synchronous scene loading
• Only MonoBehaviours
• …

We could go on here. Don’t get me wrong, nothing wrong about it. They still carry the message and instructions they intend to teach their audience. Let’s take a look at how we can implement a simple Service Locator pattern to improve on some of these aspects.

Creating a new Unity project

You can go ahead and create a new Unity project to work with or implement it right into your existing project, as you like. I am using Unity 2019.3.0f5 here.

• Open Unity Hub
• Create a new project using whichever template you like
• In your Assets folder create a new folder and name it ServiceLocator, we’ll put all the code in here

Implementing the service locator

In the first step let’s create a service interface for our service locator to work with. Inside yourAssets/ServiceLocator folder create a new script file IGameService.cs:

As you can see we leave it empty for now. We want to keep this sample really simple. Once you we got everything in place, you can use this interface to define what your services have in common and should look like and which APIs they provide.

Next we actually already get to implement the service locator. Again in Assets/ServiceLocator create a new script file ServiceLocator.cs:

A few more things going on here. Let’s step through:

• private ServiceLocator() {} we make default constructor private to avoid misuse of the service locator by creating new instances of it randomly.
• Dictionary<string, IGameService> services will hold all currently registered services using their type name as the key to retrieve them.
• public static ServiceLocator Current { get; private set; } a static property which will hold the currently active service locator instance. This should be the only Singleton in your application basically from now on.
• Initialize() will initialize the active service locator with a new instance.
• public T Get<T>() where T : IGameService is used to get a service instance from the service locator. We are restricting the generic type T to types implementing our IGameService interface.
• public void Register<T>(T service) where T : IGameService is used to register a service instance with our locator. It will then be available using Get<T> .
• public void Unregister<T>() where T : IGameService for completeness we provide a way here to unregister a service of a given type from the locator.

And that’s it regarding the locator itself! Did I mention we want to keep this locator simple? Doesn’t get any simpler I think. Worth nothing here is that this service locator will only allow one instance of a service of a given type. Which should do the trick on most cases. You can modify it to support multiple instances of a type easily.

Bootstrapping the service locator

Now that we have our locator implemented let’s see how we can initialize it when the application starts.

In Assets/ServiceLocator create a new script file Bootstrapper.cs:

Ok. Let’s see what’s going on here. I am assuming for this bootstrapper script that you have some sort of “Base Scene” in your Uniy project. This should be the scene at index 0 of your overall scenes. Use it to get your application ready and initialize everything other scenes may need to work with.

[RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.AfterSceneLoad)] using this attribute we tell Unity to run the static method Initialize() once the base scene has fully loaded. Inside Intialize() we create the service locator instance and then proceed to register all our game services with it. Once that is done, we can load our main scene which will contain appilcation content using those services eventually. You can extend this bootstrapper as needed.

What we learned and gained

In this short sample we learned how to implement our very own service locator and use it in our Unity project. But what did we gain?

• You will find that it’s much easier for you now to separate business logic code from visualization code. Your game services can take care of handling data and such, while you’ll be creating MonoBehaviours to visualize it and use it.
• This service locator is a good starting point to think about how you can implement Dependency Injection in your project next.
• You can now easily write Unit Tests out of Unity for your services since they are not MonoBehvaiours anymore.

About reinventing the wheel

Of course you don’t have to reinvent the wheel here. There is existing service locator implementations for Unity out there. If you like the idea, I can recommend taking a look at

XRTK and MRTK for two examples of existing implementations. Both provide a very good starting point for you and ways to implement your own services and register them with their service locator.

Simple service locator for your Unity project was originally published in Medialesson on Medium, where people are continuing the conversation by highlighting and responding to this story.

Step 1: Install the required extension to build Unity projects

As of now Azure Pipelines does not have a built-in task for building Unity projects. Luckily it’s extendible and we can create our own build task for use in our pipelines. “Ain’t nobody got time for this!” you say? No worries, I already went ahead and created an extension for you to use. Head to the marketplace and install it for use in your organization:

Please note: There is another Unity build extension. But that one targets specifically HoloLens Unity projects and requires the HoloToolkit to be in the project. Make sure you get the right one for this article, which is the one titled “Unity Build”.

Please note also: I’ve made the extension open source on GitHub. If you feel the need to improve on it please feel free to do a pull request. Also please report any issues over there, so I can fix them.

Step 2: Create and configure your custom build agent

While you could use the Microsoft hosted agents, I recommend setting up your own agent for Unity builds. This gives you full control over the available Unity versions and modules. Let’s get started. Navigate to your Azure DevOps homepage and in the top right corner click on your profile picture, in the menu then select “Security”.

This will open the security settings when you can create a Personal Access Token (PAT) that your agent can use to configure itself with the DevOps Organzation.

Select “New Token” right below the page header, then give a meaningful name to your PAT. Select “Custom defined” scopes and in the lower end of the panel find “Show more scopes”. Now find the “Agent Pools” scope and tick the “Read & manage” checkbox. Finally click “Create” to create the PAT. Make sure to note the PAT somewhere until we need it. You will not be able to see it again, once the panel has closed. Next we will setup the agent.

Select “Organzation settings” and then “Agent pools”. Click “New agent pool…” to create a new pool. Name it Unity Windows Agents e.g. If you are creating a Mac agent, name the pool accordingly. You will need this name later. Last but not least click “Download agent”. Please follow the instructions to download and configure your agent. You will need the PAT and Agent pool name in this process. Let me know in the comments, if you have any issues. Last but not least, please install the Unity Hub on the agent and install the Unity versions you need.

Step 3: Setting up an example build pipeline

For this article we’ll create a build pipeline targeting the Android platform. The steps may differ a bit depending on your project’s target platform but should generally be similar. So head over to your Azure DevOps project and in the pipelines section, create a new build pipeline, by clicking “New” -> “New build pipeline”:

Next you need to tell the pipeline wizard where to get the source code from. In my case, it’s all hosted on Azure Repos. You can change these settings later on, if you need to:

The wizard will now offer some templates to start from. Please select the empty job template. We want to start from scratch:

Alright. Now click the plus icon button as highlighted in the screenshot below, enter “Unity” in the task search textbox and add the Unity Build task, you installed in the beginning of this article, to the job:

Now please repeat and add the following tasks as well “Android Signing”, “Copy Files” and “Publish Build Artifacts”. Your pipeline should now look something like this:

I am not going into details with the Microsoft built-in tasks. If you have issues configuring them, please refer to the official documentation or let me know, maybe I can help out. We’ll focus on the Unity Build task and how to configure it.

Basically the only thing you need to do here, is tell the task which target platform you are building for. We are targeting Android in this example, so make sure to select “Android” in the “Build Target” dropdown. For “Unity Version” select “Project Version”. This will build the project with the Unity version it was last opened with. Make sure it’s installed on the agent.

Next unfold the “Output Variables” section of the task and enter a reference name here. The task provides a buildOutputPath variable you can use in following tasks to get the build output files.

That’s it! This pipeline will build your Unity project, sign the APK and publish it to the Azure DevOps artifact store so others can download it. In an advanced scenario you could let the pipeline publish it to the App center or some store e.g. for testing.

Conclusion

Setting up your own build agent is a one time process that will definitely pay out once you get comfortable building your projects using Azure Pipelines. Once you’ve done that, you can use it across all projects in your organization.

Please share your feedback and let me know if it worked for you or you want to discuss things for advanced scenarios. Happy building!

Continuous integration for Unity 3D projects using Azure Pipelines was originally published in Medialesson on Medium, where people are continuing the conversation by highlighting and responding to this story.

When the Windows Mixed Reality platform was introduced a couple of applications had these really cool 3D app launchers you could place somewhere in the mixed reality world and launch the app when selecting them. Unfortunately back then, there was no way for developers to use this feature for their own applications. Luckily those days are gone, Microsoft introduced new APIs in the 2017 Fall Creators Update (RS3) for their immersive headsets. Since April 2018 Update (RS4) even HoloLens devices can use them.

As you might know, most of the Windows Mixed Reality appilcations out there are made using the Unity Engine. Once you are done developing and want to deploy to a device, you’re going to do a Unity build which will create a Visual Studio Solution for us targeting the Universal Windows Platform. All good so far. Now if you’ve read through the 3D app launchers documentation you’ll notice that in order for them to work, there is some changes we need to do to the Visual Studio project file and the appmanifest file for them to work. This is where things get complicated, since those changes are gone when you create a new Unity build in a different folder or build your app using continuous integration somewhere in the cloud. So let’s fix this by creating a Unity post build script to do the required changes for us whenever we build. Credit for most part of the code goes to Edd Smith, Head of Creative Tech at Fracture Reality. I applied some minor tweaks to get it working with IL2CPP scripting backend.

First, we’ll create scriptable object to conveniently hold our 3D launcher data. So create a folder 3DLauncher in your Unity project Assets folder. Then create a new Script file and name it AppLauncher3D. Insert the following code:

Once Unity has compiled the script, you’ll get a new menu item in the Editor Assets menu under Assets->Create->3D App Launcher. Next create a new folder Editor in your 3DLauncher folder and create another script file in it, name it AppLauncher3DEditor. This script is going to use the UnityEditor namespace, that’s why it must be inside a folder called Editor, so it’s not included when building for the target platform. Anyway, here’s the code:

Let’s break this into smaller pieces:

This is our entry point. Since we added the PostProcessBuild attribute to this static method, Unity will execute it, once it has finished building the UWP solution and give as the path to the exported project. Since Unity will do this, even when not targeting the UWP platform, we have to make sure, we are targeting it. Then we lookup the AppLauncher3D asset we created to read the configuration from it. Remember it contains the path to the .glb file and eventually overrides bounding box and center.

This is where we peform the 3 steps necessary for the 3D launcher to work:

1. Add the required definition to the package manifest
2. Add the asset to the project and mark is as deployment content
3. Copy the .glb file into the projects assets folder

I’ll leave it up to you to dig into the specific methods and discover them!

And we’re done! Go ahead and create a .glb file for your 3D launcher, then add it to your Unity project assets and create the scriptable object asset to configur it. Build your project and you’ll have a shiny new 3D app launcher!

How to add a 3D app launcher to your Mixed Reality app right out of Unity was originally published in Medialesson on Medium, where people are continuing the conversation by highlighting and responding to this story.