A couple of days ago, an old post of Jeff Sutherland himself resurfaced on social media. He wrote about estimating stories and how it slows down teams instead of improving performance. One of the most common misunderstandings about agile is you need to estimate the work you will be doing. How could you otherwise predict the future? Today, I want to dive a bit deeper into why estimating your team's stories with story points, hours or apples is not doing you any good. In contrary.

With the help of your team’s historical data you can detect a lot. From certain trends in cycle times, to detecting bottlenecks which need to be resolved. Even forecasting the future is possible. Today, we’ll use that same data again to solve the question of why you should not estimate stories. And certainly not with story points or hours.

At In The Pocket, we have a lot of data. Tons of projects to analyse. Some teams working with and other teams without estimates. We were particularly interested in the ones that do use story points for estimating stories. With the help of one of our Machine Learning Engineers, I dived into some data about these teams. From one of our projects, we extracted a big list of completed stories, together with their assigned story points and the cycle time — You can read more about Cycle Time, what it is and how it can help you here — .

We plotted this on a scatter plot and calculated the correlation coefficient. This number tells you how large the statistical relationship is between two variables. When you do that, you get something like this:

Each blue dot represents a completed story. On the x-axis, you can find the story points that were used in estimating these. It is very common for teams that use story points to use the Fibonacci series. This is why there are some gaps between eg. 3 and 5. On the y-axis, you can find the cycle times of those stories. One of the first things you can notice is that there is no big difference in cycle time for a story that has 2, 3, 5 or 8 story points. We can state that for this project, there is almost no correlation at all between story points and the resulting cycle times. Perhaps only for the smaller stories. But the effort of estimating everything does not weigh up against that.

In an ideal world, if there would be any correlation at all, we would see a graph like this:

To understand why this phenomenon happens, we need to have a look to what cycle time consists of. Cycle time is the time it takes for a ticket to complete. So the time between the moment someone starts it until the moment someone completes it. The definition of the start and end status can be different for each team or context.

Between the start and the completion of a ticket, three things can happen: someone can be actively working on it. It can be sitting there, waiting for someone to be picked up (eg. a handover stage between a developer and a quality engineer), or it can be blocked. These three combined give us the cycle time of the ticket.

Cycle Time = active time + waiting time + blocked time

Nobody knows what will happen in the future. There are tons of variables that influence these times. A sudden dependency can block a ticket with a snap of a finger. Someone who gets sick can cause waiting times because the team needs that person's expertise, and so on. Thus by definition, these are impossible to estimate. This leaves us only the active time to try and estimate.

Humans are notoriously bad at estimating things. Active time consists out of development time, testing time, code reviewing time and a lot more other value adding activities. It is impossible to have insights on all these activities and know upfront how long they will take for each ticket. Moreover, the active time almost always is the smallest one of the three.

Some teams use reference stories to score new ones relative to these. As these reference stories often have a different context in which the team completes them, this also is a dangerous approach to take.

In short, estimating stories is impossible. As a development team you are ignoring waiting and blocked time. It is not that you are doing this on purpose, it is just impossible to do it. Nobody can predict the future, you merely can forecast it. Use the data you generate by completing items to gain insights and forecast the future. Don’t try to do it based on gut feeling. How? Read more about it here.

Stop estimating, start forecasting. was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

You already know your way around several metrics by now. By measuring cycle time, you spot trends in your historical data. The work item age might suggest blocked or difficult work items. To predict the future for a bunch of items, we’ll use yet another metric: throughput. As work progresses through your pipeline, it flows to the ‘done’ state. The definition of throughput is the number of completed work items per unit of time. If you display the throughput per day over a certain period of time, you get a throughput histogram.

Some days, your team completes a lot of work items. Other days, you finish none. Let’s put this data to work to predict the future. If you know how many items the desired feature has, you can use this data to create probabilities. For the sake of simplicity, we have a feature that consists of 100 items. Your client asks when it will be delivered. You already know that it is never a good idea to give an exact date nor is it to communicate in probabilities of 100%. A good forecast always contains two things: a range and a probability. You communicate to your client that there is a 85% chance you deliver the items by a certain date. The 85% being the probability. The date itself, the range.

To start forecasting, you take the number of items you want a forecast about. Pick a starting date. Then select a random value from your throughput histogram. Subtract the throughput from that number of items and add an extra day to your starting date. Repeat this process until you have no work items left. Plot that day on a new histogram. The idea here is that you repeat this whole process a couple of thousand times. The resulting histogram will look a lot like this:

On the x-axis you’ll see the various completion dates of the 100 work items. The y-axis displays the occurrence of each date in the histogram. On the graph, you can also spot the percentiles. For instance, the red box indicates the 85th percentile. This means that 85% of all the occurrences, the yellow area, happen before or on the 24th of June. That means there’s a probability of 85% that we will complete those 100 work items on or before the 24th of June. And all of this is based on historical data. Data which we collect by completing work items.

Important to mention is the fact that you need to repeat these forecasts over and over again. Both the scope (number of work items) and the historical data (throughput) will change. As soon as you have a new context to work in, your previous forecasts aren’t valid anymore. These kinds of simulations are called Monte Carlo simulations. You use them to predict the probability of certain outcomes in which random variables are present. To achieve this, the algorithm relies on random sampling of historical data. Exactly what we did above.

Selecting the correct historical data as input is more of an art than it is a science. Sometimes, people go on vacation or fall ill. From that point on, that part of the historical data might not be accurate anymore to predict the future. In any case, don’t overthink this too much. You still need to interpret what you see. Use it as a discussion tool to uncover potential underlying challenges. Challenges such as scope creep. You will find it to be a very powerful way to see the wood for the trees. Especially in the uncertain context of software development.

How you can forecast the future in software development was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

When you start tracking cycle times (a useful tool for this could be Actionable Agile), you have all the data you need to build your first flow chart. For the first graph, we will turn to what is called a scatter plot. The x-axis shows the completion date. The y-axis shows the cycle time of a certain item. Every single dot on this chart is a completed item. If you have enough data, you’ll start to see interesting trends. Trends that you can use as a discussion tool in your team.

For instance, a rising trend in cycle times will be visible over time. It’ll show you that there might be an underlying problem with eg. dependencies. Remember, you are looking at historical data. This Cycle Time Scatter Plot won’t tell you exactly why things happen. It is nothing more than a sign of an underlying challenge.

But how can you start predicting the future with this, you might ask? On scatter plots, you can mark the percentiles of your data. Predicting the future is all about taking your chances. Chances by nature are nothing more than percentages. The question you need to ask is which amount of risk you are willing to take. Is it 15%? Is it 20%? Or even more? The four most common percentiles used in flow metrics are the 50th, 75th, 85th and 95th percentile. If you plot all these percentiles on the image above, you get something like this:

Let us translate this to something more concrete by looking at the 85th percentile. It means that we have an 85% chance we deliver a single work item in 20 days or less. There is still a 15% chance we need more time for that single work item. Remember that the future does not have one single outcome? These lines are the translation of that. Thinking in a probabilistic way requires two elements: a range and a probability. In this case, our range is 20 days or less, and our probability is 85%. It does not take away all the uncertainty of the future but can help make it more bearable. And again, spark the correct discussions. These lines also are known as Service Level Expectations. As a team, you expect an item to complete within x amount of days. It is not a commitment. It is a forecast.

Keeping your cycle times under control helps your team becoming more predictable. Unfortunately, a Cycle Time Scatter Plot only shows you historical data. It shines when you want to spot certain trends, but it lacks the information to see what is happening right now. For that, you can dive into another scatter plot: the Aging WIP Chart.

This chart is ideal to use during stand-ups. On it, you see Work In Progress. We define WIP as all the work items between the starting point and endpoint in your workflow. Instead of the completion date on the x-axis, you see the stage of each item. The y-axis remains the same. On top, you have the number of WIP in each column. Finally, the percentiles of your historical data are also mapped on it. You can look at the Aging WIP Chart the same as at a Cycle Time Scatter Plot. The higher an item on it, the older it is. The big difference is that you are looking at the situation as is. Not historical data, but the now. The higher an item gets on this chart, the higher the chance it will violate our SLE (eg. the 85% percentile), and that means decreased predictability. You can even calculate the risk, but this would take us too far here.

With the Aging WIP Chart, you can spot items ageing early on. Act on them as a team. During standups, open this chart and progress from right to left, top to bottom. Try to spot the items with the highest risk of becoming outliers in your Cycle Time Scatter Plot. Ask yourself how you can resolve them. Is it an external dependency? Do you need to swarm with some developers? You might need extra people? All these are valid questions and will depend on your own context. The Aging WIP Chart is a helpful and actionable tool. One that will help you to control your cycle times.

In this article, I showed you two types of charts. The Cycle Time Scatter Plot helps you spot trends in your historical data. To track and prevent items from ageing, you use the Aging WIP Chart in your standups. The beauty of it is that the data for these charts is generated by doing the work. Not by unnecessary estimate meetings or a lot of overhead meetings. Start working on it, and change course when necessary. Next time, I’ll show you how to forecast groups of items, instead of singular ones.

How scatter plots can improve your team’s predictability was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

For the last couple of months, I have been diving into the books of Daniel Vacanti. Daniel helped to develop the Kanban Method for knowledge work. In his books, he explains how you need to answer the question ‘When will it be done?’. A question your customers often raise when they want software delivered. In this series, you’ll find some key concepts about predictability and forecasting!

A central theme in the books is probabilistic thinking. I introduced this new way of working at In The Pocket. One of my colleagues stated it would be hard to use this method with clients. According to him, people don’t think in a probabilistic way. Let’s start with a real life example to illustrate the contrary.

Everyone who ever commuted to work already knows how to think probabilistically. Let’s say you have a very important meeting with your CEO the next day. You definitely want to arrive on time. You already have a ‘kind of a notion’ of how long it takes to arrive at work whether you go by public transport, on foot or by car. But this meeting is important and you don’t want to arrive late. What do you do? You add extra time to your commute. How much depends on how confident you want to feel that you’ll arrive on time. If you go by public transport, you might take a train or bus earlier. If you go by car, you may leave half an hour earlier. All this, to make sure you don’t arrive late at that meeting with your CEO. And there you have it. You forecasted something in a probabilistic way. That wasn’t hard, was it?

So let’s extrapolate this to knowledge work. The ‘kind of notion’ I am referring to in the example above requires a definition of `commute`. It is simple. It is the difference between the moment you leave your house and the moment you enter your office. By commuting a lot, you learn to develop that feeling of how long it takes. You keep track of all the moments you ever commuted. Based on that information you can take your chances. Notice how your internal forecast exists out of two parts. First you have a range, in this case a commute time. Then you have a probability. The more time you foresee, the higher the chances you’ll arrive on time. To apply this to software delivery, all you need is a starting point and an endpoint for your work items. With your teams, decide on what these are, and start measuring the time it takes for a single work item to complete. This is what we call Cycle Time.

Soon you’ll have tracked the cycle time of enough items. Trends will emerge and you will be able to start forecasting items in a probabilistic way. You’ll develop the same ‘notion’ you develop when commuting. Based on historical cycle time data, forecasts can be generated. Only this time it won’t be a notion, the data will speak for itself. Stay tuned for more deep dives into the wonderous world of flow metrics. Next you can read all about how scatter plots can drive your team’s predictability.

The only thing you need to keep track of in your team was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

At In the Pocket, we recently got a lot more ownership over the app-building process for our team’s customers.

Previously, we shared our toolset with all teams. We used Ansible to provision our team-mac server. The server was running our Jenkins instance as well as building our iOS apps. Additionally, we shared a docker image with the company to build Android apps on the Google Cloud.

This dependency of shared tools led to 2 big problems. Firstly, it wasn’t always clear who was maintaining what. If a team wanted to upgrade to the latest Android SDK or a newer XCode version, this would have implications for everyone. Secondly, the docker image was bloated by its ability to build React Native, Flutter and native Android apps. It also contained SDK 27, 28 & 29. Simply put: more SDKs and tools meant a slower docker container.

In this situation, every now and then we had building issues that weren’t always easy to pinpoint or fix. Or we needed help from colleagues that weren’t available for immediate action.

Not so long ago, we moved to our private Gitlab and took the opportunity to use Gitlab CI and polish our toolset! This blog post focuses on what we learned about Docker and building for Android.

Docker-Android?

As we were investing time to migrate our continuous integration from Jenkins to Gitlab CI, we found out there is actually a Docker container maintained by React Native Community: Docker-Android 🥳!

We happily crafted our .gitlab-ci.yml file, only to find out that bundler (from Ruby) wasn’t pre-installed 🤕. A first ‘quick fix’ was installing it on every run. Unfortunately, it was a waste of time and only slowed down every build. At that point, we really started digging into the matter!

How to test?

Changing your .gitlab-ci.yml file, commiting & pushing changes is a slow process to see if your app is building correctly in docker.

So what can you do? You can actually build, run & log into the docker container quite easily! Once that’s over, you can set up git and checkout your project, or just copy it from your pc.

Start a docker container

First you have to build your Dockerfile:

docker build — tag itporbit/react-native-android:latest .

Now we can start it. The option — ti will make it interactive and — rm will stop the container once you exit the terminal.

docker run --rm -ti itporbit/react-native-android

Checkout, install & build

Now you are logged in the docker container you can do a git checkout, npm install & build your app!

git clone -b develop --singe-branch https://github.com/repo.git

cd app/

npm ci

cd android

./gradlew assembleRelease

Our own Dockerfile ❤️

At this point we have 2 options.

We could fork the docker-android project, and adapt the dockerfile to our needs. That means we would have to merge upstream changes when they occur.

Instead, we opted to create our own Dockerfile and extend from reactnativecommunity/react-native-android:2.1. This way we don’t depend on another git repo. We are in control of which version we extend and when we upgrade to a new release.

Now we are ready to make some changes!

Add support for Fastlane

Our first Dockerfile set some locale stuff and installed bundler.

FROM reactnativecommunity/react-native-android:2.1

# set locale to utf8 for fastlane

ENV LANG C.UTF-8

ENV LC_ALL C.UTF-8

# install bundler

RUN gem install bundler

As we use Fastlane, we don’t use ./gradlew but:

bundle install

bundle exec fastlane build_android

Non-root user

The react-native-android container runs with a root user, which comes in handy to extend from the container but poses a security risk. Therefore, we added a user (reactnative) and switched the workdir to home/reactnative.

RUN useradd -ms /bin/bash reactnative

# transfer ownership of $ANDROID_HOME (opt/android) so our new user can install additional sdk or build tools at build time. This is needed if you use libraries that target other Android SDK's.

RUN chown reactnative:reactnative $ANDROID_HOME -R

USER reactnative

WORKDIR /home/reactnative

Versioning is important!

Years ago, when we were still building apps on a self-managed build server, we definitely lost the ability to build older apps when we upgraded the software. By creating our own react-native-android docker container, every version is tailor-made for building a specific Android version. Version 1 builds React Native apps that target Android 10 (SDK 29). When we upgrade React Native to a new version that targets Android 11, we will update release version 2, made for building Android 11 (SDK 30) apps!

Thanks to this, we always target a specific version and not the latest tag in our .gitlab-ci file. When we have to do a hotfix, we will always build our app in the correct container 🎉.

Make

We use a simple make task so you don’t have to remember or document the docker commands to build, publish or run our container.

all: build

VERSION = 1.1

build:

  docker build --tag itporbit/react-native-android:${VERSION} .

push: build

  docker push itporbit/react-native-android:${VERSION}

  git tag react-native-android/${VERSION} HEAD

  git push --tags

run: build

  docker run --rm -ti itporbit/react-native-android:${VERSION}

So we can now invoke make run to test stuff out after making changes to our Dockerfile!

Resources and failing builds on mac

While testing, every now and then my docker image started failing unexplainably while building. Apt could no longer install dependencies for example. This wasn’t the case, but my reserved disk space was depleted. An easy fix, by running docker builder prune or docker image prune in the terminal so that caching of past builds was removed.

Interesting reads

• https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
• https://nickjanetakis.com/blog/best-practices-when-it-comes-to-writing-docker-related-files

Upgrading our app build process was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Let’s do the math: AR experiences are designed to only last for 1 or 2 minutes, but they require, on average, about 2 minutes and 15 clicks to download. Is it worth it? Perhaps. But can’t you create more value by cutting down the time and clicks for installation? Definitely. There’s good news: app-less AR integrations are already here. Time for a concise overview of these app-less alternatives, along with their pros and cons.

Native apps

Before diving into the app-less AR possibilities, we need to talk about the ‘classic’ first: native AR apps. To develop these you could either work on iOS via ARKit or on Android via ARcore. Most of the time, however, developers use engines like Unity or Unreal Engine to work in a hybrid way as these engines compile for both platforms while only having one codebase. It speeds up development and contains a lot of extra bells and whistles to enhance the AR experience.

A native app gives you the most options and features but is more time-consuming (read: more expensive). Moreover, downloading an extra app can be a major sticking point for users. Hurdles like going to the app store, worrying about security issues or going through too many clicks can be demotivating. These hurdles or barriers, however, can be overcome by other, clever options.

App-less AR alternatives

1. AR Viewers

Let’s talk about the app-less AR alternatives. The first on our list: native AR Viewers. Both Apple and Google have options where you can use the native power of your device without downloading an entire app. For Apple, it’s called AR Quick Look, whereas Google coined the term Scene Viewer.

Both have a functionality where the user can click on a certain object and instantly open an AR version, which you can place and view in your environment. The biggest plus? You don’t need to download an app first. It comes straight out of the box with each Android or iOS device that supports AR. The only cost for developers lies in optimising their 3D assets. The downsides? The functionality is rather limited and not as enriched as native apps.

2. Mini-apps

Other options that use native technology combined with the easy-accessible web applications are Apple’s App Clips and Google’s Instant Apps. These nifty little applications allow your customers to experience a small part of a native AR app without the hassle of downloading anything.

The mini-apps don’t need to be installed and automatically disappear after a certain time. The other side of the coin is that these mini-apps don’t have all the functionalities of a completely native app, but only give access to smaller tasks (like looking at a menu, but not being able to make a reservation).

3. WebAR

Do you want to experience AR through your browser, fully functional and without downloading requirements? Then WebAR is the way to go. Going this route requires a different set of developer skills, but platforms like 8thWall or Zappar tend to ease the process for your regular web developer.

The advantage of using the web is undoubtedly the accessibility. Your audience doesn’t need to download anything. They can use the AR functionality right away and enjoy their augmented experience on their browser. The catch here is the lack of processing power in comparison to fully native apps. WebAR experiences are more basic and limited and they need a bit more creativity to be as effective as native apps.

4. Social filters

To conclude our list, we’d like to give an honourable mention to social filters. You know these funny visual effects from Facebook, Snapchat and Instagram. This also is a form of AR that’s widely used and very accessible. Its lifespan is very short, so most of the time they are used for marketing purposes and campaigns. Functionality is rather limited, but it is quite impressive what people come up with these days. These filters can be linked to social media pages and can contain CTAs. In a way, it’s also a form of app-less AR, but as you need a social media app installed, it is kind of an outsider on this list.

Comparison

So how to pick the most appropriate AR option for your business challenge? It’s a good start to look at how they relate to each other. We mapped this out in two matrices. The first one comparing cost vs. functionality and the second one comparing cost vs. friction. Of course, there are a lot of other factors in play, but it gives you an idea of what to expect when picking your AR integration.

Cost vs. Functionality

The option of offering the highest functionality is also the most expensive one: a native app. Going app-less, the WebAR applications follow closely but have limited functionality to offer. The next three options come with a smaller price tag, but what you win in cost, you lose in functionality. The biggest cost for social filters, AR quick look and Scene Viewer is creating the content itself. App Clips and Instant Apps offer more functionality, but then again: the more requirements, the higher the cost.

Cost vs. Friction

When we look at the next graph, three options pop out immediately: the AR viewers, mini-apps and of course WebAR. None of these options require any installation of an app. They can simply be opened via a URL or by scanning a QR code. The other options, native apps and social filters do require installing an app, which naturally causes more friction.

Future forecast

AR is becoming more and more available for companies to use in their digital strategy. There are options out there for all budgets and every challenge. Looking into the future, we will steadily evolve towards easily accessible apps that are triggered by scanning a QR code of some sort. When smart glasses arrive, mini-apps will boom like never before to easily provide users with valuable information and functionality, without the barriers of native apps. The future of AR will certainly be more frictionless.

Make the right choice

Picking the right AR option always starts with understanding your challenge thoroughly. What do you want to achieve using AR? How important is visual quality? What’s your budget? If you’re struggling the right choice, don’t hesitate to reach out.

Got your AR senses tingling by now? Don’t wait to quench your thirst. Discover how the widespread LiDAR scanners can uncover new potential in your company, watch how we used AR for SMEG or simply calculate how AR can benefit your business.

This article was originally published on the blog of In The Pocket, Europe’s finest Digital Product Studio. We’re a team of +125 software engineers, architects, data scientists, product designers and digital strategists with a passion to create digital products & platforms that make people happy and businesses grow. If you have any questions about AR, don’t hesitate to contact me via thijs.morlion@inthepocket.mobi

App-less AR: beyond the barriers of native apps was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

It was that time of the year again. No, I’m not talking about a big white-bearded and friendly old man, all the mistletoe and presents by the tree, but about Apple’s WWDC. As the AR Lead at In The Pocket, these are exciting times to live in. Apple is one of the main discriminators in today's technology landscape and last Monday they treated us as it was indeed Christmas.

Spatial Computing is a human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces.

“They even didn’t mention AR…”, one of my colleagues mentioned somewhat disappointed after seeing the WWDC. As a matter of a fact, he was totally right. Apple didn’t mention it at all. Or did they? A lot is moving in the world of spatial computing. For people that are not familiar with the term, I’d love to give you a quick heads-up. It actually got coined by Simon Greenwold. He described ‘spatial computing’ as a human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces. The broader public probably is more familiar with Augmented and Virtual Reality, which are excellent applications of Spatial Computing, but it goes much further than that. It actually is a beautiful marriage of a lot of different technologies. From computer vision, to 3D scanning, to artificial intelligence and sensor fusion. Thanks to a combination of these technologies, devices start to understand the world around us. Moreover, we, as humans, can start to interact with these devices in a new, more natural way, but that is food for a whole other blogpost.

Because spatial computing entails a combination of a lot of different technologies, we need to look a bit deeper into this month’s WWDC. I’ll highlight 4 key elements that are of great importance for our spatial future: On-device speech recognition, App Clips, Spatial Audio, and a very big focus on privacy.

On-Device Speech Recognition

It’s almost a decade ago. The launch of Siri on the iPhone meant the start of having a personal assistant on that small device that fitted in your pocket. You just had to summon her by saying “Hi Siri” and you could ask her any question at which she would respond to the best of her abilities. To be able to understand human speech, and that in several languages, the audio samples had to be sent to the cloud for analysis. This often raised privacy concerns. A couple of years, Amazon and Google had to sail a rough sea when they were accused of misusing voice samples of people talking to their personal home assistants.

So why do we desperately need speech recognition? Well. One of the devices that might change the future for good, is AR Glasses. It is a pair of glasses that is able of recognizing the world around you and gives you specific and helpful information when you are at a specific place in a specific context. The most efficient way of interacting with that device is speech recognition. You can understand that when people use this technology more and more, it has to be privacy proof. To tackle this issue, it is a must-have to bring it to the device itself instead of somewhere in the cloud, and preferably to a device as small as an AR Glasses.

App Clips

The second, and personally I think it is a very neat feature, are App Clips. Not having the correct app installed at the moment you need it, is perhaps a situation you are familiar with. It then takes time to search through the App Store for it, waiting for it to be downloaded and when it is finally installed, you probably already don’t need it anymore. With App Clips, this annoying situation will be something of the past. It is actually smaller parts of an existing app that can be used via eg. a QR code or a webpage. Instead of downloading and searching for a full app, you’ll just use the smaller part of it in the specific context you’ll need it for.

This shouts spatial computing all over it. Let’s image we have one of those AR Glasses again. Our glasses will understand the world around us and the context we’re in. To be able to service you to the max of its ability, it will use these App Clips or perhaps even App Services to extend its functionality. This needs to happen all very quick and swift. It is also a plus when you, as the end-user, don’t have to look for a specific app in the App Store. No, your glasses will do al the heavy lifting for you.

Spatial Sound

Spatial computing is not only a visual thing, but it also applies to audio. Recently, we saw one of the pioneers in this field, Bose, close its Bose AR division. It was unfortunate because Spatial Sound will play a major role in building our spatial future. I was delighted to see that Apple is integrating spatial sound into their Airpods.

When you are having a VR experience, it is quite clear that surround sound is very important for being emerged in your experience. As a human, hearing is a very important sense. It is thus spatial by nature. Knowing where a sound is coming from can save your life. Although we don’t need to be aware of predators anymore, it can still be a very useful sense to have. Implementing spatial sound into technology can and will enhance the blend we are making between our real lives and the digital layer we are mixing it with.

Privacy

Last but not least, the elephant in the room when we are talking about spatial computing: privacy. The past decade companies have been building complete business models around the data of its customers. Data really was the new gold. The last couple of years some scandals were uncovered. Think of Facebook and the Cambridge Analytica scandal. Earlier in this blog post, I have been talking about Amazon and Google apparently not taking privacy too seriously with their voice assistants.

On last year’s WWDC, Apple showed us its anonymous login with Apple system in which a user’s email address was obscured for the developer, but the user could still make use of the login functionality of that specific app. It was a delight to see that Apple’s focus on privacy again in this year’s WWDC. Privacy is also one of the reasons Apple chooses to use LiDAR on its iPad Pro and probably also later in its AR Glasses. When we are entering the era of spatial computing, we all should be very aware of the implications it has on our privacy. Only companies that are taking this very seriously will prevail. Personally I’m glad to see Apple, as one of the big tech giants, is picking up its responsibility. Hopefully, more will follow.

The era of Spatial Computing is near

Although my colleague was very right. Apple did not mention AR in its keynote, but it mentioned plenty of technologies AR, and by extension spatial computing, will need to really thrive. All of these are an indication of something bigger Apple is aiming for. My humble guess would be that they are preparing for the AR Glasses I mentioned a couple of times throughout this blog post. They are combining a lot of technologies, making them smaller and most of all doing all this while taking their customers' privacy into account. I can’t wait to see what the future holds. At the end of this year, and the beginning of next year, we probably will know more.

What Apple’s WWDC means for Spatial Computing was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Managing ‘secrets’ is a pain

Credentials — API Keys, database credentials and certificates — are widely used in many applications we create at In The Pocket. Projects tend to get bigger and more complex year after year. This results in a growing ecosystem of connected managed services, each requiring its own credentials to protect the data they are storing and processing. Securing and managing those credentials becomes more and more challenging over time: as your project tends to get bigger, so does the amount of secrets you’ll need to store.

Recently Google announced its Secret Manager, a new Google Cloud service that aims to centralize, manage and secure sensitive information in a convenient and secure way. You could see this as a vault on steroids, where only the people who know the code have access to the contents of it.

So this looks really promising and it should make engineers’ lives better, but does it really? We’ve been using Google’s Secret Manager for the past few months and in this post, you’ll learn what our opinion is so far!

What is a secret manager?

A secret manager is a service that helps you protect access to your applications, services, and IT resources. This service enables you to easily rotate, manage, and retrieve database credentials, API keys, TLS certificates and other secrets needed by an application at runtime.

These are a few of our favorite things

When Google announced its new service, we were all very excited about it! We’ve been using Berglas or Sops as our go-to solution for managing secrets. But implementing those services is always a time-consuming task, so we were curious if this was the case for the secret manager as well.

Managed service

At In the Pocket, we tend to go for managed services as much as possible. There are many reasons why you should go for managed services, but we’ll discuss them in another blog post. A few of our main reasons is we can stay focused on the core business, reduce the risk of outdated software, wrong configuration, security issues, … and they are quickly implemented (which will be discussed later on).

There are many other solutions that provide managing secrets like Vault or Berglas. In the case of Vault, a team has to set up and manage the whole infrastructure. This means more budget and more time for managing something that could be done for you. Berglas on the other hand provides secret management via Google Cloud KMS. If you’re already working with Berglas, you can easily integrate it with Google Secret Manager. When starting up a new project however, you should just implement Google Secret Manager from the get-go and omit Berglas altogether, as adding Berglas to the integration provides no additional benefits.

It’s flexible

How you integrate Google Secret Manager in your application is totally up to you. You could see it as a fully decoupled service where you manage everything yourself. Another approach is to integrate it with your deploy pipeline, where secret versions could be created when building and deploying the application.

A secret could be a text string or a binary blob, so this means it could hold the value for one secret or a whole JSON file that contains all secrets for the application. How many secrets you’ll use will depend on which permissions are needed and your personal preference.

Secure access to your secrets

Google Secret Manager uses the principle of least privilege. By default, only project owners have permission to access any secret. Access control is granted through Cloud IAM where you are able to grant read & admin access to other members and make sure they have the permissions that they actually need. Permissions can be granted to individual secrets too and can be disabled easily.

A big advantage of keeping your secrets in Google Cloud is that not everyone has access to them. First you need access to Google Cloud console and on top of that you need permissions for viewing or managing secrets. Secrets could hold information that gives someone access to a service, database, … and those could hold a lot of sensitive information. When developing an application that works with medical data, you don’t want everyone from your organization to have access to it. Google Cloud and the Secret Manager helps you to prevent that! So this is a big thumbs up!

And another big plus is that audit logging comes for free when Cloud Audit Logging is enabled. For every interaction with the Secret Manager, an audit entry will be created. This could help you to identify abnormal access and potential security leaks.

Why encrypted secrets are so important

A common method for injecting configuration and secrets is the use of environment variables. Almost every application, service or platform is able to read an environment variable. This is a simple and straightforward approach, but it has one major drawback: the secrets exist in the environment in plaintext.

Any other process, library, dependency in your environment has access to the environment variables. A malicious library could result in a major security leak. Environment variables are great to store configuration settings, but not secrets!

With the help of Google’s Secret Manager, our secrets are stored in a central place, are encrypted by default and are only accessible if you have the right permissions. The encrypted part here is one of the most important ones: secrets are protected when at rest and also when they are in transit by moving them to and from Google Cloud.

When at rest, Secret Manager supports both Google-managed encryption keys (GMEKs) and Customer-managed encryption keys (CMEKs).

Versioning

The data of each secret is immutable, so you’re forced to create a new version any time the content of secret changes. From within your application, you can access a secret from a specific version (e.g. 12) or get the latest version with the ‘latest’ alias. Google advises to use the latest version only in development and staging and use a specific version for production.

Production deployments should always be pinned to a specific secret version. Updating a secret should be treated in the same way as deploying a new version of the application.

A major plus about versioned secrets is that it supports a gradual roll out of a new version of your application or that you’re able to do a rollback when something goes wrong.

Easy peasy integration

When Google announced his Secret Manager we created a proof of concept (POC) to see how much effort it is to integrate their new service. For this POC we used Cloud Run, Fastify as our Node.js framework and Node Config to handle configuration.

We extended the configuration with our secret (e.g. ORM Config) that is stored in the Google Secret Manager. First, we created a script that communicates with Google Secret Manager. You probably wonder where the credentials are to access the service? Access is done with Cloud IAM where Cloud Run has the permissions to access secrets.

Now the only thing that is left to do is create a ‘production.js’ file in the config directory so our ‘orm_config’ secret will be added to the existing configuration. Here we chose to use one secret to hold all the ORM configuration.

That’s all! Wow, that wasn’t hard right? Now in order to let this work with Fastify, there is one additional thing that should happen. When starting our Fastify server we need to resolve the config first so we are sure it’s loaded.

Beside Node.js Google provided a library for many other programming languages like GO, .NET Core, PHP, …

Conclusion

When Google announced their Secret Manager we created a POC first to test this new service. We were really surprised how smooth the integration went and certainly when you compare it with alternatives such as Berglas or Vault. It integrates really well with all other services (Cloud Run, Kubernetes, …) we use within Google Cloud, so we can only advise using Google Cloud Secret Manager!

If you’re not using Google Cloud, there are other secret managers out there like AWS Secret Manager, Vault, … It’s just so important that you don’t store your sensitive information in plain text! Using a Secret Manager will prevent access to your secrets by any installed package and these secrets are not visible to anyone that has access to the application’s infrastructure.

Original blogpost

Google Cloud Secret manager: a better place to store your secrets | In The Pocket

Google Cloud Secret manager: a better place to store your secrets was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Detecting (underwater) damage in ports is no easy feat. By using specialized point cloud scanners, trained professionals manually compare underwater snapshots through time and detect damage or other irregularities through careful inspection. This inspection of collected point clouds is very labor-intensive; that’s why at In The Pocket, we’ve devised a scalable pipeline using machine learning to automatically detect and signal irregularities in underwater point clouds.

Point cloud scanners gather a large amount of three-dimensional points to represent the environment they capture. All points in their simplest form basically consist of three numbers (the x, y and z coordinates) that together form a mathematical representation of the surfaces of the captured environment. These points can intuitively be used to visualize the captured environment, just like this point cloud of the Colosseum:

Point cloud technology is used in a variety of sectors besides traditional 3D engineering. Some examples are the manufacturing industry, mining industry, urban planning sector, construction and the gas & oil industry to monitor large-scale assets such as plants, properties and structures. The point clouds are used to inspect and determine the condition of large systems, which also make them a great asset for predictive maintenance and safety assessment.

In our case, point cloud capturing technology is used to take snapshots of the underwater environment of ports at different moments in time to subsequently use these for visual inspection by humans. If irregularities are found, divers are sent to investigate the situation. Since it is a time-consuming task to compare point clouds, comparisons for the same place only happen at timescales within an order of months or years. But what do you do when you want to catch anomalies much sooner, for example within the week? This is where our automatic irregularity detection pipeline comes into play. It can compare all available point clouds of all places at the same time, with (much) faster speeds than humans. This way it is possible to assess if an irregularity has occurred very soon after a new point cloud is available, without having it sitting idly in a queue waiting for human inspection. With a higher check-up frequency, the quality of day-to-day port operations improves noticeably.

How do we do this? At the core of our pipeline is the battle-hardened classic machine learning algorithm, called nearest neighbors search. Put very simply, this algorithm compares every point in the new point cloud with every point in the old point cloud, and calculates the distance to the nearest (old) point as an indication of irregularity. By using things like smart indexing and GPU-parallelization, this computationally time-complex process is made more efficient to adapt to the large point clouds that occur in real-life situations. Different point cloud comparisons can be run in parallel on the cloud, effectively removing the waiting queue for processing. A visual example of our algorithm finding three (artificial) irregularities looks like this:

Besides a visual presentation of the anomalies, the pipeline should provide a suggestion at which point the irregularity is actually occurring. One possibility is to find the points where the nearest-neighbor distance reaches a local maximum (using first and second spatial derivatives on the distance function), and suggesting these points as representations of the irregularities.

To avoid too many false positives caused by random small distance fluctuations (creating irrelevant local maxima), we voxelize both the old and new point cloud to the same voxel size, creating a more rigid grid of point clouds. Voxelizing is basically turning reality into Minecraft: our messy underwater point cloud with many redundant points gets transformed to a strictly rigid ‘blocky’ structure, where every point represents one volume cube with predefined size, called a voxel (volume + pixel). This effectively flattens the distance function through space, except where non-negligible irregularities that are larger than the voxel size occur, reducing the amount of false positives. This doesn’t mean that there won’t be any false positives left, but in this case it is better to have some false positives, than to miss an irregularity. A human operator during quality control can simply dismiss a falsely proposed irregularity every now and then based on his experience. This could for example be the same irregularity showing up twice in close proximity, or an irregularity that in context is nothing to worry about (e.g., a ship that was simply not there in the previous scan). The opposite case is worse: when an irregularity is not picked up, quality control will not easily look beyond the proposed irregularities, and the fault will not get addressed.

Adding this all up, we create a robust pipeline that decreases the amount of manual inspection labor, increases the inspection frequency, and overall innovates the way of working in port change detection. These techniques are not restricted to ports alone: other application domains using point clouds can benefit greatly from this pipeline as well. If you have a use case that would benefit from our algorithms, don’t hesitate to contact us!

Using machine learning to detect underwater port damage was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

The talks

There were 26 talks in total but 2 were always given at the same time, so you could only attend 13 talks in total. Sometimes it was very difficult to device which talk would be more interesting or fun. Luckily, all talks will soon be published on the Flutter Europe YouTube channel.

I’ve learned something useful from every talk I attended and I enjoyed all of them, but if I had to pick my 5 favourite talks it would be these:

• Performance: Optimizing your Flutter app
• Augmented Reality in Flutter
• Animations in Flutter Done Right
• Building games using Flame
• How to convince business to Flutter? Real-life cases

Unfortunately I missed out on some interesting talks. Apparently ‘Implementing complex UI with Flutter’ was an amazing talk. From what I’ve heard, Marcin Szałek really had the audience in the palm of his hand. He was able to explain this topic so well that creating complex UI didn’t look so difficult or intimidating anymore. This is definitely the first talk I’m going to watch as soon as they’re available online.

The Flutter community

If you’re anything like me you read Flutter articles, watch Flutter videos, listen to Flutter podcasts, … on (almost) a daily basis. A lot of the more famous Flutter community members who create this content attended Flutter Europe. I’m really glad I was able to meet so many of my personal heroes and heroines, like Emily Fortuna, Filip Hráček, Matt Rešetár, Simon Lightfoot, Rémi Rousselet, Felix Angelov, Jorge Coca, Hillel Coren, the Codemagic team, …

They say you should never meet your heroes, but that’s not true if your heroes/heroines are members of the Flutter community.

It was also amazing to meet a lot of other members of the Flutter community. On my plane to Warsaw the two people sitting next were also attending Flutter Europe, and I met a lot of others there in between the talks, during lunch or at the two parties.

I was travelling by myself, but there was never a moment where I was alone. I’m usually an introvert but it was so easy to connect with other Flutter developers. I heard a lot of amazing stories about other people’s experiences with Flutter and I’m glad I was also able to share my own.

The loot

As I’ve already mentioned in the title, this was my first developer conference. I’ve only attended gaming conferences like Gamescom so far. When I go to one of those I know I’ll bring home a backpack full of loot, a.k.a. gadgets, toys and merchandise. I didn’t expect this to happen at Flutter Europe. I wanted to attend this conference for the knowledge and the networking opportunity, but secretly I hoped I’ go home with one of those Dash plushies you see everywhere on social media…

You should be going to a conference for the knowledge instead of the loot, but I can’t be the only Flutter developer who wanted to get his hands on a Dash bird plushie, right?

Ever since I learned of their existence I knew I wanted one. I’m really glad attendees were able to earn one by completing a programming challenge. The challenge itself wasn’t very hard, the most difficult part was typing on a German keyboard (the semicolon was very well hidden).

Everyone who attended received a Flutter t-shirt and a goodiebag with a hat, a water bottle and some other things. If you visited the booths of GDG, Codemagic, LeanCode and the others, you were also able to get some other t-shirts, all kinds of stickers, …

My personal highlights

Because I wanted the complete Flutter Europe experience I decided to go to the first party, the one before the conference started. This party took place in a stupendous building called the ‘Warsaw Spire’. When I was standing in line to get my badge, someone who seemed quite familiar came to stand next to me. It was Matt Rešetár, a.k.a. Resocoder. In my opinion, his Flutter videos are the best you can find on YouTube and it was an honour to meet him.

At this party I was also finally able to meet Marie Jaksman in real life. You might know her if you’re a member of the Codemagic community. She introduced me to Łukasz Kosman, one of the organisers of Flutter Europe and also the co-founder of LeanCode. He gave the ‘How to convince business to Flutter? Real-life cases’ talk on the second day. Marie told him I was the person who wrote the article about Flutter vs. other mobile development frameworks. His reaction was amazing, but also very unexpected.

He said that he had a slide in his presentation about an experiment I conducted for my article. I was flabbergasted (or should I say fluttergasted?).

This was the first article I’ve ever written, but apparently it was quite popular. To my surprise, some of the people I met at Flutter Europe had read it. The fact that it was also a part of a presentation at Flutter Europe was definitely a highlight for me.

There’s another person worth mentioning who definitely made the whole Flutter Europe experience even better. His name is Dylan Beattie. The first time we saw him was at the party after the first day of the conference where he gave a performance. He turned Billy Joel’s ‘Piano Man’ into ‘Bug in the Javascript’, Bon Jovi’s ‘You give love a bad name’ into ‘You give REST a bad name’, … If you’re a programmer who’s into the rock classics, you should definitely visit his YouTube channel.

Dylan also gave the last talk of the conference named ‘The Art of Code’. He gave some amazing examples of why programming could/should be considered a form of art and at the end of the presentation he talked about a special project of his. Dylan is the creator of ‘Rockstar’, a programming language designed for creating computer programs that are also song lyrics. It’s difficult to explain how amazing his talk was, but you can watch it here and see for yourself. I can guarantee you that you’ll enjoy it!

Conclusion

In my opinion, and I’m convinced a lot of other attendees agree with me, Flutter Europe 2020 was an amazing experience. This was only the first edition, but the organisers already got so many things right. Hopefully Flutter Europe 2021 will be even bigger and better. I’m happy I was able to attend and I can’t wait for next year’s edition!

Flutter Europe 2020: My First Developer Conference was originally published in In The Pocket Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.