What If You Could A/B Test Physical Space? — AWS Interview [podcast] was originally published in Density on Medium, where people are continuing the conversation by highlighting and responding to this story.

Knoll has a remarkable space. Check out the video below (headphones, because the music is 👌).

New Vlog — NeoCon 2018 was originally published in Density on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building analytics is an evolving space — There’s old tech and there’s new tech. Since we often field questions about the various methods of counting people, I thought it would be helpful to document 7 technologies that make up the space.

#1 — Density Open Area

About:

Density uses depth data and machine learning to anonymously count people in open space.

Industry:

Industry agnostic. The system is designed to measure how people use any open area in real-time without invading occupant privacy.

Cost:

Hardware fee and annual data fee, variable by volume.

Benefits:

Users are able to access (from Density website):

• 60% reduction in cost to deploy (vs. camera / optical alternatives).
• Up to 20 foot range, 40 foot effective diameter (4x coverage of alternatives).
• 1,325 square feet of coverage
• Historical occupant pathing and heatmaps
• Desk and room availability (+ release)
• Touchdowns and dwell time

Includes real-time API. More info — http://density.io

Pricing:

http://density.io/pricing

Limitations:

No identity tracking; Powered device (not battery operated)

Data type:

Depth data and radar.

#2 — Density Entry

About:

Density uses depth data, machine learning, and computer vision to anonymously count people.

Industry:

Corporate offices. Companies in the Fortune 1000. The system is designed to measure how people use a company’s offices, conference rooms, and other work areas.

Average Cost:

Hardware fee and annual data fee, variable by volume.

Benefits:

Anonymous data, designed for corporate offices, real-time API. More info — http://density.io

Pricing:

http://density.io/pricing

Limitations:

No identity tracking; Powered device (not battery operated)

Data type:

Depth data and infrared lasers

#3— Cameras or “Optical Vision Sensors”

About

Most cameras see in flat color images. Smart cameras use second-by-second changes in pixel color to determine movement and identify people or other objects in a scene.

Industry

Retail, building security, corporate offices.

Average Cost

Variable based on the device’s intelligence and the analytics system behind it. Companies usually charge a hardware fee and an analytics or dashboard fee. Some examples:

Benefits

Inexpensive, impressive analytics

• Cost: Cameras are relatively inexpensive and they come in a wide variety of form factors.
• Analytics: Cameras on the cutting edge have impressive on-board computer vision. They can generate a ton of information about the areas they’re deployed in. People tracking, facial recognition (i.e. Face++), object detection, etc. As it’s become more practical to make use of machine and deep learning, camera systems have become impressive.

Limitations

“Handoff problem,” privacy, culture, and security

• The Handoff Problem: When pointing multiple cameras into a room, you have to overlap their fields of view. If you don’t, or if you’re using multiple cameras, they have to intelligently make sense of the people that disappear in-between the FOVs. This is called the handoff problem and it takes some processing.
• Lack of privacy is a camera’s biggest limitation. Some smart cameras will “anonymize data locally,” meaning they will blur or obscure a person’s face or downsample the image’s resolution so it looks fuzzy. It’s a good method but can be difficult
• Culture: In an office setting, it can be difficult to get beyond the pilot stage with camera solutions for counting people. Often, employees push back as a system looks to scale.
• Security: For buyers with a robust infosec posture, camera systems are an ideal target for hacking. Even benign systems that obscure a video stream locally can be compromised and made to do otherwise.

Data type: RGB video or flat images.

#4 — Active Infrared or “Break-beam” (AIR)

How it works

Break beam sensors have an infrared emitter and an infrared receiver. The sensors are typically placed on one side of a doorway (or both sides). The tech looks like this:

Standard AIRs count the number of times infrared light is “broken” or passed through and at the end of the day, the user (you) divides the number by 2 to determine the total number of people that came and went. It’s pretty analog.

More intelligent AIRs claim to do bidirectional movement.

Industry

Retail

Average Cost

Benefits

None, really. They’re battery operated but they’re inaccurate. Not good for counting people.

Limitations

Not accurate

• Blindness: Break beam sensors are inaccurate. The sensor becomes blind when two people enter at the same time (side-by-side) or enter and exit at the same time.
• Human movement is complex: Break beam sensors rely on signal processing to sort out when a person has entered. The signal, which looks like the figure below is hard to make sense of when lines form or people bring boxes and bags with them.

Data type:

Signal and signal processing algorithm

Each time Jordan (Density co-founder and head of algorithm) punches, this is what an AIR break beam sees:

Those two lines (blue and green) represent two AIR sensors spaced very close to one another. The spikes in values are increases in power consumption. The closer Jordan’s arms are to the sensors, the higher the values.

To a signal processing algorithm, these spikes look exactly like someone entering / exiting as it does when someone is punching, moving their arms while walking, carrying boxes, and standing in line.

Because it can’t distinguish between these different human behaviors, it makes AIRs an unreliable people counting technology. They are very inaccurate.

For more weird human behavior, see some of the odd stuff we’ve captured in the field.

#5 — WiFi Tracking or “WiFi / MAC Address Tracking”

How it works

The image above is an cartoonish version of how devices track your phone. Years ago, one of our very early prototypes was a MAC Address Tracking unit. Here’s how it works:

1. Your phone is always looking for known WiFi networks (home WiFi, work WiFi, etc). It does this out of convenience so you can automatically connect to a known network without manually selecting it.
2. The way your phone finds a WiFi network is by sending out what’s called a “probe request.” This probe request is kind of like your phone saying, “Hey my name is, Andrew.” But instead of “Andrew” it sends “40:68:AD:80:D3:A0,” which is a MAC Address unique to your phone and your phone only (it’s globally unique). You can look yours up in the Settings › About section of your smartphone. Fun fact: Bluetooth has one, too.
3. Technicalities aside, the thing to know is pretty much all WiFi routers are capable of tracking your phone. In fact, your MAC address is how a WiFi router or access point serves internet to all of your devices: your laptop, smartwatch, Fitbit, phone, Nintendo Switch, etc. They all have MAC addresses, they can all be tracked, and you do not need to be connected to the internet. All you need is to have your WiFi turned on.
4. So, you’re in a building and your phone is reaching out saying, “I’m here!” Multiple routers are listening and triangulating. They compare the relative strength of that signal to one another and can approximate where you are in the building. They also know what other devices you usually carry with you (i.e. smartwatch, iPad, etc). Most important, they can tell if they’ve seen you before (even if the last time they saw you was in another country during a business trip).
5. The routers roll up this data and send it to an analytics platform that looks like this:

*It is possible to fake or change a MAC address. This is called “spoofing.”

Average cost

Varies widely by platform. Depending on the analytics, it can vary from tens of $$ / month to tens of thousands of $$ / month. Your existing enterprise WiFi system will have this as an upgrade option.

Industry

Retail, Corporate Offices

Benefits

Low cost, widely available, and ambient.

Limitations

Privacy. Inaccurate at a room level.

Invasive: Depending on the environment you’re deploying the technology into, this can be an invasive technology. It is not “opt-in;” meaning, the users the system tracks haven’t given their permission.

Inaccuracy: The real downfall of WiFi tracking, though, is inaccuracy. The system usually isn’t granular enough to determine the use of a specific room. So you end up with heatmaps and approximation like graphic above.

#6 — Seat Sensors

How it works

Seat sensors are battery operated motion detectors + tape. You can see the removable pull tab on the side of the device above.

The sensors run in low power waiting for movement. When they see movement, they transmit “Movement!” back to another device nearby. That device is powered and usually has a 4G internet connection.

If you have 1,000 employees, you stick these beneath 1,000 desks, and however many conference room tables you might have.

Average Cost

Varies by provider. Some bundle it with their service. Others charge separately for the hardware. It’s a pretty low cost solution, though (10s-100s of $$).

Industry

Offices

Benefits

Seat specific data. Battery operated. Low cost.

Limitations

Battery operated scale. Privacy (sorta).

Battery: If this is a long-term solution and you’ve deployed them in the thousands, when a battery operated device dies, you will have to retrieve and redeploy them in the thousands.

Privacy (sorta): Technically, you can use this system to track how long someone is at their desk. The privacy issue is dependent on how you structure your analytics and how you use the data. Know that systems like this have the ability to track an individual’s time-at-desk if you associate an employee’s identity with a device. Used benevolently, they can be helpful in understanding use of certain furniture.

#7 — Thermal

How it works

Thermal uses body heat and computer vision to determine objects. Looks like this:

Industry

Retail, Offices

Average Cost

The most popular thermal people counter is about $1,500 / device. Price varies by vendor. The underlying technology is not terribly expensive (10s-100s of $$ off the shelf).

Benefits

Anonymous

Limitations

Unreliable accuracy.

Thermal systems rely on motion to distinguish humans. It has difficulty when people stand still, when they overlap, and when they carry warm things (like laptops). The image below is two people near one another but sensor only counts one.

Extra — Ultrasonic

How it works

Ultrasonic sensors bounce inaudible sound off people as they walk by. Each one has an emitter and a receiver.

Industry

Robotics

Average Cost

They are low cost. Ballpark $50–$100 (max).

Benefits

Low power

Limitations

Accuracy. Put simply, if someone wears a fuzzy coat, ultrasonic devices get confused because the sound waves bounce inconsistently. If you’re trying to count people, there are better technologies… unless everyone is wearing denim. It does very well with denim.

7 Technologies that Count People (Buildings, Offices & Agnostic) was originally published in Density on Medium, where people are continuing the conversation by highlighting and responding to this story.

Below is a practical demonstration of machine learning — a form of artificial intelligence — and computer vision applied to a product called Density — an anonymous people counting system used by workplace teams to understand how offices are used.

To appreciate the manual efforts behind AI, it helps to first understand the challenging environments we’ve asked computers to deal with and thereby the real meaning of the term “accuracy.”

For the purposes of this article, I’m going to use counting people as the practical example.

Human-spotting

In the world of people count, companies often refer to accuracy in round, blanket terms. Phrases like, “We’re 94% accurate at counting people!” make for good marketing. Unfortunately, uniform accuracy is an impossible promise in real world applications.

Human behavior and built environments are so variable from space to space or building to building, that the accuracy of one installation usually differs from all others. This reality is obvious if you operate a system at any meaningful scale.

Important

Accuracy is not just hard to achieve, it can also expose a shortcoming of a company’s technology to report on. For example, with certain technologies (like cameras), quality assurance would invade customer privacy.

The Variability Problem -or- “The Things They Carried” -or- “What The Hell Is That?”

Illustrated below are examples of the variability problem. These demonstrations are from live Density deployments in corporate offices and college campuses around the U.S..

A Spaceship Trashcan at 3am

Below is a rare event. It’s a giant trashcan that rolls through in the wee hours of a workday morning.

The trashcan is objectively hard for an autonomous computing device to distinguish. The subjects in the scene are both moving, they’re both reasonably organic shapes, and they are attached to one another.

Teamwork & Stuff on Wheels

In this scene, multiple people wrangle a large object through a doorway. The fact that there are only two people in view and just one of them exits is obvious to the human eye but an untrained algorithm can make all sorts of wrong-headed mistakes.

The reason is that this event is not “nominal” or: according to plan or design. It is “anomalous” or: deviating from what is standard or expected.

This may be self-evident but anomalous events are hard to explain to an algorithm because there are so few of them.

It’s kind of like telling a computer, “This is a cat, and that’s a cat, and this is a cat, too,” and the computer pointing at a random television and saying, “Cat?” And you saying, “Dammit John… Who put the TV there?” … but for humans.

Environments And Data Quality

Regardless of the technology: radar, millimeter wave, depth, thermal, cameras, whatever — when you decide to measure atoms instead of bits, you quickly realize you can’t normalize or control for the real world that you’re deployed in.

To teach the machine, you have to be creative but most importantly, you need a lot of data from as many of the variable and hostile environment types a machine might encounter.

Teaching the Machine

Fortunately, for modern technical companies, we have better tools today than we’ve had in the past — not the least of which is almost 70 years of research into artificial intelligence and tooling. Alan Turing proposed his eponymous test in 1950. Shortly after, a team at Dartmouth started building programs that could play checkers, solve word problems, and do math.

The time it would take, however, to realize the true value of artificial intelligence was consistently underestimated and, in more recent times, overestimated.

Under-estimation

For example, researchers believed automated, accurate language translation would be solved before the end of the 1960s. This particular story may be apocryphal but, at the time, a computer would translate the English: “out of sight, out of mind” to the Russian: “blind idiot.”

It wasn’t until some 30 years later in the 1990s that automated translation began to make some progress and become a market of its own.

Over-estimation

Most people thought the Watson-equivalent for the Chinese board game “Go” was a long way off. Go is much harder to model than Chess. There are just more potential moves at any given moment and each has cascading implications on gameplay and position.

As you may already know, AlphaGo, the Go-playing algorithm, beat the human champion, Lee Sedol, in 2016 — a decade ahead of predictions. The latest version from the team at DeepMind is considered the strongest Go player of all time.

The game is more than two thousand years old.

So, what’s the point?

The point is that it’s fun to speculate on the infinite and future potential of artificial intelligence. But we should all just get used to being off by a decade or two. Instead, I think it’s productive to look at the pragmatic applications of what early researchers have made possible for us.

Today, we are materially benefitting from nearly a century of academic work invested in symbolic systems, “Good Old Fashioned AI,” deep learning models, and extraordinary leaps in low-cost, high-powered computing.

Despite these extraordinary advances, however, something as simple as counting people is still very hard; and, I suspect, will remain so as long as humans are the subject.

Human behavior is not obvious to a computer. Even in a non-hostile physical environment (like the reasonably clear depth stream below) a computer can’t easily distinguish what’s human and what’s not human; even though you and I can.

Therein lies a popular solution for variability — the way AI companies (Google included!) solve problems like these is to teach the machine what’s obvious to us.

“Labeling:” The Adult Version of a Coloring Book

For Density, we address the variability problem by designing our own labeling tools. They are like Microsoft Paint, but funner.

This is the real, manual human work behind a lot of artificial intelligence. When there are no open-source alternatives for a dataset, like the anonymous data Density collects, you have tell a computer what’s what.

As said above, ‘labeling’ is like telling a computer, “This is a cat, and that’s a cat, and this is a cat, too,” and the computer pointing at a cat and saying, “Cat?”

And you saying, “YES! Cat! Oh my heart is full! Cat! Thank god they’re not carrying a TV.” … but for humans.

Below you’ll see how we paint what we believe to be the door and the person’s arms, head, body, etc. What is critical to training like this is that it is just as important to tell a computer what is not a human as what is.

We build our models using many tens of thousands of frames like this one.

End Result

Larger Network = Higher Accuracy = Larger Network = …

We are always feeding the machine. As Density’s network continues to expand, we learn to handle new, unexpected environments and behaviors.

And when you give high quality, labeled data to talented engineers who specialize in computer vision and machine learning, computers can learn quickly.

Plates & Carried Objects

Ignoring Large Objects:

Handling Lines:

Reliable Accuracy

For Density, we deal with accuracy door-by-door. We generate ground-truth or our “control” by annotating a statistical sample of every door and then comparing the human annotator’s results with the algorithm’s.

“Supervised Learning,” or “a Tool for Ground-truth,” or “Humans telling computers what is and what is not a ‘cat.’“

Supervised learning A custom application used by a team of Density annotators to ensure accuracy and proactively identify issues.

Slide from an accuracy deck

We deliver accuracy reports like the following. Accuracy improves datasets grows.

AI is not magic; just math and manual labor.

So, when a company makes an accuracy claim for something that has to observe and categorize atoms (image recognition, self-driving cars, robotics, sensors, or cameras) ask them to explain how they solve for behavior and environmental variability. Ask to see what their “sensor” or “optical system” sees. Ask them to produce an accuracy report. And most importantly, ask them how they generate ground-truth.

Slides on how Density does it below. Accuracy numbers are statistical samples from each door and improve as our datasets grow.

More information at density.io. Email: contact@density.io. Jobs: jobs.density.io

If this was interesting or helpful, please leave a comment or a couple claps :).

— Andrew Farah, Density CEO

AI Is Not Magic; It’s Manual Labor (& Math) — How We Built An Accurate People Counter was originally published in Density on Medium, where people are continuing the conversation by highlighting and responding to this story.