Snips was founded with a simple vision: that artificial intelligence could change the way humans and machines interact, without compromising privacy. Indeed, the more we become connected, the more we feel oppressed by technology, as each new device requires effort to learn and operate. But if we could interact with devices like we do with people –using our voice– then perhaps we could have all the benefits while we make technology disappear into the background.

But who says voice says microphone, so evidently privacy was going to be a major issue. Back when we first talked about it in 2013, few paid attention. Privacy was considered a nice-to-have at best, and a hindrance for many. Fortunately, this has changed over the last few years, with the multiplication of privacy scandals and data breaches. Privacy is now a must-have, and technology must be built following Privacy-by-Design principles.

This strong focus on privacy has led us to build our entire voice technology to run on-device, so that we could avoid sending voice recordings to the cloud. Running deep learning on the Edge however is no easy task, in particular when the goal is to have it run on devices that have limited computing power. But thanks to an incredible team, we were able to achieve our ambitions, and create the first on-device voice assistant that can handle complex use-cases.

And of all the use-cases, none is harder and more interesting than music. Given the almost infinite amount of artist and song names, it was clear that a generic “let’s build a huge model for everything” approach was not going to work, especially not on the Edge. This is where our secret sauce made a huge difference: by contextualizing the language understanding to a given domain and user, we were able to dramatically reduce the performance footprints of the models, while improving the user experience.

When it came to testing Snips, we didn’t think long about which speaker to use. As long-time Sonos fans, we had been using their products everywhere: in our first tutorial on music, in our homepage video, in our test units at home. So naturally, we were tremendously excited when the Sonos team offered to join forces to make the voice experience on Sonos even better!

This acquisition is an important validation that privacy and voice are here to stay. But it also shows that the impact of the French Tech ecosystem isn’t limited to France alone. By keeping the Snips team in Paris and integrating the technology into their products, Sonos proves that French technology can be scaled globally. To help drive this effort, our CTO Joseph and CPO Sébastien will be leading the team going forward, while I step down to focus on new privacy-related projects.

None of what we have done so far –and will do next– would be possible without our incredible team. I would like to deeply thank our engineers, researchers, product managers, business and operations people who had many opportunities to go elsewhere but decided to join our vision. In particular my co-founders Michael and Mael who took a leap of faith to start Snips, Joseph our CTO for his incredible leadership and execution skills, Yann our COO for running the company smoothly and constantly finding new business and financing opportunities and Sébastien our CPO for his foresight and good spirit. Finally, I would like to thank our clients and partners who trusted us, our investors The Hive, Eniac, MAIF, Korelya and BPI for backing us all the way, and our 38,000 developers who build amazing things with Snips!

To voice and privacy!

Snips is joining Sonos! was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

I recently discovered Snips, an amazing privacy-preserving voice assistant that runs entirely on device!! It’s also simple to set up and use.

Let’s install Snips on a raspberry pi & use it to turn on/off a light fitted with a sonoff switch.

The sonoff will be flashed with custom firmware so everything runs locally over MQTT , the same transport layer used by Snips. No cloud required, no cloud desired.

Install Snips

The Snips documentation is good with easy to follow guides at https://docs.snips.ai/getting-started but we’ll run through the install process.

The easiest way to get started with Snips is by using sam, a helper that simplifies the Snips installation. Install sam on the computer you’ll be using to connect to your pi.

sudo npm install -g snips-sam

Once sam is installed you can connect to your pi using your pi’s IP:

sam connect 192.XXX.X.XXX

This will generate a key pair, copy the public key to key to your pi, & then ssh into the pi. Now you can install Snips on your pi using:

sam init

This will also install the mosquitto MQTT broker , which will be automatically started & run as a service alongside Snips.

Install a microphone

In order to interact with Snips you need a microphone & optionally a speaker if you want to use text to speech (TTS).

I used the ReSpeaker 2 Microphone Pi HAT & I’ve been impressed with the sensitivity of this device. It’s been able to detect voice commands from up to 4 metres away, in a quiet room.

Installation is a breeze.

sudo apt-get update
sudo apt-get upgrade
git clone https://github.com/respeaker/seeed-voicecard.git
cd seeed-voicecard
sudo ./install.sh

Reboot when you’re done. After that you can setup audio for Snips with sam:

sam setup audio

Make sure you see the Seeed soundcard in the list (rebooting should take care of this, but on one occasion I needed to restart more than once in order to see the card in the list of available devices).

Test microphone & speaker

Select the Seed card for both recording & playback and then test microphone and speaker with sam test microphoneand same test speaker respectively. I found there was slight distortion with the audio at full volume so opted to reduce the volume slightly. You can do this via alsamixer command line utility. 80% volume worked well for me.

Create a voice assistant

In order to respond to voice commands you need to create & install an assistant in the Snips console. Detailed instructions can be found here.

I found the assistant naming convention & structure a little confusing at first:

An assistant is, as you’d expect, able to respond to a variety of instructions and act on them. In Snips world, a single instruction is called an intent, which is then handled by an action. You might have multiple intents that are linked to a single task, for example controlling a light might have one intent to turn the light on, and another to turn the light off. These intents and their associated actions are grouped into what’s called an app, so the hierarchy is:

assistant > app > intents & actions

We’ll create 2 intents, one for turning the lights on, and the other to turn the light off.

While you can attach python action code to your app in the console, we’re going to handle intents in a node.js application instead.

1. Create an assistant called ‘Home’ and click on it

2. Inside the home assistant, click ‘Add an app’, followed by ‘Create a new app’, naming it, ‘lightControl’.

3. Click on lightControl and then ‘Edit app’. Click on ‘Add intent’ and call it ‘turnLightOn’

4. Enter text ‘Turn on light 1’ in the textbox under training examples.

5. Create additional variations for the voice command, eg. ‘Switch light 1 on’

5. Click on ‘Create a new slot’ and call it ‘lightID’. Select Snips/number as the slot type, from the dropdown. A slot represents a variable in your voice command so you could issue commands like, ‘Turn on light 1’, or ‘Switch light 2 on’.

6. Select the number 1 in your intent text and choose lightID from the slot list.

7. Create another intent to turn off the light- it’s identical to the first one except the voice commands will pertain to switching off the device.

When you’re done, hit save. This will begin training your model! Done.

Install Assistant

Back on the command line, login into the snips console in order to install an assistant on your pi.

sam login

Login using your console email & password. Now you can install an assistant with:

sam install assistant

This will bring up a list of assistants available for installation. Choose the Home assistant we just created.

Now, if you run sam status you should see a list of services, all of which should be running (except for possibly the analytics service).

You can test the applicaton with sam watch.

Try, ‘Hey Snips, turn on light 1’.

You should hear the wakeword detection sounds and then see some info about the intent in the console.

Troubleshooting

If you don’t hear any sound, and you have a speaker attached, make sure the soundcard is configured properly, and that you’re using the correct wake word.

If you hear the wakeword detection sounds, but your intent isn’t being handled (displayed to console), make sure the Snips skill server is running:

sam status

This was one of the few problematic areas I encountered while using Snips. Sometimes manually starting the service worked, but I did resort to an entirely fresh raspbian and Snips install on 2 occasions too.

Another issue I’ve yet to resolve, is that after rebooting the pi, the skills server doesn’t restart . The only way I’ve found to remedy this is by reinstalling the assistant with Sam.

Handling voice commands with javascript.

We’ll create a simple node.js app for handling the intent and then publish an mqtt message to turn on/off a light when an intent is inferred. This will be handled by our sonoff switch after we flash and configure it.

Update node

1. SSH into your pi and check your arm version in the terminal

uname -m

2. Check the node.js download page for the correct install associated with the arm version you have. It you have pi zero it should be ARMv6 and a pi 3B+, ARMv7.

wget https://nodejs.org/dist/v10.16.0/node-v10.16.0-linux-armv7l.tar.xz
tar -xvf node-v10.16.0-linux-armv7l.tar.xz
cd node-v10.16.0-linux-armv7l
sudo cp -R * /usr/local/

Create a folder on your pi for your Snips action app. A great way of working with remote files is to mount the remote folder on your local machine so that you can edit and work with the files as though they were on your local machine — check out SSHFS.

Install the mqtt package

npm install mqtt

Then create a new javascript file and add the following. This connects to the mqtt broker, subscribes to relevant topics, and publishes messages for the lights in the home to respond too when our intents are observed.

The light switch

Sonoff switches allow us to control an appliance but they require an internet connection & connection to a cloud server. We want to keep our data local & be able to control the switch offline with mqtt, so we’ll flash the switch with the awesome tasmota firmware that enables us to do just that.

Below are a condensed set of instructions based on a fantastic SuperhouseTV video on how to flash sonoffs. Refer to the video if you’re confused about anything- you’re working with something that will be ultimately be connected to a high voltage power supply. I am not an electrical expert. Exercise caution.

1. Make sure your sonoff is not connected to mains, and remove it from it’s housing.
2. You need a USB TTL adapter to flash the device. The adapter also needs to have an output voltage of 3.3V. Mine has a jumper for toggling between 3.3V and 5V.

The EIE adapter I used has the following pins (DTR, RX, TX, VCC, CTS, & GND). The pins that we’re interested in are the RX, TX, VCC & GND.

2. Before plugging the usb into your machine, connect some female to male prototyping cables to the adapter. Flip the sonoff over to see where the male pins need to be inserted.

— Connect RX on adapter to TX on sonoff

— Connect TX on adapter to RX on sonoff

— Connect VCC on adapter to 3V3 on sonoff

— Connect GND to GND

3. Press and hold the power button on the sonoff and then insert the USB cable into your machine.

4. We’ll use esptool to flash the sonoff. Install it with pip (so you’ll need python)

pip install esptool.py

5. Download the tasmota firmware — grab sonoff.bin file.

6. Find the port of the usb adapter:

ls -l /dev/tty.usb*

This returned /dev/tty.usbserial-A5XK3RJT for me so this is the port I’ll use for flashing the device.

7. Navigate to the folder that you downloaded the sonoff.bin file to, and flash the sonoff with the instruction:

esptool.py -p /dev/tty.usbserial-A5XK3RJT write_flash -fs 1MB -fm dout 0x0 sonoff.bin

8. When it’s done, unplug the usb and plug it back in- the green LED will blink. The sonoff is advertising a wifi network that you can connect to, to configure the device. You should see it under your wifi network list. Select it and a configuration window will open.

9. Change the SSID and password to those of your local network, clear the AP2 password, and give the device a meaningul hostname. It’ll restart and join your network.

10. Find the ip of the device from your router so that you can connect to it.

11. Enter the ip address in your browser address bar, and you should be greeted with a page that looks like this.

12. We’re interested in the MQTT menu. Enter host address (this is the IP address of the pi that has the mosquitto broker installed on it, the one we installed snips on. Again, you can get this from your router’s connection list). Also enter client and topic as below.

13. Save and the device will restart.

If you have the mosquitto_pub client installed on your machine, you can quickly test that the device is working in the terminal, where -h XXX.XXX.X.XXX is the IP address of the pi that has the mosquitto broker installed:

mosquitto_pub -h XXX.XXX.X.XXX -t ‘cmnd/living-room-lamp-small/POWER’ -m ‘ON’ 

mosquitto_pub -h XXX.XXX.X.XXX -t ‘cmnd/living-room-lamp-small/POWER’ -m ‘OFF’

The green LED should turn on and off. And that’s it. You’re all done.

The only thing left to do, is to carefully wire up your sonoff to a light and then fire up the node app we created earlier.

Controlling a sonoff light switch with the snips.ai voice assistant was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Rust Neural Network library for the Edge

By Mathieu Poumeyrol and Joseph Dureau

After open sourcing Snips-NLU a year ago, Snips now shares Tract, a new piece of its embedded voice platform. Tract is Snips’ neural network inference engine.

While TensorFlow and, to a lesser extent, PyTorch dominate the ecosystem of neural network training solutions, the landscape for inference engines on tiny devices, such as mobile and IoT, is still pretty much open. TensorFlow teams are pushing TensorFlow Lite, and Microsoft ONNX runtime seems like the perfect complement to PyTorch. Hardware vendors are pushing their own solutions, be it Android NN API, ARM NN SDK or Apple BNNS.

These libraries are not really interoperable, and have limited scopes: most of them, TensorFlow Lite included, only support a small subset of TensorFlow’s operator set. This means there is no clear go-to solution for an engineering team when their machine learning colleagues hand them a neural network to run on-device. On the other hand, machine learning teams will often have to pick a network architecture to satisfy the available runtime of a given target. Mimicking the interoperability efforts done by the software industry over the past decades, we think it is time to aim for a “train once, infer anywhere” approach.

This ambition led us, nearly two years ago, to develop our own solution for embedded inference. It has been used in production for over a year as part of our Wake word engine, and it enables our machine learning team to freely explore new families of networks with the confidence that we would be able to bring these networks to production.

Even in today’s context, reverting to an off-the-shelf inference library would mean sacrificing either modelling creativity, performance, or portability. We believe the embedded neural network inference landscape still needs to progress and converge, which is why we decided to contribute to the collective effort by open sourcing our solution. In this post, we describe the library, its performance, and provide high-level motivation for the implementation choices we did to optimize an application field that is close to our hearts: voice processing.

Cross-compilation

One of the major hurdles we encounter at Snips is cross compiling our solution for all of our targets. This includes small single-board computers designed for hobbyists like the Raspberry Pi Zero, 2 and 3, as well as industrial ones like NXP’s i.MX7 and i.MX8, Samsung’s Artik platforms, NVIDIA’s Jetson TX2, etc. The two main mobile platforms, iOS and Android, are also natural targets. This means we support ARMv6, ARMv7 and ARMv8 hardware platforms, on three major operating systems: GNU/Linux, iOS, and Android. We also support the regular non-IoT Intel systems, so that developers can work comfortably.

Snips Flow, our voice platform, includes a Wake word engine. This engine listens continuously, and triggers the speech recognition component when it hears the user say the Wake word (“Hey Snips”, “Alexa”, etc). The Wake word detector relies on a neural network that our machine learning team trains using TensorFlow. Two years ago, before TensorFlow Lite came to existence, the natural approach we tried was to embed TensorFlow as a library to execute our models on device.

However, TensorFlow is a big, complex framework, and, nearly two years ago, cross-compiling it as a library — to Android specifically — gave us a lot of pain. So much that we realized we needed a plan B.

At Snips, we chose Rust as the main language to develop the Snips Flow platform. Our team got used to the comfort brought by working with a modern software environment where cross-compiling is not an obscure afterthought but a solid design principle. So we gave it a try. We used the protobuf library to parse the TensorFlow format, ndarray to operate on tensors, and implemented a handful of operators that were part of our first Wake word models. This was the birth of Tract.

While we finally managed to get TensorFlow working on all our target platforms, the sheer size of it motivated us to actually switch to shipping Tract instead of TensorFlow a few months later.

If the origin story can sound anecdotal, the reasons behind it are not. As of today, the continuous integration infrastructure of the Snips platform targets dozens of operating systems and hardware combinations, with more added every month. This leads to some strong guiding principles in the evolution of Tract:

• it is written In Rust
• it is trivial to cross compile
• it is free of external dependencies: instead of linking computing libraries, like BLAS, we integrated the small bits that we actually use (which in turn gave us a few opportunities for optimizing even more)
• at runtime, Tract detects the device it is running on in order to adapt and optimize its performance appropriately. This allow us to ship the same binary for ARMv6 and ARMv7 devices without sacrificing performance on the smarter chips.

Today, some of these principles are actually vindicated by TensorFlow Lite. The latter is much easier to cross-compile than TensorFlow, and also takes the form of a static library free of external dependencies.

Streaming aware

Snips is focused on voice assistant technology. Neural networks are the natural solution for several tasks involved: Wake word detection, recognition of voice commands, speaker identification (finding out who is speaking) and acoustic modelling (translating sounds to phones or words, as part of a speech-to-text engine).

All of these tasks have to be performed in the context of interactive sessions with a human user: time is a central element of the problem description, and minimizing latency is key in making the interaction comfortable.

Wake word detection, for instance, has to happen “live”. Such a detector is always-on: there is no natural end to the input signal that it should wait for before processing the entire signal. In other words, the detection needs to happen in a streaming fashion, which means the engine needs to make a decision at every step in time based on the signal captured in the immediate past. This puts hard constraints on the neural network architecture, forbidding, for instance, reductions over the entire time axis.

This streaming constraint pushes the inference frameworks to their limits: most of the popular frameworks are designed with image classification in mind. They only work naturally over the entire signal, and become very awkward to work with when streaming is required, if they work at all. Kaldi — probably the most popular open-source speech-to-text framework — is a notable exception: its neural network inference engine is designed around streaming.

In TensorFlow, network graphs are “frozen” at train time, and contain a series of training-related idiosyncrasies. They must be optimized to run inference efficiently. Tract transforms networks after loading them, first by decluttering the network of those idiosyncrasies, then by adapting the network to the runtime environment. While TensorFlow Lite does this during a network translation stage, we chose to do it just before runtime, to be able to perform machine-dependent tweaks.

One of the critical transformations Tract performs is translating convolutional models to a streaming form. Indeed, our preferred architecture for Wake word detection and user identification consists of stacks of convolutional layers, in the same fashion as ResNet and WaveNet. While recurring networks are relatively natural to run in a streaming fashion, convolutional networks need a bit more work: a lot of operations can be skipped by implementing caching around each convolution operator.

Convolutional networks transformed naively to frame-by-frame streaming stateful networks would suffer from operation and data dispersion: a lot of flow control logic and data access work would have to happen for each incoming audio frame while performing a relatively modest number of useful computing operations. To run efficiently, it is necessary to re-introduce a grouping of frames, that we nicknamed pulses. By processing a reasonable number of frames together, in the order of 8 or 16, we amortize the flow control overhead. Additionally, this provides vectorization opportunities to optimize the execution even further.

Once networks run through these transformations, they can be fed in a streaming fashion and output decisions in real time. While we still consider it experimental, this “streaming and pulsing” transformation is applicable to all convolutional networks. We have successfully used it to run our WaveNet Wake word networks and ResNet speaker identification ones. Acoustic models will be next.

Beyond voice

With the necessity of reusing efficiently cached data from previous iterations, our main use cases does not fit well in regular frameworks. But rather than solely working on voice-related applications, we felt like we could learn a lot from experimenting with more common use cases, in order to get a complimentary assessment of the performance of Tract itself. We chose to implement enough operators to make a few popular models run, in order to get a fair comparison of the intrinsic performance of Tract against TensorFlow Lite. Obviously, supporting these operators also widens the field for possible third-party use cases of the library. Today, for instance, Tract can run unmodified pretrained Inception v3, MobileNet, and the acoustic model from DeepSpeech.

We also implemented about 85% of ONNX, initially to take advantage of its extensive test-suite that helps us cover the internal operations it shares with TensorFlow. But now, as a “free” side effect, our machine learning teams can choose to work with TensorFlow or PyTorch on a per-project basis.

On top of ONNX and TensorFlow, we are also considering new importers: we are specifically interested in importing the Kaldi format, as our speech-to-text engine uses it. We are also paying close attention to the NNEF format: despite a large support from hardware and software vendors and a very elegant specification, it has received very little attention from the machine learning community.

How good is Tract ?

There are several answers to this question.

The first angle is “will Tract run this network”? Supporting TensorFlow’s immense operator set is not, and will probably never be a goal for Tract. On the TensorFlow front, we implement missing operators on a per-application basis. As of today, Inception v3, several ARM Keyword spotting networks, Mozilla’s DeepSpeech, MobileNet networks and others run. Adding operations is not difficult. Or at least, not much more so than implementing the computation itself.

On the ONNX front, things are a bit simpler: the operator set has a reasonable size, and comes with a test-suite. We cover about 85% of version 9 of the operator set. Recurring operators are coming soon, and the last remaining flow control features are also on the roadmap. 100% coverage of ONNX is an objective for Tract.

Beyond the perimeter of supported operators, comes the question of performance. Tract does not try to compete on the big-hardware side of things: Snips’ main interest lies in running neural networks on small devices. For instance, we have no support for GPU acceleration but we want to be as efficient as we can on ARM CPUs. We pay attention to devices ranging from ARMv5 or v6 with VFP (like a Raspberry Pi Zero) to the bigger ARMv8 with Extended SIMD instruction sets that can be found in most modern smart phones and may become the workhorse of the IoT industry. We also keep an eye on Intel’s CPUs as it is the native environment for most developers.

On the chart below, we compare Tract and TensorFlow Lite’s performance on two neural network architectures that are supported by both libraries. The first one is an early version of the Snips Wake word detector, which we’ll call v1 for the sake of clarity. We then switched to the very convolutional networks that we prefer today, but which aren’t supported by TensorFlow Lite. Snips Wake word v1 relies on a 1D CNN architecture. The second architecture we consider here is a 2D CNN network from the “Hello Edge” paper by ARM.

On these two examples, Tract outperforms TensorFlow Lite on all devices considered. There is a bigger bonus on the two before-NEON ARM devices: we have found that TensorFlow Lite, as many other players, is neglecting these older and smaller chip variants.

As a matter of fact, Tract also outperforms TensorFlow Lite on these pre-NEON targets for other challenging tasks like image classification. For example, Tract brings a 2.5x speedup on the Inception network on Raspberry Pi Zero. TensorFlow Lite’s performance on this network is harder to match on bigger devices.

In the figure below, we show the performance of Tract on all kinds of voice processing applications (user identification, Wake word, voice commands), including neural network architectures not supported by TensorFlow Lite. This figure shows that a lot can be done on the edge, on tiny devices, with CPU to spare. Gaining in efficiency is like a double-edged sword with only good sides: between the two iterations of our Wake word model, we took advantage of the extra efficiency gained in Tract to run a significantly bigger and more accurate neural network architecture.

Yet another library?

Today, there are comparatively more libraries available off-the-shelf than when we started working on Tract two years ago. Google, ARM, Apple and Microsoft are actively developing their own offerings. While there is overlap between them, each one has its own angle, or operator subset covered, and it does not look like any will be an obvious go-to solution anytime soon. Each one has a bias towards the type of applications and the network architectures each institution is primarily exploring.

Tract brings specific attention to voice processing applications like Wake word detection, voice commands, speaker identification, and soon acoustic modelling, on low- to mid-end CPUs. Anyone working in these areas or other streaming-based applications may be interested in looking at Tract.

As an Artificial Intelligence company, we made the choice to be in control of the software we are deploying in our users’ environment, be it a home, office or factory. Concretely, instead of developing and maintaining Tract, we could have hard-coded every new architecture we adopted, which would probably have refrained us from exploring new architectures. We could also have made massive contributions to TensorFlow Lite to extend its perimeter. For such a critical piece of the stack, our engineering team feels better owning the neural network engine, controlling how it is built, and keeping its performance and size in check.

In future posts, we’ll be diving deeper into some experiments and optimizations we explored with Tract, what worked, and what didn’t. Follow us to keep informed.

The point of Tract is not to directly challenge TensorFlow or PyTorch as a generic go-to solution, but we did learn a lot in the process. Moreover, Tract’s performance is significantly better than that of TensorFlow Lite for the networks we deploy in production. We now have an angle on the problem of running deep models on device, we have built a tremendous amount of experience, we have found a few issues and a few original solutions. We are also open sourcing Tract to support the global effort of moving inference and in particular voice processing on device. We are hoping that sharing code, experience and thoughts will contribute to the landscape converging to more compatibility, and more consistent choices. Only then will we be able to “train once, run anywhere”.

Lastly, we are now making use of this experience to prepare for the next big wave that is coming to the IoT landscape: Neural Processing Units. Semiconductor manufacturers are now bringing dedicated chips that are specialized to run deep learning operations. These chips will literally unleash the potential of what computations can be run on the edge. They will significantly contribute to edge computing becoming even more of a reality. They will naturally bring their share of new questions on what paradigms to follow, what operators and architectures to prioritize, how to serve efficiently all application fields, from vision to voice, etc. This is only the beginning.

Interested in learning more?

Check out our open roles and explore our solutions. Or clone tract on GitHub!

If you liked this article and want to support Snips, please share it!

Follow us on Twitter: @Snips

Snips Open Sources Tract was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Too many remotes? Simplify! Use your voice to control infrared devices around your home using Snips voice platform. A completely on-device assistant that does not share anything it hears, or what it is you do. Privacy by design.

Background

My young 3-year-old likes to hide my TV remote and I can never find it when I need to use it. I already use Snips voice platform in my home, so I started thinking about how I could remove the remotes completely and just ask my home assistant, with my voice, to control my devices. Turns out it was very easy, and this project is so simple, easy to setup, and manage, that you can too.

This project not only benefits me, it also fills a requirement for other users out there. My elderly parents, for example, can never find the right button to press on the TV remote. This also benefits people with disabilities who are not able to manipulate a remote physically, but are able to say to an assistant what they want to do.

Once upon a time… I had 4 remotes, just for the devices in the lounge room alone. A remote each for the TV, amplifier, DVD, and aircon. They are now in the bottom draw, and I just speak aloud what I want to control.

Some basics

This project uses Snips. So a very quick run down of “what is Snips?”. Snips is natural language understanding and speech recognition software. Your interaction with Snips as a developer is as a set of applications that together create an AI assistant and is able to run on a tiny Raspberry Pi 3B or a range of other devices. Snips listens for the wake word to be spoken for the assistant to know you want it to listen to you. It then converts what you say into text (TTS and ASR), then parses the text into meaningful intents (NLU). Applications, such as this project, listen for a specific intent notification from Snips. When they receive an intent, they go to work and “do something”, whatever it is they are programmed to do.

“Hey Snips, turn my Samsung TV on”
’hey snips’ = wake word — start listening to my command
’turn my Samsung TV on’ = Intent — IR Control
Specific things about the intent that are important. (1) ‘Samsung TV’ and (2)‘on’. The IR Control application (this project) sees that you want to use the Samsung TV remote and press the power ON button, so that’s what it does.

There are two parts to this project. The IR Control and the IR Client.

The IR Control is part of a Snips assistant application.

The IR Client uses a RPi Zero with an IR receiver and emitter connected to GPIO pins. This project shows you how build and interface the RPi with an IR receiver and emitter, configure the remotes you use in your home, and then apply this information to your own Snips assistant.

If you don’t yet have a Snips setup, Snips with Seeed provide a Maker Kit to get you up and running with Snips.

For more information and an in-depth look into Snips, check out the Snips Docs.

Building the IR Client

The IR Client uses a small, inexpensive Raspberry Pi Zero W (to use wifi to communicate with the IR Control in the Snips assistant). You can however use any RPi you wish.

Items you will require to build the IR Client:
- Raspberry Pi Zero (W version for wifi connection)
- Infrared Receiver Module — Generic 38kHz
- Breadboard (generic)
- 2x 940nm Infrared Emitters
- General Purpose Transistor NPN (PN2222A will do)
- 10k ohm Resistor
- Jumper wires (3x female-female, 3x female- male, 2x male-male)

Lay them out on a breadboard, as shown in the below images and circuit diagrams.

Follow the layout, paying attention to the Tx and Rx signal pins. 3.3v is used by the IR receiver, and the IR emitters use 5v.

Installing the IR Client software

The IR Client runs a small python application, which auto starts at boot. It listens for instructions from the IR Control in the assistant, and then transmits IR codes via the infrared emitters.

To install the IR Client on the RPi Zero, ssh into the device and run these commands:

git clone https://oziee@bitbucket.org/oziee/irclient.git
cd irclient
sudo bash install.sh

The install.sh script:
- creates a venv environment
- installs the python library requirements
- installs LIRC
- sets up the IR Tx/Rx pin assignment
- creates the service to start the app on boot

Once the install completes, it will prompt you to edit the settings.ini file in the current directory.

nano settings.ini

Enter in your Snips connected MQTT broker address and port, and a unique name for the IR Client device.

Testing the LIRC

You can now test the LIRC install on the IR Client RPi. Use the commands below and a remote control.

sudo /etc/init.d/lircd stop
mode2 --driver default --device /dev/lirc0

Press some buttons on the remote control while it is pointed at the receiver and you should see the RPi console outputting multiple lines like the below:

Adding remote codes to LIRC

Adding your own remote controls yourself is simple.

sudo /etc/init.d/lircd stop
sudo irrecord -d /dev/lirc0 ~/lircd.conf

Follow the on screen instructions. The gist of it is, you enter in a name for the remote you are recording, press random buttons on the remote over and over a few times while LIRC learns and detects pulses, signals, pause timing etc.

You will be prompted to enter in names of the function button to record such as KEY_POWER, KEY_VOLUMEDOWN. LIRC uses capitals and a list of valid values can be found here.

Once you have completed recording your remote, you need to copy the created file to the lirc directory which contains all the conf files. You can do a `ls` to check for the name of the file that was created. It will be something like <what you named the remote>.lirc.conf

sudo cp ~/<name of remote>.lirc.conf /etc/lirc/lircd.conf.d
sudo /etc/init.d/lircd start

Create as many remote conf as you like, making sure they are all uniquely named, and are all copied over to the “/etc/lirc/lircd.conf.d” directory.

LIRC has a repo of remote configurations created by others that can be downloaded and used. Of course, not every remote on earth has been documented and available for download. Check out the repo for a list of remotes that can be downloaded for your use.

LIRC also includes command line tools to search this repo and install them, you can read how to here.

Setting up the IR Control in the Snips assistant

In your Snips assistant console https://console.snips.ai, click “add an app”.

Use the search field to look for “ir control” and then add this app to your assistant:

Once the app is installed, and the Snips skill server has taken control of running the Snips apps, you can log into the IR Control web management page from your browser by going to <url of snips server>:7777 (the default port is 7777, but you can change this to anything from inside the Snips config.ini for the app.

The management page will communicate with all the IR Clients you have built on the network, and gather a list of all the known remotes for each device and their function button mappings.

Here you are able to add common word and names to remotes and buttons which you say out loud to your assistant. Snips will then do its magic to understand what you ask and, in turn, inform the IR Controller of which device, or remote and which button IR code you want transmitted.

In the web page enter in synonyms, or words that describe or name something. In the image example above, it would be hard to ask your assistant in a sentence that you want to use the N2QAYB000352 remote, so entering in a common word or name that you would say in conversation with the assistant makes sense. You can then say “TV” or “Samsung TV” and Snips and the IR Control app will know you mean the N2QAYB000352 remote. Same goes for the function buttons. Saying “on” or “off it knows to use the KEY_POWER button that is mapped.

“Turn the TV on” = N2QAYB000352 | KEY_POWER

Creating Your Own Actions

Actions are a group of remote commands to run in series, but instead of saying each item to preform one at a time to the assistant with your voice, you can say one command to the assistant and all the grouped function buttons of each remote in the action will be performed in sequence.

Asking your assistant, “let’s watch a movie”, the TV will turn on, the amplifier turns on. Yet the amp takes 8 seconds to start before it will react to any other remote commands, so a pause value is set, as in the example image above. A 10-second pause occurs before the next function is performed, which is to set the amp to the TV input.

Check the bitbucket repo README for extra information on the web interface.

Telling Snips assistant your setup

Over in the Intent Builder (selected in the menu) all the properties for all the IR Client devices, each remote, each button, and all the synonyms are displayed in such a way so you can easily copy them over to the Snips Console assistant to fill the slot values for the training data.

One at a time, you’ll copy the items between the *****’s and import it into the corresponding Intents slots in the Snips Console. In the above image the highlighted items are copied and imported to the `ir_control` intent, into the slot button (shown below).

Once all the slots for ir_control and ir_action have been imported, save your assistant, and install the newly updated assistant to your Snips setup.

In future, if you rename a remote, delete a remote, add or remove a remotes function buttons, created new actions — you will need to use the Intent Builder to gather the changes to import into the Snips assistant. You should remove all the current values in the Snips console intent slots before importing new items, as this will remove old items that may no longer be valid.

Other Features

IR Control is also able to start a sequence of button events. If you find the need to open the TV guide on the TV and you want to move around the listings of channels and shows if would be annoying to continuously have to repeat the wake word and say a command over and over. The sequence event solves this as you can just announce function after function.

“Hey Snips, start a sequence for me” — listening for your commands.
“press the TV guide button”
“press up button”
”up”
”up”
”select”

“ok stop listening for commands” — this ends listening. If you do not let the assistant know you want to stop announcing commands, it will just keep listening until you do.

You need to say either the device or the remote once to set a base of where to start. From then on you can just say the button commands to be completed which are mapped to that device or remote. If you want to then start interacting with a different remote, say that remotes name so that IR Control knows you now will be referring to a different remote to operate. When you want to stop using commands, just ask it to stop.

In Action

Quick demo in action. The video was made in a darkened room to demonstrate the IR emitters lightly light up as they transmit.

What next

Design and print a case to hold the RPi Zero and provide IR transmitter receiver mounts.

Start thinking about what you too could automate. The possibilities are endless, limited only by to your imagination. If you can think it, code it, make it, then use it with your own home assistant. Check out the Snips store in the console for many other applications designed by other Snips developers for you to use.

Credits

Thanks to Snips (https://snips.ai) | LIRC team (http://www.lirc.org) | icons8 (https://icons8.com)

Infrared Voice Controlled With Snips was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Snips VP of Legal Marion Bergeret and CTO Joseph Dureau untangle some of the most widely-made claims about voice assistants, privacy, performance, and security.

Nearly a year into the GDPR’s debut, we’re analyzing some of the foundational claims made by cloud-based voice assistant companies along four dimensions: performance, trust, security, and privacy. In so doing, we’ll arm you with the logical tools to dispel the most commonly held misconceptions about the need for cloud-based voice recognition technology, and to demonstrate the power of edge computing and value of Privacy by Design.

Read the full post on the Snips Blog.

If you liked this article and want to support Snips, please share it!

Follow us on Twitter: @Snips

Debunking the Unnecessary Compromise: Users Can Have a Voice Assistant and Their Privacy Too was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

We launched our Partner Program to bring voice-first technologies and products to a wide variety of consumer and industrial markets. The Snips Partner Program builds a network of key Technology and Solutions companies to provide clients with full access to building blocks and capabilities needed to integrate embedded, natural voice recognition into products and services. Read more about our launching partners Smile and DSP Group on the Snips blog, as well as how Snips is enabling access to groundbreaking voice technologies.

Find out more about this Program here and become a partner!

If you liked this article and want to support Snips, please share it!

Follow us on Twitter: @Snips

If you want to add offline and private voice recognition to your products, Let’s Talk!

Snips launches Partner Program to help OEMs bring voice-enabled products to market faster was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introducing the people building Snips, one of the largest AI voice companies in Europe.

Let us reintroduce ourselves. Over the past year, Snips has more than doubled our team size and opened an additional office in Tokyo. Here’s what that momentum looks like in numbers:

— Over 60% of Snips employees are engineers;
— 15% hold PhDs;
— 8 patents filed for different bricks of our technology;
— 25% of our workforce is female.

Read more about each team and some of the rising stars making technology disappear on the Snips blog.

Interested in joining us in Paris?

Check out our open roles and explore our technologies.

If you liked this article and want to support Snips, please share it!

Follow us on Twitter: @Snips

Inside Snips: Meet the team behind the private-by-design voice assistant was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Snips has partnered with Find & Order, a mobile intelligence company, to reinvent the the shopping experience with embedded in-store navigation. Soon you’ll be able to tick items off your shopping list simply by asking where your items are located, alleviating the hassle of searching endlessly for a product in a large store. We recently interviewed Find & Order’s CEO Mickael Carvalho to get his take on the future of retail and why Snips was the best voice solutions partner. Read the full interview and watch our technology in action on the Snips blog.

If you liked this article and want to support Snips, please share it!

Follow us on Twitter: @Snips

If you want to add offline and private voice recognition to your products, Let’s Talk!

Interview with Mickael Carvalho, CEO of Find&Order was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Snips is excited to announce our technology partnership with DSP Group, a partnership that unlocks entirely new possibilities for voice recognition in the local environment and makes even more voice recognition functionality available on the edge. By combining our voice recognition software, running on-device and offline, together with DSP Group we give OEMs a new possibility to create a local grid of mutually-aware devices, lowering the BoM per device and giving opportunities to diversify their product offerings. As our COO Yann Lechelle puts it, “This new voice architecture will be applicable in smart home and smart building verticals where performance, cost effectiveness and data privacy play an important role for end consumers.”

Explore what this partnership unlocks on the Snips blog.

If you liked this article and want to support Snips, please share it!

Follow us on Twitter: @Snips

If you want to add offline and private voice recognition to your products, Let’s Talk!

Snips Partners with DSP Group was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

A guest blog post by Michael Ritchie.

Voice Panel is an open source Android Voice Assistant for Home Assistant powered by the Snips Voice Platform. Snips provides a private, powerful, and customizable voice assistant technology that processes all language input on-device, with no data ever sent to the cloud.

Voice Panel uses Snips to act as a voice interface for Home Assistant. You can control your alarm system, lights, windows, blinds, switches, check status, get the date/time, and retrieve the weather information. Here is a list of supported Snips apps:

Currently the application has a few limitations. The Snips Android SDK does not work as a satellite, so all messages must be forwarded to Home Assistant using MQTT, and speech is processed on the Android device using TTS. The Snips Android SDK does not support custom wake-words at this time. To initiate a conversation, you must say “Hey, Snips” or you can try the face detection wake-word feature which starts listening when the device camera recognizes a face.

Features

• Face activated wake-word (no need to say “Hey, Snips”).
• Control Home Assistant components using voice commands (“Turn on the kitchen lights”).
• Stream video, detect motion, detect faces, and read QR Codes.
• Support for MQTT Alarm Panel Control to control your alarm system.
• MQTT commands to remotely control the application (speak text, play audio, send notifications, alerts, etc.).
• Device sensor data reporting over MQTT (temperature, light, pressure, battery, etc.).
• MQTT Day/Night mode based on the sun value reported from Home Assistant.
• MQTT weather data to display weather reported from Home Assistant.

Installation & Requirements

• Android Device running Android OS 5.0.1 (SDK 21) or greater. Though performance on older devices may vary.
• Google Play services may also be required, this may affect some Amazon Fire Tablet devices.
• Ideally you want to keep your device unlocked and use Daydream as a screensaver, waking the device when you either say the wake-word or with motion or face detection.
• You can download and install the latest release from Github project release section or if you prefer, install the application from the Google Play.

Assistant Setup

You first need to setup a MQTT Broker in Home Assistant by adding the HASS MQTT Broker add-on to Home Assistant. This allows the two-way communication between Home Assistant and Voice Panel using the MQTT messaging protocol.

For the Voice Assistant to control Home Assistant components, you need to install either the Snips component or the Snips Add-On.

Home Assistant already has bundled scripts included when you add the Snips platform to Home Assistant. These scripts will allow you to turn on/off components (lights, switches, etc.), open/close components (garage, blinds, etc.), set the color of lights, add items to your shopping list, and retrieve items from your shopping list. So you can say “Turn on the kitchen lights” to control a the component lights.kitchen.

For Snips you basically need to add snips: to you configuration.yaml file and then add the custom intents you want to work with. The basic intents bundled with Home Assistant will already allow you to work with your HA components.

Custom Assistant Commands

Voice Panel also has custom scripts that extend the basic functionality. Currently Voice Panel can control your MQTT alarm panel in Home Assistant, get the status for any component by name (“What’s the status of the front door?”), get the time/date, get the weather, and control the thermostat.

To use custom scripts you need to create a file to store your intent scripts and customize the assistant’s behavior. Create an “intents.yaml” file in your configuration directory, then link this from the configuration.yaml file by adding this line at the bottom:

intent_script: !include intents.yaml

You will place intent scripts within the new “intents.yaml” file to work with the various components currently supported by application. Home Assistant already has bundled scripts included when you add the Snips platform to Home Assistant. Voice Panel also has custom scripts to extend upon the basic functionality. The additional scripts can be added to your intents yaml file to use these features.

HA Alarm Control

To control a Home Assistant alarm control panel using voice, you need to add the alarm intents scripts to your “intents.yaml” file. This will allow you to set the alarm status, disarm the alarm, and set the alarm to “home” or “away” modes by voice. Here are some sample voice commands:

• “Set the alarm to home.”
• “Disable the alarm.”
• “Set the alarm to away mode.”
• “What’s the status of the alarm?”

Alarm Panel Intents Scripts

HaAlarmStatus:
  speech:
    type: plain
    text: >
      {% if is_state("alarm_control_panel.ha_alarm", "disarmed") -%}
        The alarm is not active.
      {%- endif %}
      {% if is_state("alarm_control_panel.ha_alarm", "armed_home") -%}
        The alarm is in home mode.
      {%- endif %}
      {% if is_state("alarm_control_panel.ha_alarm", "armed_away") -%}
        The alarm is in away mode.
      {%- endif %}
HaAlarmHome:
  speech:
    type: plain
    text: > 
      OK, alarm set to home.
  action:
    - service: alarm_control_panel.alarm_arm_home
      data:
        entity_id: alarm_control_panel.ha_alarm
HaAlarmAway:
  speech:
    type: plain
    text: >
      OK, the alarm set to away, you have 60 seconds to leave.
  action:
    - service: alarm_control_panel.alarm_arm_away
      data:
        entity_id: alarm_control_panel.ha_alarm
HaAlarmDisarm:
  speech:
    type: plain
    text: >
      The alarm has been deactivated.
  action:
    - service: alarm_control_panel.alarm_disarm
      data:
        entity_id: alarm_control_panel.ha_alarm

Weather Updates

If you would like to get weather updates, you will need to add a weather component to Home Assistant. Voice Panel can report the weather by adding a new intent that will speak the current weather conditions to your “intents.yaml” file. Here is a sample voice command:

• “What’s the weather today?”

Example Dark Sky Weather

searchWeatherForecast:
  speech:
    type: plain
    text: >
      The weather is currently 
      {{states('sensor.dark_sky_temperature') | round(0)}} 
      degrees outside and 
      {{states('sensor.dark_sky_daily_summary')}} 
      The high today will be 
      {{ states('sensor.dark_sky_daytime_high_temperature') | round(0)}}
      and 
      {{ states('sensor.dark_sky_hourly_summary')}}

Status Updates

You can get the status of Home Assistant components using voice commands. First you need to add the intent scripts to the “intents.yaml” file to handle script information. The status intent has two slots, one for the entity location (such as kitchen, living room, deck, upstairs, etc.) and one for entity id (door, camera, light, etc.). These slots can be used to provide a specific response for the item status.

• “What’s the status of the main door?”
• “Main door state?”
• “Are the doors secure?”

Example Status Intent Script

getStatus:
  speech:
    type: plain
    text: >
      {% if entity_locale == 'main' and  entity_id == 'door'  %}
        The {{ entity_locale }} {{ entity_id }} is {{states(sensor.main_door)}}
      {% endif %}

In the above example, “entity_locale” and “entity_id” are used to provide a specific sensor status. In this case we asked “what is the status of the main door?” and Snips responds with “The main door is closed”, which is the status of the sensor.main_door entity.

Time/Date

You can get the current time and date from Home Assistant components using voice commands. Add the following intent scripts to the “intents.yaml” file to handle intents for Time/Date.

• “What’s the date?”
• “What day is it?”
• “What time is it?”

Example Status Intent Script

getCurrentTime:
  speech:
    type: plain
    text: > 
      The time is {{ now().hour}}  {{ "%0.02d" | format(now().strftime("%-M") | int) }}
getCurrentDate:
  speech:
    type: plain
    text: > 
      It is {{ now().weekday }}, the {{ now().day }} of {{ now().month }}, {{ now().year }}
getCurrentDay:
  speech:
    type: plain
    text: > 
      Today is { {now().weekday }}

MQTT Alarm Panel Control

The alarm panel can be controlled using only voice, however, it also comes included with a manual way to set and disable the alarm using MQTT messaging. To use this feature, you need to install the Manual Alarm Control Panel with MQTT Support. This component allows for two-way control of the Home Assistant alarm panel component using MQTT messaging.

To enable the MQTT alarm feature, under settings (the gear icon) select Alarm Settings. Once active, you will see a lock icon at the bottom of the main screen which displays the current alarm mode and provide a manual means for arming and disarming the alarm in addition to the voice controls. The Alarm Settings has options to change the MQTT topic and commands if you do not wish to use the defaults.

Supported Command and Publish States

• Command topic: home/alarm/set
• Command payloads: ARM_HOME, ARM_AWAY, DISARM
• Publish topic: home/alarm
• Publish payloads: disarmed, armed_away, armed_home, pending, triggered (armed_night not currently supported).

Example Home Assistant Setup

alarm_control_panel:
  - platform: manual_mqtt
    state_topic: home/alarm
    command_topic: home/alarm/set
    pending_time: 60
    trigger_time: 1800
    disarm_after_trigger: false
    delay_time: 30
    armed_home:
      pending_time: 0
      delay_time: 0
    armed_away:
      pending_time: 60
      delay_time: 30

— If I set the the alarm mode home, the alarm will immediately be on without any pending time. If the alarm is triggered, there will be no pending time before the siren sounds. If the alarm mode is away, I have 60 seconds to leave before the alarm is active and 30 seconds to disarm the alarm when entering.

— Notice that my trigger_time is 1800 and disarm_after_trigger is false, this means the alarm runs for 1800 seconds until it stops and it doesn’t reset after it’s triggered.

MQTT Weather

You can also use MQTT to publish the weather to the Voice Panel application, which it will then display on the main view. To do this you need to set up an automation that publishes a formatted MQTT message on an interval. Then in the application settings, enable the weather feature. Here is a sample automation that uses Dark Sky data to publish an MQTT message:

- id: '1538595661244'
  alias: MQTT Weather
  trigger:
  - minutes: '/15'
    platform: time
  condition: []
  action:
  - data:
      payload_template: '{''weather'':{''summary'':''{{states(''sensor.dark_sky_summary'')}}'',''precipitation'':''{{states(''sensor.dark_sky_precip_probability'')}}'',''icon'':''{{states(''sensor.dark_sky_icon'')}}'',''temperature'':''{{states(''sensor.dark_sky_apparent_temperature'')}}'',''units'':''{{states.sensor.dark_sky_apparent_temperature.attributes.unit_of_measurement}}''}}'
      topic: voicepanel/command
      retain: true
    service: mqtt.publish

The resulting payload will look like this:

{"topic": "voicepanel/command","payload":"{'weather':{'summary':'Partly Cloudy','precipitation':'0','icon':'partly-cloudy-day','temperature':'22.5','units':'°C'}}

MQTT Day/Night Mode

Similar to how weather works, you can control the Voice Panel to display the day or night mode by sending a formatted MQTT message with the sun’s position (above or below the horizon). To do this add the sun component to Home Assistant, then setup an automation to publish an MQTT message on an interval:

- id: '1539017708085'
  alias: MQTT Sun
  trigger:
  - minutes: '/5'
    platform: time
  condition: []
  action:
  - data:
      payload_template: '{''sun'':''{{states(''sun.sun'')}}''}'
      retain: true
      topic: voicepanel/command
    service: mqtt.publish

The resulting payload will look like this:

{
  "payload": "{'sun':'below_horizon'}",
  "topic": "alarmpanel/command"
}

You can also test this from the using the “mqtt.publish” service under the Home Assistant Developer Tools:

{
  "payload_template": "{'sun':'{{states('sun.sun') }}'}",
  "topic": "voicepanel/command"
}

If you wish, you can use an offset to change the day or night mode values or send a MQTT message at the desired time with “above_horizon” to show day mode or “below_horizon” to show night mode. If you wish to always be night, you need only send one MQTT message with “below_horizon” and the app will not switch back to day mode. Be sure to turn on the Day/Night mode under the Display settings in the application.

MQTT Sensor and State Data

If MQTT is properly configured, the application can publish data and states for various device sensors, camera detections, and application states. Each device requires a unique base topic which you set in the MQTT settings, with the default being“voicepanel”. This distinguishes your device if you are running multiple devices.

Device Sensors

The application will post device sensor data per the API description and Sensor Reading Frequency. Currently device sensors for Pressure, Temperature, Light, and Battery Level are published.

Sensor Data

Sensor values are constructed as JSON per the above table and Voice Panel publishes all sensors to MQTT under voicepanel/sensor. Each sensor publishes to a subtopic based on the type of sensor. An example topic would be voicepanel/sensor/battery .

NOTE: Sensor values are device-specific; not all devices will publish all sensor values.

Home Assistant Examples

sensor:
  - platform: mqtt
    state_topic: "voicepanel/sensor/battery"
    name: "Alarm Panel Battery Level"
    unit_of_measurement: "%"
    value_template: '{{ value_json.value }}'
    
 - platform: mqtt
    state_topic: "voicepanel/sensor/temperature"
    name: "Voice Panel Temperature"
    unit_of_measurement: "°C"
    value_template: '{{ value_json.value }}'

  - platform: mqtt
    state_topic: "voicepanel/sensor/light"
    name: "Alarm Panel Light Level"
    unit_of_measurement: "lx"
    value_template: '{{ value_json.value }}'
    
  - platform: mqtt
    state_topic: "voicepanel/sensor/magneticField"
    name: "Alarm Panel Magnetic Field"
    unit_of_measurement: "uT"
    value_template: '{{ value_json.value }}'

  - platform: mqtt
    state_topic: "voicepanel/sensor/pressure"
    name: "Alarm Panel Pressure"
    unit_of_measurement: "hPa"
    value_template: '{{ value_json.value }}'

Motion, Face, and QR Codes Detection

In additional to device sensor data publishing, the application can also publish states for Motion Detection and Face Detection, as well as the data from QR Codes derived from the device camera.

Sensor values are constructed as JSON per the above table and Voice Panel publishes all sensors to MQTT under voicepanel/sensor. Each sensor publishes to a subtopic based on the type of sensor voicepanel/sensor/motion.

Home Assistant Examples

binary_sensor:
  - platform: mqtt
    state_topic: "voicepanel/sensor/motion"
    name: "Motion"
    payload_on: '{"value":true}'
    payload_off: '{"value":false}'
    device_class: motion 
    
binary_sensor:
  - platform: mqtt
    state_topic: "voicepanel/sensor/face"
    name: "Face Detected"
    payload_on: '{"value":true}'
    payload_off: '{"value":false}'
    device_class: motion 
  
sensor:
  - platform: mqtt
    state_topic: "voicepanel/sensor/qrcode"
    name: "QR Code"
    value_template: '{{ value_json.value }}'

Application State Data

The application can also publish state data about the application such as the current dashboard url loaded or the screen state.

State values are presented together as a JSON block {"currentUrl":"http://hasbian:8123/states","screenOn":true}.

Voice Panel publishes a state to topic with a default topic of voicepanel/state.

MQTT Commands

Interact and control the application and device remotely using either MQTT commands, including using your device as an announcer with Google Text-To-Speech. Each device required a unique base topic which you set in the MQTT settings. This distinguishes your device if you are running multiple devices.

Commands

The base topic can be changed in the application settings. Commands are constructed via valid JSON. It is possible to string multiple commands together {"clearCache":true, "relaunch":true}. Voice Panel subscribes to default topic voicepanel/comma with a JSON payload.

Google Text-To-Speach Command

You can send a command using either HTTP or MQTT to have the device speak a message using Google’s Text-To-Speech. Note that the device must be running Android Lollipop or above.

Example format for the message topic and payload:

{"topic":"voicepanel/command", "payload":"{'speak':'Hello!'}"}

MJPEG Video Streaming

Use the device camera as a live MJPEG stream. Just connect to the stream using the device IP address and endpoint. Be sure to turn on the camera streaming options in the settings and set the number of allowed streams and HTTP port number. Note that performance depends upon your device (older devices will be slow).

Example format for browser URL:

http://192.168.1.1:2971/camera/stream

Home Assistant Example:

camera:
  - platform: mjpeg
    mjpeg_url: http://192.168.1.1:2971/camera/stream
    name: Voice Panel Camera

Additional Resources

Snips Voice Platform
Home Assistant
Issues

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A version of this blog post originally appeared on Michael Ritchie’s GitHub Page, available here. To learn about Voice Panel and other android hacks from Michael, follow his GitHub.

A Snips-Operated Home with Android Voice Panel was originally published in Snips Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.