nanoQ (nQ for short) was made for streaming real-time data: audio, video, clicks, sensors — where it’s okay to lose data to get the most up to date messages quicker.

The main goal is instant “fire and forget” publish with a best-effort delivery guarantee and support for many (hundreds, thousands) simultaneous streams.

If you’re “show me the code” kind of programmer, without further due github.com/aigent/nq

Rationale

We firmly believe that good software products were made to solve one specific problem (and get bloated and abused for non-intended purposes only afterward).

At Aigent we use nQ to transfer VoIP audio and signaling streams through our backend infrastructure. Existing message brokers don’t cover this scenario too well. If a service wants to transfer data to another service, a scheme:

Pub → broker → Sub

creates additional roundtrip with increased latency and hardware load.

In some cases, a central broker may become a bottleneck, and especially in mixed workload: fast and slow clients, big and small messages, attempts to scale a broker may result in pain in the neck.

Instead, using nQ publishers connects directly to subscribers.

    /→ Sub1

Pub  → Sub2

    \→ Sub3

This approach is very simplistic but powerful: you can build scalable pipelines of arbitrary complexity. Combine this with Kubernetes or cloud service mesh routing and auto-scaler, and you’re golden.

In the scenario above, service Pub publishes to Sub1, Sub2, and Sub3 simultaneously, imagine services Sub1 and Sub2 are fast but Sub3 is being slow or even inaccessible. What options do we have?

1. Slow down processing for all subscribers and create backpressure.
2. Buffer incoming messages until Sub3 will be able to keep up.
3. Deliver messages to Sub1 and Sub2 at their pace, and drop messages directed to Sub3 — this is what nanoQ does.

This is an easy implementation of graceful degradation — overall pipeline continues to run, despite some services are underperforming or malfunctioning, all with very low maintenance cost (you don’t need to monitor growing queues or chase the slowest component).

It works in our case — we want that received data to be fresh anyway. At Aigent we send hints to call center agents in real-time during a conversation, and to receive a hint about something you spoke about a few minutes ago is not helpful — it’s irritating at best.

How nanoQ works

I wish every software project has one-two paragraph description of its internal kitchen and design decision instead of thousands of pages of marketing material. So here we go.

There are some spectacular failures, trials, and errors behind nanoQ that eventually made it.

In a Nutshell

TCP vs. UDP

You may think that UDP is low latency and “fire and forget” by design, so it has to be an obvious choice. There are two reasons against it: packet loss and congestions.

UDP drops packet on too many occasions: send rate is too high, CPU is busy with another process, a kernel queue is full. We want to drop packets as a last resort when a receiver is inaccessible but we don’t want to deal with cracks, hisses, and missing pieces of audio because a CPU on of the machines was busy.

Congestions are even harder to tackle. Imagine you have a 100Mbit/s network, and you send at the 80Mbit/s rate because, hey, hardware must work for the money. All of a sudden, bandwidth drops to 15Mbit/s. The first question is: how do we know that it dropped? The second question is: what to do? We can not continue to send at the previous rate of 80Mbit/s. In the worst-case case scenario, we’ll have congestive collapse, when the link is available but due to exceeding incoming traffic, no packets can reach the destination before timeout. The worst of all is that bandwidth may return to the previous number of 100Mbit/s but due to the existing congestion destinations may seem unreachable, or data may arrive too broken to be meaningful.

The network protocols built on top of UDP usually have complementary protocols or messages for bandwidth control, jitter avoidance, QoS control, retries on packet loss. For instance, RTP — Real-time Transport Protocol has RTCP — RTP Control Protocol.

These protocols are quite complex. But then — TCP does all of this, and often very efficient on the hardware level.

In our case, we can tolerate hundreds of milliseconds delay, and we want to transfer thousands of streams through the same link so we let TCP to take care of all the heavy lifting, and use all the existing diagnostic tools.

We use TCP/IP, and every stream has its own connection to utilize bandwidth.

Wire Protocol

The protocol has only one message: data transfer, and it’s as simple as:

+--...---+---------...
|<length>| payload ...
+--...---+---------...

The <length> is Varint 1–5 octets depending on the payload size.

Varints are a method of serializing integers using one or more bytes. Smaller numbers take a smaller number of bytes.

Each byte in a varint, except the last byte, has the most significant bit (msb) set — this indicates that there are further bytes to come. The lower 7 bits of each byte are used to store the two’s complement representation of the number in groups of 7 bits, least significant group first.

So a payload of 0–127 bytes long requires only 1 additional byte of metadata.

We use nanoQ in the existing backend infrastructure where integrity checking, routing, encryption are already in place. This minimalistic protocol is easy to implement using almost any programming language.

API v0 and Implementation details

At the very beginning, we decided that nanoQ should be data oblivious: some of our services speak JSON, some — Protobuf or msgpack — whatever works best.

The publisher’s API is:

Publish(ctx context.Context, payload []byte, streamKey interface{}) error

So our payload is simply a slice of bytes. The streamKey defines whether to use an existing connection or to create a new one — every new streamKey is a new connection.

Our subscriber’s API v0 was:

type IncomingMsg struct {
    Payload            []byte
    StreamDescriptor   interface{}
}

Subscribe(ctx context.Context) <-chan IncomingMsg

And it was a no-brainer to use go channels as internal queues. It was chan []byte all the way.

It turned out that this design has several critical flaws.

Let’s start with the publisher.

payload := []byte(“Example payload”)
// We store the message into the internal queue
Publish(ctx, payload, “example”)
payload[3] = 0 // Wait! What?! This is illegal!!!

Because slice is essentially a pointer, we can not just send payload down the channel — user can still modify the contents, and then:

1. it’s a race condition, and
2. we’re transferring spoiled data

So at this point, we had to either explain to every user not to do this, pretty please, or to allocate a copy of each incoming message (guess what we’ve chosen, and write a comment).

From the subscriber’s part, we had the same problem — API promised that every incoming message will be delivered as its own []byte slice, so we had to allocate memory for every incoming packet too.

Is there a problem with allocations? Go is not magical. Nothing is magical. Our job as engineers is to fight magic and to explain how things actually work, a plausible explanation at least — no magic.

Go goes a great length to convince us that garbage collector and runtime don’t exist (and it’s amazingly convincing). But when you allocate here and there a couple hundred thousand times per second, suddenly the garbage collectors would like a word with you.

Oh man, CPU usage was high as a kite.

Redesign and the current API

The pillar of the new design became circular buffer — we used it for internal queues instead of chan []byte.

So that we can read and write using the same pre-allocated memory arena.

Publisher’s API stayed the same:

Publish(ctx context.Context, payload []byte, streamKey interface{}) error

We just copy the contents of the payload into a circular buffer already in the wire protocol format. Background thread will flush the buffer depending on a FlushFrequency option. This and NoDelay (disable or enable Nagle’s algorithm) options control latency vs. bandwidth “slider”.

Subscriber’s API became more io.Reader -like:

Receive(ctx context.Context, p []byte) (payload []byte, stream StreamDescriptor, err error)

The payload is the same slice as p but of the length of the received message.

Having this API, a user can decide whether to reuse the same memory for each incoming message or to allocate.

Built-in Prometheus metrics

This part may seem like an overkill for a library, calling itself minimalistic.
nanoQ can lose data, and we want a user to know if this is the case. The usual way of a program to programmer communication — logging doesn’t work well. If you’re losing thousands of messages per second it’s not the best idea to produce thousands of log lines. And that’s exactly the reason why metrics were invented.

We hope that nanoQ will solve some of your problems too. Fork us on GitHub.

Meet nanoQ — high-performance brokerless Pub/Sub for streaming real-time data with Golang was originally published in Aigent on Medium, where people are continuing the conversation by highlighting and responding to this story.

To get the quality of transcriptions required to train such a model we built an extremely resource-heavy Transcription Process. For a transcription to be defined as Gold Standard, we required that at least four human transcribers came into contact with it. Such a resource-heavy process was an option for us because we have a huge Data Acquisition Team in the Philippines (see here for more information on how we set this team up).

Now, we want to streamline this entire process and utilize our state-of-the-art Speech Model to support our Data Acquisition Team. Ultimately, we hope to reduce the 30.5 active human working hours it takes for one hour of audio to pass through the entire process. This will not only allow us to get transcriptions in a more timely manner, but also free up our human resources to do a wider range of data acquisition tasks, from labelling text to annotating audio.

The Plan

Our current time-consuming Transcription Process looks as follows:

Audio (in our case complete two-channel phone conversations) is passed through this process from left to right.

• First the audio is manually labelled in our Audio Labelling Tool, with unusable audio weaned out and other labels such as emotion and speaking rate added.
• Next, the audio is split into sentence-sized segments by an automated tool.
• The audio is then routed to a human transcriber who transcribes the audio from scratch.
• To avoid bias, the very same audio is then transcribed again from scratch by a different transcriber.
• A human Quality Assurance (QA) round is then completed. An experienced transcriber is presented with both of the previous round transcriptions and has to make a decision on which is correct or make adjustments where both are wrong.
• A final human QA verification round is completed before the audio can be classified as Gold Standard.

For this initial investigation into utilizing Automated Speech Recognition (ASR) to support our Transcription Process, we decided to simply replace one of the most resource-heavy initial transcription rounds (Transcription Round 2) with an automated transcription completed by our Speech Model.

To give our model the ultimate test, we created a fifteen-hour dataset containing audio recordings from a call center program that our Speech Model has not been trained on. Within ten days we had our results.

To evaluate the success of replacing a human transcription round we accounted for a number of quantitative and qualitative metrics. They are as follows:

Transcription Speed

• How is the speed of the overall process affected?
• How is the speed of the specific stages affected?

Transcription Quality

• What is the WER between QA Round 1 and the ASR Transcription Round? How much worse is this WER than that of a human round?
• Is the QA Round 1 quality affected? (this is determined by calculating the WER between QA Round 1 and QA Round 2).

User Observations

• What do the impacted users (particularly the QA Round 1 Team) think of the change? I.e. do they use ASR transcriptions? Or do they just ignore them and focus on the human round?

Transcription Speed

We were certain going into this task, that by substituting one of the two human transcription rounds with a transcription completed by our ASR model that we would see a significant decrease in the overall time taken for audio to be transcribed.

We were, however, less sure about the impact of this on the QA rounds, particularly the first QA round. Would introducing a non-human transcription round make these subsequent QA rounds more difficult and thus increase the time taken to QA?

The below table outlines the time taken for one hour of audio to pass through each of the 5 stages. In the Normal Transcription Process (1 hour) column you can see how long it takes for 1 hour of audio to be transcribed using just human labor. In the ASR Supported Transcription Process (1 hour) column you can see how long it took for 1 hour of audio to be transcribed after incorporating ASR transcriptions.

The results are conclusive and clearly show that by incorporating ASR transcriptions into the process, Aigent will significantly reduce the time taken for audio to pass from non-transcribed to Gold Standard. For each hour of audio we would save over nine hours of manual human work.

However, it should be noted that the time taken to transcribe in QA Round 1 does increase substantially after introducing ASR transcriptions. This is partly because the first QA stage is now harder and requires more effort from these users, who can rely less upon a second round transcription. This spike can probably also be partly explained by the QA Team making observations about the ASR quality while transcribing this dataset. In all likelihood, as the QA Team adapt to this change, the time spent will converge with the Old Transcription Process.

Transcription Quality

When evaluating the benefits of incorporating ASR transcriptions into the Transcription Process it is vital to take into account the impact on transcription quality- i.e. the WER of transcriptions. For instance, if the overall speed of the Transcription Process increases, but we also witness a significant dip in overall transcription quality, then the costs of implementing such a change may outweigh the benefits.

ASR Transcription Quality

We calculate the quality of the transcriptions by comparing the ASR Transcription Round and the Human Transcription Round to the first human Quality Assurance Round.

In the below table, the WER percentage, where the lower the percentage the more accurate the transcription, is displayed. For this 15 hour dataset of 250 calls, you can see the Average WER for a human transcriber and the ASR round. You can also see the Max WER and Min WER to get a grasp of the quality range within and between the two rounds.

In line with expectations, the ASR transcriptions are on average three times less accurate than human transcriptions. The accuracy range (difference between Max and Min WER) of these ASR transcriptions also varies much more than with human transcribers. This is because when a human makes a mistake it generally is a word here or there. The ASR on the other hand, although often accurate, can get whole speech segments incorrect.

Other interesting observations in terms of the quality difference between the human and ASR transcriptions are that:

• on one occasion the ASR Transcription round has a lower WER than the Human Transcription round (12% vs 18%).
• the average difference between the WER score of the ASR and Human transcription is 16.5%. Normally the WER difference between the Round 1 & Round 2 transcriptions is closer to 4%.

QA Round 1 Transcription Quality

As mentioned, we expected the ASR transcription quality to be significantly worse than that of a trained human transcriber, particularly as the Speech Model is not trained on this audio. However, we were unsure how strong the impact of introducing this new round would be on the quality of the first QA Round. It is certainly feasible that the added distraction of this new, less accurate and non-human round will negatively impact the quality of the first QA transcription (which is measured by calculating the WER between this round and the second QA Round).

To evaluate this we compared the WER between the QA Round 1 and QA Round 2 for a normal 15- hour dataset and the WER between the QA Round 1 and QA Round 2 for the dataset which included ASR transcriptions.

The above table shows that the quality output for the first QA Round does not seem to be particularly impacted by the introduction of the ASR Transcription Round. The Average WER does increase slightly but the spike is negligible. Also, the Max WER is actually lower for this the ASR supported dataset and in both datasets the Min WER is 0% (i.e. the QA Round 2 transcription is identical to the QA Round 1 transcription).

User Observations

Lastly, it’s important to look at the observations that the users most affected by this change — the QA Round 1 Team — noted while correcting the transcriptions.

As the higher WER would imply, the team pointed out that they were much less likely to select the ASR transcription than the human round. They reported that for audios with minimal background noise and clear speech the ASR transcription was more accurate and thus more helpful. But when the audio became noisier or the speaker had a stronger accent, the Speech Model struggled. On a number of these occasions the Speech Model produced truncated transcriptions, missing large parts of speaker turns.

As a result of this a number of the QA Round 1 Team admitted that they lost confidence in the ASR transcription. Many began ignoring this round’s transcription altogether, making the conscious decision to simply focus on the human round and their own judgement. Our user behavior metrics confirmed that users were considerably more likely to click on the human transcription round and correct it, even when both rounds contained similar transcriptions.

Next Steps

By simply examining the hard facts from the above investigation it would seem obvious that permanently replacing one of the resource-heavy human transcription rounds with an ASR transcription round would be a logical next step. Such a move would shred over nine hours of human labor from the overall transcription speed and have little detriment to the overall transcription quality output.

But, these quantitative metrics fail to represent the full story. The observations from our QA Team suggest that these improvements are more a consequence of removing the human transcription round, rather than a positive consequence of adding an ASR transcription round.

Due to these observations we have decided to investigate a further shake-up of our Transcription Process. Rather than directly replacing a human transcription with an ASR transcription, we want to cease manually transcribing audio from scratch and alternatively have a human correct an initial ASR transcription. Image 2 outlines how this future flow could look.

We expect such a change to have a profoundly greater impact on our Transcription Process. Further reducing the time taken for Aigent’s Speech Team to get the vast amount of transcriptions required to train the model.

Nevertheless, there are certainly risks that such a method will cement and exasperate the model’s inherent bias. Potentially, transcribers will subconsciously fail to correct some of the model’s errors, which will then be fed back into the model through its training.

We look forward to finding out and sharing the results with you!

Using Automated Speech Recognition to Support Human Transcriptions was originally published in Aigent on Medium, where people are continuing the conversation by highlighting and responding to this story.

The recent pandemic has changed this overnight. Leaders have been forced to transition large parts of their workforce away from the office and to their own homes. This rapid transition is causing sleepless nights for executives, who first had to make this possible by setting up their new remote workforce and now have the arguably more difficult task of maintaining current service standards.

After speaking to a number of industry experts, from executives to team leads and to agents themselves, we uncovered some of their main concerns. These range from the suitability of the home environment to questions of security and compliance. Aigent’s state-of-the-art speech recognition can alleviate many of the challenges related to both of these concerns.

Security & Compliance

Transitioning to WFH has led to a lot of executives and team leads feeling as though they’ve lost the oversight of their agents which they normally had in the office. Without colleagues and supervisors in earshot, agents may be more willing to go off-script, cut corners in the verification process or even worse, ask the customer for sensitive information which they do not need.

Aigent analyses calls in real-time and immediately detects when an agent goes off-script. For instance, if an agent asks for a customer’s card expiry date, Aigent will send a warning message to the agent. This call will also be flagged in our Client Dashboard for team leads to quickly monitor and take necessary action.

This in-call guidance also helps with less critical, but more common compliance issues, such as agents leaving customers on hold for too long or failing to offer a client survey at the end of the call. Many of these problems existed previously but have proliferated since agents have begun WFH. In these cases, Aigent will send agents reminder messages if they fail to follow the program’s policy.

These compliance features compliment Aigent’s guidance features, which provide agents with tips on how to solve a customer’s concern in real time. For example, if a caller is requesting for a password reset to be done on their account, Aigent will send the agent a message with step-by-step by instructions on how to do this. This improves call-handle time and first-call-resolution metrics, therefore helping to address a range of customer challenges in a more timely manner.

By combining these two methods, Aigent can address the root cause of many compliance problems and maintain a safety net for edge cases.

Suitable Working Environment

While working from home serves convenience, many contact center agents are worried about this setting, especially those who live with others or in a noisy neighborhood. Can the customer hear the kids in the other room or the dog barking outside? Aigent’s background noise detector uses acoustics to determine the level of background noise stemming from the agent’s line and gives agents tips on how to decrease its impact on the call. When background noise levels creep up, we remind agents to “adjust their headset speaker”, or if the noise gets too disruptive, to “find somewhere quieter”.

Junior agents also expressed increased concerns about transferring calls to supervisors who they no longer have regular personal interaction with. Aigent’s Regulatory Complaint Prevention feature determines when an agent is in danger of losing control of a conversation and should transfer the customer to their supervisor. Ultimately, we help provide confidence that Aigent is here to help, 24/7.

The switch to remote working is forcing the contact center industry to rethink their operations and take drastic measures to continue providing critical service to their customers. Aigent offers solutions through the lens of change and uncertainty.

Contact us to learn more about building and maintaining a successful work from home environment.

Working from home? Aigent is here to help was originally published in Aigent on Medium, where people are continuing the conversation by highlighting and responding to this story.

Makefiles are quite unpopular in the dynamic languages world. Some Python and JavaScript developers consider GNU make an ancient, deprecated, outdated, and dying tool, used by some dinosaurs.

The truth is: make is a wonderful and often misunderstood tool. It is fairly simple yet very powerful. Hell, source-based Linux distros and BSD operating systems heavily rely on make.

There are many application for Makefiles outside of C/C++ world, and I think that make deserves more attention and love.

Why does Python even need a Makefile?

The short answer is: to avoid the “It worked on my machine” situation. Python does a very good job to abstract OS layer, although Python application still needs some kind of packaging before it can be distributed, or deployed to a server.

In general, Makefile is a good alternative to a bunch of bash scripts used to automate tedious tasks.

First of all, instead of remembering and typing ill ./do_this_one_thing.sh ./do_the_other_one (was it dash or underscore?), you can confidently strike: make one two.

But that’s a joke, of course.

The real reason is: as soon as there is more than one bash script, the hidden dependencies may arrive. You have to run script_1 before running script_2. Another problem is copy-paste code (each bash file tends to hold the same paths, artifacts names, and whatnot).

Make for the rescue

Feel free to skip this intro if you’re already familiar with make.

All make does is tracks files’ dependency graph, and “last modified” timestamps of the files. If one of dependencies timestamp is greater than the one of the original file, the original file needs to be rebuilt.

Every time, one file has to be built from another, make is a good alternative for the job. You may think of building a .jpg from .dot files, or .mpeg4 from video, audio and subtitles files, or .html files build from .j2 or .pug templates. Python itself does the very same thing with .pyc files.

A Makefile consists of “rules” in the following form:

target: dependency1, dependency2, …
<TAB>recipe line
<TAB>recipe line
…

target is a file needs to be built; dependency1 and dependency2 are source files used to build the target, and recipe is a shell command used as a build step.

If you want to just copy a file, you may write something like:

dist/config.yaml: src/config.yaml
    cp -f src/config.yaml dist/config.yaml

What is all this have to do with Python again?

Okay, let us make a simple application.

Let’s say we want to make an application that shows random cat gifs (the example is shamelessly hijacked from the glorious Elm tutorial)

So now you want to show this application to your buddies. No — the whole World must to see it in its magnificence.

The application should render something like this:

The only problem is:

Traceback (most recent call last):
 File “./app.py”, line 3, in <module>
 import requests
 ModuleNotFoundError: No module named ‘requests’

Okay, apparently simple python3 app.py may not work.

Let’s make a virtual environment, and install requests module in it. requirements.txt file with the long list of all the requirements:

requests

Long, long list.

(Trying to pull out real-world example out of the thin air here).

And the virtual environment itself:

$ python3 -m venv venv
$ ./venv/bin/pip install -r requirements.txt

We may even want to save this into some build.sh file.

Now all we need is:

$ ./venv/bin/python3 app.py

And we also may want to save this one-liner into some run.sh.

So this is the very common situation I was talking about in the beginning: we need to run build.sh before run.sh, and this dependency is hidden and nowhere stated.

Let’s see if this make stuff will be more clear.

The first rule in our Makefile will be the virtual environment, but “venv” is a directory, and we need a plain file, so let’s take any file from the “ venv”. For instance, “venv/bin/activate” will do:

venv/bin/activate: requirements.txt
    python3 -m venv venv
    ./venv/bin/pip install -r requirements.txt

Now if we run make in the project’s directory, we’ll get virtual environment build with requests installed. We had it before, with bash, but now dependency is tracked, so running make the second time will give us:

$ make
make: `venv/bin/activate’ is up to date.

If we will edit requirements.txt — add or remove dependencies, make will rebuild the virtual environment for us automagically:

$ touch requirements.txt  # simulate edit
$ make
python3 -m venv venv
./venv/bin/pip install -r requirements.txt
Requirement already satisfied: requests in ./venv/lib/python3.7/site-packages…

Now, let’s make the second target called run to run our application:

run: venv/bin/activate
    ./venv/bin/python3 app.py

run explicitly depends on the virtual environment target, so now you only have to type make run and dependencies will be created or updated before running the application.

So far so good, but we need to go deeper.

First of all, we don’t have the file named run; make thinks that this file will be created after executing the recipe, so it all works. Although, if someone would decide to created a file named run the behavior of make would become unexpected.

Let’s fix it by explicitly saying that we don’t expect a file to be produced.

.PHONY: run

Now run is a phony target, and it will be re-evaluated every time it is called (that’s exactly what we need).

We may want to add the clean target to remove what we don’t need:

clean:
    rm -rf venv
    find . -type f -name ‘*.pyc’ -delete

Don’t forget that the clean target also doesn’t produce a file:

.PHONY: run clean

By default, running make without arguments will execute the first rule from the Makefile. It is considered a good style to define an entry target called all that should define a default behavior. In our case, it will create the virtual environment.

all: venv/bin/activate

And again:

.PHONY: all run clean

Want more make?

This is where some people draw the line in the sand, but make also has variables, functions, and macros.

Variables are really simple:

NAME := VALUE

defines a variable, that can be refered by ${NAME} or $(NAME)

There is also some useful shortcuts:

NAME ?= VALUE

Assign VALUE to the variable NAME if it’s not yet defined.

This is especially useful because you may define variables while executing make:

$ make BUILD=RELEASE ENVIRONMENT=PRODUCTION

There is also a possibility to declare an alias, like this:

NAME = VALUE

In this case, right hand side will be evaluated every time variable is accessed.

This can be explained easily by the following example

v1 := $(shell ls)
v2 = $(shell ls)

Both variables hold the result of the shell ls command execution, but v1 will be evaluated only once, and will never show any new files.

By the way, shell is a make functions. Hooray, we’ve also learned the make functions!

Let’s wrap up everything we’ve learned just now:

Easy, right?

What else?

You may want to add your favorite linter and formatter: pep8, yapf, pyflakes — you name it. You can generate gRPC or swagger stub, you can generate documentation from .rst or .md files.

A shiny new hammer

Why not use make for everything? Literally everything?!

It is very good tool for tracking dependencies between files with benefits, but when targets are dynamics, like docker images, or deployments… Well, things may get ugly and unpredictable. Builds may be haunted.

TL;DR

Makefiles still rocks in 2020! Feel free to use make outside of the C/C++ world having fun joy and pleasure.

Every time a file needs to be created from other files, GNU make is a good candidate for the job.

Makefiles for Python and beyond was originally published in Aigent on Medium, where people are continuing the conversation by highlighting and responding to this story.

In order to create features which make the above possible we needed to train our own Automatic Speech Recognition system. As anyone in the field will testify these features and products require a lot of data. This data comes in the form of transcriptions (in our case the literal verbatim transcription of speech into text) and annotations (i.e. the labelling of both audio and text).

This blog post presents the reader with a comprehensive overview of Aigent’s evolving data acquisition process. Detailing how we got to where we are today and offering a glimpse of what future steps we expect to introduce.

The Journey

Conveniently, I find myself writing this blog post on my way home from Bacolod, Philippines after visiting Aigent’s main data acquisition source. This is the third time I have visited the aptly nicknamed “City of Smiles”, located on Negros Island, in the Western Pacific. Since the first time I visited Bacolod in 2018 a lot has changed. In terms of numbers, the program has witnessed exponential growth. From the eleven employees (ten Transcribers and one Team Lead) that we had in November 2018 we now have forty-eight Annotators, three Team Leads and one Operations Manager.

To accommodate this growth, Aigent has relocated to a bigger office in a new building (with plenty of room to accommodate further growth). In this large space, the team’s identity can really be observed. Aigent posters line the wall and Annotators who have recently been rewarded for their performance, show off their new Aigent t-shirts and coffee mugs.

However, before we established our own Data Acquisition Team, we pursued a number of other data sourcing methods.

Method 1: Online Datasets

We utilized online datasets to satisfy our search for data, such as the extensive Switchboard Corpus. These datasets, due to their high word accuracy, have certainly been pivotal in the evaluation of our speech models.

But the lack of domain bias in these datasets limits their use when it comes to training a model for our specific business context.

Method 2: Limited Internal Drive

When it comes to raw data, we found ourselves in the unique and advantageous position of having access to an unlimited pool of domain specific audio files. However, we spent the best part of a year attempting to find a reliable, effective and cost-efficient way of converting this raw audio into usable transcriptions and annotations.

Our initial efforts to solve this problem led us to employ a small number of part-time students. Aigent’s prime location in Amsterdam, makes it relatively straightforward to attract native English-speaking students looking for part-time work to support their studies and gain work experience. These students certainly helped us create and define our own data acquisition process. The fact that these students worked in the office allowed developers to quickly test out new transcription tools and get instant feedback. Nevertheless, the nature of part-time work meant this couldn’t provide us the quantity of data needed to train our models. Also in terms of return, it is a relatively expensive data acquisition method.

Method 3: Outsourcing

The third method we explored involved outsourcing this transcription task to a third-party. This multinational company was contracted to provide us 100 hours of high-quality transcription, using our client’s audio. Undoubtedly, these transcriptions will prove to be very useful in the development of our domain biased speech model.

But to yield 100 hours of transcription a lot of work has been logged on our own end. For instance, the pre-processing of the initial datasets, to align with their particular format, took weeks of work.

On the return of the transcriptions we also found that the quality did not meet the expected standard and guidelines were only loosely adhered to. Thus, these transcriptions required an internal quality check from our side. Due to this quality lapse, some data was even returned to the third-party to be transcribed again.

Essentially, the outsourcing of these transcriptions involved a significant amount of insourcing. Yet it was this insourcing that motivated us to seek alternative solutions. The work logged here, laid the groundwork and developed the internal infrastructure that allowed Aigent to realistically pursue the following method of establishing our own, permanent, Data Acquisition Team.

Method 4: Large-scale Internal

The above experiences, as well as our insider access to Ubiquity’s (Aigent’s parent company and a large US call center) resources, led us to make the leap of establishing the Data Acquisition Team in Bacolod in November of 2018. Thus far, having our own full-time team has proved by some distance the most successful way of satisfying our data deficiency. Having our own large team means we can quickly gather domain specific transcriptions and annotations, without sacrificing direct control over the process.

In the first twelve months, roughly 3000 hours of audio were transcribed by the team in Bacolod. To put this in perspective, these figures not only dwarf what our part-time students produced, but also the third-party company. In half the time it took us to receive 100 hours of transcriptions from the third-party, our Data Acquisition Team in Bacolod have produced over ten times as many transcriptions.

Challenges

This is not to say that having our own team has magically solved all of our data sourcing problems. The balance between transcription quality and transcription quantity, is one which still defines our process. However, the direct control we have over a large team allows us to quickly implement changes, adapt our processes accordingly and witness instant results. As mentioned above, this flexibility is absent when it’s outsourced to a third-party.

It would be disingenuous not to mention the logistical obstacles in running an annotation center on the other side of the world. Unlike many careers in the Philippines, Aigent offers its employees a day-time shift. This six to seven-hour time difference, depending on the time of the year, means there is just a two to three-hour window where the Amsterdam team and the Bacolod team are both working. Therefore, technical hiccups can result in a few hours of lost work.

Regular formal and informal communication channels help to address many of the above challenges. In fact, these logistical problems have been noticeably less damaging than our initial estimates suggested.

There are also a number of benefits associated with the time difference. The time delay allows us to differentiate our processes. Meaning our team in Bacolod can work independently of our Amsterdam office and vice versa. This space facilitates positive change as the Amsterdam office can analyse the work of the annotators and have clear concrete feedback ready for when the Bacolod team begin working again the following day.

A good example of this time difference working to our benefit is in regards to a pilot we are currently running with one of our US based clients. This client takes calls during the US daytime, and by the time they finish receiving calls our Bacolod team arrive to their office to begin their daytime shift. A sub-section of the team then annotate all the calls from the previous day, usually 1,200 recordings, which have been automatically placed in a purpose-built tool using a script developed by one of the developers here in Amsterdam. By the time the Amsterdam team arrive to the office, they can observe these annotations and, if needed, have the necessary time to make adjustments to the model or process. These changes are implemented before the client starts taking calls. This process is repeated daily and allows for clear and accurate analysis which results in positive change.

Similarly, the threat of outages has led to improvements in internal troubleshooting steps, the creation of monitoring logs and driven general tool enhancements. Without the threat of outages and lost working time, it is arguable that these permanent solutions would have been sidelined in the place of more ad-hoc workarounds.

Finally, these sociable working hours help to maintain a positive working environment in Bacolod and ensures that Aigent remains an attractive employment option. These unseen benefits are by their nature difficult to measure, but their importance shouldn’t be underestimated.

Mission Accomplished?

The first fourteen months of the program have been a success. The target of establishing a sizeable skilled and motivated workforce has been completed. As a result of this, Aigent now has a stable, efficient and scalable way of converting raw audio into usable transcriptions and annotations.

Looking ahead, 2020 will be a big year for Aigent and the Data Acquisition Team. The reality of onboarding and integrating multiple new clients will test the speed that raw data can be transformed into usable data.

Yet, we’re confident that Aigent’s infrastructure and personnel can handle this pressure. Bring on 2020.

Building a Data Acquisition Team from Scratch was originally published in Aigent on Medium, where people are continuing the conversation by highlighting and responding to this story.