When I first started programming I spent a lot of time scraping the internet for tips and tricks to make me a better programmer. I thought that if I could learn as much as possible from others, the less mistakes I would have to make myself.

Over the last decade I have learned a lot, so I’ve compiled a list of tips and tricks that I want to share with all of you.

Use Variables for Constants

When working on a team, or coming back to something you wrote months later is always a challenge. By taking advantage of variables and naming them properly, you can save yourself some headaches. This one is pretty simple. Anytime you are about to use something that is constant, make it a variable. The reason for this is it helps you to avoid mistakes and makes your code more maintainable. Don’t worry about memory, a lot of languages are smart enough to optimize the usage. Here’s an example:

Instead of:

balance = 1000 – 50 – 400 – 200

Do:

income = 1000
internet = 50
savings = 400
car_payment = 200
balance = income - internet - savings - car_payment

Instead of:

if queue.size() > 10:

Do:

MAX_QUEUE_SIZE = 10

if queue.size() > MAX_QUEUE_SIZE:

By not repeating constants, the code is easier to maintain, easier to read, and less prone to bugs; especially as your code grows. Instead of having to go through all of your files and find & replace a constant, you can store it in one place and change it when you need to. When someone else who is less familiar with your code comes along, they can read it and understand it faster. If you do have to change the value, you don’t have to worry that you missed a place and introduced a bug.

Don’t Over Comment

Comments are important. They can help programmers navigate through code and add to documentation. However, they can also be a maintenance nightmare.

Imagine you come across a function that appears to do one thing, but has a comment that describes something completely different. Which one is correct? Which one wasn’t updated properly? To avoid this, don’t write comments that can be better said in code. Comments should provide insight into things you can’t see in the code, not describe the code. They should provide new information, not just repeat what the code is already telling you. For example describing the intent of the function (not what it does, but what the goal is), or writing a reason for doing something a certain way.

Don’t:

# Commit session to database
db.session.commit()

Do:

# We need to commit at this point to avoid a race condition with
# another thread that is also trying to access the database.
db.session.commit()

Learn Keyboard Shortcuts

The faster you are at getting things done, the better (unless you’re just putting out garbage, then that’s too fast). Especially in a business setting. Remember, people don’t pay for code they pay for the value the software provides. The only person who will really appreciate your code is you and your development team. So the faster you can write code, the faster you can test it, and the faster you can ship it.

By learning and using keyboard shortcuts, your workflow will improve as you don’t have to spend time moving the mouse around or doing things in five steps that can be done in one. However let me be clear, this does not mean to sacrifice code quality for speed. That is a balance and is a different topic, this is just about improving your productivity so you can focus more on writing quality code and less on other supplemental tasks.

Automate, Automate, Automate

Before I go further, you should not automate every little thing. Think about the task you are doing and see if the amount of time it will take to automate will give you a positive return. There is no sense in automating a task you will only do once. If you’re doing a task, and you know you (or others) will be doing it multiple times in the future, or if you (or others) find yourself doing the same things over and over it might be worth automating. Some examples include setting up a development environment, building your software, testing/running software, or running the same commands over and over.

Use Functions Liberally

Use functions to make your code more readable. Initially you might think that having a bunch of function calls everywhere would impact performance, and you would be right if we were still in the days of dumb compilers and little to no compile time optimizations. In modern times this is not much of a concern except for in the most extreme cases. For 99% of the time, the pros of using a lot of little functions outweigh the cons.

By using functions liberally, you can better adhere to the single purpose principle and really simplify your functions. This makes for easier-to-understand logic, better readability, better traceability, and better testability. Trying to follow the flow of a 200 line function with 7 nested for-loops inside 16 different if-else cases with 3 try-catch statements is awful. If you do this, stop.

Functions aren’t just for the DRY (Don’t Repeat Yourself) principle. You can use them for code that you will only ever use once and that’s totally OK. This has the added benefit that if you ever do need to use it again, you’ve already set yourself up nice to do so.

However, as with most things don’t go overboard and make more functions than you really need. If you’re starting to spread your code across many directories and files you might be going too far. The point of this tip is to help prevent spaghetti code, not make it worse.

Use the Language

In today’s world most popular languages are very feature-full and have been worked on for some time. This means that a lot of common use cases and tasks have been considered and the standard library or language syntax comes with these things baked in. Knowing what the language you use can do is very important to increasing your productivity and doing things the best way possible.

For instance, most languages today come with sorting algorithms built in. Before you go off and start to write a bubble sort or heap sort algorithm yourself, check if your language already has an algorithm that will solve your problem. For example in Python there are two built-in ways to sort a list: using the sorted() function and the list.sort() function. They both use the Timsort algorithm, which performs fairly well on most real-world data. Obviously in this example there may be cases where you want to use a very specific algorithm, but for most of the time when you don’t care how the sorting happens it will suffice.

Using the language is even more important if you’re considering pulling in third party libraries. If can you accomplish your task using what’s available to you in the standard libraries, it will help keep your project small, simple, and stable. The less code you have to manage the better.

Write Automated Tests

Are you writing unit tests? Integration tests? Any kind of tests? If you are, I hope they are automated. If they aren’t, or you aren’t writing them, you need to start. Automated tests are the best defense against introducing bugs and breaking things. Anything going into production or that will be used by others who expect it to work, needs automated tests.

There are tons of things out there on how to write unit tests, integration tests, etc. so I won’t go into all of that as there are books written on the subject, but I will try to convince you that it is worth your time.

How many times have you modified code either to integrate a new feature or to refactor existing code? How many times have you done this and found out that your changes broke something in some other part of the code? Now imagine that you didn’t catch this but one of your customers did and it caused them to lose their customers. That’s not a very fun place to be in now is it? How nice would it have been to catch that break before you rolled out your changes to production? This is where automated tests come in.

If you have any sort of development pipeline, automated tests should be a part of it. For example where I work any time we build our software or push it to our repository, all of our tests are ran to ensure that the software still works. If it doesn’t pass a test, it can’t be merged into the baseline.

Obviously getting perfect code coverage (testing every little thing for every possible case) is practically impossible. That’s why you should focus on the main goals of your software and always test that the most important things work every time, at least.

By using automated tests your software will be more reliable, easier to maintain, and of higher quality. Yes there is a cost included by spending time writing tests, but the time you save in the long run by not debugging complex edge cases and fixing them is more than worth it.

Simple and Explicit Over Clever

Many times, people who are new to programming will get excited and try to find a way to do complex things in as little lines of codes as possible. Usually the less code the better, but this like all things can be easily abused and done incorrectly.

Often when you try to do complex things in a small amount of code, you’re making some trade offs, usually readability. By compressing a lot of logic down you run the risk of making your code hard to understand and follow at first glance. This can lead to a lot of issues when working in teams or when you revisit code later on.

Your code should accomplish the goal in the simplest way possible such that it is easy to understand and easy to maintain by not only yourself but your team. Even if this means turning that one cool little one liner into multiple lines, it may be worth it. Code is executed by computers but is read by humans; not only that but code is almost always read more than it is written, so optimize for the common case.

Don’t:

return val1 if (val2 if (reduce(lambda x, y : x + y, my_list) <= 10) else val3) % 2 == 0 else val4

Do:

def foo(my_list):
  MAX_SUM = 10
  sum = reduce(lambda x, y : x + y, my_list)
  val_to_check = val2 if sum <= MAX_SUM else val3
  is_even = val_to_check % 2
  return val1 if is_even else val4

Don’t Solve a Problem You Don’t Have

Once you build up confidence in your ability to program, you’ll come across things that you will think need solving. For example you’ll be writing a function for a specific reason, but you’ll think “oh if I write it this other way instead we can also use it for this one case that we may or may not come across in the future, but if we do come across it it’ll work!”. Or perhaps you’ll be designing a class and decide to add multiple different kinds of constructors “just in case we need them”. Welp, just a heads up you will probably never need them.

It can be tempting to try to solve problems in advance, and sometimes that is worth it, BUT only when you know for sure that you will need to do those things. If there is any uncertainty about it, you’re better off just not doing it now. If you write your code with the open-closed principle in mind, it shouldn’t be a big deal to add things later when you actually need them.

“Move fast and break things”

“Move fast and break things”, once the motto of Facebook CEO Mark Zuckerberg, really is good advice. When developing and learning don’t be afraid of breaking things. The best way to learn is to fail fast and correct. You can spend all day thinking about “will this work” or you can jump straight to the answer and try it.

If your development environment is setup appropriately, you won’t have to worry about impacting others while you break everything (do NOT try this in production, you will quickly regret it). So give things a shot and see what happens.

Learn Your Tools

As a programmer you will be spending a lot of time using other software. IDEs, terminal shells, SSH clients, etc. are all things you will eventually come across. When you do, you should be prepared to learn how to use them.

If you use something like Eclipse for your development, you should take the time to learn about its features and capabilities beyond just using it as a text editor. You don’t have to use everything, but you should learn what tools are available to you.

As another example, do you use PuTTY everyday to SSH into a machine? Are you typing your user password every time? Well you should really learn how to configure PuTTY to use a SSH key instead (here is a link incase you’re curious).

If you take some time to learn your tools, I promise you will have a better experience and see productivity gains as you learn.

Programming Languages are aTool in a Toolbox, not the Toolbox

Often times I see people asking questions like “should I learn language X or Y?” or saying things like “language A is way better than language B, why would you ever use that?”

Well let me tell you a secret, saying things like the above is kind of pointless. Languages are a tool in your toolbox, not your toolbox. You should know multiple languages and more importantly know when to use one over the other. They were all designed to solve different problems and thus have different strengths and weaknesses.

Need your software to execute safety-critical tasks in a real-time environment? Well you probably don’t want to use Python for that then. Want to write a quick script to automate something trivial? Python is probably a good answer in this case.

If you are just starting off and are trying to decide what languages to learn, pick languages that solve different problems. C++ for performance applications, Python for scripting, JS/HTML/CSS for web development, SQL for databases, just to give some examples.

Learn a CLI Editor

Inevitably at some point you will be forced to work in an environment where you don’t have access to a normal GUI. For instance if you have to SSH into a linux server to do your work.

In situations like the one mentioned above, it helps to know how to work in a Command Line Interface (CLI) editor such as Vim, Emacs, Nano, etc. Picking one is not super important, as long as it’s fairly common and you learn it well enough to get around.

Personally I would recommend Vim because it is very popular and comes standard, or is at least easily accessible via package managers like Yum.

By knowing how to use a CLI editor, you not only know a new tool but you will be able to work in nearly any environment and be productive.

While going through some of my documents, I came across a paper I was a part of in college that I thought would be perfect to share. For context, this paper was written for one of my classes where we had a project sponsored by a company. In my case, I worked with Matthew Bussing, Kai Nichols, and Sidney Johnson on a project sponsored by ProKarma to try to automate customer satisfaction surveys. This paper was originally published on our course website here. It has been adapted to fit this format and the original is embedded at the end of the article. Enjoy!

INTRODUCTION

Natural language processing (NLP) is a broad discipline concerning the problems of capturing, analyzing, and interpreting human
language. In particular, NLP can be used to evaluate human speech
in order to determine different qualities about its speaker, context,
and sentiment. These evaluations can be used to inform human-computer interaction, accessibility, automation, and statistical analysis of natural language.

This NLP project was proposed by the Edge Intelligence group at
ProKarma, an IT solutions company headquartered in Nebraska offering digital transformation and analytics consultancy. ProKarma’s
Edge Intelligence group is researching ways to improve customer
service by incorporating machine learning to augment customer
service calls in the telecommunications industry. One approach to
improve customer service satisfaction includes determining customer attitudes toward particular products and by performing sentiment and topic analysis on customer call data, effectively allowing for automated customer service surveys.

Automating customer satisfaction data collection would provide
actionable information about consumer attitudes toward products
and services at much higher response levels than simply relying on
on customers staying on the line to finish a phone survey. This automated system could run parallel to the traditional phone surveys
currently in place at many wireless carriers. Automated sentiment
analysis on products mentioned during the call could be validated
against instances where the customer finished the post-call phone
survey.

Because of the huge number of variables involved in natural language, traditional analytic methods would be prohibitively difficult
to adapt to this purpose. Machine learning algorithms allow simple linear interpolation of audio data as well as powerful pattern
finding tools. For this reason, neural networks were developed for
both topic and sentiment classification. These models find patterns
between product types and consumer attitude which provides the
information required to generate automated customer satisfaction
surveys from customer call audio.

METHODS

Three models were required to accomplish the ultimate goal of an
automated survey process: speech-to-text transcription, sentiment
modeling, and topic modeling.

• Speech-to-text transcription: In order to automate transcription of call data, a strategy for automated speech recognition (ASR) was required. An accurate and high-performance implementation of ASR was crucial in order to perform later topic and sentiment analyses, which take plain-language transcripts as input. Transcripts were preferable to pure audio as inputs for two reasons: firstly, topic modeling cannot be performed directly on audio, as topics are strings which must be pulled directly or interpreted from within a text source; secondly, audio files can be useful in detecting vocalized emotion but cannot detect unemotional sentiment (such as a monotonous customer saying “thank you for your help” or a bubbly customer happily saying “my phone won’t turn on”).
• Sentiment model: As sentiment analysis is important to determine customer satisfaction, it was required to develop a machine learning model to detect sentiment (positive, negative, or neutral) towards the customer service representative based on transcribed text input. ProKarma’s Edge Intelligence group previously implemented a bi-directional long short-term memory recurrent neural network (LSTM RNN) trained using open-source datasets such as the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database in order to perform emotion and sentiment classification on acoustics features extracted from audio clips. During this project, our transcript based model is intended to capture overall sentiment rather than from vocal features like the previous acoustic-based model captured.
• Topic Model: To cross-analyze customer sentiment with topics of conversation, classification models were produced to identify concepts and specific subjects in transcripts. Trends in conversational topics can be used to understand both individual customer attitudes toward products and services as well as to understand those of the general customer base.

These models culminated in an engine for end-to-end audio analysis of customer call recordings. All project code was written using
Python 3.6, used Google’s TensorFlow machine learning library via
the Keras interface, and used the Python/R Anaconda distribution.
Version control was performed using git and GitHub was used for
hosting all project repositories. The project was developed using an
Agile workflow, and team collaboration and communication was
facilitated using the software, Slack.

SYSTEM ARCHITECTURE AND DESIGN

The aim of this project was to prove it is possible to take a customer
service call as input and get data relevant for a customer satisfaction survey as output. The system constructed to accomplish this task, pictured in figure 1, consists of three main models: a speech-to-text model, a topic model, and a sentiment model. The speech-to-text model was trained on web-scraped customer service call audio files. The topic model was trained on a web-scrapped corpus of a wireless carrier’s FAQs and forum posts. The sentiment model was trained on a combined dataset of annotated IEMOCAP transcripts and electronics reviews. The customer service call audio was then passed through the speech-to-text program to generate transcripts, which were then fed into the topic and sentiment models. These two latter models output predictions of overall topics and sentence-by-sentence sentiment tagging. These two outputs were then aggregated to produce visualizations of customer satisfaction indicators such as sentiment over time during the call and sentiment associated with specific topics.

TECHNICAL DESIGN

The use of Convolutional Neural Networks (CNN) for natural language
processing is a fairly recent development. CNNs are most
well known for their success in image classification from filters’
natural proficiency for identifying components of an image like
lines and shapes. Whereas a CNN used on images can successfully
find patterns in shapes, the filters of a CNN used on language is
analogously able to find patterns in n-grams.

The network used for testing, pictured in figure 2 and elaborated
on in figure 3, has two convolutional layers. The first convolutional
layer has 64 filters with a filter size of 4, and the second has 32 filters
with a filter size of 6. These were starting points to test that the
model has functionality. These parameters could be further tuned
to improve the accuracy of the model. A larger number of filters
and filter sizes in the range of 3–5 are recommended for sentiment
analysis using a CNN in order to capture a large amount of contextual
information based on the relations between closely adjacent
words. The dropout layer in the diagram is intended to prevent
overfitting.

Latent Dirichlet Allocation (LDA) topic models are widely used for
identifying underlying topics in a set of documents. Given K topics
to find, an LDA model will sort words in a document into various
topics based on the words they appear most frequently with and
their respective topics. This works through a generative process
where the LDA model assumes the following: a document is composed
of an assortment of topics and each topic is composed of
an assortment of words. It then reverse engineers this process in order
to identify the topics in a corpus. Figure 4 is the graphical plate
notation where:

• M is the number of documents
• N is the number of words in a document
• α is the per-document topic distributions
• β is the per-topic word distribution
• m is the topic distribution for document m
• φ_k is the word distribution for topic k
• Z_mn is the topic for the nth word in document m
• W_mn is the specific word

For the purposes of sorting customer service transcript data into
topics, an online technical support discussion board and FAQ section
of a wireless carrier was used. By using this dataset, the topics
could be predetermined and the model could be tuned to identify
specific topics. Figure 5 provides examples of topic information
which can then be used to identify topics. In order to do this,
β was defined in the following way:

• For each topic, all words were initially given a 1 / corpus
  vocabulary size distribution
• Then words that were predetermined to be more frequent
  for a given topic were heavily biased (1000 times more weight)

This enabled the model to bias toward finding specific topics in a
document rather than determining its own.

DESIGN DECISIONS

Some tools were across all sections of the project. The general purpose
data science tools used throughout the project include:

• Python: Python is a powerful, general-purpose programming language
  with a robust set of libraries. It is an industry standard
  with wide support networks.
• TensorFlow: TensorFlow is an open source library for deployment of high
  performance numerical computation across different platforms.
• Anaconda: The Anaconda distribution contains a plethora of data science
  packages, including conda, which is an OS-independent
  package and virtual environment manager, allowing for easy
  package installation across machines.
• Jupyter Notebook: Jupyter notebooks are documents which contain text and embedded code which can be easily run within the document.
  They are simple to share and provide high-quality,
  interactive output for our python code. The prose-like format
  of Jupyter notebooks’ interpreter, markdown, and visualizations
  environment facilitates code and concept collaboration.
• Pandas: Pandas is a powerful python library which is ubiquitous in
  data science. In this project it was used for managing training
  and testing data.
• Matplotlib: Matplotlib is a python library for data visualization. In this
  project it is used for final visualization of sentiment analysis
  and topic modeling data.
• Pickle: Pickle is a python library for object serialization and data
  storage. In this project it is useful for exporting model configurations
  and related dataframes.

As this was a far-reaching project with many moving parts, a variety
of technical tools were required to specifically accomplish the
goal of each major utility:

Speech-to-text transcription

• ffmpeg: Due to the way speech-to-text engines process audio
  files, it is important to use audio encoding that is consistent
  with that expected by the engine. FFmpeg is a free, lightweight
  command-line audio encoder that fits seamlessly into
  our audio processing workflow. The final encoding pattern
  consisted of mono-channel audio with a 16 kHz sampling frequency,
  a 16-bit bit depth, and 300/3000 Hz high/low pass
  filtering to specifically capture voice audio.
• Audacity: Beyond encoding, our data required a small amount of editing, mostly in the form of phrase partitioning. Audacity’s
  “Silence Finder” utility allowed us to break long audio
  clips into phrase-long sound bites and export them easily
  for immediate transcription processing in DeepSpeech.
• DeepSpeech: DeepSpeech is a free speech-to-text engine with
  a high accuracy ceiling and straightforward transcription
  and training capabilities. DeepSpeech also comes with pretrained
  models that can be refined via transfer learning to
  improve accuracy on any particular type of audio, avoiding
  the need for extended training times or an extremely large
  dataset. Our transfer learning process involved removing
  the final interpretive layer of DeepSpeech, replacing it with
  a randomly-initialized layer, freezing all sublayers and training
  the final layer.

Topic model

• gensim: gensim is a general topic modeling library for python
  that has many useful features. Gensim was used primarily
  for its LDA model as it is very powerful and easy to train
  and tweak to our needs.
• NLTK: NLTK stands for Natural Language Toolkit and is a
  python library used for natural language processing. This library
  was used for our topic model to normalize documents
  before running them through our LDA model.
• pyLDAvis: pyLDAvis is a python package used for visualizing
  LDA models and was used for this project because it
  interfaces well with gensim. pyLDAvis allowed us to easily
  visualize our LDA model’s performance.
• Stanford NER: Stanford Named Entity Recognizer model was
  used for identifying named entities in a corpus such as persons,
  locations, and organizations. This helped us identify
  the subject of a document.
• Stanford POS Tagger: Stanford Part of Speech Tagger was used for identifying the part of speech for each word in a corpus. This helped us identify the subject, verb, and object of call transcript sentences.

Sentiment model

• Word Embedding: Tensorflow_hub’s word embedding modules
  were chosen as input for our neural network classification
  models since they allow us to map sentences to numerical
  values and preserving word order information. These
  modules are easy to implement through tensorflow_hub
  and come pretrained.
• Manual Feature Selection (MFS): Manual feature selection
  was used for the SVM and Random Forest classifier models
  because it allowed faster training, and therefore could handle
  a larger number of features. This allowed us to train our
  model to recognize vocabulary specific to the topic at hand.
• (MFS) NLTK Stemming: In manual feature selection, words
  were stemmed so that words with the same root would be
  recognized as the same word which reduced our feature
  space. NLTK’s Stemming module was chosen for ease of use.
• (MFS) Filtering: Words longer than 25 characters, words shorter
  than 3 characters, words appearing more than 2000 times,
  and words appearing less than 5 times were all filtered out
  to reduce the feature space.
• Bigrams: Bigrams were added to the feature space to embed
  some information on word order.
• Classification Models: A variety of classification algorithms
  were explored to have comparisons for the accuracy of our
  models on the training data. The models explored were an
  SVM classifier, Random Forest classifier, LSTM Recurrent
  Neural Network, and Convolutional Neural Network.
• Sklearn, Tensorflow, and Keras: Tensorflow and Keras were
  used for neural network training. Sklearn was used for the
  other classification algorithms.
• NLTK and NLTK-trainer: NLTK was used for sentiment analysis,
  since it has several prebuilt sentiment models. This lets
  us quickly set up a standard model and compare against our
  own model iterations.

PERFORMANCE RESULTS

Speech-to-text transcription

DeepSpeech, the automated speech recognition (ASR) engine
preferred by our client, is an end-to-end trainable, character level
LSTM recurrent neural network. DeepSpeech has been
used to produce a very low word error rate (WER), particularly
a 6% WER on LibriSpeech, an open-source 1000 hour
dataset of natural speech. 6% is approximately the human
error rate. Out of the box, DeepSpeech’s generic models returned
a 65.4% WER on our customer service call testing
data, more than 10x the WER of human processing.
This WER was ultimately reduced to approximately 18.7%
on the custom model generated using a very small corpus
of training data.

DeepSpeech functions best when translating sentence-length,
<10 second audio clips. The processing time required by Deep-
Speech to create a transcript per audio clip is between 4 and
9 seconds, with a mean of 7.14 seconds. Therefore, 5 minutes
of audio could take DeepSpeech almost 4 minutes of
processing time.

Sentiment model

Accuracy of various models were compared on 80/20 split
training/testing data made up of the IEMOCAP dataset and
a web-scraped dataset of electronic reviews with the sentiment
classification accuracies contained in figure 6. These
were 3-class classifiers where the sentiment classes were
positive, negative, and neutral. Accuracies of different feature implementations were then compared, with results contained
in figure 7. Some feature implementations did not easily
lend themselves to being run on different classification
models, so that data is absent.

Among the different datasets and implementations the best
results were on Neural Network Classifiers using the tf_hub
word embedding module. Of the word embedding modules,
the universal sentence encoder outperformed the other modules.
A deeper neural network outperformed sklearn’s DNNClassifier
under the same feature implementation, and was the
highest performer of all the implementations at about 10%
higher test set accuracy than the second contender. The best
model iteration was an LSTM RNN that used tf_hub’s universal
sentence encoder as feature input. This model had a
test set accuracy of 77.6% on a combined IEMOCAP and
electronic review dataset. Baseline accuracy for randomly
choosing a class for 3 classes is 33.3%.

Topic Model

Using data scraped from a wireless carrier’s online discussion
forums and FAQ section, the training data was organized
into 5 topics: accounts and services, android products,
apple products, network and coverage, and other products.
Each of these topics also included some keywords that were
seeded into the LDA model with bias (such as “iphone” for
apple products, “galaxy” for android products, etc). Using
this training data to train the LDA model, it was able to successfully
predict the topics on the testing data 57.6% time. Baseline accuracy for randomly choosing a class for 5 classes is 20%. A visualization of this LDA can be found in figure 10.

The part-of-speech subject identifier (PoSSI) model was trained
using corpora included with the Natural Language Toolkit
(NLTK) and then tested on the web-scraped wireless carrier
data described above. This model was able to produce
coherent subject-verb-object (SVO) results, but because the
web-scraped dataset is unlabeled, there is not a quantitative
way of analyzing the accuracy unless all web-scraped data
was hand-labeled for part-of-speech. Hand labeling the entire
web-scraped dataset was not in the scope of this project.

Visualization and analysis of customer call data

Visualizing sentiment over the course of a call showed trends
we might expect: a green, positive, first sentence introduction
followed by a red, negative, introduction of the problem,
and darker greens (i.e. more strongly positive sentiment) towards
the end of the call.

These are promising results that suggest sentiment analysis
could be applicable in the context of automating customer
satisfaction surveys.

CONCLUSION

Speech-to-text transcription

DeepSpeech is highly trainable but, with 120 million parameters,
requires massive training corpora, hundreds of hours
of training, and expensive equipment. Transfer learning was
performed in the form of freezing all sublayers of the model
and retraining the final layer. Experimentation with altering
previous layers was unable to create improvements. Transfer
learning is a viable course of action to customize Deep-
Speech for future transcription requirements but should be
trained using a greater number of high-quality audio clips.
DeepSpeech performed significantly better on our testing
data than alternative ASR engines such as Kaldi and CMUSphinx
as well as traditional ASR approaches such as Hidden
Markov Models.

Future work
Further improvements could likely be made in preparing
low-quality call data for transcription, as well as in post-processing
the transcriptions to produce more correct speech
(as DeepSpeech is character-level, it sometimes produces
nonsense). Retraining the weights of DeepSpeech on a large,
industry-specific copora would also likely produce greater
accuracy, but would be significantly more expensive — both
in computation time and amount of data required.

Sentiment model

The two biggest impacts on the accuracy
of the tested sentiment analysis models were data preparation
and model selection. Data preparation and input formatting
have a larger impact on a smaller dataset. Model selection
has a larger impact on a larger data set. Throughout
the various model iterations we tried, three different types
of formatting for data input were used:

• Manual Feature Selection (used in SVM and RandomForest-
  Classifier): Manual Feature Selection was done by using the techniques
  stemming, filtering, and tokenization. Words were
  stemmed using NLTK’s word stemming module, so all
  words of different tenses would correspond to the same
  token. Words were not included in the training/testing
  sets when their length was less than 3, greater than 25,
  when they appeared more than 2000 times or less than 5
  times in the full corpus. Tokens were then created for all
  viable unigrams and bigrams in the corpus.
• Pre-trained Word Embedding Modules (used in DNNClassifier)
  - Pre-trained word embedding modules from tensorflow_hub
  were used for the DNN Classifier.
  - The NNLM embedding is based on a neural network
  language model with two hidden layers. The least frequent
  tokens are hashed into buckets.
  - Word2vec is a token based embedding module trained
  on a corpus of English Wikipedia articles.
  - Universal-Sentence-Encoder encodes sentences for use
  in text classification. It was trained using a deep averaging
  network and has the feature of encoding semantic
  similarity.
• Tokenization: Words with a length less than one were removed, then
  sentences were vectorized using Keras’s text preprocessing
  module. A token was chosen for every word in the
  training input. All sentences were padded to be of equal
  length, and words were replaced with tokens.

Given the relatively small vocabulary size of our data set,
using a Pre-trained word embedding module gave the best
results, outweighing the effect of what model was used. Alternative
models were also developed, such as:

• SVM and Random Forest: These two models were the first
  to be run as a way to gauge what sort of accuracy was
  attainable, and what more complex models should be expected
  to beat. The sklearn SVM Classifier and Random-
  ForestClassifier were used for this task. These are two
  types of commonly used high performing classifiers.
• DNN: This model is a prebuilt Deep Neural Network from
  Tensorflow. This model was used to evaluate the performance
  of a Neural Network on the dataset, and to test use
  of tf_hub’s word embedding modules.
• LSTM: LSTMs are currently an industry standard model
  for sequential data like natural language.
• CNN: Convolutional neural networks have been shown
  to provide excellent results on sentence classification by
  capturing information about n-grams through the use of
  convolutional layers.
• CNN-LSTM: a combined CNN-LSTM model has been shown
  to work well by encoding the regional information with
  convolutional layers and long distance dependencies across
  sentences using LSTM layers.

With a small-to-mid sized training dataset, there is a much higher risk of overfitting to your training set when using neural networks with thousands of parameters, such as RNN and CNN neural networks. This can be seen by large discrepancies between testing and training set accuracies in our performance results. The ideal point at which to stop training is right as the training and testing scores begin to diverge where the training score continues going up, but the testing score starts going down. In the context of training a model, this means it is important to save checkpoints as you train, so the model can be reverted to an older checkpoint which is not yet overfitting. See Appendix B for an example of training and testing scores over the epochs of training. Overfitting can also be reduced by adding a regularization parameter to the loss function or to introduce dropout layers. Adding dropout to layers has seen success in most types of neural networks. However, this method does not work well on RNNs with LSTM units, which is the type of model that showed the greatest performance in this project. To mitigate overfitting on our small training dataset we apply dropout only to a non-recurrent layer.

Future work

• Manual Feature Selection: When doing manual feature
  selection the NLTK stemming module was used to find
  word roots. Lemmatization would likely give better results,
  but requires part of speech tagging. Implementing
  this would create a more accurate feature space. This could
  be done using NLTK’s WordNetLemmatizer and NLTK’s
  pos_tag in conjunction.
• Training Data: Use a larger training dataset. The IEMOCAP
  data set has a vocabulary size of 3,000 words, which
  is relatively small, and testing data could easily contain
  unseen words. This was improved by the inclusion of the
  electronic review dataset which increased the vocabulary
  size to around 8,000 words. The data, furthermore, is not
  as domain specific as could be. IEMOCAP very general
  purpose, and performed by actors reading scripts about
  everyday situations. The electronic reviews are for electronic
  products similar to those talked about in a technical
  troubleshooting wireless carrier customer call, but are not
  recent. More data specific to the content discussed in wireless
  carrier customer service calls could be introduced,
  such as more recent electronic device reviews or manually
  annotated wireless carrier discussion forum posts.
• Pre-Trained Word Embedding Modules: Transfer learning
  could be implemented on these modules using a corpus
  of text relevant to the type of text data on which classification
  is being attempted. In this case the word embedding
  modules could be further trained on a corpus of
  wireless carrier related text, such as data scraped from a
  specific wireless carrier’s online FAQ section and discussion
  forum posts.
• CNN and LSTM Neural Networks: The fine tuning of layers
  and their parameters, and feeding the model more
  data could be improved on the neural network models.
  These efforts would be for the purpose of increasing accuracy
  and decreasing overfitting.

Topic Model

The low accuracy result of the LDA model is expected as
LDA models are not meant to be forced into predefined topics
and instead should be used for finding hidden topic information
in the corpus. SVO extraction using POS and NER
tagging was able to produce coherent results, but these results
are not consistent across topics which presents a challenge
when trying to sort documents by topics.

Future work

• The LDA model could be improved by using a more domain-specific
  part-of-speech tagger (rather than the default POS
  tagging function from NLTK) as well as better refined alpha
  (document-topic density) and beta (topic-word density) parameters.
• The PoSSI model could be improved in a few ways:
  - Using a POS tagger and a Named Entity Recognition
  (NER) tagger trained on domain-specific corpora
  - A more domain-specific model for the POS tagger (currently
  uses a research model from Stanford)
  - A more domain-specific model for the NER tagger (currently
  uses a research model from Stanford)
  - A better method of extracting the subject from a tagged
  document such as a parser in combination with a context-free
  grammar for the domain-specific corpora
  - Speed improvements as the model can be very slow for
  large documents.
  - Both of these models could be combined to identify topics
  in the generated SVOs.

BIBLIOGRAPHY

A. G. Tripathi and N. S, ”Feature Selection and Classification Approach
for Sentiment Analysis,” Machine Learning and Applications:
An International Journal, vol. 2, no. 2, pp. 01–16, 2015.

B. C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim,
J. Chang, S. Lee, and S. Narayanan, ”IEMOCAP: Interactive emotional
dyadic motion capture database,” Journal of Language Resources
and Evaluation, vol. 42, no. 4, pp. 335–359, December 2008.

C. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, ”A Neural
Probabilistic Language Model,” Journal of Machine Learning Research
, vol. 3, pp. 1137–1155, Mar. 2013.

D. T. Mikolo, K. Chen, G. Corrado, and J. Dean, ”Efficient Estimation
of Word Representations in Vector Space,” CoRR, vol. abs/1301.3781,
Jan. 2013.

E. D. Cer, Y. Yang, S.-yi Kong, N. Hua, N. Limtiaco, R. St. John,
N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, Y.-H. Sung, B.
Strope, and R. Kurzweil, ”Universal Sentence Encoder,” CoRR, vol.
abs/1803.11175, 2018.

F. W. Zaremba, I. Sutskever, and O. Vinyals, ”Recurrent Neural Network
Regularization,” CoRR, vol. abs/1409.2329, 2014.

G. M. Hu and B. Liu. ”Mining and summarizing customer reviews.”
Proceedings of the ACM SIGKDD International Conference on Knowledge
Discovery Data Mining, Aug. 2004.

H. N. Jindal and B. Liu. ”Opinion Spam and Analysis.” Proceedings
of First ACM International Conference on Web Search and Data
Mining, Feb. 2008.

I. Qian Liu, Zhiqiang Gao, Bing Liu and Yuanlin Zhang. Automated
Rule Selection for Aspect Extraction in Opinion Mining. Proceedings
of International Joint Conference on Artificial Intelligence,
July 2015.

J. Y. Kim, ”Convolutional Neural Networks for Sentence Classification,”
CoRR, vol. abs/1408.5882, 2014.

K. TensorFlow. (2018). Text Modules | TensorFlow. [online].

L. X. Ma and E. Hovy, ”End-to-end Sequence Labeling via Bi-directional
LSTM-CNNs-CRF,” Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics, vol. 1, 2016.

M. J. Cheng, L. Dong, and M. Lapata, ”Long Short-Term Memory-
Networks for Machine Reading,” Proceedings of the 2016 Conference
on Empirical Methods in Natural Language Processing, 2016.

N. Y. Goldberg, ”A Primer on Neural Network Models for Natural Language Processing,” Journal of Artificial Intelligence Research,
vol. 57, Nov. 2016.

O. J. Wang, L.-C. Yu, K. R. Lai, and X. Zhang, ”Dimensional Sentiment
Analysis Using a Regional CNN-LSTM Model,” Proceedings
of the 54th Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers), 2016.

P. A. Y. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E.
Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Y. Ng,
”Deep Speech: Scaling up end-to-end speech recognition,” CoRR,
vol. abs/1412.5567, 2004.

Q. J. Kunze, L. Kirsch, I. Kurenkov, A. Krug, J. Johannsmeier, and
S. Stober, ”Transfer Learning for Speech Recognition on a Budget,”
Proceedings of the 2nd Workshop on Representation Learning for
NLP, 2017.

R. S. Bansal and Natural Language Processing and Machine Learning,
”Beginners Guide to Topic Modeling in Python,” Analytics Vidhya,
29-Aug-2016. [Online].

S. J. Brownlee, ”How to Develop an N-gram Multichannel Convolutional
Neural Network for Sentiment Analysis,” Machine Learning
Mastery, 14-Feb-2018.

T. S. Axelbrooke, ”LDA Alpha and Beta Parameters — The Intuition,”
LDA Alpha and Beta Parameters — The Intuition | Thought Vector
Blog, 21-Oct-2015. [Online].

U. A. Crosson, ”Extract Subject Matter of Documents Using NLP,”
Medium, 08-Jun-2016. [Online].

V. J. R. Finkel, T. Grenager, and C. Manning, ”Incorporating nonlocal
information into information extraction systems by Gibbs
sampling,” Proceedings of the 43rd Annual Meeting on Association
for Computational Linguistics — ACL 05, pp. 363–370, 2005.

W. K. Toutanova and C. D. Manning, ”Enriching the knowledge
sources used in a maximum entropy part-of-speech tagger,” Proceedings
of the 2000 Joint SIGDAT conference on Empirical methods
in natural language processing and very large corpora held in
conjunction with the 38th Annual Meeting of the Association for
Computational Linguistics, pp. 63–70, 2000.

X. S. Bird, E. Klein, and E. Loper, Natural language processing with
Python: Analyzing Text with the Natural Language Toolkit. Beijing:
OReilly, 2009.

Y. Smoothed LDA. Wikipedia, 2009.

Z. S. Ruder, ”Deep Learning for NLP Best Practices,” Sebastian Ruder,
25-Jul-2017. [Online].

A Machine Learning Approach to Automated Customer Satisfaction Surveys was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the web development community it seems like there is a new framework every other week that is supposed to “revolutionize” the way we build websites and change the entire landscape, except they don’t. For the most part the only really successful frameworks are Angular and React (though React is really a library, but for the purposes of this article I’m going to wrap it in with frameworks). This is because of a few reasons that are often overlooked by the developers.

Learning Curve

One of, if not the biggest, reasons frameworks fail is because of the learning curves associated with them. There are so many options and to learn all of them (or even most of the useful ones) to the point of being able to fully take advantage of them is impractical. Things are often complex and every framework has its own design philosophy and style deeply rooted in the developers’ own personal beliefs that it’s often a mental exercise to navigate all of the intricacies within. The developers are usually so deep into their designs that they often lose sight of what it’s like coming into the framework with no knowledge of how it works or what it really does. Because of this developers often stick with what they know and have a hard time fully converting to one framework over another.

So Powerful They’re Almost Useless.

With every new framework coming out they always have to find their own way of making them standout and worth using. Often this is achieved through promises of “plug and play” or “extensibility” so much to the point that it can nearly build the damn website for you. Now this sounds great on paper, but what I have noticed is that the more complex and “powerful” a framework is the less likely it is to be used. Why is this? Well when things do most of the work for you it comes with some consequences, one of which being making debugging a lot harder when you finally have to do some work yourself. In order to fix an issue, not only do you need an understanding of your own code (which I pray you do) you also have to have an understanding of the framework’s code and what it’s doing so you can figure out how to fix your issue. For me this is reason enough to not use frameworks all together as this often slows me down more than if I just wrote the application from scratch (for disclaimers sake my go-to is React (mostly because it’s not a framework)). However in the beginning I did mention that there are some successful frameworks, so there must be some reasons for this right?

Why Not All Front-End Frameworks Fail

In the first paragraph I mentioned two frameworks (really one and a library) that are successful, and I believe it comes down to achieving really what all frameworks are trying to achieve: making the developer’s life easier. React does this by not doing everything for you and not trying to make things black-boxy. Instead it simply provides more of an outline of how to build an application and then gives you the tools to use this outline. You still have to make everything yourself, and you can use the library however you want, so it doesn’t try to make all of your applications fit into a box. It’s more like a useful assistant hanging around with some tools to make common problems easier to solve. It’s a support structure not another medium between you and your application. Angular is a little different as it does have some more structure, but it didn’t start out this way and that’s why I think it’s successful. It started out small and basic which allowed it to build a community of people who knew how to use it so that when updates came out with new features the learning curve was drastically reduced for the majority of the community. Then because of its wide-use it was able to establish a footing in the community and was able to hang around. Both of these examples also have the backing of tech giants (Google for Angular and Facebook for React) in addition to a large open-source community that they can pull from. This makes a huge difference as people can work on them full-time and constantly improve and innovate while some smaller frameworks and libraries may not be able to sustain that sort of development.

Closing Remarks

In the end the web development environment is constantly changing so what’s popular today won’t necessarily be popular tomorrow. Whether it’s a new trend or a new platform, the needs of users and developers will keep evolving and with them new ways of building web applications. For now it looks like React and Angular are here to stay, but soon there will be another contender and the race for the best web framework will continue on.

I discovered this when I was having an issue with the RadSideDrawer component not recognizing a property update event after I navigated away and returned to the page. The “tap” property on a button that was supposed to open the drawer worked fine and I could view all of the data I was trying to update just fine through console log and alerts, but I was not seeing the property update reflected on the front-end. Apparently when I set the bindingContext after returning to the page the binding for this particular component broke and I couldn’t update it anymore.

Enough with the words though, here’s some code!

main-page.xml

<Page xmlns=”http://schemas.nativescript.org/tns.xsd" loaded=”onLoaded”>
...
</Page>

main-page.ts

export function onLoaded(args: EventData): void {
 const page = <Page>args.object;
 
 // Notice how I am checking for a truthy bindingContext value 
 // before assigning a new one. If you navigate away and return to 
 // this page this will evaluate to false as page.bindingContext is 
 // preserved from the last time it was set.
 if (!page.bindingContext)
   page.bindingContext = new MainPageViewModel(page);
}