POV: waiting for your pull requests to get reviewed

You just finished writing the new comments feature for your web app: you wrote the code, tested it, and put up the PR. You’re under a deadline and your next highest priority task is to add reactions to those comments.

But wait a minute — the reactions code will depend on the comments code, which isn’t yet merged to main. Which means you're blocked.

If you’ve ever found yourself in this position, you know you have three options:

1. Work on something lower-priority while you wait for your comments PR to be approved (and risk missing the deadline).
2. Interrupt your teammate to ask for a faster review.
3. Make your comments PR bigger by merging what should have been a separate reactions PR into it (which will make it harder to review and increase the chance of bugs).

Stacking your changes

“But what about the fourth option?” — you might ask. Why can’t you write the reactions branch on top of the comments branch to stay unblocked without compromising on the size of your PRs?

The blocker here isn’t Git. In fact, the reason you can’t go with option #4 is that your code review tools probably don’t support it:

• How do you tell reviewers that your reactions PR depends on the comments PR?
• How will reviewers see the relevant diff for the reactions PR (excluding the comments PR changes)?
• How can you propagate changes to the reactions PR if you ever need to update the comments PR before you land it (i.e. to address review comments)?

But suppose we did have the right tooling to make all of this easy — would you stop at just 2 PRs stacked on top of each other? Should comments really be just one PR? Or would you break it into the series of atomic changes you originally wrote: a DB change, a new endpoint to the API schema, the implementation of that endpoint, and the front-end changes that call that endpoint?

No one likes a 4,000-line pull request

I think most developers agree that smaller PRs are easier to work with:

I Am Devloper on Twitter: "10 lines of code = 10 issues.500 lines of code = "looks fine."Code reviews. / Twitter"

10 lines of code = 10 issues.500 lines of code = "looks fine."Code reviews.

Smaller changes help you get faster, more thorough reviews on your code, be more targeted in who you tag (why do backend engineers and designers both need to review the same change?), and experience fewer merge conflicts.

By stacking your changes, you can get all of these benefits while staying unblocked and not needing to thrash your teammates.

Introducing Graphite

This idea isn’t new, and if you’ve worked at a company like Facebook, Google, Uber, Dropbox, or Lyft (among others), you’ve either had first-class support for stacked changes built into your code review platform (i.e. Phabricator at Facebook, Critique at Google) or scripts built on top of it to enable stacking. Many engineers who come from those companies seem to miss this workflow, but it has yet to make its way into the public toolchain.

Dan on Twitter: "I like GitHub announcing new features but really all we need is Stacked Diffs. Please? / Twitter"

I like GitHub announcing new features but really all we need is Stacked Diffs. Please?

vjeux ✪ on Twitter: "Stacked diffs are the biggest feature I'd love to see support added on github! https://t.co/sijkqhsGLc / Twitter"

Stacked diffs are the biggest feature I'd love to see support added on github! https://t.co/sijkqhsGLc

Alexandra on Twitter: "The experience I probably miss most from writing code at FB is stacked diffs. It was SO easy to split diffs/PRs and have "1 idea" per change set. I has no clue this was not a widely available feature in industry. / Twitter"

The experience I probably miss most from writing code at FB is stacked diffs. It was SO easy to split diffs/PRs and have "1 idea" per change set. I has no clue this was not a widely available feature in industry.

We’re among the engineers who want stacked changes back — so much so that we’re building the tooling to make it accessible for everyone else.

Graphite — our platform for stacked changes — is built directly on top of Git and seamlessly syncs with GitHub. You can use it without needing anyone else on your team to change their workflows — they’ll see normal PRs from you in GitHub, which they can comment on and review as they always have.

Graphite is currently in closed beta, but it’s already used by engineers at some of the largest companies every day. If you’re interested in being among the first to try it out, you can sign up for our waitlist starting today.

tl;dr This post summarizes and discusses part of my undergraduate senior thesis (a research project that took place between September 2016 and March 2017): compiling Google’s TensorFlow library into JavaScript.

In case you missed it, Google just announced TensorFlow.js: a largely hand-ported version of TensorFlow to JavaScript. This is super exciting for me for two reasons:

1. Generally speaking: I think it is a super interesting, and difficult, problem, so it’s really cool to see a big company tackling it and releasing their work.
2. Closer to home for me: almost exactly a year ago (March 31st, 2017), I handed in my undergraduate senior thesis: Machine Learning in the Browser. It was an overview of what was possible, theoretical blockers, and a largely focused on my efforts compiling TensorFlow into JavaScript (which I, confusingly, also called TensorFlow.js). Needless to say, this is a space I care for and have thought about a lot.

I’ll start off by saying my work is not production-ready and is still very much a research project; however, given Google’s announcement, I wanted to pause and share what I had found, in hopes that it might motivate or inform someone else’s work. I was hoping I would be able to share all of this after I had had better luck compiling XLA output to JS (more on that later); however, some combination of it being a hard problem and life happening (graduating / moving to a new city / starting a new job), prevented me from making much progress, so there is still much work to be done.

Note: Almost all of this work is a year old, meaning the TensorFlow I compiled is pinned at version 1.0.0 (which was the most recent version when I submitted my thesis). If there is enough interest, I can look into bumping the version; when I was actively working on this, I bumped from 0.10 to 0.11, 0.11 to 0.12, and 0.12 to 1.0 all without any major issues.

Why would anyone want this?

When I was doing my thesis there were two questions I would commonly get: how is this research, and not just an engineering problem? (To which I still don’t have a good answer.) And why would anyone want to do machine learning (and specifically inference) in the browser? To me, it seemed like the obvious fusion of two trends: machine learning becoming ubiquitous and web applications becoming more powerful; concretely:

(Reason #1) Privacy Guarantees: This is fairly straight-forward, people want to be able to have the advantages of machine learning such as image classification, hot-word activation for speech recognition (e.g. “Ok Google…” or “Alexa…”), and smart reply to messages without needing to give up their raw data to companies. On the flip side, companies don’t always want user data, in a lot of cases it can pose a liability for them to be storing this data on their servers.

(Reason #2) Offline Mode: A lot of web apps are not supporting some limited, offline version of their application. It would be nice if these offline modes weren’t totally unable to run any ML at all.

(Reason #3) Free / Local Computation: Imagine a world where every ML model open-sourced on github came with a web demo where you could input an image to classify, or upload your own test set to see how it performs. Right now this doesn’t happen because no one wants to pay for the compute. With the capacity to run ML in the browser, no one would have to. More generally, this would enable a class of unhosted web applications to use machine learning.

Note: It is important to call out that: (1) this is not about running TensorFlow in JavaScript for some node-like environment. TensorFlow is a C++ core with bindings to many languages (Python being the most popular), so for that we could simply create a JS binding to the core; (2) I think inference is a much more valuable use case than training (with the exception of transfer learning to fine-tune models, I can’t imagine many usecases for full-scale ML model training in the browser). This is about being able to deliver and run machine learning models to people’s browsers.

Approaches

As far as I’m concerned, there are two real ways to get TensorFlow running in the browser: rewriting it (which would take a massive engineering force, something that large companies have at their disposal, but individual developers / students doing an undergraduate thesis do not) or compiling it. MXNet.js is an example of a major machine learning library being compiled into JS; Google’s TensorFlow.js / Keras.js / Tensor Fire are examples of handwritten implementations.

Lack of Google-scale engineering resources aside — and even with a working handwritten port of TensorFlow in JS — compiling TensorFlow to JS seems like an approach worth exploring for 3 major reasons:

(1) TensorFlow is rapidly changing, and having to constantly port it by hand seems like it will take a massive, sustained engineering effort.

(2) The Web is rapidly changing. Since the submission of my thesis and writing this blog post (almost exactly a year apart) the standards for the web are already different. APIs I was excited for have been deprecated, and new technologies have been proposed. Compiling it would abstract away these changes.

(3) Compiled JavaScript has some speed benefits over hand-written JavaScript (which is critical for performance-sensitive things, like machine learning), but more on this later.

Compiling C++ into JavaScript isn’t actually as crazy as it seems. Emscripten is an open-source project developed to compile JavaScript into ASM.js. ASM.js is a subset of javascript meant to be used as a compile target. Some browsers even fast-track the execution of asm code (by compiling it into machine instructions), allowing it to run at only 2x native speed, even for large benchmarks.

A primer on TensorFlow

Note: This writing is largely distilled from my research — which is almost a year old — so the following may be a little outdated.

TensorFlow — along with almost every machine learning framework — faces a tradeoff between needing a high-level language and needing speed. Without a high-level language, it is difficult to architect complex models and express abstract concepts in a readable way. Without speed, it is difficult to run the actual neural nets. TensorFlow — along with many other libraries — reconciles these opposing requirements by allowing developers to construct a computation graph in Python, and sending the graph along with any inputs to a C++ core to evaluate. (While Python is the most common front-end to TensorFlow’s core, the SWIG bindings allow for TensorFlow bindings in many languages that all interface back to the same common core.)

A TensorFlow graph consists of Ops (nodes) and Tensors (edges), imagine the following TensorFlow program:

import tensorflow as tf

# (Part 1) create graph
y = tf.add(
    tf.multiply(
        tf.Variable(2, name="m"), # initial value of 2
        tf.placeholder(tf.int32, shape=(1,), name="x")
    ),
    tf.Variable(1, name="b") # initial value of 1
)

# (Part 2) create and initialize session 
sess = tf.Session()
sess.run(tf.global_variables_initializer()) # sets m and b

# (Part 3) run inputs through the graph
print sess.run("Add:0", feed_dict={"x:0": (5,)}) # returns 11

Part 1 in the code creates a TensorFlow graph with the following shape:

As you can see, it has both ops (the circles) and tensors (the edges). Tensors are simply multi-dimensional arrays (a rank-0 tensor is a scalar, a rank-1 tensor is a vector, a rank-2 tensor is a matrix, etc). Ops take in 0 or more tensors and produce 0 or more tensors. For example, the add op takes in 2 tensors, and produces 1 tensor. The names for tensors are op_that_produced_them:index. For example, the result of the add operator is denoted as “add:0”.

Part 2 (in the code) loads this graph in the C++ core, and part 3 says fetch the value for add:0, initializing (or “feeding”) the tensor x:0 with a value of 5. The first op that can be evaluated in the graph is “multiply”, as all of its inputs are defined (m:0 and x:0). After evaluating multiply, multiply:0 is set to the result of the op. With multiply:0 set, all of add’s inputs are defined so it is evaluated next, which sets add:0 and returns it to the user.

TensorFlow compute graphs can be serialized and distributed as Protobufs (a serialization format developed and used extensively by Google). A common approach for deep learning in TensorFlow is to architect a graph in python, train it (i.e. set the variables in the graph), export that graph along with values for the variables to protobufs, and distribute those protobufs to servers or mobile devices.

A primer on Emscripten

TensorFlow is capable of running on many devices architectures (i.e. x86 on your laptop and ARM on your smartphone) through the magic of compilers. TensorFlow’s source code (and, specifically, the C++ core) can be compiled into machine-specific instructions for a variety of compile targets.

In broad strokes, compilers work by taking source code, “lowering” it to an intermediate representation (that is somewhere between source code and machine code), optimizing the intermediate representation, and then lowering that into machine code. For example, the popular compiler Clang compiles C++, by lowering it into LLVM-IR (the intermediate representation), optimizing that, and then lowering it into machine code. The first part of this process (source code to LLVM-IR) is referred to as the front end of the compiler, and the latter (LLVM-IR to machine code) is the backend of the compiler. Emscriten works by providing an alternative backend to Clang, and compiling the LLVM-IR to ASM.js (a subset of JS intended as a compile target).

Recently, web standards have even gone further in defining a good compile target. WebAssembly (WASM) is a binary protocol that is supported by every major browser (albeit behind a flag in some of them) that’s intended to be the compile target for the web. Because WASM is a binary protocol defined from scratch, it is smaller and can run faster than ASM.js. Emscripten can optionally generate WASM from C++ targets, simply bypassing a flag to the compiler.

Coercing TensorFlow to Compile

Even using Emscripten, there are still challenges to compiling TensorFlow. The first challenge I encountered was literally using Emscripten. Emscripten is invoked as a drop-in replacement for gcc, emcc. It is also distributed with two tools, emmake and emconfigure, to make it easier to integrate with pre-existing makefile build systems. TensorFlow doesn’t use a makefile. Rather, it uses Google’s open-sourced build tool, Bazel. I explored updating the bazel files to use Emscripten, but soon realized that (1) the bazel crosstool functions are largely undocumented (or poorly documented); (2) the requisite files would effectively require maintaining a separate build system that would require significant upkeep.

Fortunately, TensorFlow maintains a makefile for use with their mobile framework. One of the themes of this project is that in many ways compiling for JavaScript is similar to compiling for mobile (as resources and computational primitives are limited). By modifying the mobile makefile, we we embarked on a journey of debugging C++ compiled into a JS binary. That means we had few (read no) debugging tools, and limited insight into what is happening in the binary.

The first hurdle was that TensorFlow’s dependencies must be compiled into JavaScript. At the time, there are three critical dependencies for TensorFlow’s core: libmath, zlip, and libprotobuf. Because libmath is ubiquitous, it ships with Emscripten. ZLib has already been ported to Emscripten (although we discovered this after we already ported it). LibProtobuf had prior work in compiling it to JavaScript, and so all we had to do was fork the repository and bump the version (dealing with minor bugs). If you want to use the image ops (e.g. DecodeJPEG) you also need to compile JPEG, PNG, and GIF libraries. I have done this, but exclude it from here for the sake of brevity (feel free to contact me for more details if this is interesting).

The next issue was a type error, and to resolve it we would have to make TensorFlow work on 32-bit systems. Because all JavaScript numbers are implemented as IEEE754 double precision floating points (Section 6.1.6), they only have 53 bits of integer precision. This means that Emscripten can only reliably emulate 32-bit systems (as it could not address into a full 64-bit address-space). This is problematic because TensorFlow only supports 32-bit systems. Fortunately, I only had to make minor modifications to correct this.

TensorFlow uses Eigen as it’s linear algebra library — As an aside, this was incredibly lucky for me because many popular linear algebra libraries are written in Fortran (Blas, Lapack, Atlas) and could not be easily compiled with Emscripten — Eigen defines scalars to be of type std::ptrdiff_t. The problem is that it is common in the TensorFlow code base to do something akin to the following:

int64 a = 5;

// ...other stuff happens...

Eigen::Index b = a;

This will cause the compiler to (rightly) complain about type narrowing, because in JS Eigen::Index is only an int32. In order to fix this, we simply need to replace instances of int64 with the more correct “Eigen::Index”. This was a surprisingly small amount of changes.

At this point, TensorFlow will compile. It will just hang when it is run. This is because JavaScript currently has no way to create POSIX-style threads (although there are experiments to create posix-style threads using webworkers with a shared memory array in FireFox nightly). Vastly oversimplifying some details, the way in which TensorFlow processes graphs is by creating a threadpool and dispatching op kernels (the implementation of the ops) to other threads. On Emscripten, no threadpool is created, and the main thread is left waiting for a task that will never complete. To correct this, I had to dive into the concurrency model. TensorFlow’s scheduler effectively takes a function closure and dispatches it to a thread, so to correct this, simply replace the scheduler with a function that executes the closure immediately and on the current thread. This ensures that by the time any thread barrier is invoked all prerequisite tasks have already been completed.

TensorFlow will now compile and run in JavaScript.

DevX

This creates a JavaScript library that we can link other Emscripten-compiled C/C++ programs against. The problem is that I don’t believe developers want to write their machine learning for the web in C/C++, they tend to avoid it even for native programs. Taking some inspiration from the python frontend, my goal was to create a library that could take in a graph file, and return a class that the developer could pass inputs into and get outputs from.

This library would require a C++ component to actually use the compiled TensorFlow library, and a JavaScript component to wrap the C++ component and pass in inputs. Emscripten allows one to expose and call functions in the C++ using embind. Embind is incredibly powerful, but only really supports simple argument types. In order to pass graphs and input tensors to the C++ library, I needed to someway to serialize them into basic types. Fortunately, TensorFlow defines a Protobuf to serialize tensors! This meant that I could serialize the tensor in JavaScript, convert it to a string, pass it into the C++ layer (through embind), and deserialize the Tensor. I wrote a library to create tensors from multidimensional javascript arrays.

I then wrote the wrapper that loaded in the compiled C++ core and provided convenience methods to invoke it. For example, to run the simple y=mx+b graph from above:

const TFJS = require('tfjs');
const tensorjs = require('tensorjs');

// load in the graph protobuf, maybe through a string
// const example_graph_pb = ...

TFJS.for_browser('/tensorflowjs/').then(lib => {
  const sess = new lib.Session(example_graph_pb);
  const results = sess.run(
    {
      "x:0": tensorjs.intTensor([5]),
    },
    ["add:0"]
  );

console.log(results[0]); // prints 11
});

On top of that, I wrote a quick library to make encoding images as tensors easier:

const TFJS = require(‘tfjs’);
const tensorjs = require(‘tensorjs’);

// const mnist_graph_pb = ...

TFJS.for_browser(‘/tensorflowjs/’)
  .then(lib => {
    const sess = new lib.Session(mnist_graph_pb);

// pull out handwriting image from a canvas
    const canvas = document.getElementById(‘handwriting’);
    const context = canvas.getContext(‘2d’);
    const img_data = context.getImageData(0, 0, canvas.width, canvas.height);
    const img_array = lib.image_ops.get_array(img_data, true, 0, 255);

const results = sess.run(
      {
        "Reshape:0": tensorjs.floatTensor(img_array),
        "dropout:0": tensorjs.floatTensor(1.0)
      },
      ["prediction_onehot:0"]
    );
  });

Benchmarks

Given my naive approach, I did not expect to be fast; however, our JS execution times were within an order of magnitude of singlethreaded, CPU-only TensorFlow. For running full TensorFlow graphs in the web browser, this seems quite fast, and it backs up Emscripten’s claim that we might be able to get native code running in the web browser at near native speeds (something that always make me think on The Birth and Death of JavaScript). I expect these metrics will continue to improve as WASM becomes more powerful.

While speed was good, size was not. The library was 30MiB, with another 4.7MiB needed to initialize Emscripten. Compressed they were better (4.2MiB library, and 408KiB to initialize), but still large for web libraries. We can improve this by selectively including op kernels in the compiled code (op kernels are the code to necessary to evaluate nodes in the computation graphs); for context, at the time, TF defined 216 unique ops, and Inception v3 only took 11 to run. However, even if we were to improve the size of the runtime, model graphs can still be 100s of MiB (which would be unreasonable to ship to a web browser); while we can make that better with tools like quantization, we still have a way to go before we can use that in production.

Takeaways

Firstly, one of the advantages of waiting almost a year to publish this is that I have the advantage of hindsight. It’s funny reading the discussion section of my thesis to see a lot of predictions I made either come true or fall flat on their face. It does seem to be the case that people want machine learning libraries in the browser, something that was non-obvious when I was doing this. It also seems to be the case that the web is only continuing to get faster (WASM has continued support, and we have working groups for better use of the GPUs and other forms of hardware acceleration on the web).

Secondly, I’m starting to think that for the time being, compiling an entire machine learning runtime to the web is unfeasible. It is likely the case that we will have breakthroughs that enable this in the future, but for now we should probably direct our efforts elsewhere.

However, I don’t think this means that we should discount looking further into compilation as an approach; if anything, it shows that it’s possible. One of the experiments I’ve been most interested to try would be to take Google’s XLA (a compiler from TensorFlow graph to bytecode) and make it compile the graph to JS. This is possible because XLA uses LLVM-IR as an intermediate step, and, as you may remember, Emscripten compiles LLVM-IR to JS. Like XLA programs in general, these compiled-js programs might be faster and lighter-weight than the full runtime.

If you’re interested in this work, and want to talk about it or pick up where I left off, feel free to reach out! I would love to chat.

Resources

Website with Working Examples: https://tensorflowjs.github.io/

Repositories:

• Where I did most my work: https://github.com/tomasreimers/tensorflow-emscripten
• Compiled TensorFlow: https://github.com/tomasreimers/tensorflowjs
• Wrapper to interface with the compiled assets: https://github.com/tomasreimers/tfjs
• Library to read and write tensors to protobufs (in JS): https://github.com/tomasreimers/tensorjs

P.s. If there is anyone from Google reading this, congrats on the project launch!! It’s really cool! Also, if you would like the tensorflowjs github organization name —I would be happy to transfer ownership; you seem to have a better use for it than I do now :)

Note: This post was co-authored by Greg Foster (Medium won’t let us add co-authors), definitely check out his profile!

TL;DR This post motivates and describes an attack where accelerometer/gyroscope readings and machine learning are used to develop a keylogger for mobile devices. While previous research has been conducted in this space, we hope that our narrative is useful for someone tackling an unintuitive machine learning problem (also the results and graphs are just really darn cool).

Hypothesis Formation

In Fall 2016, we were tasked with creating a final project for CS263 (Harvard’s Systems’ Security Class): implementing some attack. Mostly to spite our hype-hating professor, we committed to integrating the greatest buzzword of all into our project — machine learning. We had been intrigued by previous attacks that used machine learning on audio data of keyboards and pin pads to extract passwords. This type of attack is called a side channel attack and takes advantage of the physical properties of the real-world implementation of a security system. Side-channel attacks seem like good candidates for machine learning because they often involve picking up on subtle, non-evident patterns. We decided to devise a side-channel attack for mobile devices, which are chock-full of incredibly-accurate, and permission-less, sensors. Specifically, we wanted to develop an attack that exploited the accelerometer (measures movement) and gyroscope (measures rotation), as they don’t require permissions and can be read by apps running in the background. On iPhone, apps can run for up to 10 minutes in the background after they’ve been closed.

After a bit of brainstorming, we formed our 4-part hypothesis:

(1) When you tap your phone, it moves.
(2) This movement can be picked up by the gyroscope and accelerometer.
(3) This movement is distinct from other movement and can be identified by machine learning and statistical techniques.
(4) The movement is sufficiently unique to identify where on-screen it occurred.

While we were fairly certain about (1) and (2), we were not sure about either (3) or (4). Previous work in this space quickly lead us to believe it was possible, but we weren’t sure how robust these methods were or if we’d be able to recreate them. Generally, we’ve learned to be weary with academic descriptions of attack vectors — as sometimes they only work in a lab setting, expect certain conditions, or simply aren’t as practical as their authors make them out to be (of course, this does not describe all papers — but there are certainly a good number for which this is the case).

Evil: Predicting Touches

Repository here: https://github.com/tomasreimers/axolotl

With any deep learning problem, the first question you need to ask yourselves is what data do you care about. Neural nets are very specific devices that learn mappings from some well-defined input to some well-defined output. They do this by learning statistical correlations between labeled inputs and outputs. Many people, especially those newer to machine learning, seem to think that stellar algorithms are where people really differentiate themselves. While that’s true, a large majority of modern machine learning problems are solved with fairly canonical algorithms, and a really awesome data set.

On the iPhone, one accesses accelerometer and gyroscope readings by listening to events emitted by iOS at seemingly-random intervals (accelerometer and gyroscope readings are different events, and emitted separately). We developed an app to continuously log these events along with touch data. We used this code to collect 8 data sets to experiment on. Our data sets contained rows of data, where each row represented one of the data events, including the unix timestamp it was collected at, whether this data was an accelerometer or gyroscope event, the x/y/z of the data (both accelerometer and gyroscope data are 3 dimensional: accelerometers can move along the x/y/z axis, and gyroscopes have 3 degrees of rotational freedom); additionally, we also collected whether the user was touching the screen at the time of the accelerometer / gyroscope event, and, if so, where the touch had started (the beginning of the touch likely started with the user tapping the phone).

We then graphed the data by time.

This was super encouraging because it showed us that taps began with a distinct spike in readings. When doing machine learning on sources where you’re not sure how much signal there is, it is usually a good idea to start with small predictions and get progressively more complex. For this reason, we decided to see whether a neural net could predict whether a randomly sampled 200ms window was a tap or not (from visual inspection, it seemed all the spikes indicating a spike fell within a 200ms span).

Traditional neural networks require fixed-sized inputs, and our windows contained variable amounts of data events (emitted at the whim of the operating system). In order to overcome this, we decided to interpolate N points at equal distances in windows. Then we could feed our neural net an input with 6N values (3 gyroscope and 3 accelerometer values for each of the N points). We tried various values for N and ultimately settled on N=20 (the trade-off is more information vs slower training times).

After a little bit of experimenting, we settled on a 5-layer fully-connected neural network with relu activations as our binary classifier. This method leads to the following prediction accuracies on held-out testing sets:

Sample 0: 97.5% correct
Sample 1: 94.7% correct
Sample 2: 81.8% correct
Sample 3: 96.7% correct
Sample 4: 91.2% correct
Sample 5: 87.7% correct
Sample 6: 83.2% correct
Sample 7: 92.6% correct
Source: learn_touches.py

Note: another, possibly “more correct,” way to do this would use an RNN to analyze the sequence data. We chose not to do that primarily because densely connected neural nets tend to be less expensive to train than RNNs over long streams of data.

Eviler: Predicting Location

Being able to detect taps reliably gave us faith that we might be able to detect the location of the taps on the screen. We modified our neural net to output an X, Y coordinate for the predicted location and retrained it. We got the following mean error (the average distance between our predicted touch location and the true location of the touch).

Sample 0: 0.366 inches
Sample 1: 0.373 inches
Sample 2: 0.4077 inches
Sample 3: 0.4190 inches
Sample 4: 0.4552 inches
Sample 5: 0.3935 inches
Sample 6: 0.3923 inches
Sample 7: 0.3946 inches
Source: learn_location.py

However, as is usually the case with aggregate statistics, the number is only part of the story. When we graph the distribution of errors, the majority are fairly close (with some outliers skewing the mean).

Evilest: A Unified Pipeline

Having these two building blocks, we can now assemble a full app that could key-log. A potential attack vector might look something like this:

Unsuspecting user downloads “Evil Flappy”, an app where they have to tap on the screen mindlessly to advance some objective. During this tapping, the app uses transfer learning to tailor the model to the user and test its own predictive capacity. Once the app detects it has sufficient accuracy, it lets the user win and prompts them to log into Facebook (or any other social network) to share their high score. At this point, the app will be sent to the background, where it can begin key logging the known process: type in a password and press log-in.

In order to do this, the app collects accelerometer and gyroscope data, and then either analyze it locally or sends it to a server for analysis. Whether it’s happening locally or remotely, a sliding window can be run over the data to detect when taps happen, and, if one did, where that tap occurred.

The accuracy of this attack could be increased by using a Markov model over what letters might follow other letters in user’s passwords, replacing the neural network to predict letters instead of location (thus giving us the probability it is each individual letter and allowing us to create some kind of Kalman filter over all possible potential passwords), or asking the user to input their password multiple times and building a statistical model to integrate the multiple inputs.

Some potential fixes that smartphone operating systems manufacturers could implement are: requiring permissions to access the accelerometer / gyroscope, fuzzing the data from those sensors while typing into password sensitive fields, or simply downsampling the rate at which apps can read those sensors while running in the background.

General Lessons about Data Science and Machine Learning

Throughout this project, we found ourselves grappling with a problem that’s not super intuitive and with too much data to analyze by hand. Lessons learned are:

(1) Start small, scale up.

(2) Visualize regularly.

(3) Aggregate statistics don’t necessarily tell the whole story, make sure you’re choosing correctly.

There are a lot of blog posts out there about this along with a bunch of StackOverflow posts, all with varying instructions. The official ones by Google seem outdated, and the only thing I could find on Let’s Encrypt side was a bug saying something along the lines of “plz add support for GAE”. I tried a few, and by combining steps in some of them (and a lot of frustration and eventually-resolved confusion) I got it to work. I’m writing up this blog post primarily for myself (so that when I need to re-encrypt in a few months I know how), but hopefully someone out there might find it useful.

(Update: Automated certificate management is coming soon: https://issuetracker.google.com/issues/35900034)

Preliminaries:

1. LetsEncrypt issues SSL certificates by automatically verifying that you have ownership of the domain you claim you have. (More on how to verify later.)
2. LetsEncrypt client has since been renamed to certbot and is maintained by the EFF: https://certbot.eff.org/
3. LetsEncrypt allows you to cover up to 100 domains / subdomains in one certificate.
4. LetsEncrypt implements a protocol called ACME (Automated Management Certificate Environment) that allows you to perpetually renew their certificates.
5. LetsEncrypt only issues 3 month certificates.
6. When LetsEncrypt ‘supports’ a hosting provider, it means you can use their automated software to renew your certificates. If your provider (*cough* Google App Engine *cough*) is not supported, you can still get a LetsEncrypt certificate (it just takes a little more effort), and you’ll need to manually renew it every three months.
7. (Not necessary info, but good to know) Amazon apparently has a certificate manager that automates this whole process (and renewal) for you.

Instructions:

1. Install CertBot, I used (running on Mac OSX):

brew install certbot

Although there are many ways to install it: https://certbot.eff.org/docs/install.html

2. Decide whether you want to verify your domain through DNS (adding a TXT record to your DNS, same thing you do to verify ownership for Google Analytics or Google WebMaster tools) or HTTP (uploading a file to your app).

For HTTP:

sudo certbot certonly --manual

For DNS (note: DNS can take a while to update, so this may be slower than HTTP):

sudo certbot certonly --manual --preferred-challenges dns

3. Follow instructions! CertBot will ask you to enter a list of space or comma separated domains. Then it will go one by one asking you to verify them by the method you chose in the previous step (DO NOT close the terminal while you complete these steps, then it will issue a new challenge).

For HTTP:

It will ask you to create a file at yourdomain.com/.well-known/acme-challenge/ABC with the value XYZ (note: ABC and XYZ are just placeholders, not actual values). Create folder called letsencrypt in your google app directory, place a file in it called ABC with the contents XYZ, and add the following handler to your app.yaml:

handlers:
- url: /.well-known/acme-challenge
  static_dir: letsencrypt

...

Then deploy the app (gcloud app deploy app.yaml)! Finally verify that it works (by visiting yourdomain.com/.well-known/acme-challenge/ABC and making sure it doesn’t 404). If that works, go ahead and press enter on the waiting certbot terminal.

For DNS:

It will ask you to add a TXT record to _acme-challenge.yourdomain.com or _acme-challenge.yoursubdomain.yourdomain.com with the contents XYZ (Note: XYZ is a placeholder, not a real value). Go to your domain name provider and add the record, for namecheap it was

type: TXT
host: _acme-challenge (or _acme-challenge.yoursubdomain)
value: XYZ

Then save it and wait for it to propagate. You’ll be able to tell when it’s done because the value XYZ will show up in the output of the DIG terminal command:

$ dig -t txt _acme-challenge.yourdomain.com

OR 

$ dig -t txt _acme-challenge.yoursubdomain.yourdomain.com

Once it appears press enter in the certbot terminal.

4. Congrats! Certbot has just congratulated you on the fact that you have a shiny new certificate waiting for you at: /etc/letsencrypt/live/yourdomain.com/fullchain.pem

Head over to Google’s App Engines SSL Certificates Pages (App Engine > Settings > SSL Certificates, or just https://console.cloud.google.com/appengine/settings/certificates), where you’ll be prompted to copy and paste a certificate and private key (note: you can also upload a file, but this is harder as only root has read permissions on /etc/letsencrypt/live/ and Chrome doesn’t have root permissions). The certificate can be uploaded as is, you can use the following command to copy it to your clipboard:

$ sudo cat /etc/letsencrypt/live/yourdomain.com/fullchain.pem | pbcopy

Before you can upload the private key though, you must convert it to an RSA private key (for the difference see here: https://stackoverflow.com/a/20065522/781199). To do this, run the following command:

$ sudo openssl rsa -inform pem -in /etc/letsencrypt/live/yourdomain.com/privkey.pem -outform pem > /etc/letsencrypt/live/yourdomain.com/rsaprivatekey.pem

(Note: if you want to write out to a file like that, you’ll need root permissions as only root has write permissions on /etc/letsencrypt/live/, to do that you can use “$ sudo su root”).

And then copy and paste it into the private key box on Google App Engine:

$ sudo cat /etc/letsencrypt/live/yourdomain.com/rsaprivatekey.pem | pbcopy

Done! Your site is now HTTPS protected! Don’t forget to renew your cert in 3 months 🔐

Let’s Encrypt and Google App Engine in 2017 was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

(This post motivates and describes applications of findings from Han, Mao, and Dally’s paper on neural net compression on TensorFlow Graph Files. Huge shout out to Michael Shumikhin, Anish Athalye, and Shai Szulanski — who helped motivate and advance this work.)

TL;DR Neural nets are big. Sometimes it’s useful when they’re smaller (so you can store more, or deploy them with less bandwidth). This paper sets forth ways to do theoretical compression of NNs. You can use off the shelf compression algorithms and our library to achieve 3x size reductions in TF graphfiles, without having to retrain the graph and with no (noticed) significant loss in accuracy.

Artificial neural networks have been gaining popularity over the past few years…

… is an understatement. Machine learning is possibly the most hyped field in computer science right now, and neural nets are used for almost all things ‘machine learning’: sentiment analysis, decision making, image classification, etc. Neural networks work by taking in a ‘training’ set of data (input and output pairs), and learning to predict the output based on the input; once trained, a neural network can predict the outputs for a ‘test’ set (data that it hasn’t previously seen). One example of an input and output pair might be the pixels of an image, and the output could be a value 0 or 1 indicating whether that image contains a dog. Another example might be an input of pixels from an image and the output is a 3-dimensional vector, where the first dimension is 0 or 1 to indicate the presence of a dog, the second dimension indicates the presence of a cat, and the third dimension indicates the presence of a bird. Neural networks in general have been shown to be be incredibly robust and are currently deployed in production at many top companies including Google/Facebook/Amazon/Etc.

The essence of neural networks is a sequence of weighted linear regressions, combined with transformations on the outputs of regressions (these transformations range from the identity function, to tanh, to ReLUs). The neural network can ‘learn’ these weights through an algorithm called back propagation, which relies on gradient descent to constantly tweak the weights to improve accuracy over the training set of data.

Note: This is an oversimplification, and CNNs and RNNs can complicate this — but it will work for demonstrative purposes. If you’re interested in learning more more, I highly recommend reading the intro from Stanford’s CS231n.

Remembering back to 8th grade algebra, linear regressions take in a value, and produce a value by a coefficient and adding a bias. This is the classic y=mx+b equation. Where m is the weight, and b is the bias. In multiple dimensions, this works much the same way, the only difference is that x and m are vectors.

In a neural network, where one is doing many, many high dimensional regressions —the values for weights and biases can take up a lot of memory. Using TensorFlow, a popular machine learning library open-sourced by Google, as an example: a trained deep mnist graph (which classifies handwritten digits, and is used as the getting started demo for TensorFlow) takes up 12MiB. The Google Inception V3 Graph (used for image classification) takes up 91MiB (found here, WARNING: large download). As we continue to iterate on these nets and store multiple versions, the size becomes problematic. It would be nice to have a smaller representation to store multiple versions, send over the network to deploy on servers (or self driving cars…), and store on mobile devices.

Crash Course in TensorFlow Graphs

TensorFlow structures computation in graphs. These graphs store operations (ops) and the relationship between the inputs of an operation and outputs from previous operations.

These relationships are stored as ‘tensors’, which are basically n-dimensional arrays whose values are not necessarily known and whose shape may be partially known. You can run a TensorFlow graph by feeding values into the graph for some Tensors and running them through operations until you’ve reached the desired output. For example, if I want to model the equation y=mx+b in TensorFlow, I can do it with the following Python:

import tensorflow as tf

y = tf.add(
    tf.mul(
        tf.Variable(2, name="m"),
        tf.placeholder(tf.int32, shape=1, name="x")
    ),
    tf.Variable(1, name="b")
)

As you can see the ops are: “Add”, “Mul”, “m”, “b”, and “x”. Tensors are named by OPERATION:OUTPUT_INDEX, so for example Mul has one output (the result of m*x), and it would be named Mul:0. Variables and placeholders (placeholders are Variables whose value must be fed on any graph run) are also operations with one output tensor (their own value). In this example, the value of m:0, x:0, b:0, are fairly self evident, the Mul operation takes in m:0 and x:0 as inputs to produce Mul:0, and Add takes in Mul:0 and b:0 to produce Add:0. You may notice there is no y in this graph, that is because y is effectively Add:0, although if we really wanted to you could also create a y variable and assign it to Add:0 (although that seems like overkill). To run this graph, one might feed in a value for x:0 and ask for Add:0, using a command like sess.run(["Add:0"], feed_dict={"x:0": (5,)}).

These graphs are encoded and stored in a GraphDef protobuf. Values for variables are not stored in a graph def (as they can be assigned over multiple executions) and are therefore not part of the graph; however, variables can be saved in a checkpoint file (which is basically a dictionary containing variable name and value… at least according to Google, the file type is much more opaque than GraphDefs). Variables are useful when we’re fitting a regression (or training neural net), but it doesn’t make much sense to keep them as variables once we know we will no longer be changing them. Instead we can convert them to constant values (either using the convert_variables_to_constants function or the freeze_graph tool distributed with TensorFlow) and simplify our graph. An example GraphDef after freezing our toy mx+b model contains a collection of nodes and (possibly) the value of the tensor they contain:

node {
  name: "m"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_INT32
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_INT32
        tensor_shape {
        }
        int_val: 2
      }
    }
  }
}

...

node {
  name: "Add"
  op: "Add"
  input: "Mul"
  input: "b/read"
  attr {
    key: "T"
    value {
      type: DT_INT32
    }
  }
}

Note: Tensors with values that have more than one scalar are stored as packed bytes in a string. They can be transformed to/from arrays with the tf.contrib.util.make_ndarray and .make_tensor_proto functions.

To read more on the variety of TensorFlow model files, see here.

Compression

https://github.com/tomasreimers/tensorflow-graph-compression

Wanting to compress TensorFlow Graph files, the first thing we did was try gzip, a popular compression tool. Gzip was not super effective. It did not significantly reduce the file size of the saved deep mnist graph (rounds to 12MiB before and after compression) and reduced the file size of the inception graph from 91MiB to 85MiB (even using gzip -9 compression). Facing similar results with other lossless compression algorithms, I decided to consider lossy neural net compression.

TensorFlow graph files are largely weights. Stripping the tensors containing the various weights from the GraphDef files, the deep mnist graph is only 3.7KiB and the inception graph is only 120KiB. Knowing that, I began to look for ways to lossy compress the weights.

Neural net weight imprecision has started to emerge as an area of research for a variety of reasons: hardware manufacturers are interested in it because it might allow them to make more imprecise hardware that can run neural nets faster (NVidia’s TensorRT has hardware level support for INT8 and Float16 modes to run nets faster), and software engineers are interested in it because it might allow them to reduce the space taken up by neural nets and possibly even software level speedups or energy efficiency.

The seminal research in the field of weight compression was conducted by Han, Mao, and Dally and showed that neural networks could be reduced in size with pruning, quantization, and Huffman coding. This research assumed that one still had access to the training set of the neural net; however, for our purposes we’re going to assume that we don’t have access to that dataset (for example, if we have a pre-trained net from a model zoo). This means that we can not do pruning (which built on Han’s previous work), as that would require being able to train the network; however we still can do quantization.

The insight behind quantization is that weights in a neural network are fundamentally imprecise by the very nature of the back-propagation algorithm, meaning that they don’t necessarily need all the precision we allocate for them. Currently, weights (assuming they’re standard 32b floats) can take on any one of 2³² different values, meaning that their representation must have a minimum of 32 bits. However, if we only let the weights take on 2⁸ values (8 is arbitrary, we can pick any integer < 32), we would only need 8 bits to represent them (you could imagine replacing each weight with an integer 0–255, and having a codebook of negligible size to translate the integer to a floating point value for that weight).

So now the problem becomes, how can we select 2⁸ values to appropriately represent all of our weights? Well this becomes a clustering problem, and Hans et al. did this using K-Means clustering (which finds a local optima) on the collection of scalar values from all of the weights, and replacing each scalar with its cluster centroid. We implemented this algorithm, and realized we could actually improve it (shout out to Shai), because the collection of the scalars in all the weights exists in one-dimensional space, and we can use dynamic programming to find the global optima for clustering in 1D in O(kN²) time and O(kN) space, where k is the number of clusters and N is the number of points being clustered. We implemented optimal clustering here, although it will likely need to be reimplemented in C++ if we want it to be fast enough to use on real nets.

Note: We have been minimizing the distance between each scalar and the value we replace it with, using a predetermined number of clusters; however if you have more knowledge of your net and want to use another loss function, you should check out this paper which proposes an O(N²) for clustering without a predetermined number of clusters.

Having replaced all the weight scalars with the centroid of their cluster, we should have the exact same file size because the cluster centroids still have the same amount of bits as the values they replaced. In order to see a size reduction we would need to implement a codebook and replace each weight with a lower bit representation indexing into the codebook. If we replace 32 bit weights with a 8 bit index into the codebook, this should achieve a 4x reduction. In addition, Han et al. point out that we could use Huffman coding to achieve even greater compression ratios (assuming the weights are not evenly distributed).

However, implementing a codebook with Huffman coding is painful (or at least time-consuming) and there is no reason we have to do it ourselves (Shout out to Michael for pointing this out): by replacing the weights with their centroids, we’ve increased redundancy in the file; this means that we can use off the shelf compression algorithms and they will create a codebook with Huffman encoding for us (but possibly less with a worse compression ratio than if we did the codebook manually).

Using GZip on the new files, we compressed mnist from 12MiB to 5MiB (using a global codebook) and inception from 91MiB to 37MiB (using a per layer codebook). We also found Facebook’s zstd to be exceptional and further reduce mnist to 4MiB and inception to 34MiB; while these are less than the 4x we should see in theory, it is still a higher compression ratio than we originally had and we’re not dependent on a custom encoding.

The nets with new weights did not show significant increases in error. Mnist with weight replacement achieved an accuracy of 0.9644(compared to 0.9642 before weight replacement… the improvement in accuracy is likely due to random chance). An interesting area of further research would be to try and encode the weights with even fewer bits (imagine a 2⁶ entry codebook, where each weight can be encoded in 6 bits) and graph the size/accuracy tradeoff.

I’ve open sourced the weight compression script, and it is available here. Improvements and pull requests welcome :)

Conclusions

Neural nets are incredibly powerful and being deployed in more and more places. Currently, the representation of neural nets is space intensive, and this might be a problem as we begin to export neural nets to more and more platforms (both mobile and web, consider projects like Keras.js which is incredibly exciting and pushing the frontiers of what’s possible, but whose demos requires the user to download ~100MiB of weights to run inception in the browser). We demonstrate that we can cluster weights to encode them to reduce file sizes. Additionally, we show there is no need to implement custom decoders because by simply replacing the weights with some similar value we can increase redundancy in the representation of the net and thus increase the compression ratios we can expect with off-the-shelf compression algorithms (added win: gzip is implemented in most servers and browsers by default ;)); all without too much loss of accuracy.

P.s. Throughout this project, Apple Finder’s notion of base-10 MB vs Terminal’s notion of base-2 MiB could be wildly confusing at times -_- Relevant XKCD.

Update: You can now also do something similar using TensorFlow’s quantize_weights graph transform.

When Smaller’s Better was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hi! We’re Tomas & Joe: student partners from two student-run venture capital firms, Rough Draft Ventures (RDV) and Dorm Room Fund (DRF). We spend a lot of time meeting with founders, learning about different markets, and listening to dozens of pitches.

After collectively meeting with hundreds of companies, we’ve found that there’s a pretty small set of simple questions we need ask to structure our thoughts. We wanted to post some of these questions online, with hopes that they might be helpful to current and aspiring founders.

1. Who are you?

This should be no surprise. When we are looking at early stage student-run companies, typically the most important thing to us is the team, since it’s often hard to assess the market and product potential at such an early stage.

2. What is the pain point you’re addressing?

You’ve probably heard some version of Maslow’s law of the instrument before: “If you have a hammer, everything looks like a nail.”

Reinterpreted for startups: Don’t fit a product to a problem. Lead with a real problem faced by real people and show us how your company arises from that. Can you quantify how bad the problem is? How big is the market for your solution?

3. What are you building?

Having covered team and pain point, we’re going to want to hear about the solution. Having a concise description is important, and being able to explain what that will be now, and a little while into the future is also helpful.

4. Is this problem best solved as a company? And is this a feature or a product?

A friend of ours once opined that “doing a company should be the last option you pursue when solving a problem.” If the problem can be solved by shifting a habit or writing up a good tutorial online, do that instead; even if it can be better solved by writing a plugin to a popular piece of software, do that instead.

If, and only if, the problem will be best solved by creating a company to develop and sustain a solution, that is when you should pursue that.

5. What have you achieved in the last 3 months?

We want to hear about how hard you’re hustling. If you come to us and say you have a great idea, don’t really need capital, but aren’t going to start until you have it … that shows us a wild lack of conviction. Prove to us that you’re driven by this problem and that you will solve it with or without us.

6. Why are you the one to build it?

Building a company is hard. With few exceptions, you probably are not the first to either consider this space or build a company in it. Why do you believe you’ll be able to?

This is especially important for college students wanting to build companies in spaces most college students know little about it. If you’re trying to build a product for advertisers, why are you qualified? With that said, there are many examples of college students disrupting fields they had no formal experience in, but explain to us why you’re passionate or what is your unique insight or experience.

7. Can you build it yourself?

If you’re building a technical product, one of your cofounders better be technical. If you’re building a company that uses “machine learning”, we’re going to ask what algorithms you’re using, who your ML person is, how long they’ve been doing ML, and so on…

8. If you have a cofounder, how is your relationship?

So many great company ideas go up in flames because the co-founders started fighting when running the company became difficult (i.e. user acquisition struggles, hiring struggles, difference in leadership styles, difference in visions, etc). How long have you two known each other? How’d you meet? And have you successfully worked on any projects before? Most importantly, are you actively working on the health of your relationship? What steps have you taken to build a sustainable and trusting partnership?

Some quality reading: How to pick a co-founder.

9. What’s your go-to-market?

Getting someone to initially trust your company is hard. How are you going to get your first few users? This is orders of magnitude harder if you’re building a two sided marketplace. Think about how you can start with individuals or small groups of users and get usage to grow from there (and no, “all their friends will also join the service at the same time” is not a good answer).

10. Why is it defensible?

Assuming your company goes well, and you start to scale and gather users, and are making 100s of billions a year, what’s to stop a large competitor from rolling out the same product? How will you build defensibility in that time so that someone else can’t straight up copy you? Does your product benefit from network effects? Is there a regulatory moat?

If you are a student team: both DRF & RDV are made up of students and invest around $20,000 in student-led startups. We’d love to chat and talk about your company more.

P.S. The startup launch list by Panda is a great resource of blog posts and articles written by thought leaders in technology and venture capital.

10 Questions for Student Startups was originally published in Rough Draft Ventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

Rough Draft Ventures is excited to announce our support of Wolfepack in their mission to supercharge personal computers through the development of external graphics solutions and supporting software.

Graphics cards are specialized pieces of hardware in almost every computer that perform computation related to the processing and rendering of images, videos, and animations. Their performance has huge implications on gaming, video/photo editing applications, and machine-learning processes. Despite their importance, many laptops (most notably, many MacBooks) lack powerful graphics cards because they’re large and drain the battery relatively quickly. That means graphic designers, serious gamers, programmers (basically, anyone who nerds out on computers) currently sacrifice quality graphics unless they’re on a desktop.

Wolfepack provides a level-up for consumers by giving their graphics a significant boost with the Wolfe and the Wolfe Pro. These external devices attach through thunderbolt and increase the compute power of your machine. The Wolfe gives users the portability of a laptop with the power of a high end desktop opening up new worlds for gaming and photo/video editing.

Early prototypes of the Wolfe have seen frame rate increases from 14fps (on the newest 15-inch MacBook) to 52.4fps (using the Wolfe) and 75.8fps (using the Wolfe Pro, all measured using the Heaven Benchmarking Suite).

The Wolfepack consists of three phenomenal Harvard founders: Dino, Christian, and Chase. All of them are passionate for building computers and gaming. The pack began working on the Wolfe in summer 2015 and has been working full time this summer.

We’re super excited to welcome them to the Rough Draft family! The Wolfe will be launching on Kickstarter in the next two weeks — signup on their website to stay updated!

Visit theWolfe.io to learn more and follow them on Facebook to keep up to date. Learn more about Rough Draft here.

The Wolfe: Supercharge your Laptop was originally published in Rough Draft Ventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

Rough Draft Ventures is excited to support Nikhil Srinivasan and Alex Kern as they build Pavlov, an artificial intelligence company using deep learning to help structure image and video data.

Machine learning is a field of study that gives computers the ability to learn and recognize patterns that are complex and hard to describe. While the technology is incredibly useful when making sense of large and unstructured data, the tooling that currently exists leaves a lot to be desired.

Pavlov is focused on training computers to understand what it sees in image and video data, helping businesses make sense of their rich media. Their first product, Pavlov Spot, helps companies detect copyright and trademark at scale via a simple API and web interface. Over the coming months, the company plans to help enterprise customers with drone, medical, and creative content gather and query insights.

The cofounders bring incredible domain expertise to the team, having previously built distributed data infrastructure at Apple, NASA, and Harvard Medical School. The company went through the inaugural YC Fellowship batch and is heads down taking their product to market. For a glimpse of what they’re up to check out GitHub.

We’re excited to welcome Pavlov to the Rough Draft family and support their their journey ahead!

Visit Pavlov to learn more and follow them on twitter to keep up to date. Learn more about Rough Draft here.

Pavlov: Image and Video Understanding was originally published in Rough Draft Ventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

My photography teacher, who I was very close with — and who is largely responsible for my interest in photography — used to joke that I talked too much. He told me that he wanted me to consolidate my thoughts, and so he assigned me the homework of writing journal entries on various topics: famous photographs, artistic styles, and techniques. The assignments weren’t graded on anything other than completion. He just wanted me to go through the exercise of writing down my thoughts.

To my surprise, I loved that assignment. The journal entries themselves were little more than my transcribed thoughts and the writing was awful; however, the process was helpful. It forced me to think through otherwise nebulous ideas. Articulating something — either through speech or text — forces you to consider the details. It’s easy when thoughts exist only in thought-space to deceive yourself into believing that you understand something or that your idea is fully formed.

My photography teacher loved the assignment because it meant that I talked less in class. I loved the assignment because I found that it helped me form my thoughts. Also, once they were on paper, it was easy to revisit and rework them.

Recently, a close friend and mentor gave me similar advice. I was describing a recent idea for a startup, and he told me to write a blog post about it. I told him I didn’t particularly want to share the idea with the world, and he responded that wasn’t the purpose of writing a blog post. It was to consolidate my thoughts.

The act of writing your thoughts down consolidates them, gives you a way to share them, and a medium to manipulate them in.

I think ‘blogging’ (and when I say that I don’t necessarily mean publishing — Medium and many other blog platforms have the ability to let you share drafts without making the draft public) is awesome for a variety of reasons. Two of the most obvious ones are that it forces you to consolidate your thoughts before you share them, and it also gives you a way to share your thoughts with other people. This is why my photography teacher and college mentor both encouraged me to write my thoughts down.

The other benefit that I think many people overlook is that writing your thoughts down gives you a medium to manipulate them in. In many ways, writing something down makes it tangible. I can edit, rework, and rewrite a paper. No one tries to do complicated math in their head, and the same should apply to other forms of thought: write them down so you can think through and work with specific parts.

A while ago I got into the habit of writing down my thoughts as ‘blog posts’ on Medium. Only recently have I started to revisit and edit these drafts. I don’t know how many I’ll publish yet, but I thought I would share the process because it’s been valuable to me so far.

Let me tell you a story:

Once upon a time, there were a group of friends in college interested in technology and entrepreneurship. In this city, there were many restaurants, and these restaurants sympathized with the crazy college-student work hours and stayed open late into the night to provide food after the dining halls closed. Unfortunately, these businesses didn’t have the capacity to deliver the barrage of 1am orders, and the students didn’t have an integrated and easy way to order delivery.

Equipped with coding skills, our main characters believed they could carve a niche in the space. While the market was dominated by companies such as GrubHub and Seamless, they believed that their product could target the local audience in a way that these companies didn’t. They built out their product, and within the first week of launching they got many orders…

At this point, the story has (at least) two possible endings:

(Ending #1) …within the first week of launching they got many orders, and that number grew, and grew, and grew. The company partnered with more and more restaurants and opened in more and more cities and eventually became a nationally-known company.

(Ending #2) …within the first week of launching they got many orders, and that number grew, and grew, and then the operation was abruptly halted by the university administration, which demanded the students stop business or face disciplinary action.

Those stories aren’t made up.

The first ending is the story of DoorDash, a Stanford-based startup that partnered with local businesses to deliver late-night food to students at Stanford. Eventually expanding to SF, Boston, DC, Chicago, LA, etc. They have raised almost $60M in 4 rounds and have been wildly successful as a company.

The second ending is the story of Instanomz, a Harvard-based startup that partnered with local businesses to deliver late-night food to students at Harvard. Once they gained traction, they were approached by the administration and told that they had to shut down, strike a deal with the HSA (Harvard Student Agencies, a monopoly on businesses that serve Harvard students), or face disciplinary action.

Now, let me start by saying that: I am not presenting this as insult or criticism, simply an observation about differences on campuses. I agree that these two aren’t identical cases. It is true that Instanomz was breaking a Harvard rule of a entering dorms as a 3rd party delivery service, but I can’t imagine DoorDash didn’t break some obscure rule in the student handbook. Or, if they truly didn’t, it’s possible that those kind of rules are stifling to innovation, and simply shouldn’t exist.

I am presenting this because I do not believe that all campuses are equally welcoming to innovation. And I think that when so many administrators are concerned about how to bring innovation to campus, a lot of them simply miss the point.

In some respects it’s almost paradoxical. On one hand, these are great universities that are doing their part to encourage innovation. At Harvard this means: having a new building for innovation; it means having a showcase for innovation; it means having an award for innovation in certain sectors.

On the other hand, in some respects it’s almost Social Network esq, that the administration acts as a barrier to innovation rather than a support, forcing students to choose between innovating and facing disciplinary action, or backing down. And that’s not how it should work. When universities are trying to increase innovation, they should encourage it, not hinder it. Encouraging innovation means that they need to be willing to let it change the way the world works.

It reminds me of a story that gained a lot of publicity about a year ago. A handful of students at Yale built a better way to index course data, called Yale Blue Book Plus (Yale Blue Book is the name of their course evaluations). The administration blocked their site and told them that if they didn’t remove it, it would result in disciplinary action.

If universities want more “innovation,” they need to be aware that innovation comes at the cost of changing the way students explore course data; they need to be aware that it causes students to run websites from their dorm room; and they need to be aware that sometimes it leads to disruption of monopolies and late night food delivery. Innovation comes at the cost of potentially disrupting the way we currently do things and changing our perspectives on the world. And so, if universities want to encourage more innovation, they need to focus on first building a culture that is open to growth and trying new things.