With industrialized agriculture, the world’s food production has changed from being biodiverse to focusing more on monoculture - a few selected crops. With a smaller and simpler product line, the food producers are able to specialize, and hence produce larger quantities at lower costs. This kind of production is often sold to only one or a few wholesalers who control the regional food market.

Even though an optimized and monocultural production may yield good return on investment today, it is a short sighted strategy that our ecosystems already have started to pay the price for. Loss of biodiversity is one of the biggest environmental challenges we face today, and the current food system is making it worse. A biodiverse food production is more resilient to rapid and unexpected changes, including climate change. Yet, only 9 plant species account for 2/3rd of the total crop production.

Small scale food producers who support biodiversity are going out of business every day. Because today’s system is rigged for buying and selling few products in large volumes, bigger and specialized production is taking over. You do not need to be a farmer, or in any way be schooled in this topic to see the evidence, just look at the shelves in your local grocery store. All vegetables are within the weight and size measure of what is expected for each category. For meat and dairy, only products the consumer is familiar with and demands are found in the shelves. The season is global.

In economics theory, there is a concept known as the long tail. Where a regular product company or shop may sell a limited number of unique products (let’s say less than thousand different products), marketplaces like Ebay, Craigslist or Airbnb have millions of listings, most of them completely unique.

A marketplace with a long tail distribution of products can give buyers endless opportunities to find exactly what they are looking for, and can revive unused products or increase profitability on low valued products. Take Airbnb as an example; the supply of housing has always been there, but it took a combination of engineers, designers, luck and timing to build a marketplace that made it possible to check in to someone’s apartment anywhere on the planet. By designing a product that made travelers belong anywhere, and hosts feel safe that their houses were in good hands, it is now possible to check in to this unexploited supply of housing.

To utilize the diverse and unexploited food resources, we have spent two years building a full stack marketplace enabling producers to sell directly to consumers. We want any food producer, big or small, to be able to compete on equal terms as wholesalers, and we want consumers to have the opportunity to buy directly from a wide variation of produce, and from the producers they trust and value.

In our first market, Norway, around 50 producers and 150 HoReCa consumers are trading food every week on our platform. By 2020 we plan on multiplying these numbers by 10x, and we are going to need world class engineers and designers to join our mission. Our 2020 roadmap is packed with essential functions that need to be built, and because we hope some of you reading this will join our team this year, we will give you a sneak peek into two of our top priorities on the product roadmap.

We believe the only way towards a more transparent food system is bringing consumers and producers closer together. The consumers should have as much insight into the production, as if they were sitting on the tractor together with the producers. By building planning tools that let producers and consumers collaborate from seed to fork, we help reducing the producers’ production risk. In particular, producers with highly seasonal products, take a huge risk when they start their production every year, not knowing if their crops will be sold. Let 2020 be the year of food futures.

To enable smaller food producers to compete with the whales, we have to build a system that lets them move aggregated parcels instead of single pallets. Today’s system is designed to sell huge volumes in single transactions to wholesalers who get to decide what produce is sellable and at which prices. We will decentralize the system and let more actors enter the market with many small transactions. Last year we built a logistics system for our last mile logistics partners and for internal operations, but soon all producers on our platform will get access to a logistics system where they can drop of parcels at hubs instead of spending time driving to every single customer. By aggregating drops and owning nothing but the digital infrastructure, we are able to do this at fraction of a wholeseller’s cost.

Our mission is to facilitate a fair market for all food producers, because we believe the world needs a transparent and resilient food system. As we enter 2020 we are looking for developers and designers to join our team. If you have read this story and got exited about redesigning a broken food system, send us your CV and a cover letter explaining your motivation to jobs@dagensmat.no or check our positions at Startupmatcher.

Når denne bloggposten skrives har Epigram allerede rukket å fylle ett år. I året som har gått har vi forsøkt å navigere i et hav av produktidéer og mulige forretningsmodeller, mens vi samtidig har lurt på hvilken selskapsform som lar seg best kombinere med visjonen om å bli landets fremste fagmiljø innen maskinlæring.

Da vi startet Epigram var vi enige om å starte et selskap som skulle være et inkluderende miljø for studenter og andre som ønsket å lære om maskinlæring. Vi var også enige om at Epigram skulle være en motvekt til all hypen rundt maskinlæring og kunstig intelligens, og kommunisere at bak all den tilsynelatende magien befinner det seg et håndfast og forståelig fag. I tillegg til å være inkluderende og faglig relevante, var det også viktig for oss at vi skulle bidra positivt til samfunnet.

Epigram består av et team teknologer som brenner for faget maskinlæring. Vår jobb er å være best mulig på vårt felt, og vi stoler på at våre kunder brenner like mye for sitt. Vi tror at summen av de to blir en bra match.

I Epigram er vi opptatt av at studenter skal få komme innom for å leke og lære slik at de blir best mulig i maskinlæring. Denne faglige styrken tar studentene med seg videre til kunder når de i fremtiden utfører konsulentoppdrag som ansatte i Epigram.

Maskinlæring er ingeniørkunst på samme måte som å bygge biler, hus og apper. På samme måte som at det ikke finnes en app for alt, så finnes det heller ingen generell kunstig intelligens for alt. Ihvertfall ikke ennå.

Det må fremdeles utvikles løsninger for spesifikke problemer, og det er disse løsningene menneskene i Epigram er flinke på å utvikle.

I året som har gått har vi fått lov til å vise frem hvor flinke vi er til kunder i næringslivet og hos forskningsinstitusjoner. Akkurat hva vi hjelper kundene med varierer med kundens behov.

I de fleste tilfeller handler det om å utvikle fullstendige løsninger som kunden kan plugge inn i sin eksisterende teknologi. Andre ganger bidrar vi som rådgivere for å gjøre kunden best mulig i stand til å utvikle løsningen selv.

Selv om vi i året som har gått har vært svært uenige om hvordan selskapet skal drives og hva vi skal tjene penger på, så har vi alltid vært helt enige om at det maskinlæringsfaglige skal stå i sentrum. Ved inngangen av 2018 ble vi også helt enige om at Epigram skal være et maskinæringsbyrå som hjelper næringslivet og forskningsinstitusjoner å ta i bruk maskinlæring for å løse kompliserte problemer.

Denne bloggen skal være stedet der både teknologer og ikke-tekniske kan svippe innom for å lære om maskinlæring. Mye av innholdet vil være rettet mot teknologer både til lands og utlands, men det skal også skrives poster som ufarliggjør maskinlæring og tar konsepter som Big Data, AI og Deep Learning ned på jorda igjen.

Vi synes også det er veldig kult om lesere har lyst til å skrive innlegg til Epigram-bloggen. Om du sitter inne med en bloggpost i magen, eller kanskje allerede har skrevet noe som passer her, ta kontakt med oss så skal vi få det publisert 😃

Besøk oss på vår nettside eller ✉️ oss på hello@epigram.ai 💻

Epigram AI — hvem, hva og hvorfor? was originally published in epigramAI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome back! In this part I’ll take you through the process of sending requests to your model. If you haven’t finished part 1 yet, I strongly recommend you do that first.

As I mentioned in part 1, the model server is not a regular HTTP endpoint that you can just send a POST request with your data to. It’s a gRPC service, and even though it may seem complicated at first, it’s actually a pretty cool concept.

What you need to know about gRPC

From gRPC’s own pages:

In gRPC a client application can directly call methods on a server application on a different machine as if it was a local object, making it easier for you to create distributed applications and services. As in many RPC systems, gRPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types.

This basically means that if you have an app where a user wants to see all his or her friends, when the user clicks “My Friends” in the UI, the client calls the function getFriends(). No need to send a GET request to the endpoint /getFriends. That sounds like a great concept! The server side code is automatically remotely available in the client. No more GET, POST, PUT, just call the methods! But wait, how can the client know that there’s a method called getFriends on the server? Is the client running the server side code too?

Actually, the answer is (almost) yes. The gRPC client side has a stub that provides all the methods the server has. Of course, when the data travel over the wire, it’s no more magic than HTTP going on. But it’s the way you as a developer write the communication that is different!

We’re not aiming to become gRPC experts in this tutorial (and I am not an expert on gRPC), but if you want to learn more I can recommend this 30 minute video. Don’t watch it now, do it when you have finished the tutorial.

For now, let’s focus on using a client I’ve already built. You don’t need to understand or know anything about gRPC to complete the next section.

Send requests to the model server

The client works without TensorFlow installed. So if you are using conda or virualenv (or similar environment tool), create a new environment (without TensorFlow of course…) if you don’t believe me.

The reason we don’t want TensorFlow client side is because anyone who wants to integrate with your model shouldn’t have to deal with TensorFlow to use it.

Imagine you wrote a kick ass image classifier that can separate hot dogs from not hot dogs. Now your friends who developed an awesome food app wants to use your model, but they know nothing about TensorFlow. So they say: “Hey, what’s the endpoint? Should we just POST an image to your API or …”.

Because you are serving the model with TensorFlow Serving and you know that none your app making friends have ever heard about gRPC, you hand them the client and show them how to use it. Give them a hostname, a model name and a model version number, and they’re all set!

Now, Go ahead and install the client. I should probably submit the client to PyPI, but for now:

pip install git+ssh://git@github.com/epigramai/tfserving-python-predict-client.git

❗️Some readers have had trouble installing the client using the ssh link, try pip install git+https://github.com/epigramai/tfserving-python-predict-client.git, if the ssh link does not work in your case.

Make sure the model you saved in part 1 is running. Start it again if you stopped the container when we cleaned up in the end of part 1. Then start Python and test the client:

>>> from predict_client.prod_client import ProdClient
>>> client = ProdClient('localhost:9000', 'simple', 1)
>>> req_data = [{'in_tensor_name': 'a', 'in_tensor_dtype': 'DT_INT32', 'data': 2}]
>>> client.predict(req_data)

I’ll go through what’s happening here in detail:

• The first line imports the client. ProdClient is the client you would use to send requests to an actual model server. I’ve written two other clients as well, they serve different purposes. More on those clients later.
• Pass the host, model name and version to the client’s constructor. Now the client knows which server it’s supposed to communicate with.
• To do a prediction we need to send some request data. As far as the client knows, the server on localhost:9000 could host any model. So we need to tell the client what our input tensor(s) are called, their data types and send data of the correct type and shape (exactly what you do when you pass a feed_dict to sess.run). Now, in_tensor_name ‘a’ is the same ‘a’ that we used in the signature definition in part 1. The input tensor’s data type must also match the one of the placeholder a from part 1. Note that we use the string ‘DT_INT32’ (and not tf.int32), I will show you where to find this mapping in the next section.
• Finally call client.predict and you should get this response:

{'add': 12}

Perfect! a + b = add, or in this case with actual values: 2 + 10 = 12.

That’s pretty much all you need to know to send requests. The next section is for those of you who wants dig deeper and understand how the client uses gRPC.

More about gRPC and the client

I am not going to make this section very long, but I’ll try to take you through the the process of sending a request and getting a response in more detail. I hope that will answer some of your questions (the same questions I had) on how to use TensorFlow Serving and gRPC.

All client code can be found here: https://github.com/epigramai/tfserving-python-predict-client. If you choose to clone the repo and play around with the code, make sure to uninstall the client we installed with pip earlier on.

You don’t need to understand the stuff we go through in this section to use the client. It’s just a more detailed explanation for those who want to learn more.

Protocol Buffers

Look at the files in the /proto folder. These files define a gRPC service for sending and receiving requests. They don’t look much like Python code, and that’s because they aren’t. These files can be used with any language to create a client, you just need the protocol buffer compiler protoc to compile them.

Python gRPC tools include the protoc compiler, and it can be installed with with pip:

python -m pip install grpcio-tools

To generate Python code, run:

python -m grpc_tools.protoc -I protos/ — python_out=predict_client/pbs — grpc_python_out=predict_client/pbs protos/*

❗️The generated files already exist in the repo, so you don’t have to do this! If you run this command you may have to fix some import statements to make the client work again :)

The files this command generates end up in predict_client/pbs/. I know some of these generated files are almost empty, and you are probably wondering what all this generated code does. The files in this folder are used by the production client, that’s all you need to know. Think of the content in this folder as compiled code, so don’t touch it.

Sending a request

Open predict_client/prod_client.py, and take a look at these two lines:

stub = PredictionServiceStub(channel)
...
request = PredictRequest()

What we’re doing here is creating the PredictionService which is defined in prediction_service.proto. The service object on the client is also known as a stub in the gRPC world. Then we create a PredictRequest message defined in predict.proto.

We assign the model spec in PredictRequest:

request.model_spec.name = self.model_name

if self.model_version > 0:
  request.model_spec.version.value = self.model_version

Now, if you peak into predict.proto you can see that the PredictRequest has the field model_spec and in model.proto you find the ModelSpec with fields name and version.

Create tensor proto that is compatible with the model we are serving:

tensor_proto = make_tensor_proto(d[‘data’], d[‘in_tensor_dtype’])

In the function predict_client.util.make_tensor_proto, you can see how I also assign DataType and TensorShapeProto to the TensorProto message. This is where the upper case DT_INT32 data type comes in. Look in types.proto to see all data types.

Add the tensor proto to the inputs map of the PredictRequest:

request.inputs[d[‘in_tensor_name’]].CopyFrom(tensor_proto)

The line map<string, TensorProto> inputs = 2; in predict.proto corresponds to the parameter inputs that we send to the signature definition back in part 1. We chose ‘a’ as our input tensor, that’s why the client must send a tensor proto with ‘a’ as the key in the inputs dictionary.

Finally we call stub.Predict() in the Python code and the request is sent.

Receiving the response

I‘ll be a little less verbose as we go from server and back to client:

predict_response = stub.Predict(request, timeout=request_timeout)

In the bottom of predict.proto, PredictResponse is defined. This message has one field: map<string, TensorProto> outputs = 1;. As you probably guessed already, the key in this map is our ‘add’ from part 1.

The rest of the process is simply converting this response to a dictionary.

InMemoryClient and MockClient

There are two other clients you can use. Those clients are for test and mock usage, they don’t send the request to a server. Documentation on how to use them can be found in the readme.

The mock client is basically just mock. Initiate it with some response on the same format you would expect from a real model. In cases where you are developing locally and you don’t have the model served and correct predictions doesn’t matter, use this client. For instance when you are developing an API on your laptop and the API expects a response from a heavy model, you can use the mock client to pretend to get the response from a real model server.

The in memory client is another way to pretend you have the model served. Instead of using fake data, we can “serve” the model instead. The in memory client loads the model into memory and acts as both client and server. This is convenient if you don’t have a model server at hand, but want to simulate how the client would work in a production setting.

❕ NB: TensorFlow must be installed to use the in memory client.

Wrapping it up

There are a few other prediction clients in open source GitHub repos out there. There are a few reasons I wanted to build my own:

• I wanted a client that is generic and supports all types of models where you send a dict and get a dict in response. And it should be installable and ready to use out of the box.
• Until now I’ve only found Python client code that use tf.contrib.util.make_tensor_proto to make the tensor proto. In my opinion the client should be TensorFlow agnostic. You shouldn’t have to install and import TensorFlow if you want to send requests to an already trained model.
• When I found out I had to understand how this actually works, I wanted to build my own and share my knowledge with the community.

Alright, that’s it for now. I hope you learned something about TensorFlow Serving and how you can use this client in your own projects even outside TensorFlow environments :)

Thank you for reading. I really hope you learned something! Leave a comment if you have questions or comments 😃

In the end I want to thank a few people:

• This is where I started about a year back: https://github.com/tobegit3hub/tensorflow_template_application. Thanks to tobegit3hub and this github issue.
• There’s a node client here that’s also TensorFlow agnostic. I used the .proto files from here to generate my own client. Thanks to shygiants for writing a node client.
• Thanks to dayzhang for open sourcinng dict-to-protobuf. Really helpful when converting a dictionary to protobuf without using tf.contrib.util.make_tensor_proto: https://github.com/davyzhang/dict-to-protobuf.

TensorFlow Serving 101 pt. 2 was originally published in epigramAI on Medium, where people are continuing the conversation by highlighting and responding to this story.

My goal with this tutorial is to explain, as straightforward as possible, how you can:

• Save a TensorFlow model so it can be loaded with TensorFlow Serving ModelServer and used in production.
• Serve your model with the TensorFlow Serving ModelServer.
• Send requests to your model (and get responses).

To get the most out of this tutorial you should be familiar with Python and have written a TensorFlow model before. This is not an introduction to TensorFlow itself, but to the serving system TensorFlow Serving.

I found it convenient to split the tutorial into two parts. In Part 1 you will learn how to save a simple model and serve it with TensorFlow Serving ModelServer. In Part 2 we will send requests to the model using a Python client.

Background

Many guides and blog posts have been written about how to save models for production and serve them using TensorFlow Serving. So, why would I write a tutorial when there are numerous guides and examples already written?

Because I have (and I believe others have too) spent countless hours reading tutorials, documentation, examples and blog posts on how to deploy TensorFlow models for production. Although it’s fairly easy to get your models up and running with TensorFlow Serving, there are many ways to do it and a lot of concepts to digest. These two (many ways to do stuff + new concepts) usually lead down the following paths:

• You spend way too much time exploring all the options and learning new concepts instead of just building what you were supposed to build in the first place.
• You copy some stuff you don’t really understand, but it works. When it eventually breaks or you want to build a new feature, you spend all your time figuring out how it works (again).

I have been doing a combination of these two for almost a year now. So here’s a straightforward way to save, serve and send requests to your model. This is how we do it at Epigram. We found a way that works for us, but keep in mind that TensorFlow Serving is a flexible serving system, and the way I see it, there’s no right or wrong in this space.

Alright, let’s get started. All code for part 1 of the tutorial can be found here: https://github.com/epigramai/tfserving-simple-example.

1. Write a simple model

In the snippet below I have written a simple model. Most of you have probably never written such a primitive TensorFlow like this (and why would you, it’s LSTMs and convolutional neural networks that rocks!). But all we need to save our model for production is a simple graph like the one below, so I am not going to do it more complicated that necessary.

Open export_simple_model.py if you want to get the full picture.

placeholder_name = 'a'
operation_name = 'add'

a = tf.placeholder(tf.int32, name=placeholder_name)
b = tf.constant(10)

# This is our model
add = tf.add(a, b, name=operation_name)

with tf.Session() as sess:

    # Run a few operations to make sure our model works
    ten_plus_two = sess.run(add, feed_dict={a: 2})
    print('10 + 2 = {}'.format(ten_plus_two))

    ten_plus_ten = sess.run(add, feed_dict={a: 10})
    print('10 + 10 = {}'.format(ten_plus_ten))

If you’re not completely new to TensorFlow, the code above should look more or less like the most primitive model you’ve ever written. You feed the model with a number, the model adds that number to 10 and returns the sum.

2. Save the model

In the example above we created a placeholder, a constant and an operation on the default graph. Then we started a session and ran the add operation. We could wrap this code in an API endpoint written in a Python framework like Flask, Falcon or similar, and voilá we have an API. But there are some really good reasons you don’t want to do it that way:

• If your model(s) are complex and run slowly on CPU, you would want to run your models on more accelerated hardware (like GPUs). Your API-microservice(s), on the other hand, usually run fine on CPU and they’re often running in “everything agnostic” Docker containers. In that case you may want to keep those two kinds of services on different hardware.
• If you start messing up your neat Docker images with heavy TensorFlow models, they grow in every possible direction (CPU usage, memory usage, container image size, and so on). You don’t want that.
• Let’s say your service uses multiple models written in different versions of TensorFlow. Using all those TensorFlow versions in your Python API at the same time is going to be a total mess.
• You could of course wrap one model into one API. Then you would have one service per model and you can run different services on different hardware. Perfect! Except, this is what TensorFlow Serving ModelServer is doing for you. So don’t go wrap an API around your Python code (where you’ve probably imported the entire tf library, tf.contrib, opencv, pandas, numpy, …). TensorFlow Serving ModelServer does that for you.
• Most importantly, the TensorFlow team wrote TensorFlow Serving and the ModelServer for a reason. They are probably better than you when writing a high performance serving system. Use it!

Great! Now that we’ve agreed on using TensorFlow Serving ModelServer to serve our model. It’s time to save the model. Here’s an outline for the rest of this section:

• First we have to grab the input and output tensors.
• Create a signature definition from the input and output tensors. The signature definition is what the model builder use in order to save something a model server can load.
• Save the model at a specified path where a server can load it from.

First we have to figure out which nodes are input and output nodes. Our simple math model is a + b = add. If we replace the constant, we get a + 10 = add. From placeholder a and operation add, we can grab input tensor a:0 and output tensor add:0.

In the snippet below we grab our input and output tensors, and build a signature definition that we will use to save the model.

# Pick out the model input and output
a_tensor = sess.graph.get_tensor_by_name(placeholder_name + ':0')
sum_tensor = sess.graph.get_tensor_by_name(operation_name + ':0')

model_input = build_tensor_info(a_tensor)
model_output = build_tensor_info(sum_tensor)

# Create a signature definition for tfserving
signature_definition = signature_def_utils.build_signature_def(
    inputs={placeholder_name: model_input},
    outputs={operation_name: model_output},
    method_name=signature_constants.PREDICT_METHOD_NAME)

We’re going to pause here for a few seconds. There is an important thing about input and output naming I want to point out.

The names placeholder_name and operation_name are the strings ‘a’ and ‘add’. You can use whatever strings you like to name the input and output of your models; ‘inputs’ and ‘outputs’ are also fine names. In fact TensorFlow has defined some constants for us that we can use! These constants are defined in signature_constants.py, and there are three sets of constants: predictions, classification and regression. If you peek into signature_constants.py you’ll see that the input and output constants are ‘inputs’ and ‘outputs’.

I am going to use ‘a’ and ‘add’ to show you that you don’t have to specify TensorFlows constants here. If you ever write a model with multiple inputs and outputs you have to come up with names yourself. In most cases using the constants makes sense. I am just showing you that it’s not required for your models to work with TensorFlow Serving ModelServer.

A place where you actually have to use a string TensorFlow has defined is the third keyword parameter called method_name. It must be one of tensorflow/serving/predict, tensorflow/serving/classify, tensorflow/serving/regress. They also defined in signature_constants.py as CLASSIFY_METHOD_NAME, PREDICT_METHOD_NAME and REGRESS_METHOD_NAME. I am not sure exactly why we need this to save the model, and the documentation is quite poor here. The model server will give you an error if you don’t use one of these constants.

Alright, let’s finish saving the model. We pass the path where we want the model stored to the builder. The last part of the path is the model version. For a model that is actually trained on real data, use this number to increase the model version when you retrain them.

Again, we will pass some constants that TensorFlow has defined for us to the builder. Pass the signature definition we defined in the previous snippet and save the model.

builder = saved_model_builder.SavedModelBuilder('./models/simple_model/1')

builder.add_meta_graph_and_variables(
    sess, [tag_constants.SERVING],
    signature_def_map={
        signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
            signature_definition
    })

# Save the model so we can serve it with a model server :)
builder.save()

Run the code and you should have your model ready for serving. If the code runs without errors you can find the model in models/simple_model/1.

3. Serving the model

If you finished the two previous steps without trouble, we are ready to serve the model now.

First, there are many ways to serve your models with TensorFlow Serving. Feel free to dive into TensorFlow Serving source code and build your own model server if you like to. I’ll stick to a simple CPU compiled server for the rest of the tutorial.

You can find the model server we’re going to use here. It can be installed with apt-get, but I already built a Docker image with the server so I’ll use that. If you are not familiar with Docker or use a Linux system with apt you can install the server on your system.

To serve the model all you need to do is run the file run_model.sh. I’ll take you through what’s going on when running that file.

docker run -it -p 9000:9000 --name simple -v $(pwd)/models/:/models/ epigramai/model-server:light --port=9000 --model_name=simple --model_base_path=/models/simple_model

The Docker image is publicly available and Docker wiIl download it for you when you run the script. If the script runs fine, you now have a running container. A container is basically an instance of an image. We pass a lot of options and flags here, so I’ll explain what they all do.

• When you do docker run, you run the image epigramai/model-server:light. The default entrypoint for this image is tensorflow_model_server. This means that when you run the container, you also start the model server.
• Because the model is not built into the image (remember, the image is just the model server) we make sure the container can find the model by mounting (-v) the models/ folder to the container.
• The -it option basically tells docker to show you the logs right in the terminal and not run in the background. The name option is just the name of the container, this has nothing to do with TensorFlow or the model.
• Then there’s the -p option, and this one is important. This option tells docker to map its internal port 9000 out to port 9000 of the outside world. The outside world in this case is your computer also known as localhost. If we omitted this option, the model server would serve your model on port 9000 inside the container, but you would not be able to send requests to it from your computer.
• The three last flags are sent all the way to the model server. The port is 9000 (yep, the port we are mapping out to your machine). With the model_name flag we give our model a name. And with the last flag we tell the model server where the model is located. Again, the model is not in the image, but because we used the -v option and mounted the folder to the container, the model server can find the model inside the running container.

❕ If it looks like I am mixing flags and options here, I am not. The options are for Docker (the ones with a single -) and the flags (port, model_name, model_base_path) are parameters to the model server.

If you see an output like this, you are now serving the model with TensorFlow Serving!

❕The model server installed in epigramai/model-server:light is compiled for newer CPUs. Most computer these days should be able to use it. If you get an error when you run the container, you could try this image instead: epigramai/model-server:light-universal. If you the read the install instructions on TensorFlow Serving pages, you can also find tensorflow-model-server-universal there. That’s the same one I’ve built into epigramai/model-server:light-universal.

Congrats! You are now serving a model with TensorFlow Serving. I wish I could tell you that the model server is a normal HTTP server and that you can POST your data and get a response. Unfortunately, it’s not that simple. That’s why I chose to split this tutorial in two parts. The model server runs something called a gRPC service, and I’ll tell you need to know about it (and how to send requests) in part 2.

You probably want to clean up a bit. If you are familiar with Docker, you know that hitting ctrl + c should stop a running container. Unfortunately, it doesn’t work in this case (at least for me, and I am not really sure why). So open a new terminal tab and run:

docker stop simple && docker rm simple

If you cannot open a new terminal tab, it’s possible to hit ctrl + pq to detach from the container to get a free terminal window.

I like to point things out, so I am going to point out that simple in the two commands above is related to Docker and is the container name. We gave the model and container the same name.

Is this really production ready? What happened to GPUs?

The short answer to this question is: Yes, the model you saved can be hosted on any TensorFlow Serving system. Yay! If your model is complex and should be served on accelerated hardware like GPUs, you cannot use the Docker image / the apt-get version of the model server we use in this tutorial. I am not going to include GPU serving in the scope of this tutorial, but I may write up a new blog post on that topic if I get any requests from readers.

Thank you for reading, I really hope you learned something! Leave a comment if you have questions or comments. Looking forward to see you in part 2 😃

TensorFlow Serving 101 pt. 1 was originally published in epigramAI on Medium, where people are continuing the conversation by highlighting and responding to this story.