If you’re interested in my occasional posts about AWS, Python, iOS, Mac, …, you can continue reading them at https://robertvojta.com. It’s a Wordpress blog and you can follow me via:

• https://robertvojta.com/feed/rss2
• https://twitter.com/robertvojta

Or replace rss2 with rss, atom or rdf.

All Medium posts were imported to the new blog. Also I asked Medium support to remove custom domain robertvojta.com from this publication. If you have bookmarks to my posts, don’t worry, I did setup redirects for all of them. They still work.

Cu elsewhere :)

P.S. Will keep this publication alive for a while before I trash it.

Leaving Medium was originally published in Zrzka’s adventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

My career is kind of colourful. I was working on aircraft systems, VoIP HW & SW, developing Linux kernel drivers, Linux distributions, OpenOffice.org and many other low level (RTOS) & high level (desktop applications) stuff. Linux nerd. I said enough one day. I wanted to just work and wasn’t willing to continue wasting my time with searching how to workaround this and that. Bye bye Linux, did spend nice 10 years with you. Not saying it was a bad experience. Nope, I learned a lot and enjoyed it. But …

Fast forward, I’m with Apple for more than 10 years now. Hmm, another 10 years, time to leave? Nope! Honestly, not excited as I was when I switched, but still pretty happy. Remember the first iPhone? iPad? Magical devices. Not mentioning first SDK. Exciting times. I did develop lot of applications for iPhones, iPads and for the desktop. Learned a lot about Objective-C, Objective-C runtime and all these Apple platforms.

I also bought every iPad Apple did release. But I didn’t know what to do with it. Kind of device for content consumers. I tried, really hard, to use it as a device where I can create something. But I always struggled, I always gave up and go back to my Mac. Apparently I was trying to use it for something it wasn’t designed for. Or it was, but iOS itself wasn’t ready for it.

My needs changed few months ago. I no longer work as a full time developer, but rather trying to lead people and working on new product specifications. Hard change. Lot of new things to learn.

Do I need to travel with my 15" MacBook Pro for this kind of work? Am I willing to try iPad again? Yup, I am. Bought the iPad Pro 12.9", external keyboard and pencil few months ago. iOS 11, Files, drag & drop, wonderful applications like Workflow, OmniGraffle, Editorial, Ulysses, MindNode, etc. I have to say that the iPad Pro makes sense as a device where I can create stuff finally. Took quite some time, but it seems it’s already here. Suits my needs perfectly.

What all this has to do with Python IDE on the iPad Pro? I’m no longer working on our products, but I didn’t left programming. And I never will. It’s quite refreshing to hide myself with all these devices in some place where no one can disturb me from time to time. We do use Amazon Web Services a lot. I’m still kinda involved in scripting, support tools, … and we do write all these things in Python. Amazon provides nice package named boto3:

Boto is the Amazon Web Services (AWS) SDK for Python, which allows Python developers to write software that makes use of Amazon services like S3 and EC2. Boto provides an easy to use, object-oriented API as well as low-level direct service access.

I wrote lot of tools, scripts, … leveraging boto3 power to maintain our AWS services. Also wrote some django applications for internal use. Can I do this on my iPad Pro? Quite some effort, but yes, I can! And now, you're finally going to learn how.

Pythonista for iOS

Hats off to Ole for writing Pythonista for iOS. Seriously, tremendous amount of work. Quote from the Pythonista website:

Pythonista is a complete development environment for writing Python™ scripts on your iPad or iPhone. Lots of examples are included — from games and animations to plotting, image manipulation, custom user interfaces, and automation scripts.

In addition to the powerful standard library, Pythonista provides extensive support for interacting with native iOS features, like contacts, reminders, photos, location data, and more.

Pythonista supports Python 2.7, 3.5 (beta includes 3.6), standard library, lot of bundled packages and neat support for native iOS features.

Release cycle of Pythonista is quite long. Version 3.0 was released an year ago, version 3.1 just a few days ago. This release cycle leads to a situation, where bundled packages can be outdated and native iOS features doesn’t support latest & greatest stuff — drag & drop, Files.app, etc.

What can be done about this? There’s project named StaSh — Shell for Pythonista. Do I need shell you say? Bet you do. StaSh does contain useful commands like pip, which allows you to install additional packages from PyPI. Not all packages are supported. Package must be pure Python, because of the way how binaries, signatures, … work on iOS. It's more complicated, but I'm not going to explain it here. Just try it and you’ll see if your package will be installed or not.

Remember the first iPhone OS SDK? First apps? Did you ever think this will be possible? And we just started.

You should really buy Pythonista if you’re into Python.

Dark side of the moon

You’re excited, you started to love Pythonista, but you slowly do realize that you miss something. Keyboard shortcuts, features like Jump to definition, Open quickly and lot of other things. Stuff which makes great IDE great.

I started filling issues as a first thing. There’s more than 330 open issues. Hmm, how quickly will Ole solve them? Am I willing to wait weeks, months, …? Isn’t my use case minor one? Will they ever be fixed? Lot of questions. I’m a problem solver guy, so, I decided to fix it by myself. That’s how the Black Mamba project started.

Prerequisites

I assume that you’re familiar with following topics:

• Objective-C Runtime — messaging, signatures and type encoding,
• Event handling, responders and the responder chain — UIResponder, UIKeyCommand and UIEvent.

Also I highly recommend to look at ctypes and objc_util in advance.

Keyboard shortcuts

Pythonista provides small amount of shortcuts and I can’t work without them. Still consider screen tapping as a slow thing. Let’s add some of them. We have two basic requirements:

• be at the end of the responder chain, just to avoid clashes with Pythonista view controllers shortcuts,
• have them working even if the view controller hierarchy changes over time.

It seems that the keyCommands property of the UIApplication perfectly suits our needs. When compared to the UIViewController, there’s no addKeyCommand: and we have to found another way how to modify keyCommands result. Time to poke Objective-C runtime, especially class_addMethod and method_exchangeImplementations functions. First one allows us to add new methods and second one allows us to exchange their implementations. It’s called swizzling. We will need to use these functions more than once, so, here are Python wrappers:

• add_method — adds new method to the class,
• swizzle — adds new method to the class and exchange implementations with the original one.

Ready for swizzling? Let’s write custom keyCommands function in Python:

def _blackmamba_keyCommands(_self, _cmd):
    obj = ObjCInstance(_self)
    commands = list(obj.originalkeyCommands() or [])
    commands.extend(_key_commands)
    return ns(commands).ptr

Simple one, which gets the original list of key commands and adds our ones from the _key_commands list (module global variable). We do not need to add new method explicitly, our swizzle function does it automatically:

swizzle('UIApplication', 'keyCommands', _blackmamba_keyCommands)

What happened? A picture is worth a thousand words:

The only remaining thing now is to introduce function, which allows us to register keyboard shortcuts — register_key_command. Signature of this function follows.

def _register_key_command(input, modifier_flags, function, title=None)

• input - it can be any string (like letter a for example) or UIKeyInput enum, which does contain values for special keys like left arrow.
• modifier_flags - it can be int (if you know the correct value) or you can use UIKeyModifier enum.
• function - Python function, no arguments, to call when user presses the shortcut.
• title - optional str, discoverability title.

How to print Hallo with Cmd H?

from blackmamba.uikit.keyboard import (
   register_key_command, UIKeyModifier
)
def hallo():
   print('Hallo')
register_key_command(
   'h',
   UIKeyModifier.command,
   hallo
)

Passed function hallo is wrapped with a function, which does meet Objective-C runtime specs.

def key_command_action(_sel, _cmd, sender):
        function()

Based on input and modifier_flags, method name is generated. blackMambaHandleKeyCommandH: in this case.

blackMambaHandleKeyCommandH: method is added to the UIApplication with implementation pointing to the key_command_action. UIKeyCommand object is created and stored in the _key_commands global list.

And that’s it. We have a way how to register custom keyboard shortcut, assign it to a function written in Python.

Pretty awesome and I can’t still believe it’s possible.

Keyboard events

Imagine you have a dialog and you would like to use keyboard shortcuts for it as well. Like arrow keys to change selection, enter to confirm selection, Cmd . to close dialog, etc. Global shortcuts can be reused for this task, but it will bring another sort of issues. We will be forced to remember which dialog is active, pass events to it from our shortcut handler, forget the dialog when it’s closed, … Sounds complicated to me.

Let’s solve this task in a similar way, by swizzling handleKeyUIEvent: (UIApplication). This method receives physical keyboard events (type = 4) and we can handle them there.

This is private API, don’t do this in your applications.

Because it’s very similar to global keyboard shortcuts, here’re just links to specific implementations:

• _blackmamba_handleKeyUIEvent — our implementation of handleKeyUIEvent:,
• register_key_event_handler — event registration,
• unregister_key_event_handler — event unregistration.

And an example how to register and unregister keyboard events.

Drag & drop

There was no drag & drop support prior to Pythonista (311013, beta). I desperately did want to use it and wrote another script — drag_and_drop.py.

This script demonstrates how to create custom classes, classes that implements protocols like UITableViewDragDelegate and how to use blocks. I think it’s pretty self explanatory except one thing.

There’s experimental support of blocks in Pythonista. You can write a Python function and create block from it. Here’s an example:

def block_imp(_cmd):
   print('Hallo')
block = ObjCBlock(
   block_imp,
   restype=ctypes.c_void_p,
   argtypes=[ctypes.c_void_p, ctypes.c_void_p]
)

If you need to pass block as an argument somewhere, you can just pass block variable now and that’s it. Works. But what if you have to implement block, which receives another block as an argument. There’s no support for it in Pythonista.

I wrote the ObjCBlockPointer class, which can be initialised with the block pointer. And then you can call it. Here’s an example how to use this class. Drag session delegate, which is supposed to load data and then call another block when the operation completes. It’s achieved via ObjCBlockPointer class.

Conclusion

Do you need additional packages like me? Just install StaSh and use pip. I did install django, boto3 and many other packages. Do you miss some functionality in Pythonista? Install Black Mamba.

I like Pythonista for iOS a lot. I do use it almost on a daily basis even when it’s not perfect (nothing’s perfect) and lacks some features. But because it’s Python IDE, we have an access to the Objective-C runtime, nothing can stop you from enhancing it on your own. Like I did. See the gallery of scripts I wrote.

Can the iPad Pro replace my desktop computer? Yup, it can. There’re still some edge cases where I rather use my MacBook Pro, but the number of them is very low.

Did you ever think that it will be possible to do these kind of things? In an App Store application, no jailbreak, … Me not, never, until now. And I can’t still kind of believe it.

Learn or die and remember, sky is the limit :)

Again, many thanks to Ole for writing this wonderful piece of software! Please, buy it to support Ole’s ongoing effort to make it better.

Links

• Pythonista for iOS
• Working Copy for iOS
• StaSh
• Black Mamba documentation
• Black Mamba repository

Django notes

If you’d like to run Django application on your iPad Pro, don’t forget to pass following arguments:

dir = os.path.abspath(os.path.join(os.path.dirname(path), '../application'))
sys.path.append(dir)
# import Django execute_from_command_line
arguments = list(sys.argv)
arguments.append('runserver')
arguments.append('--noreload')
arguments.append('--nothreading')
arguments.append('-v')
arguments.append('0')
logging.disable(logging.CRITICAL)
try:
    execute_from_command_line(arguments)
except KeyboardInterrupt:
    logging.disable(logging.NOTSET)
    sys.path.remove(dir)

Why these arguments? Django is very talkative and it is constantly filling your console with messages. Every single time there’s a new message, Pythonista will reveal console, tab with your app becomes inactive and you have to tap on it to reveal your application. Kind of … That’s the reason for -v 0 and disabled logging.

No threading & reloading is here, because Pythonista supports only one Python thread.

Still some limitations, but better than nothing. Otherwise it works perfectly.

Python IDE on the iPad Pro was originally published in Zrzka’s adventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

• Does increased memory size help?
• Is this huge cold start time VPC related?
• Is there a difference when we do use another language?
• When the lambda container is reused?
• Does some kind of keep alive requests help?

Internet is full of answers, but they’re not clear. Wild guesses. So, I decided to quickly hack benchmark to get more accurate (still not precise) answers.

Scroll down if you’re not interested in details, but want to see results only.

Lambda

Limits

Always learn limits of your enemy. What’s interesting for now is number of concurrent executions. Default limit is 100.

Cold vs Hot

Lambda function runs in a sandbox environment; container. Containers can be reused. Container created for lambda A can’t be reused for lambda B, just for lambda A. There’s no clear answer when they’re reused. But I would like to know.

Every container has ephemeral disk capacity (0.5GB) mounted to /tmp. Whatever lambda stores there, it stays there if container is reused. My reuse test is very simple. Container is reused if file /tmp/perf.txt exists. If not, lambda is going to create it. Check Python, JavaScript and ClojureScript lambda function handlers.

More info about container reuse on the AWS blog. Old, but enough to get a clue.

Variants

I would like to see if there’s a difference between languages, memory sizes and VPC [not] set. I prepared several variants:

• Lambda functions in Python, JavaScript and ClojureScript
• Each configured with 128, 256, 512, 768, 1024, 1280, 1536MB of memory
• Each with and without VPC

All possible combinations, 42 lambda functions.

NOTE: There’s no way how configure CPU for lambda function. It’s tightly tied with memory size. Bigger memory size, faster CPU. This is the reason why I made different memory variants as well.

API Gateway

Limits

Again, your enemy limits. Requests per seconds default limit is 1,000. Burst 2,000. This limit is per account; don’t kill your other APIs with stress testing.

VPC

Limits

We’re interested in ENI limits. You can have 350 network interfaces per account. And here are limits per instance. I’ve got t1.micro for the benchmark purpose, so, the limit is 2.

Lambda

Lambda function is not able to access VPC resources by default. You have to set at least one subnet and at least one security group from the same VPC to allow it to access VPC resources.

NOTE: If you’re using ClojureScript, there’s pull request with VPC support. Already merged, 0.6.3-SNAPSHOT published, you can test it.

Time to learn more about ENI. It’s a virtual network interface attached to EC2 instance in your VPC. ENI can be created, deleted, attached, detached, … How does it work with Lambda container? ENI is created (or reused) during lambda/container initialisation process. This ENI is used within lambda container to allow it to communicate with your instance. Lambda ends, container is suspended, but ENI is still there. ENI is deleted when the lambda container is deleted. It takes some time and I want to know when approximately.

Also think about 350 ENIs limit when you’re designing your infrastructure. If there’s no ENI available, lambda ends with internal server error and API Gateway returns 500:

Lambda was not able to create an ENI in the VPC of the Lambda function because the limit for Network Interfaces has been reached.

Lambda role must have policy allowing to create ENI, delete ENI, … Do not create your own policies, but stick with the service one created and maintained by the AWS guys — AWSLambdaVPCAccessExecutionRole:

arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole

BTW I tried to create my own policy. I did forget to add delete interface action and drained ENI pool pretty quickly. Managed to create 352 ENIs (limit is 350).

Benchmark

Methodology

None. I just wanted to see some numbers and I quickly hacked Python script which sleeps for 0, 5, 10, 15, 30, 60, 300, 600, 900, 1800, 2700 and 3600 seconds. And for every sleep cycle:

• Sequentially benchmarks predefined endpoints (/js/128, /js/256, …)
• Fires 30 requests per endpoint via 10 workers (parallel requests)
• Gathers each request-response duration and container reuse info

It fires 15,120 requests in total and it runs for more than two hours. Sleep is there to approximate time when the container is reused and when it isn’t.

Should be enough to get some data and it also shouldn’t trigger throttling on the AWS side.

Technical

Always run redeploy.sh script before you run this benchmark. Lambda functions are redeployed and all containers are trashed. You can check that the first endpoint requests (up to 10) should return no container reuse.

Output

Script generates CSV file with following values:

• sleep — in which sleep cycle (group) were requests issued (seconds)
• start — exact date and time when the batch of 30 requests was fired (IOW first request start date time from the particular batch)
• path — API Gateway path
• lang — lambda function language (js, python, cljs)
• vpc — lambda with VPC? (1 yes, 0 no)
• memory — lambda memory size (MB)
• workers — number of workers in this batch (parallel requests)
• min — fastest request-response time (milliseconds)
• max — slowest request-response time (milliseconds)
• mean — mean request-response time (milliseconds)
• d1, d2, …, d30 — first, second, … request-response duration (milliseconds)
• r1, r2, …, r30 — first, second, … request-response lambda container reuse info (1 reused, 0 not reused)

Environment conditions

No lab, no equal conditions, … Do not compare absolute values. I’m benchmarking it before lunch, in the afternoon, during night, … On my VDSL lines, over the air (LTE), at home, at work, … The only thing which is constant is region (eu-west-1) and benchmark settings (endpoints, memory sizes, VPC, number of requests per endpoint and number of workers).

Analysis

We’re going to focus on vdsl-home-morning data set for the purpose of this analysis.

Script fired 15,120 requests. Simple sort in Numbers and we already know that the slowest response took 13,106ms (13s) and the fastest one 191ms. There’re …

• 177 requests (1.2%) slower than 10s
• 403 requests (2.7%) slower than 5s
• 539 requests (3.6%) slower than 2s
• 723 requests (4.8%) slower than 1s
• 1,921 requests (8.5%) slower than 0.5s

Pretty big deviation and kind of disappointment. CSV table, raw data, … all these things are not clear. Let’s dive in, use R, plot some charts, check what’s wrong and try to fix it.

Following part was done with cooperation with Tomáš Bouda. Our math, stats, … magician. Many thanks Tobbi!

Overview

First thing to check is histogram. It shows that lot of requests are quick, but there’s something weird going on around 10 seconds.

Let’s zoom in left side (limit time to 150–500ms) to check typical request duration. It’s somewhere in the 200–300ms range. Which is not super quick, but it’s not 10 seconds or more.

Histogram is not going to help us more, we’re going to try box plot. Much better information.

Left half is without VPC. Right half is with VPC. As you can see, we have lot of requests with duration greater than 5s. To be more precise, with duration in the 7.5-12.5s range.

We have lot of input variables and we have to get rid of them. Let’s limit time to 200-500ms in this box plot.

What does it say? If everything works as expected, there’s no difference between languages and memory sizes.

We can safely ignore language and memory size variables. What do we have now? VPC and lambda container reuse variables.

Lambda containers reuse

X axis shows when requests were fired and Y axis shows their duration.

First batch has several durations around 10s. That’s “okay”, because lambda functions were redeployed before benchmark and there were no containers to reuse. But what we can see is that containers weren’t reused about one hour after the benchmark start.

Let’s find more precise time and use more colours. Sleep time of 900s (15m) looks okay. Sleep time of 1800s (30m) doesn’t. We can say that containers are not reused after 15 minutes (rough estimate).

Some numbers and explanation from Tobbi:

To ensure ourselves let’s check the numbers. 2nd and 3rd columns contain sample mean and sample standard deviation. We can see that most of the means are below 300ms. The bold rows have higher means since right-skewed distributions tend to pull mean off.

4th to 6th columns contain a probability that call remains below given time limit using Gaussian distribution, e.g. norm.1000 represents probability the call is below 1000ms. Let’s ignore the bold rows since these distributions are bi-modal and not Gaussian.

The last two columns contain empirical probabilities based on the test data, e.g. emp.500 show the probability the call remains under 500ms.

As we can see, we have pretty reasonable times (below 1 second) for up to 900s gaps.

sleep  mean    sd norm.500 norm.1000 norm.2000 emp.500 emp.1000
    0   939  2324    0.425     0.510     0.676   0.816    0.906
    5   282    69    0.999     1.000     1.000   0.999    0.999
   10   282    82    0.996     1.000     1.000   0.991    0.998
   15   277    43    1.000     1.000     1.000   0.998    1.000
   30   283    74    0.998     1.000     1.000   0.990    0.998
   60   275    38    1.000     1.000     1.000   1.000    1.000
  300   287   123    0.957     1.000     1.000   0.985    0.998
  600   293   204    0.843     1.000     1.000   0.960    0.991
  900   294   153    0.909     1.000     1.000   0.933    0.994
 1800   618  1757    0.473     0.586     0.784   0.867    0.960
 2700  1423  2945    0.377     0.443     0.578   0.756    0.832
 3600  1652  3089    0.355     0.416     0.545   0.679    0.751

VPC

Last thing we have to find is how VPC configuration influences all these durations. We’re lucky with another box plot.

X axis shows if the lambda container was (1) or wasn’t (0) reused. Y axis shows the request duration.

There’s no difference if container is reused. VPC can be set or not. Huge difference is when the container is not reused.

We can say that these unacceptable durations come from lambda functions with VPC when their containers are not reused.

Keep alive containers

We already found that these containers are not reused after ~15 minutes. I tried to keep them alive with quickly hacked keep alive script.

This script fires 10 requests with 10 workers per endpoint to keep alive 10 containers. That’s because our benchmark script also do use 10 workers. These requests are fired in 15 minutes interval. What happened?

Third data set (ka-15-r-10-vdsl-home-afternoon, keep alive time interval 15m, 10 containers) has one duration just below 5s and rest is in the 0-2.5s range.

Let’s look at the ka-15-r-10-vdsl-home-afternoon data set directly.

Huge improvement. 10s requests are gone. Compare against vdsl-home-morning data set.

We can say victory ;-) Not victory actually, but huge progress at least.

Conclusion

Be aware that our testing lambda functions are very simple. They just returns lambda container reuse info. These numbers and findings can differ if your lambda has lot of dependencies, does lot of other things, etc.

Based on these facts, we can say that:

• Lambda function language (Python, ClojureScript or JavaScript) has no influence on lambda container initialisation time
• Memory size has no influence on lambda container initialisation time
• VPC has huge impact on lambda container initialisation time when the lambda container is not reused, but have no impact at all (or negligible) when the lambda container is reused
• Lambda containers are not reused after ~15 minutes
• If we issue keep alive requests in the 15 minutes interval, very long request durations (> 5s) are rare

Recommendation?

Use CloudWatch to issue keep alive functions in the 15 minutes interval if you’re using VPC, API Gateway & Lambda functions.

UPDATE: Configurable keep alive function in Python, CloudWatch event rule and CloudFormation template can be found here.

This will not solve your issue completely, but will help a lot. Especially if your lambda function is rarely used and containers are not reused. It’s a nightmare to explain to your user that he has to wait for 15s to get some response from your application.

Not sure if AWS guys will be happy, but what else can we do?

Do you want to play with this benchmark, roll your own analysis, …? Here’s the repository link with all lambda functions, scripts, data sets and HTML outputs.

AWS journey — API Gateway & Lambda & VPC performance was originally published in Zrzka’s adventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hooray (irony), we have to switch to the EC2 Container Service. Try to explain this to our business department. Additional time for infrastructure. Can handle it, but it’s tough sometimes.

Docker images repository

Cloud Formation Template

This step is optional, but we’re trying to leverage as many as possible AWS services. You need a repository for your Docker images. And because I like Cloud Formation, here’s the template:

"AthenaRepository": {
  "Type": "AWS::ECR::Repository",
  "Condition": "DevelopmentEnv",
  "Properties": {
    "RepositoryName" : "athena",
    "RepositoryPolicyText" : {
      "Version": "2008-10-17",
      "Statement": [
        {
          "Sid": "ProductionAccountPullAccess",
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::$ANOTHER-ACCOUNT-ID:root"
          },
          "Action": [
            "ecr:GetDownloadUrlForLayer",
            "ecr:BatchGetImage",
            "ecr:BatchCheckLayerAvailability"
          ]
        }
      ]
    }
  }
}

Why RepositoryPolicyText? We have several AWS accounts, but we want just one repository. This policy gives pull access to $ANOTHER-ACCOUNT-ID account.

Why DevelopmentEnv condition? Same reason, just one repo per several AWS accounts. We do use same templates for all accounts. They’re parametrised.

Push images

There’s nothing new. Just replace docker login … with $(aws ecr get-login).

$(aws ecr get-login)

docker build -t athena .

docker tag athena $AWS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/athena:build-$TRAVIS_BUILD_NUMBER

docker push $AWS_ACCOUNT_ID.dkr.ecr.eu-west-1.amazonaws.com/athena:build-$TRAVIS_BUILD_NUMBER

EC2 instance

Cluster

First of all, you need cluster. There’s no way how to deploy your services without it. Cloud Formation template:

"ECSCluster" : {
  "Type" : "AWS::ECS::Cluster",
  "Properties" : {
    "ClusterName" : "Athena"
  }
}

Now we have Athena cluster. But it doesn’t contain EC2 instance(s). How to assign them? Fun is coming.

Role

We have to set EC2 instance profile, which can contain roles. And one of our roles must allow communication with EC2 Container Service.

"ECSRolePolicy": {
  "Type": "AWS::IAM::ManagedPolicy",
  "Properties": {
    "PolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "ecr:BatchCheckLayerAvailability",
            "ecr:BatchGetImage",
            "ecr:DescribeRepositories",
            "ecr:GetAuthorizationToken",
            "ecr:GetDownloadUrlForLayer",
            "ecr:GetRepositoryPolicy",
            "ecr:ListImages",
            "ecs:CreateCluster",
            "ecs:DeregisterContainerInstance",
            "ecs:DiscoverPollEndpoint",
            "ecs:Poll",
            "ecs:RegisterContainerInstance",
            "ecs:StartTask",
            "ecs:StartTelemetrySession",
            "ecs:SubmitContainerStateChange",
            "ecs:SubmitTaskStateChange"
          ],
          "Resource": [
            "*"
          ]
        }
      ]
    }
  }
},
"ECSRole": {
  "Type": "AWS::IAM::Role",
  "Properties": {
    "ManagedPolicyArns": [
      { "Ref": "ECSRolePolicy" }
    ],
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "ec2.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  }
},
"InstanceProfile" : {
  "Type" : "AWS::IAM::InstanceProfile",
  "Properties" : {
    "Path" : "/",
    "Roles" : [ { "Ref": "ECSRole" } ]
  }
}

We can assign it in this way:

"Athena": {
  "Type": "AWS::EC2::Instance",
  "Properties": {
    ...
    "IamInstanceProfile" : {"Ref" : "InstanceProfile"},
    ...

ECS configuration

Still not done. EC2 instance must know the cluster name. Here we can leverage UserData property:

"Athena": {
  "Type": "AWS::EC2::Instance",
  "Properties": {
    ...
    "UserData": { "Fn::Base64": { "Fn::Join" : [ "\n", [
      "#cloud-config",
      "repo_update: true",
      "repo_upgrade: all",
      "package_upgrade: true",
      "packages:",
      " - aws-cli",
      " - ecs-init",
      "runcmd:",
      { "Fn::Join" : [ "", [
        " - echo ECS_CLUSTER=",
        { "Ref": "ECSCluster" },
        " >> /etc/ecs/ecs.config"
      ]]},
      " - service docker start",
      " - start ecs" ]]}},
      ...

And …

… victory ;-) Registered Container Instances shows 1.

Service

Task definition

I’m not going to cover task definition. You can study it here. It’s pretty straightforward. Kind of stack file for Docker Cloud, but in JSON. Very similar.

Travis CI script

First part is task registration (or update with the same command). New revision is created if task already exists. It’s a simple sequence 1, 2, 3, …

Sad is that I didn’t find a way how to specify revision number manually. Just to keep it in sync with $TRAVIS_BUILD_NUMBER.

aws ecs register-task-definition \
  --cli-input-json file://athena-task-definition.json

ARN=$(aws ecs describe-services --cluster Athena --services athena | jq '.services? | .[0]? | .serviceArn?' -r -M)

if [ "$ARN" = "null" ]; then
    aws ecs create-service \
        --cluster Athena \
        --service-name athena \
        --task-definition athena \
        --deployment-configuration maximumPercent=100,minimumHealthyPercent=0 \
        --desired-count 1
else
    aws ecs update-service \
        --cluster Athena \
        --service $ARN \
        --task-definition athena \
        --deployment-configuration maximumPercent=100,minimumHealthyPercent=0 \
        --desired-count 1
fi

Next part is more interesting. We try to get service ARN. If not found, we’re going to create it otherwise we’re going to update it.

There’s one issue with this script — if you delete service manually, it’s still accessible via AWS CLI (state is inactive). Service ARN is found, but update-service is going to fail; create-service must be used in this case. Better add state check to this script as well.

Focus on minimumHealthyPercent. Value 0 says — feel free to stop my service during deployment. It’s useful in development environment where you can have one tiny instance for example. If you do not set it to 0, deployment can fail:

• one tiny instance in cluster,
• not enough memory on your EC2 instance, because old revision is still running,
• port in use, because of the same reason,
• etc.

Do not use this strategy in production environment. You should have more than one instance, setup load balancing, etc.

That’s it. You have pretty decent idea how EC2 Container Service works. Not all issues were covered, but should be enough to start with it. Hope that helps. It took me some time to find all these things.

AWS journey — EC2 Container Service was originally published in Zrzka’s adventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

Quite huge, isn’t it? Learning curve is pretty steep and you’re going to make lot of mistakes. Sometimes fatal ones. We, especially I, made them. I’m going to share some of them, some tips, so, you can avoid head scratching and long night shifts fixing these mistakes when your customers are asleep.

CloudFormation

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.

Don’t start with CloudFormation if you’re new to the AWS platform. It slows you down, because you have to learn template system, how to use Fn::Join & friends, nested stack, outputs, parameters, … You have to learn it along with other AWS services and it’s not an easy task.

Don’t take me wrong. I like CloudFormation, I do use it and I recommend it. What I’m trying to say is — don’t try to learn everything at once. One by one. Learn whatever service you choose, then jump into the Cloud Formation and create templates. Why?

• You do not want to learn how to write template when you have no clue how API Gateway & Lambda works for example.
• You do not want to wait for deployment (it’s fast, but not enough sometimes), you want to tweak settings, play with it and test it immediately.

Also you can easily shoot yourself:

• Resources deployed via CF templates shouldn’t be manually tweaked at all. You can easily update your templates, CF can process these updates and it updates resources automatically. If you touch them manually, CF can be puzzled (state differs) and you can end up in limbo.
• Never delete resources manually. Update template and let CF process it. If you do it in reverse order, CF can be puzzled. You can fix it by recreating resource manually sometimes. Better learn how to work with DeletionPolicy attribute.
• What’s more important. Never ever delete nested stack manually. This action is not reversible. You can recreate nested stack with the same name, but it has different PhysicalID and thus the stack is there, but it’s not linked to your parent stack.

I deleted nested stack manually (frankly, I don’t know why I did it) and I’m angry with myself.

Hope AWS support guys can help me (tweak resource PhysicalID) and if not I’ll be forced to delete stack, resources, … and create everything “from scratch”.

“From scratch” in quotes, because we have templates for 99.9999% of our resources. But there are other things to fix with this redeployment like Lambda functions configuration (another beast, will dive into details in another article), Continuous Integration (we do use Travis CI), etc.

Nested stacks

You can split one stack into several stacks and link them together. You can pass output from one stack as a parameter value to another one, which is really convenient. This is not possible with standalone stacks. I’m tempted to recommend one root stack for all your stacks. You never know when you’ll need output of one stack as an input for another one.

We have several root stacks, several nested stacks, but these root stacks are not linked together with some kind of top level stack for everything. My hope is recently announced Fn::ImportValue function. But there’s one limitation which bothers me little bit:

• You can’t modify or remove the output value as long as it’s referenced by another stack.

I didn’t find time to test this function. But if I understand it correctly, I can’t change the output value with template update, because the value is kinda “copied” to another stack and then the initial stack is locked. If so, not sure how useful this feature is. Because nested stacks, outputs & parameters, handle this very well. Whenever I change something in one stack, all other dependent stacks are updated accordingly.

What I’m going to do? Will test Fn::ImportValue and will use it. Or I’ll refactor our stacks and will add 3rd level (top for all stacks).

My Cloud Formation workflow

• I manually create resources, tweak them and make them working together. Especially when I’m trying to learn new AWS service and I don’t know how to do it properly yet.
• If they’re in acceptable state, they work, I do not expect huge changes, work on CF templates is started.
• Then I delete manually created resources, put these templates in our repo where Travis CI handles them automatically (stack creation, updates, …).
• I no longer care about these resources in the AWS Console and I work with templates only.

Skipping first step when I know what I would like to achieve and mainly how to do it.

Conclusion

Use it, highly recommended, but try to learn other services first. Then learn how to describe them in Cloud Formation templates.

Think twice:

• Don’t create huge, not easily maintainable stacks. Use nested stacks.
• Is there a tiny chance that you’ll need the output value of one stack as an input value for another stack? Use nested stacks. (or Fn::ImportValue, not tested yet)
• Don’t tweak resources created via CF manually. Don’t delete them as well.
• Learn how DeletionPolicy attribute works.
• Finally, don’t delete nested stack manually = limbo.

Templates allows me to deploy everything in under 15 minutes without worrying if I did or didn’t make a mistake. Who likes manual and repetitive work? Me not. So, use them.

AWS journey — Cloud Formation was originally published in Zrzka’s adventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

• macOS client (web wrapper) performance is terrible, especially in the full screen mode. Same applies to web application as well.
• Search sucks! It’s almost impossible to find what others want from me. Try to quit Slack, do not open it for one day and then try to find mentions of your name, tasks what you should do, etc. Without reading all unread messages — impossible.
• No threads! Remember news groups? Threads, read / unread, … It was a pleasure. It’s not uncommon scenario that you start a thread about specific topic, people step in, chit chat and your topic is lost.
• Both things are forcing me to check Slack channels regularly and react almost immediately (or completely ignore it).

This behavior has consequences:

• concentration and focus killer,
• you don’t have enough time to thing about the problem (question, proposal, …),
• you’re judging problems too quickly,
• can lead to lot of short “no” answers or no answers at all,
• can lead to frustration of your team members.

I agree that it’s all about rules and how you (and your team) do use Slack. But hey, people do not follow rules, they have tendency to step into any discussion, kill it, etc. They’re selfish and they don’t care about you.

My experience says that all #channels are quiet (almost no messages) or they’re full of unreadable and time consuming mess. Or, which is even much more worse, direct messages / private channels — you’re not sharing knowledge, problems, … with others.

But you know what? I actually like Slack, because all these reply-to-all email chit chats are gone. Guess where they’re now? Yes, in #general.

Another problem is that written text can’t never replace face to face communication. Emotions are missing and emoji characters can’t simply replace them. Also you can pretend that you’re in a good mood even if you’re upset. Picking smiling emoji character costs nothing.

Actually we’re experimenting with other technologies like good old Team Speak for example. It has similar issues, like all tools for distributed teams. But it’s much better in terms of emotions — you can hear them in your buddy’s voice and it’s much much quicker when compared to typing. macOS client is ugly, messages sucks, but it’s actually a good thing, because no one writes them :)

Slack, in it’s current state, is overwhelming and is killing focus. Period.

P.S. They have a plan to release threads later this year. Who knows, maybe I’ll change my opinion, but I doubt it.

Yes, I don’t like Slack was originally published in Zrzka’s adventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

(defn- timeout [ms]
 (let [c (chan)]
 (js/setTimeout (fn [] (close! c)) ms)
 c))

(defn- error
 [code message]
 (.stringify js/JSON (clj->js {:error-code code :error-message (pr-str message)})))

(def EventSchema
 {:action (s/enum “data” “exception” “timeout”)})

(def ^:export lambda
 (async-lambda-fn
  (fn [event context]
   (go
    (if-let [e (s/check EventSchema event)]
     (fail! context (error “INVALID_PARAMS” e))
     (try
      (let [action (:action event)]
       (cond
        (= “data” action) (succeed! context {:some “data”})
        (= “timeout” action) (<! (timeout 5000))
        (= “exception” action) (throw (js/Error. “Oops”))))
     (catch :default e
      (fail! context (error “EXCEPTION” e)))))))))

I’m not going to describe how to deploy Lambda function or API Gateway basics. I assume you know how to do it. I’m going to focus on error handling only.

Our function is deployed, API Gateway resource (/lambda) and GET method created and deployed as well. All with default settings, just for now.

What happens when we try to use it?

curl -sw “\n%{http_code}” -H “x-api-key: ...” -H “Accept: application/json” https://...execute-api.eu-west-1.amazonaws.com/latest/lambda

{“errorMessage”:”{\”error-code\”:\”INVALID_PARAMS\”,\”error-message\”:\”{:action missing-required-key}\”}”}
200

JSON in JSON and status code is 200. That’s not what we want. At all.

Pass action from query to lambda function

POST method handles this automatically. Body is passed via event argument. But GET is a slightly different beast.

We have to add action to the URL Query String Parameters section. Then we have to add Body Mapping Template (BMT) in the Integration Request section.

{"action" : "$input.params('action')"}

Redeploy API and test.

{“errorMessage”:”{\”error-code\”:\”INVALID_PARAMS\”,\”error-message\”:\”{:action (not (#{\\\”data\\\” \\\”exception\\\” \\\”timeout\\\”} \\\”\\\”))}\”}”}
200

Much better, we can see validation response. Let’s try some [in]valid options. Status code 200 for all of them with following responses.

action=data
{“some”:”data”}

action=timeout
{“errorMessage”:”2016–04–14T12:20:20.584Z 3f0e0ca9–023b-11e6-a561-e58695fc8058 Task timed out after 4.00 seconds”}

action=exception
{“errorMessage”:”{\”error-code\”:\”EXCEPTION\”,\”error-message\”:\”#object[Error Error: Oops]\”}”}

NOTE: Our lambda function is deployed with timeout set to 4 seconds. Just to make it working, check the code, we’re waiting 5 seconds.

This is not good. We want 200 for success, 400 for invalid input parameters and 500 if something bad happens (exception, timeout, …) and better structure of our response.

Mapping status codes

We have to add 400 and 500 in the Method Response section. Otherwise we’ll not be able to select these status codes in the Integration Response section.

If lambda function fails, it returns JSON in the following format:

{“errorMessage” : ”whatever”}

And the errorMessage is a string. Do you want to return more than just a string? You have to encode it.

Open the Integration Response section and you’ll see something like this:

Default mapping uses 200 for the status code, there’s no regex and Body Mapping Template (click on the triangle) is empty = passthrough. And because we know what our Lambda function returns, we can add some regular expressions there.

What’s this all about? Lambda Error Regex column contains regular expressions that are matched against errorMessage from the lambda response if it indicates it failed (fail! in the code). For INVALID_PARAMS we would like to return 400, for any other error message (.+) we would like to return 500 and 200 for the rest.

Redeploy and test.

action=exception
{“errorMessage”:”{\”error-code\”:\”EXCEPTION\”,\”error-message\”:\”#object[Error Error: Oops]\”}”}
500

action=timeout
{“errorMessage”:”2016–04–14T12:41:09.348Z 27638414–023e-11e6-bcf3–89b1797fdc7b Task timed out after 4.00 seconds”}
500

action=hallo
{“errorMessage”:”{\”error-code\”:\”INVALID_PARAMS\”,\”error-message\”:\”{:action (not (#{\\\”data\\\” \\\”exception\\\” \\\”timeout\\\”} \\\”hallo\\\”))}\”}”}
400

action=data
{“some”:”data”}
200

Better response structure

Status codes solved. Now we have to focus on the response structure. It’s quite simple. We can do it in the Integration Response section as well. We have to add Body Mapping Template for each regular expression we already added.

As you can see, BMT for 500 is more fancier that the one for 400. Why? AWS can kill our lambda function because of timeout for example and then there’s just plain string in the errorMessage. Also our lambda can catch exception, send back custom error map and then there’s encoded JSON in the errorMessage. And because we can’t add two (or more) regular expressions for the same status (500 in this case), we have to handle it in the BMT.

Redeploy and test.

action=exception
{“error-code” : “EXCEPTION”,
 “error-message” : “#object[Error Error: Oops]”}
500

action=timeout
{“error-code” : “AWS”,
  “error-message” : “2016–04–14T13:50:09.212Z caf1daed-0247–11e6–91f8-d9f9a69fef2c Task timed out after 4.00 seconds”}
500

action=hallo
{“error-code” : “INVALID_PARAMS”,
 “error-message” : “{:action (not (#{“data” “exception” “timeout”} “hallo”))}”}
400

action=date
{“some”:”data”}
200

Goal reached. Status codes and better response structure for errors. It’s not perfect, there’s always room for improvement, but it works.

It’s a bare minimum you have to do. Pretty rough, okay, terrible, but when you know how to do it … Hope it helps, because I spent quite huge amount of time on this.

AWS Lambda & API Gateway Error Handling was originally published in Zrzka’s adventures on Medium, where people are continuing the conversation by highlighting and responding to this story.

Not that I don’t like Jekyll and GitHub Pages. But the workflow is kinda crazy when I want to publish new post and the only device I have is my iPad.

I’ll give it a try for a while and then I’ll merge them. Either on GitHub Pages or here. We’ll see.

Giving Medium a Try was originally published in Zrzka’s adventures on Medium, where people are continuing the conversation by highlighting and responding to this story.