The Road to Serverless

It would not be wrong if I tell you the next awakening step of cloud computing is serverless. The evolution to serverless architecture from physical servers has not been an easy journey. In early years software was installed in mainframe type server computers and used their own data stores. With the time these physical servers were replaced by the virtual machines where it became possible to make copies of an entire physical machine.

And the next big thing in this journey was containerization. With the uprise of containers developer needed not to worry about the operating system and environment. As a result, they could focus more on the software, application code and dependencies.

Serverless, The New Buzzword

With the new serverless architecture, you just need to focus on the applications’ business logic, and should not need to worry about any infrastructure needs. And this leads to many more advantages which we will discuss later in this article.
Serverless architecture, the new buzzword has been around for the past few years and now it has been a reality. The adaptation rate for this tech is slower than any other new technologies. But with the time number has been increased with the awareness of the benefits of serverless architecture.

The term, Serverless Architecture may be confusing. Someone may think we are building an application which runs completely on the client side or probably distributed within different clients, without any serverside operation.

But is that a reasonable definition for an application builds on serverless architecture. No… Cloud Architecture specialists define this in another way.

Serverless architectures are application designs that incorporate third-party “Backend as a Service” (BaaS) services, and/or that include custom code run in managed, ephemeral containers on a “Functions as a Service” (FaaS) platform.

As he explains more on serverless architecture, it denies the need of an always-on, pre-purchased compute capacity behind the application, unlike the traditional architecture. The containers which server-side logic has been written are stateless, and they are event triggered. With these facts its clear to us there is a server component in somewhere connected to these serverless applications.

There are many cloud vendors who provide these hosted services. And these companies have hugely invested in serverless. Amazon Web Services(AWS), Google Cloud Platform(GCP), Microsoft Azure, IBM OpenWhisk, Auth0 Webtask, Oracle Fn Project could be named as the giants in the serverless industry.

FaaS & BaaS

Two main components of the serverless architecture are FaaS and BaaS. As in the definition of serverless architecture describes the part of applications that are fully or significantly incorporated third-party, cloud-hosted services that are capable of managing server-side logic and state is known as BaaS. For an example, the developer gets unified lean access to services like database services like firebase, authentication services like Auth0, AWS Cognito. As a benefit with these services, developers can bring rich client, fully functional application to the market in a very short period of time.

For some scenarios in application development, the developer needs to declare logics that needs much more computational power, unlike authentication or data querying. And also if these logics are more complex and time-consuming, it will not fit in the client side as well. To fulfill this kind of requirement serverless has introduced FaaS. So the developer could write and upload the backend logic and well-defined interface to invoke the backend logic with certain inputs. These functions run in stateless compute containers which are event triggered. Once these functions are implemented and deployed, cloud providers take care of the rest, which is where to run it and scale it up necessarily.

One of the most popular FaaS in the present serverless universe is AWS Lambda. And there are many more. As serverless become more and more popular cloud providers have given the capability to write these functions in several languages like Node js, Python, Java, and .NET.

Some could think that FaaS is the same as Platform as a Service(PaaS). Well, although there are some similarities between these two, there is a huge concept difference. FaaS takes PaaS concept into a next level. One of the main difference between FaaS and PaaS is when an application deployed as PaaS, it typically runs on the server all the time, but in FaaS it provides the capability to deploy single function or a part of an application. And it won’t be running at all until the function needs to be executed.

Pros & Cons

Serverless is not scary. There are many benefits when programmers develop with serverless architecture. As there is nothing perfect in this world, serverless is also has its own dark side. Although serverless has many pros than cons when getting into consideration.

Pros of serverless

• Scalable services — It is the ideal designing architecture for a scalable application. It automatically scales endlessly as it takes separate functions on your FaaS and runs them in separate containers
• Billing per invocations — Not like in traditional architecture models you ain’t paying for constant server availability. You just pay for the number of invocation made to your lambda. Thus your cost depends on the traffic on your serverless application
• Lower cost on human resource and hardware resource — As there is no much expensive hardware, no need of a human taskforce to maintain this hardware
• Focus much more on user experience — As there are lots of serverside services are provided with serverless computing, hence the engineering team could more focus on user experience features
• Fast delivery to market — Achieved by deploying smaller units of functions as features to end user

Cons of serverless

• Vendor lock-in — It’s not easy to migrate from one vendor to another, or you aren’t independent in choosing the coding language.
• Architectural complexity — Developer need to go through a learning curve and follow best practices to understand and develop flawless applications
• Lack of operation tools — Developer depends on different third-party vendors for monitoring and debugging, and still difficult in debugging distributed system and requires a significant amount of relevant metrics to identify the root cause

Originally published at slappforge.com on November 1, 2018.

Serverless Architecture: Learn it, Perfect it and Evolve with it was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.

Even long before serverless was born, message queues had been a popular weapon in the arsenal of software developers and architects to integrate software systems. First, when there were large software systems which were incompatible with each other, message queues acted as the intermediary communication channels between them. Then the world moved into distributed systems and message queues became more and more popular as an integration medium due to some of their inherent capabilities such as message broadcasting, guaranteed delivery, in-order delivery and so on. Now, in an era where almost all the software systems are moving into the serverless landscape, where do the serverless message queues fit-in?

Serverless Message Queues

Not surprisingly, they are EVERYWHERE! Why? Because naturally, a complete serverless system is a combination of several smaller moving parts that processes messages and integrate with other resources. And these parts are required to pass messages and events between them. For example, such a system may contain multiple AWS Lambdas, Google Cloud Functions, Microsoft Azure Functions or any combination of them. Each one of these smaller execution parts operates with its own latency, own throughput and even scales up and down by its own. So when you have to move messages between such volatile subparts, one of the best and easiest approach is the use of message queues.

All the serverless service providers such as AWS, Azure, and GCP provide several queuing solutions to choose from. Therefore the next problem will be which is the best and optimal queue service for your application. So the only approach to select the correct service would be to analyze the requirements of your application and choose a solution that meets them. Therefore this article discusses some of the main aspects of serverless queuing services that should be taken into consideration when choosing the optimal service.

Things to consider when selecting a Queuing Solution

Maximum Message Size

Maximum Message Size is the maximum size of the payload a single message on a queue can hold. Most of the serverless queues such as AWS SQS, AWS SNS, Azure Queues allow only up to several hundred kilobytes per message. But some of them such as AWS Kinesis allow up to a couple of megabytes as well. Generally, serverless systems deal with smaller sized messages that can be processed within a limited time period. Therefore this size limitations would not be an issue in most use cases. But if the size of messages generated by the source system exceeds approximately 1MB, it is important to select a queue that is capable of accepting such messages.

Maximum Throughput

Maximum Throughput is the maximum number of message related operations that can be handled by a queue per second. Although some of the queuing solutions such as AWS SQS (standard queues) and AWS SNS do not impose any limitation on the throughput, most of the other queuing solutions do. And the definition of this limit varies from one solution to another as well. For example, an AWS SQS FIFO queue supports up to 300 message operations, i.e. the total number of read, write or delete operations per second. In contrast, an AWS Kinesis shard can support up to 1,000 put record operations per second. Because of these limitations, it is better to estimate the expected throughput of your application (message enqueuing rate as well as dequeuing rate) beforehand and choose a queuing solution which can cater to that throughput level.

Retention period

The retention period is the maximum time duration that the queue storage will keep a message until the next processing stage consumes it. This next processing stage most probably can be a serverless function such as AWS Lambda or an Azure function. Most, if not all message queues have a defined retention period to prevent unconsumed and obsolete messages from piling up on the queue storage. Otherwise, such a pile-up will make it impossible for the publishers to enqueue new messages to the queue due to the shortage of storage capacity. This aspect is very important especially in the serverless landscape because most serverless systems are a combination of multiple independent processing stages.

So if the processing stage which is responsible for retrieving messages from the queue has gone down, the queue should be able to retain the messages for a reasonable time period until that system becomes available again. On the other hand, what if your system is dealing with time-sensitive messages i.e. messages that become obsolete unless they are processed within a certain time period? In such a case, the queue should be able to discard them or send them to a Dead-Letter-Queue without handing them over to the next processing stage.

Guaranteed Ordering

Guaranteed Ordering means that a subscriber of a queue will receive the messages in the exact same order that the publisher had published them to the queue. Some applications require this capability when it is important to preserve the message order between each processing stage. Although some queuing solutions such as AWS SQS Standard Queues and Azure Queue Storage do not guarantee this behavior, there are many solutions such as AWS SQS FIFO Queues and AWS SNS that do provide this guarantee.

Delivery Method

Most of the serverless queuing solutions fall into one of the two categories based on the message delivery method. When there are multiple subscribers for a queue, some queuing solutions such as AWS SQS deliver a particular message to only one of those subscribers. In contrast, other solutions such as AQS SNS deliver a copy of the same message to each and every subscriber aka broadcasting. Therefore based on the requirement of the serverless application, it is vital to select a queue which supports the correct delivery method.

Summary of Operational Limits on Popular Serverless Queuing Services

In conclusion, we can see that all of these services provide exceptional performance and reliability guarantees for very reasonable pricing, but only the behavior will differ. Therefore it is the developer’s responsibility to choose the correct service or service based on the requirements of the application.

Call To Action

• Clap. Appreciate and let others find this article.
• Comment. Share your views on this article.
• Follow me. Udith Gunaratna to receive updates on articles like this.
• Keep in touch. LinkedIn, Twitter

Originally published at slappforge.com on March 18, 2019.

Serverless message queues and how to leverage the best was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.

Can every storage service or database service be called a serverless storage or a serverless database? What makes one storage service being put into the serverless category and others drop out? This article will cover the basic properties a persistence service should have, for it to be called a serverless storage or a serverless persistence service. Next up, we’ll give examples of different serverless storage and serverless database categories along with a few sample use cases of them in serverless computing. Lastly, we’ll go on to discuss the serverless storage categories which are yet to emerge and look towards the future of serverless storage.

Originally published at notebookbft.wordpress.com on January 1, 2019.

Serverless — no servers at all?

Source — http://www.commitstrip.com/en/2017/04/26/servers-there-are-no-servers-here/

The term serverless is a misnomer. Serverless does not mean that servers are no longer required and it basically goes with the concept of abstracting technology away from the user. The users no longer have to worry about servers and someone else (the cloud provider) is taking care of all that for the user.

Serverless Computing

Serverless is quite a buzz word these days. Even though the term serverless has been there for some time, the major reason for it becoming such a buzz word was the launch of AWS Lambda in 2014. The term further gained popularity with the launch of Amazon’s API Gateway in July 2015. In general, the term serverless — or rather serverless computing — refer the applications where server-side logic written by the application developer is run in stateless compute containers that are event-triggered, ephemeral (may only last for one invocation), and fully managed by a third party. This is also called Function as a Service (FaaS). Read more about serverless architecture in one of our previous articles.

The statelessness of Serverless Computing and the need for Storage/Persistence

One of the features of serverless computing or FaaS is that it is stateless. Therefore persisting state between two executions of the same function is not possible in the general design. This is where serverless storage/persistence comes into the picture. In cases where the state needs to be persisted between two FaaS function executions, it should be stored externally in a storage/persistence service.

The Problem with Traditional Storage

Traditional storage solutions are designed to run continuously on a fixed set of servers at a single geographic location. To use these storage solutions properly, application developers need to know a lot about configuration specifics. In the worst cases, high availability and performance are dependent on developer knowledge of database internals. Developers need to figure out and configure things like regions, zones, volumes, memory, compute capacity, and software versions. Teams have to spend hours after hours thinking about capacity planning, provisioning, sharding, backups, performance tuning, and monitoring. When you spend 80% of your time setting up and operating databases just to support serverless functions, you know something is amiss. While serverless computing adds elasticity to the compute layer, you cannot fully absorb its advantage when your persistence layer does not offer a comparable level of elasticity.

A Truly Serverless Storage

A truly serverless storage should ideally have the following properties.

• No provisioning
• Truly elastic (scale up and scale down without operator intervention)
• Pay-as-you-go

When using a serverless storage, developers should not need to worry about any infrastructure details, like node size, memory, or storage size. In the same way that AWS Lambda charges per single-function invocation, consumption in a serverless storage should be measured by the amount of compute and storage used by different workloads. This is the essence of a serverless database. Users are never charged for idle capacity. Storage usage reflects function invocation and workload.

With traditional storage offerings, over-provisioning is the only strategy available to prepare for traffic spikes. A serverless storage’s elasticity prevents over-provisioning, which can be wasteful. Since a serverless storage has the capability to scale elastically without user intervention, developers can launch with no capacity planning and the applications will always have enough capacity. Further, the pay-as-you-go pricing model would ensure that no idle resources drain the user’s bank account. The cost of using serverless storage simply scales with usage. You never pay for unused capacity.

Serverless Storage/Persistence Options

Nowadays, there are various serverless storage categories; often, more or less the same service is provided by multiple cloud vendors. In this section we’ll go through those different serverless persistence categories, examples for each category, and some use cases of them as well.

Serverless Object Storage

Object storage is a hierarchy-free method of storing data, typically used in the cloud. Unlike other data storage methods, object-based storage does not use a directory tree. Discrete units of data (objects) exist at the same level in a storage pool. Each object has a unique, identifying name that an application uses to retrieve it. Additionally, each object may have metadata that is retrieved with it. Some common examples of serverless object storages are Amazon S3, Google Cloud Storage, Azure Storage and IBM Cloud Storage. One common example of the use of serverless object storage in the context of serverless computing would be the thumbnail generation.

1. The mobile application uploads an image to an object store.
2. Serverless object storage fires a change event after the image was uploaded, resulting in an execution of a serverless function.
3. The serverless function creates a thumbnail based on the uploaded image.
4. After uploading the thumbnail on to the serverless storage, the serverless function sends a push notification to the mobile application.
5. Mobile application downloads the thumbnail and updates the user interface accordingly.

Serverless Relational Databases

An example of a serverless relational database is Amazon Aurora Serverless which was launched in the last quarter of 2017. It comes in two different editions compatible with both MySQL and PostgreSQL, but it is also compatible with other known systems like MariaDB, Oracle, etc. Amazon Aurora Serverless is an on-demand, auto-scaling database service, where the database will automatically start up, shut down, and scale capacity up or down based on the application’s needs. It enables users to run their databases in the cloud without actually managing any database instances. It’s a simple, cost-effective option for infrequent, intermittent, or unpredictable workloads, and hence is ideal for serverless computing.

Serverless NoSQL Databases

Serverless Key-value and Document Stores

There are different kinds of serverless NoSQL databases available today. The most common ones are key-value stores and document stores. A key-value database, or key-value store, is a data storage paradigm designed for storing, retrieving, and managing associative arrays, a data structure more commonly known today as a dictionary or hash table. Dictionaries contain a collection of objects, or records, which in turn have many different fields within them, each containing data. These records are stored and retrieved using a key that uniquely identifies the record, and is used to quickly find the data within the database. A document-oriented database or document store is a storage designed for storing, retrieving and managing document-oriented information, also known as semi-structured data. A few examples of serverless key-value and document stores are Amazon DynamoDB, Google Cloud Datastore and IBM Cloudant NoSQL DB. Recently Amazon announced DynamoDB on-demand which provides the option for on-demand read/write capacity for DynamoDB with the pay-for-request model. There are various uses of such serverless key-value and document stores; out of those a simple use case is handling a contact us form with a serverless function.

Serverless Time Series Database

A time series database (TSDB) is a software system that is optimized for handling time series data, arrays of numbers indexed by time (a datetime or a datetime range). Time-series data has specific characteristics such as typically arriving in the time order form, data is append-only, and queries are always over a time interval. While relational databases can store this data, they are inefficient at processing this data as they lack optimizations such as storing and retrieving data by time intervals. An example of a serverless time series database is the recently announced Amazon Timestream which is a purpose-built time series database that efficiently stores and processes records by time intervals.

Source — https://aws.amazon.com/timestream/

Serverless Graph Databases

Very simply, a graph database is a database designed to treat the relationships between data as equally important as the data itself. It is intended to hold data without constricting it to a predefined model. Instead, the data is stored like we first draw it out — showing how each individual entity connects with or is related to others. When it comes to a serverless graph database, we have FaunaDB which is a general purpose, transactional, temporal, geographically distributed, strongly consistent and relational database. FaunaDB also supports document store and time series database features.

What is Missing?

Serverless In-Memory Data Stores/Caches?

One of the storage types which are not yet available in the serverless flavour is in-memory data stores. One of the options we have at the moment for caching is caching at the API Gateway level. Then we have Amazon DynamoDB with single digit millisecond latencies. If a user needs to reduce the latency a bit more, then even though not yet fully serverless, DynamoDB Accelerator (DAX) can be used. Further, we do have AWS ElastiCache which offers Redis and Memcached but they do not yet offer them as fully managed services. Then from Google App Engine, we have Memcache as an in-memory caching service, however, this can only be accessed within the App Engine itself.

Conclusion

Serverless storage support is growing rapidly along with the growth and high adaption of serverless computing in the IT industry. We expect these services to grow, allowing a truly serverless application development.

Call To Action

• Clap. Appreciate and let others find this article.
• Comment. Share your views on this article.
• Follow me. Rajind Ruparathna to receive updates on articles like this.
• Keep in touch. LinkedIn, Twitter, Personal Blog

What makes a Storage Service Truly Serverless? was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ack: This article is a remix of stuff learned from personal experience as well as from multiple other sources on serverless security. I cannot list down or acknowledge all of them here; nevertheless, special thanks should go to The Register, Hacker Noon, PureSec, and the Serverless Status and Serverless (Cron)icle newsletters.

We all love to imagine that our systems are secure. And then…

BREACH!!!

A very common nightmare shared by every developer, sysadmin and, ultimately, CISO.

Inevitable?

One basic principle of computer security states that no system can attain absolute security. Just like people: nobody is perfect. Not unless it is fully isolated from the outside; which, by today’s standards, is next to impossible — besides, what’s the point of having a system that cannot take inputs and provide outputs?

Whatever advanced security precaution you take, attackers will eventually find a way around. Even if you use the most stringent encryption algorithm with the longest possible key size, attackers will eventually brute-force their way through; although it could be time-wise infeasible at present, who can guarantee that a bizarre technical leap would render it possible tomorrow, or the next day?

But it’s not the brute-force that you should really be worried about: human errors are way more common, and can have devastating effects on systems security; much more so than a brute-forced passkey. Just have a peek at this story where some guys just walked into the U.S. IRS building and siphoned out millions of dollars, without using a single so-called “hacking” technique.

As long as systems are made and operated by people — who are error-prone by nature — they will never be truly secure.

So, are we doomed?

No!

Ever seen the insides of a ship?

How its hull is divided into compartments — so that one leaking compartment does not cause the whole ship to sink?

People often apply a similar concept in designing software: multiple modules so that one compromised module doesn’t bring the whole system down.

Combined with the principle of least privilege, this means that a component will compromise the least possible degree of security — ideally the attacker will only be able to wreak havoc within the bounds of the module’s security scope, never beyond.

Reducing the blast radius of the component, and consequently the attack surface that it exposes for the overall system.

A security sandbox, you could say.

And a pretty good one at that.

PoLP: The Principle of Least Privilege

Never give someone — or something — more freedom than they need.

More formally,

Every module must be able to access only the information and resources that are necessary for its legitimate purpose.

— Wikipedia

This way, if the module misbehaves (or is forced to misbehave, by an entity with malicious intent — a hacker, in English), the potential harm it can cause is minimized; without any preventive “action” being taken, and even before the “breach” is identified!

It never gets old

While the principle was initially brought up in the context of legacy systems, it is even more so applicable for “modern” architectures; SOA (well, maybe not so “modern”), microservices, and FaaS (serverless functions, hence serverless security) as well.

The concept is pretty simple: use the underlying access control mechanisms to restrict the permissions available for your “unit of execution”; may it be a simple HTTP server/proxy, web service backend, microservice, container, or serverless function.

Meanwhile, in the land of no servers…

With increased worldwide adoption of serverless technologies, the significance of serverless security, and the value of our PoLP, is becoming more obvious than ever.

Server-less = effort-less

Not having to provision and manage the server (environment) means that serverless DevOps can proceed at an insanely rapid pace. With CI/CD in place, it’s just a matter of code, commit and push; everything would be up and running within minutes, if not seconds. No SSH logins, file uploads, config syncs, service restarts, routing shifts, or any of the other pesky DevOps chores associated with a traditional deployment.

“Let’s fix the permissions later.”

Alas, that’s a common thing to hear among those “ops-free” devs (like myself). You’re in a haste to push the latest updates to staging, and the “easy path” to avoid a plethora of “permission denied” errors is to relax the permissions on your FaaS entity (AWS Lambda, Azure Function, whatever).

Staging will soon migrate to prod. And so will your “over-permissioned” function.

And it will stay there. Far longer than you think. You will eventually shift your traffic to updated versions, leaving behind the old one untouched; in fear of breaking some other dependent component in case you step on it.

And then come the sands of time, covering the old function from everybody’s memories.

An obsolete function with unpatched dependencies and possibly flawed logic, having full access to your cloud resources.

A serverless time bomb, if there ever was one.

Yes, blast radius; again!

If we adhere to the least privilege principle, right from the staging deployment, it would greatly reduce the blast radius: by limiting what the function is allowed to do, we automatically limit the “extent of exploitation” upon the rest of the system if its control ever falls into the wrong hands.

Nailing serverless security: on public cloud platforms

These things are easier said than done.

At the moment, among the leaders of public-cloud FaaS technology, only AWS has a sufficiently flexible serverless security model. GCP automatically assigns a default project-level Cloud Platform service account to all its functions in a given project, meaning that all your functions will be in one basket in terms of security and access control. Azure’s IAM model looks more promising, but it still lacks the cool stuff like automatic role-based runtime credential assignments available in both AWS and GCP.

AWS has applied its own IAM role-based permissions model for its Lambda functions, granting users the flexibility to define a custom IAM role — with fully customizable permissions — for every single Lambda function if so desired. It has an impressive array of predefined roles that you can extend upon and has well-defined strategies for scoping permission to categories of resources or principals; merging rules that refer to the same set of resources or operations; and so forth.

This whole hierarchy finally boils down to a set of permissions, each of which takes a rather straightforward format:

{
    "Effect": "Allow|Deny",
    "Action": "API operation matcher (pattern), or array of them",
    "Resource": "entity matcher (pattern), or array of them"
}

In English, this simply means:

Allow (or deny) an entity (user, EC2 instance, lambda; whatever) that possesses this permission, to perform the matching API operation(s) against the matching resource(s).

(There are non-mandatory fields Principal and Condition as well, but we'll skip them here for the sake of brevity.)

Okay, okay! Time for some examples.

{
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::my-awesome-bucket/*"
}

This allows the assignee to put an object (s3:PutObject) into the bucket named my-awesome-bucket.

{
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::my-awesome-*"
}

This is similar, but allows the put to be performed on any bucket whose name begins with my-awesome-.

{
    "Effect": "Allow",
    "Action": "s3:*",
    "Resource": "*"
}

This allows the assignee to do any S3 operation (get/put object, delete object, or even delete bucket) against any bucket in its owning AWS account.

And now the silver bullet:

{
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*"
}

Yup, that one allows oneself to do anything on anything in the AWS account.

Kind of like the AdministratorAccess managed policy.

And if your principal (say, lambda) gets compromised, the attacker effectively has admin access to your AWS account!

A serverless security nightmare. Needless to say.

To be avoided at all cost.

Period.

In that sense, the best option would be a series of permissions of the first kind; ones that are the least permissive (most restrictive) and cover a narrow, well-defined scope.

How hard can that be?

The caveat is that you have to do this for every single operation within that computation unit — say lambda. Every single one.

And it gets worse when you need to configure event sources for triggering those units.

Say, for an API Gateway-triggered lambda, where the API Gateway service must be granted permission to invoke your lambda in the scope of a specific APIG endpoint (in CloudFormation syntax):

{
  "Type": "AWS::Lambda::Permission",
  "Properties": {
    "Action": "lambda:InvokeFunction",
    "FunctionName": {
      "Ref": "LambdaFunction"
    },
    "SourceArn": {
      "Fn::Sub": [
        "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/*/${__Method__}${__Path__}",
        {
          "__Method__": "POST",
          "__Path__": "/API/resource/path",
          "__ApiId__": {
            "Ref": "RestApi"
          }
        }
      ]
    },
    "Principal": "apigateway.amazonaws.com"
  }
}

Or for a Kinesis stream-powered lambda, in which case things get more complicated: the Lambda function requires access to watch and pull from the stream, while the Kinesis service also needs permission to trigger the lambda:

  "LambdaFunctionExecutionRole": {
    "Type": "AWS::IAM::Role",
    "Properties": {
      "ManagedPolicyArns": [
        "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
      ],
      "AssumeRolePolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Action": [
              "sts:AssumeRole"
            ],
            "Effect": "Allow",
            "Principal": {
              "Service": [
                "lambda.amazonaws.com"
              ]
            }
          }
        ]
      },
      "Policies": [
        {
          "PolicyName": "LambdaPolicy",
          "PolicyDocument": {
            "Statement": [
              {
                "Effect": "Allow",
                "Action": [
                  "kinesis:GetRecords",
                  "kinesis:GetShardIterator",
                  "kinesis:DescribeStream",
                  "kinesis:ListStreams"
                ],
                "Resource": {
                  "Fn::GetAtt": [
                    "KinesisStream",
                    "Arn"
                  ]
                }
              }
            ]
          }
        }
      ]
    }
  },
  "LambdaFunctionKinesisTrigger": {
    "Type": "AWS::Lambda::EventSourceMapping",
    "Properties": {
      "BatchSize": 100,
      "EventSourceArn": {
        "Fn::GetAtt": [
          "KinesisStream",
          "Arn"
        ]
      },
      "StartingPosition": "TRIM_HORIZON",
      "FunctionName": {
        "Ref": "LambdaFunction"
      }
    }
  },
  "KinesisStreamPermission": {
    "Type": "AWS::Lambda::Permission",
    "Properties": {
      "Action": "lambda:InvokeFunction",
      "FunctionName": {
        "Ref": "LambdaFunction"
      },
      "SourceArn": {
        "Fn::GetAtt": [
          "KinesisStream",
          "Arn"
        ]
      },
      "Principal": "kinesis.amazonaws.com"
    }
  }

So you see, with this granularity, comes great power as well as great responsibility. One missing permission — heck, one mistyped letter — and it’s 403 AccessDeniedException.

No easy way; you just have to track down every AWS resource triggering or accessed by your function, look up the docs, pull out your hair, and come up with the necessary permissions.

But… but… that’s too much work!

Yup, it is. If you’re doing it manually.

But who drives manual these days? :)

Fortunately there are quite a few options, if you’re already into automating stuff:

serverless-puresec-cli: thanks PureSec!

If you’re using the famous Serverless Framework — which means you’re already covered on the trigger permissions front — there’s the serverless-puresec-cli plugin from Puresec.

The plugin can statically analyze your lambda code and generate a least-privilege role. Looks really cool, but the caveat is that you have to run the serverless puresec gen-roles command before every deployment with code changes; I couldn't yet find a way to run it automatically - during serverless deploy, for example. Worse, it just prints the generated roles into stdout; so you have to manually copy-paste it into serverless.yml, or use some other voodoo to actually inject it into the deployment configuration (hopefully things would improve in the future :))

AWS Chalice: from the Gods

If you’re a Python fan, Chalice is capable of auto-generating permissions for you, natively. Chalice is awesome in many aspects; super-fast deployments, annotation-driven triggers, little or no configurations to take care of, and so forth.

However, despite being a direct hand-down from the AWS gods, it seems to have missed the word “minimal” when it comes to permissions; if you have the code to list the contents of some bucket foo, it will generate permissions for listing content of all buckets in the AWS account ("Resource": "*" instead of "Resource": "arn:aws:s3:::foo/*"), not just the bucket you are interested in. Not cool!

No CLI? go for SLAppForge Sigma

If you’re a beginner, or not that fond of CLI tooling, there’s Sigma from SLAppForge.

Being a fully-fledged browser IDE, Sigma will automatically analyze your code as you compose (type or drag-n-drop) it, and derive the necessary permissions — for the Lambda runtime as well as for the triggers — so you are fully covered. The recently introduced Permission Manager also allows you to modify these auto-generated permissions if you desire; for example, if you are integrating a new AWS service/operation that Sigma doesn’t yet know about.

Plus, with Sigma, you never have to worry about any other configurations; resource configs, trigger mappings, entity interrelations and so forth — the IDE takes care of it all.

The caveat is that Sigma only supports NodeJS at the moment; but Python, Java and other cool languages are on their way!

(Feel free to comment below, if you have other cool serverless security policy generation tools in mind! And no, AWS Policy Generator doesn’t count.)

In closing

Least privilege principle is crucial for serverless security, and software design in general; sooner or later, it will save your day.

Lambda’s highly granular IAM permission model is ideal for the PoLP.

Tools like the Puresec CLI plugin, all-in-one Sigma IDE and AWS Chalice can automate security policy generation; making your life easier, and still keeping the PoLP promise.

Originally published at randomizd.blogspot.com on November 14, 2018.

Serverless Security: Putting it on Autopilot was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.

Google is catching up fast on the serverless race.

And we are, too.

That’s how our serverless IDE Sigma recently started supporting integration with Google Cloud Platform, pet-named GCP.

What the heck does that mean?

To be extra honest, there’s nothing new so far. Sigma could interact with GCP (and any other cloud service, for that matter) right from the beginning; we just didn’t have the scaffolding to make things easy and fun — the nice drag-n-drops and UI pop-ups, and the boilerplate code for handling credentials.

Now, with the bits ’n’ pieces in place, you can drag, drop and operate on your GCP stuff right inside your Sigma project!

(*) Conditions apply

Of course, there are some important points (well, let’s face it, limitations):

• GCP access is confined to one project, meaning that all cloud resources accessed in a Sigma project should come from a single GCP project. You can, of course, work around this by writing custom authorization logic on your own, but drag-n-drop stuff will still work only for the first project you configured.
• You can only interact with existing resources, and just as operations (no trigger support). New resource creation and trigger configurations require deeper integration with GCP, which will become available soon (fingers crossed).
• Only a handful of resource types are currently supported — including Storage, Pub/Sub, Datastore, and SQL. These should, however, cover most of the common scenarios; we are hoping to give more attention to big data and machine learning stuff in the immediate future — which Google also seems to be thriving upon — but feel free to shout out so we can reconsider our strategy!

So, where can I get this thing?

• Open up your Sigma project (or create a new one, if you haven’t already).
• You’ll notice a new set of buttons at the top of the Resources pane; click on GCP Resources (the second one, with the GCP logo).

• Now you can provide a service account key from the desired GCP project to access its resources via Sigma. Click Authorize, paste the JSON service account key, and press Save.

• The Resources pane will now display the available GCP resource types. You can drag-n-drop them into your code, configure new operations and edit existing ones, just like you would do with AWS resources.

(Disclaimer: The list you see may differ, depending on how old this post is.)

But you said there’s nothing new!

How could I have done this earlier?

Well, if you look closely, you’ll see what we do behind the scenes:

• We have two environment variables, GCP_SERVICE_TOKEN and GCP_PROJECT_ID, which expose the GCP project identity and access to your Sigma app.
• Sigma has an Authorizer file that reads this content and configures the googleapis NodeJS client library, using them as credentials.
• Your code imports Authorizer (which runs the abovementioned configuration stuff), and invokes operations via googleapis, effectively operating on the resources in your GCP project.

See? No voodoo.

Of course, back in those days, you won’t have seen the cool UI pop-ups or the drag ’n’ drop capability; so we’ve had to do a bit of work, after all.

Not cool! I want more!!

Relax, the really cool stuff is already on the way:

• Ability to define new GCP resources, right within the IDE (drag-drop-configure, of course!)
• Ability to use GCP resources as triggers (say, fire a cloud function via a Cloud Storage bucket or a Pub/Sub topic)

And, saving the best for the last,

• Complete application deployments on Google Cloud Platform!

And…

• A complete cross-cloud experience: operating on AWS from within GCP, and vice versa!
• With other players (MS Azure, Alibaba Cloud and so forth) joining the game, pretty soon!

So, stay tuned!

Originally published at randomizd.blogspot.com on July 16, 2018.

Google, Here We Come! was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.

Yep, it’s not what the title says it is.

Okay, let’s face it: this article has nothing to do with robbery, banks or, heck, ballpoint pens; but it’s a good attention grabber (hopefully!), thanks to Chef Horst of Gusteau’s. (Apologies if that broke your heart!)

Rather, this is about getting your own gossip feed — sending you the latest and the hottest, within minutes they become public — with just an AWS account and a web browser!

Maybe not as exciting as a bank robbery, but still worth reading on — especially if you’re a gossip fan and like to always have an edge over the rest of your buddies.

TL,DR;: It’s all right here, ready for action.

Kicking out the server

Going with the recent hype, we will be using serverless technologies for our mission. You guessed it, there’s no server involved. (But, psst, there is one!)

Let’s go with AWS, which offers an attractive Free Tier in addition to a myriad of rich serverless utilties: CloudWatch scheduled events to trigger our gossip seek, DynamoDB to store gossips and track changes, and SNS-based SMS to dispatch new gossips right into your mobile!

And the best part is: you will be doing everything — from defining entities and composing lambdas to building, packaging and deploying the whole set-up — right inside your own web browser, without ever having to open up a single tedious AWS console!

All of it made possible thanks to Sigma, the brand new truly serverless IDE from SLAppForge.

The grocery list

First things first: sign up for a Sigma account, if you haven’t already. All it takes is an email address, AWS account (comes with that cool free tier, if you’re a new user!), GitHub account (also free) and a good web browser. We have a short-and-sweet writeup to get you started within minutes; and will probably come up with a nice video as well, pretty soon!

A project is born

Once you are in, create a new project (with a catchy name to impress your buddies — how about GossipHunter?). The Sigma editor will create a template lambda for you, and we can start right away.

Nurtured with ❤ by NewsAPI

As my gossip source, I picked the Entertainment Weekly API by newsapi.org. Their API is quite simple and straightforward, and a free signup with just an email address gets you an API key with 1000 requests per day! In case you have your own love, feel free to switch just the API request part of the code (coming up soon!), and the rest should work just fine!

The recipe

Our lambda will be periodically pulling data from this API, comparing the results with what we already know (stored in DynamoDB) and sending out SMS notifications (via SNS) to your phone number (or email, or whatever other preferred medium that SNS offers) for any already unknown (hence “hot”) results. We will store any newly seen topics in DynamoDB, so that we can prevent ourselves from sending out the same gossip repeatedly.

(By the way, if you have access to a gossip API that actually emits/notifies you of latest updates (e.g. via webhooks) rather than us having to poll for and filter them, you can use a different, more efficient approach such as configuring an API Gateway trigger and pointing the API webhook to the trigger endpoint.)

Okay, let’s chop away!

The wake-up call(s)

First, let’s drag a CloudWatch entry from the left Resources pane and configure it to fire our lambda; to prevent distractions during working hours, we will configure it to run every 15 minutes, only from 7 PM (when you are back from work) to midnight, and from 5AM to 8 AM (when you are on your way back to work). This can be easily achieved through a New, Schedule-type trigger that uses a cron expression such as 5-7,19-23 0/15 ? * MON-FRI *. (Simply paste 0/15 , 5-7,19-23 (no spaces) and MON-FRI into the Minutes, Hours and Day of Week fields, and type a ? under Day of Month.)

But wait! The real value of gossip is certainly in the weekend! So let’s add (drag, drop, configure) another trigger to run GossipHunter all day (5 AM — midnight!) over the weekend; just another cron with 0/10 (every ten minutes this time! we need to be quick!) in Minutes, 5-23 in Hours, ? in Day of Month and SAT,SUN in Day of Week.

Okay, time to start coding!

Grabbing the smoking hot stuff

Let’s first fetch the latest gossips from the API. The requests module could do this for us in a heartbeat, so we'll go get it: click the Add Dependency button on the toolbar, type in requests and click Add once our subject appears in the list:

Now for the easy part:

request.get(`https://newsapi.org/v2/top-headlines?sources=entertainment-weekly&apiKey=your-api-key`,
  (error, response, body) => {

    callback(null,'Successfully executed');
  })

Gotta hide some secrets?

Wait! The apiKey parameter: do I have to specify the value in the code? Since you probably would be saving all this in GitHub (yup, you guessed right!) won't that compromise my token?

We also had the same question; and that’s exactly why, just a few weeks ago, we introduced the environment variables feature!

Go ahead, click the Environment Variables ((x)) button, and define a KEY variable (associated with our lambda) holding your API key. This value will be available for your lambda at runtime, but it will not be committed into your source; you can simply provide the value during your first deployment after opening the project. And so can any of your colleagues (with their own API keys, of course!) when they get jealous and want to try out their own copy of your GossipHunter!

(Did I mention that your friends can simply grab your GossipHunter’s GitHub repo URL — once you have saved your project — and open it in Sigma right away, and deploy it on their own AWS account? Oh yeah, it’s that easy!)

Cool! Okay, back to business.

Before we forget it, let’s append process.env.KEY to our NewsAPI URL:

request.get(`https://newsapi.org/v2/top-headlines?sources=entertainment-weekly&apiKey=${process.env.KEY}`,

And extract out the gossips list, with a few sanity checks:

  (error, response, body) => {
    let result = JSON.parse(body);
    if (result.status !== "ok") {
      return callback('NewsAPI call failed!');
    }
    result.articles.forEach(article => {

    });

    callback(null,'Successfully executed');
  })

Sifting out the not-so-hot

Now the tricky part: we have to compare these with the most recent gossips that we have dispatched, to detect whether they are truly “new” ones, i.e. filter the ones that have not already been dispatched.

For starters, we shall maintain a DynamoDB table gossips to retain the gossips that we have dispatched, serving as our GossipHunter's "memory". Whenever a "new" gossip (i.e. one that is not already available in our table) is encountered, we shall send it out via SNS, the Simple Notification Service and add it to our table so that we will not send it out again. (Later on we can improve our "memory" to "forget" (delete) old entries so that it would not keep on growing indefinitely, but for the moment, let's not worry about it.)

What’s thaat, Dynamo-DB?

For the DynamoDB table, simply drag a DynamoDB entry from the resources pane into the editor, right into the forEach callback. Sigma will show you a pop-up where you can define your table (without a round trip to the DynamoDB dashboard!) and the operation you intend to perform on it. Right now we need to query the table for the gossip in the current iteration, so we can zip it by

• entering gossips into the Table Name field and url for the Partition Key,
• selecting the Get Document operation, and
• entering @{article.url} (note the familiar, ${}-like syntax?) in the Partition Key field.

      result.articles.forEach(article => {
        ddb.get({
          TableName: 'gossips',
          Key: { 'url': article.url }
        }, function (err, data) {
          if (err) {
            //handle error
          } else {
            //your logic goes here
          }
        });

      });

In the callback, let’s check if DynamoDB found a match (ignoring any failed queries):

        }, function (err, data) {
          if (err) {
            console.log(`Failed to check for ${article.url}`, err);
          } else {
            if (data.Item) {  // match found, meaning we have
                              // already saved it
              console.log(`Gossip already dispatched: ${
                article.url}`);
            } else {

            }
          }
        });

Compose (160 characters remaining)

In the nested else block (when we cannot find a matching gossip), we prepare an SMS-friendly gossip text (including the title, and optionally the description and URL if we can stuff them in; remember the 160-character limit?). (Later you can tidy things up by throwing in a URL-shortener logic and so on, but for the sake of simplicity, I'll pass.)

            } else {
              let titleLen = article.title.length;
              let descrLen = article.description.length;
              let urlLen = article.url.length;

              let gossipText = article.title;
              if (gossipText.length + descrLen < 160) {
                gossipText += "\n" + article.description;
              }
              if (gossipText.length + urlLen < 160) {
                gossipText += "\n" + article.url;
              }

Hitting “Send”

Now we can send out our gossip as an SNS SMS. For this,

• drag an SNS entry from the left pane into the editor, right after the last if block,
• select Direct SMS as the Resource Type,
• enter your mobile number into the Mobile Number field,
• populate the SMS text field with @{gossipText},
• type in GossipHuntr as the Sender ID (unfortunately the sender ID cannot be longer than 11 characters, but it doesn't really matter since it is just the text message sender's name; besides, GossipHuntr is more catchy, right? :)), and
• click Inject.

But…

Wait! What would happen if your best buddy grabs your repo and deploys it; his gossips would also start flowing into your phone!

Perhaps a clever trick would be to extract out the phone number into another environment variable, so that you and your best buddy can pick your own numbers (and part ways, still as friends) at deployment time. So click the (x) again and add a new PHONE variable (with your phone number), and use it in the Mobile Number field instead as (you guessed it!) @{process.env.PHONE}:

            } else {
              let titleLen = article.title.length;
              let descrLen = article.description.length;
              let urlLen = article.url.length;

              let gossipText = article.title;
              if (gossipText.length + descrLen < 160) {
                gossipText += "\n" + article.description;
              }
              if (gossipText.length + urlLen < 160) {
                gossipText += "\n" + article.url;
              }

              sns.publish({
                Message: gossipText,
                MessageAttributes: {
                  'AWS.SNS.SMS.SMSType': {
                    DataType: 'String',
                    StringValue: 'Promotional'
                  },
                  'AWS.SNS.SMS.SenderID': {
                    DataType: 'String',
                    StringValue: 'GossipHuntr'
                  },
                },
                PhoneNumber: process.env.PHONE
              }).promise()
                .then(data => {
                  // your code goes here
                })
                .catch(err => {
                  // error handling goes here
                });
            }

(In case you got overexcited and clicked Inject before reading the but… part, chill out! Dive right into the code, and change the PhoneNumber parameter under the sns.publish(...) call; ta da!)

Tick it off, and be done with it!

One last thing: for this whole contraption to work properly, we also need to save the “new” gossip in our table. Since you have already defined the table during the query operation, you can simply drag it from under the DynamoDB list on the resources pane (click the down arrow on the DynamoDB entry to see the table definition entry); drop it right under the SNS SDK call, select Put Document as the operation, and configure the new entry as url = ${article.url} (by clicking the Add button under Values and entering url as the key and @{article.url} as the value).

                .then(data => {
                  ddb.put({
                    TableName: 'gossips',
                    Item: { 'url': article.url }
                  }, function (err, data) {
                    if (err) {
                      console.log(`Failed to save marker for ${
                        article.url}`, err);
                    } else {
                      console.log(`Saved marker for ${
                        article.url}`);
                    }
                  });
                })
                .catch(err => {
                  console.log(`Failed to dispatch SMS for ${
                    article.url}`, err);
                });

Time to polish it up!

Since we’d be committing this code to GitHub, let’s clean it up a bit (all your buddies would see this, remember?) and throw in some comments:

let AWS = require('aws-sdk');
const sns = new AWS.SNS();
const ddb = new AWS.DynamoDB.DocumentClient();
let request = require('request');

exports.handler = function (event, context, callback) {

  // fetch the latest headlines
  request.get(`https://newsapi.org/v2/top-headlines?sources=entertainment-weekly&apiKey=${process.env.KEY}`,
    (error, response, body) => {

      // early exit on failure
      let result = JSON.parse(body);
      if (result.status !== "ok") {
        return callback('NewsAPI call failed!');
      }

      // check each article, processing if it hasn't been already
      result.articles.forEach(article => {
        ddb.get({
          TableName: 'gossips',
          Key: { 'url': article.url }
        }, function (err, data) {
          if (err) {
            console.log(`Failed to check for ${article.url}`, err);
          } else {
            if (data.Item) {  // we've seen this before; ignore it
              console.log(`Gossip already dispatched: ${
                article.url}`);

            } else {
              let titleLen = article.title.length;
              let descrLen = article.description.length;
              let urlLen = article.url.length;

              // stuff as much content into the text as possible
              let gossipText = article.title;
              if (gossipText.length + descrLen < 160) {
                gossipText += "\n" + article.description;
              }
              if (gossipText.length + urlLen < 160) {
                gossipText += "\n" + article.url;
              }

              // send out the SMS
              sns.publish({
                Message: gossipText,
                MessageAttributes: {
                  'AWS.SNS.SMS.SMSType': {
                    DataType: 'String',
                    StringValue: 'Promotional'
                  },
                  'AWS.SNS.SMS.SenderID': {
                    DataType: 'String',
                    StringValue: 'GossipHuntr'
                  },
                },
                PhoneNumber: process.env.PHONE
              }).promise()
                .then(data => {
                  // save the URL so we won't send this out again
                  ddb.put({
                    TableName: 'gossips',
                    Item: { 'url': article.url }
                  }, function (err, data) {
                    if (err) {
                      console.log(`Failed to save marker for ${
                        article.url}`, err);
                    } else {
                      console.log(`Saved marker for ${
                        article.url}`);
                    }
                  });
                })
                .catch(err => {
                  console.log(`Failed to dispatch SMS for ${
                    article.url}`, err);
                });
            }
          }
        });
      });

      // notify AWS that we're good
      // (no need to track/notify errors at this stage)
      callback(null, 'Successfully executed');
    })
}

All done!

3, 2, 1, ignition!

Click Deploy on the toolbar, which will set a chain of actions in motion: first the project will be saved (committed to your own GitHub repo, with a commit message of your choosing), then built and packaged (fully automated!) and finally deployed into your AWS account (giving you a chance to review the deployment summary before it is executed).

Once the progress bar hits the end and the deployment status says CREATE_COMPLETE (or UPDATE_COMPLETE in case you missed a spot and had to redeploy), GossipHunter is ready for action!

Houston, we’re GO!

Until your DynamoDB table is primed up (populated with enough gossips to start waiting for updates), you would receive a trail of gossip texts. After that, whenever a new gossip comes up, you will receive it on your mobile within a matter of minutes!

All thanks to the awesomeness of serverless and AWS, and Sigma that brings it all right into your web browser.

Originally published at randomizd.blogspot.com on May 15, 2018.

How to rob a bank: no servers - just a ballpoint pen! was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.

Resources, triggers, and heck, operations

With serverless, you stopped caring about the server.

With Sigma, you stopped (or will stop, if not already) about the platform.

Now all you care about is your code — the bliss of every programmer.

Or is it?

I hold her (the code) in my arms.

If you have done time with serverless frameworks, you would already know how they take away your platform-phobia, abstracting out the platform-specific bits of your serverless app.

And if you have already tried out Sigma, you would have noticed how it takes things further, relieving you of the burden of the configuration and deployment aspects as well.

Leaving behind just the code.

Just the beautiful, raw code.

So, what’s the catch?

Okay. Now for the untold, unspoken, not-so-popular part.

You see, my friend, every good thing comes at a price.

Lucky for you, with Sigma, the price is very affordable.

Just a matter of sticking to a few ground rules while you develop your app. That’s all.

Resources, resources.

All of Sigma’s voodoo depends on one key thing: resources.

The concept is quite simple: every piece of your serverless app — may it be a DynamoDB table, S3 bucket or SNS topic — is a resource from Sigma’s point of view.

If you remember the Sigma UI, the Resources pane on the left contains different resource types that you can have in your serverless app. (True, it’s pretty short; but we’re working on it :))

Behind the scenes

When you drag a resource from this pane, into your code, Sigma secretly creates a resource (which it would later deploy into your serverless provider) to track the configurations of the actual service entity (say, the S3 bucket that should exist in AWS by the time your function is running) and all its usages within your code. The tracking is fully automated; frankly, you didn’t even want to know about that.

“New” or “existing”?

On almost all of Sigma’s resource configuration pop-ups, you may have noticed two options: “new” vs “existing”. “New” resources are the ones that would be (or have already been) created as a result of your project, whereas “existing” ones are those which have been created outside of your project.

Now that’s a tad bit strange because we would usually use “existing” to denote things that “exist”, regardless of their origin — even if they came from Mars.

Better brace yourself, because this also gives rise to a weirder notion: once you have deployed your project, the created resources (which now “exist” in your account) are still treated by Sigma as “new” resources!

And, as if that wasn’t enough, this makes the resources lists in Sigma behave in totally unexpected ways; after you define a “new” resource, whenever you want to reuse that resource somewhere else, you would have to look for it under the “existing” tab of the resource pop-up; but it will be marked with a “ (new)” prefix because, although it is already defined, it remains “new” from Sigma’s point of view.

Perhaps we should have called them “Sigma” resources; or perhaps even better, “project” resources; while we scratch our heads, feel free to chip in and help us with a better name!

Rule o’ thumb

Until this awkwardness is settled, the easiest way to get through this mess is to stick to this rule of thumb:

If you added a resource to your current Sigma project, Sigma would treat it as a “new” resource till the end of eternity.

Bottom line: no worries!

Being able to use existing resources is sometimes cool, but it means that your project would be much less portable. Sigma will always assume that the resources referenced by your project are already in existence, regardless of whatever AWS account you attempt to deploy it. At least until (if) we (ever) come up with a different resource management mechanism.

If you want portability, always stick to new resources. That way, even if a complete stranger gets hold of your project and deploys it in his own, alien, unheard-of AWS account, the project would still work.

If you are integrating with an already existing set of resources (e.g. the set of S3 bucket in your already-running dev/test/prod environment), using existing resources is the obvious (and the most convenient) choice.

Anyways, back to our discussion:

Where were we?

Ah, yes. Resources.

The secret life of resources

In a serverless app, you basically use resources for two things:

• for triggering the app (as an event source, a.k.a. trigger)
• for performing work inside the app, such as invoking external services

Resources? Triggers?? Operations???

Sigma also associates its resources with your serverless app in a similar fashion:

In Sigma, a function can have several triggers (as long as the application itself is aware of tackling different trigger event types!), and can contain several operations (obviously).

Yet, they’re different.

It is noteworthy that a resource itself is not a trigger or an operation; triggers and operations are associated with resources (they kind of “bridge” functions and resources) but a resource has its own independent life. As a result, a resource can power many triggers (to be precise, zero or more) and get involved in many operations, across many (again, zero or more) functions.

A good example is S3. If you want to write an image resizer function that would pick and process images dropped into a S3 bucket, you would configure a S3 trigger to invoke the function upon the file drop, and a S3 GetObject operation to retrieve and process the file; however, both will point to the same S3 resource, namely the bucket where images are being dropped into and fetched from.

Launch time!

At deployment, Sigma will take care of putting the pieces together — trigger configs, runtime permissions and whatnot — based on which function is associated with which resources, and in which ways (trigger-mode vs operation-mode). You can simply drag, drop and configure your buckets, queues and stuff, write your code, and totally forget about the rest!

That’s the beauty of Sigma.

When a resource is “abandoned” (meaning that it is not used in any trigger or operation), it shows up in the “unused resources” list (remember the dustbin button on the toolbar?) and can be removed from the project; remember that if you do this, provided that the resource is a “new” one (rule of thumb: one created in Sigma), it will be automatically removed from your serverless provider account (for example, AWS) during your next deployment!

So there!

if Sigma’s resource model (the whole purpose of this article) looks like a total mess-up to you, feel free to raise your voice on StackOverflow — or better still, our GitHub space, FB page or Twitter feed; we would appreciate it very much!

Of course, Sigma has nothing to hide; if you check your AWS account after a few Sigma deployments, you would realize the things we have been doing under the hood.

All of it, to make your serverless journey as smooth as possible.

And easy.

And fun. :)

Welcome to the world of Sigma!

Originally published at randomizd.blogspot.com on April 22, 2018.

Sigma, the first Truly Serverless DevKit! was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.

A no-brainer!

Few years ago, containers swept through the dev and devops lands like a category-6 hurricane.

Docker. Rkt. others.

Docker Swarm.

K8s.

OpenShift.

Right now we are literally at the epicenter, but when we glimpse at the horizon we see another one coming!

Serverless.

The funny thing is, “serverless” itself is a misnomer.

Of course there are servers. There are always servers. How can programs execute themselves in thin air, without the support of the underlying hardware and utility modules? So, there are servers.

Just not where you would expect them to be.

Traversing the timeline of computing, we see the turbulent track record?? of servers: birth in secret dungeons of vaccum tubes and city-scale power supplies; multi-ton boxes; networks; clusters; cloud datacenters and server farms (agriculture just lost its royalty!); containers.

Over time, we see servers losing their significance. Gradually, but steadily.

And now, suddenly, puff! They are gone.

Invisible, to be precise.

With serverless, you no longer care about the server. It may be a physical machine, a cloud VM, a K8s pod, an ECS container… heck, even an IoT rig.

Nobody cares, as long as the job gets done.

In this sense, we realize that serverless is nothing new; the concept, and even some pracical implementations, have been there since as far back as 2006. You yourself may have benefited from serverless (or conceptually serverless) architectures; while one may argue them to be PaaSes, Google App Engine and Google Apps Script (especially) are good examples.

Just like touchscreens, serverless resemblances had always been there, but never has the marketing hype been this intense — obviously it is growing, and we’ll surely see more of it as time flies by.

AWS had an early entry to the arena and currently owns a huge market share, bigger than all the others combined; Azure is behind, but catching up fast; and Google still seems to be more focused on Kubernetes and related containerization stuff although they too are on the track with Cloud Functions and Firebase.

Streaming and event-driven architectures are playing their part in bringing value to serverless. We should also not forget the cloud hype that made people go every-friggin-thing-as-a-service and later left them wondering how they could pay only for what they really use, only while they use it.

All ramblings aside, serverless is growing in popularity. Platforms are evolving to support more event sources, better integration support for other services and richer monitoring and statistics. Frameworks like Serverless are striving to provide a unified and generifier serverless development experience while IDEs like Sigma are doing their part in helping newbies (and sometimes even professionals) get going with serverless with minimum hassle and maximum speed.

Being new and shiny does not necessarily mean that serverless is the silver bullet for all your dev issues; in fact, right now it fits into only a few enterprise use cases (primarily due to the lack of strong guarantees, which are quite commonplace in the bureaucratic enterprise atmosphere). Nevertheless, providers are already working on this, and we can expect some disruptive — if not revolutionary — changes in the not-too-distant future. However it is always best to re-iterate your requirements before officially stepping into the serverless world, because serverless demands quite a shift in your application architecture, devops, as well as the very core of your developer mindset.

And, of course, the best way to pick the cake is to taste it yourself.

Originally published at randomizd.blogspot.com on April 22, 2018.

Serverless: a no-brainer! was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.

Towards a Faster Serverless IDE

TL;DR: The QuickBuild/QuickDeploy feature described here is pretty much obsoleted by the test framework (ingeniously hacked together by @CWidanage), that gives you a much more streamlined dev-test experience with much better response time!

In case you hadn’t noticed, we have recently been chanting about a new Serverless IDE, the mighty SLAppForge Sigma.

With Sigma, developing a serverless app becomes as easy as drag-drop, code, and one-click-Deploy; no getting lost among overcomplicated dashboards, no eternal struggles with service entities and their permissions, no sailing through oceans of docs and tutorials — above all that, nothing to install (just a web browser — which you already have!).

So, how does Sigma do it all?

In case you already tried Sigma and dug a bit deeper than just deploying an app, you may have noticed that it uses AWS CodeBuild under the hood for the build phase. While CodeBuild gives us a fairly simple and convenient way of configuring and running builds, it has its own set of perks:

• CodeBuild takes a significant time to complete (sometimes close to a minute). This may not be a problem if you just deploy a few sample apps, but it can severely impair your productivity — especially when you begin developing your own solution, and need to reflect your code updates every time you make a change.
• The AWS Free Tier only includes 100 minutes of CodeBuild time per month. While this sounds like a generous amount, it can expire much faster than you think — especially when developing your own app, in your usual trial-and-error cycles ;) True, CodeBuild doesn’t cost much either ($0.005 per minute of build.general1.small), but why not go free while you can? :)

Options, people?

Lambda, on the other hand, has a rather impressive free quota of 1 million executions and 3.2 million seconds of execution time per month. Moreover, traffic between S3 and Lambda is free as far as we are concerned!

Oh, and S3 has a free quota of 20000 reads and 2000 writes per month — which, with some optimizations on the reads, is quite sufficient for what we are about to do.

2 + 2 = …

So, guess what we are about to do?

Yup, we’re going to update our Lambda source artifacts in S3, via Lambda itself, instead of CodeBuild!

Of course, replicating the full CodeBuild functionality via a lambda would need a fair deal of effort, but we can get away with a much simpler subset; read on!

The Big Picture

First, let’s see what Sigma does when it builds a project:

• prepare the infra for the build, such as a role and an S3 bucket, skipping any that already exist
• create a CodeBuild project (or, if one already exists, update it to match the latest Sigma project spec)
• invoke the project, which will:

1. download the Sigma project source from your GitHub repo,
2. run an npm install to populate its dependencies,
3. package everything into a zip file, and
4. upload the zip artifact to the S3 bucket created above

• monitor the project progress, and retrieve the URL of the uploaded S3 file when done.

And usually every build has to be followed by a deployment; to update the lambdas of the project to point to the newly generated source archive; and that means a whole load of additional steps!

• create a CloudFormation stack (if one does not exist)
• create a changeset that contains the latest updates to be published
• execute the changeset, which will, at the least, have to:

1. update each of the lambdas in the project to point to the new source zip file generated by the build, and
2. in some cases, update the triggers associated with the modified lambdas as well

• monitor the stack progress until it gets through with the update.

All in all, well over 60–90 seconds of your precious time — all to accommodate perhaps just one line (or how about one word, or one letter?) of change!

Can we do better?

At first glance, we see quite a few redundancies and possible improvements:

• Cloning the whole project source from scratch is overkill, especially when only a few lines/files have changed.
• Every build will download and populate the NPM dependencies from scratch, consuming bandwidth, CPU cycles and build time.
• The whole zip file is now being prepared from scratch after each build.
• Since we’re still in dev, running a costly CF update for every single code change doesn’t make much sense.

But since CodeBuild invocations are stateless and CloudFormation’s resource update logic is mostly out of our hands, we don’t have the freedom to meddle with many of the above; other than simple improvements like enabling dependency caching.

Trimming down the fat

However, if we have a lambda, we have full control over how we can simplify the build!

If we think about 80% — or maybe even 90% — of the cases for running a build, we see that they merely involve changes to application logic (code); you don’t add new dependencies, move your files around or change your repo URL all the time, but you sure as heck would go through an awful lot of code edits until your code starts behaving as you expect it to!

And what does this mean for our build?

80% — or even 90% — of the time, we can get away by updating just the modified files in the lambda source zip, and updating the lambda functions themselves to point to the updated file!

Behold, here comes QuickDeploy!

And that’s exactly what we do, with the QuickBuild/QuickDeploy feature!

Lambda to the rescue!

QuickBuild uses a lambda (deployed in your own account, to eliminate the need for cross-account resource access) to:

• fetch the latest CodeBuild zip artifact from S3,
• patch the zip file to accommodate the latest code-level changes, and
• upload the updated file back to S3, overriding the original zip artifact

Once this is done, we can run a QuickDeploy which simply sends an UpdateFunctionCode Lambda API call to each of the affected lambda functions in your project, so that they can scoop up the latest and greatest of your serverless code!

And the whole thing does not take more than 15 seconds (give or take the network delays): a raw 4x improvement in your serverless dev workflow!

A sneak peek

First of all, we need a lambda that can modify an S3-hosted zip file based on a given set of input files. While it’s easy to make with NodeJS, it’s even easier with Python, and requires zero external dependencies as well:

Here we go… Pythonic!

import boto3

from zipfile import ZipFile, ZipInfo, ZIP_DEFLATED

s3_client = boto3.client('s3')

def handler(event, context):
  src = event["src"]
  if src.find("s3://") > -1:
    src = src[5:]
  
  bucket, key = src.split("/", 1)
  src_name = "/tmp/" + key[(key.rfind("/") + 1):]
  dst_name = src_name + "_modified"
  
  s3_client.download_file(bucket, key, src_name)
  zin = ZipFile(src_name, 'r')
  
  diff = event["changes"]
  zout = ZipFile(dst_name, 'w', ZIP_DEFLATED)
  
  added = 0
  modified = 0
  
  # files that already exist in the archive
  for info in zin.infolist():
    name = info.filename
    if (name in diff):
      modified += 1
      zout.writestr(info, diff.pop(name))
    else:
      zout.writestr(info, zin.read(info))
  
  # files in the diff, that are not on the archive
  # (i.e. newly added files)
  for name in diff:
    info = ZipInfo(name)
    info.external_attr = 0755 << 16L
    added += 1
    zout.writestr(info, diff[name])
  
  zout.close()
  zin.close()
  
  s3_client.upload_file(dst_name, bucket, key)
  return {
    'added': added,
    'modified': modified
  }

We can directly invoke the lambda using the Invoke API, hence we don't need to define a trigger for the function; just a role with S3 full access permissions would do. (We use full access here because we would be reading from/writing to different buckets at different times.)

CloudFormation, you beauty.

From what I see, the coolest thing about this contraption is that you can stuff it all into a single CloudFormation template (remember the lambda command shell?) that can be deployed (and undeployed) in one go:

AWSTemplateFormatVersion: '2010-09-09'
Resources:
  zipedit:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: zipedit
      Handler: index.handler
      Runtime: python2.7
      Code:
        ZipFile: >
          import boto3
          
          from zipfile import ZipFile, ZipInfo, ZIP_DEFLATED
          
          s3_client = boto3.client('s3')
          
          def handler(event, context):
            src = event["src"]
            if src.find("s3://") > -1:
              src = src[5:]
            
            bucket, key = src.split("/", 1)
            src_name = "/tmp/" + key[(key.rfind("/") + 1):]
            dst_name = src_name + "_modified"
            
            s3_client.download_file(bucket, key, src_name)
            zin = ZipFile(src_name, 'r')
            
            diff = event["changes"]
            zout = ZipFile(dst_name, 'w', ZIP_DEFLATED)
            
            added = 0
            modified = 0
            
            # files that already exist in the archive
            for info in zin.infolist():
              name = info.filename
              if (name in diff):
                modified += 1
                zout.writestr(info, diff.pop(name))
              else:
                zout.writestr(info, zin.read(info))
            
            # files in the diff, that are not on the archive
            # (i.e. newly added files)
            for name in diff:
              info = ZipInfo(name)
              info.external_attr = 0755 << 16L
              added += 1
              zout.writestr(info, diff[name])
            
            zout.close()
            zin.close()
            
            s3_client.upload_file(dst_name, bucket, key)
            return {
              'added': added,
              'modified': modified
            }
      Timeout: 60
      MemorySize: 256
      Role:
        Fn::GetAtt:
        - role
        - Arn
  role:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      - arn:aws:iam::aws:policy/AmazonS3FullAccess
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Action: sts:AssumeRole
          Effect: Allow
          Principal:
            Service: lambda.amazonaws.com

Moment of truth

Once the stack is ready, we can start submitting our QuickBuild requests to the lambda!

// assuming auth stuff is already done
let lambda = new AWS.Lambda({region: "us-east-1"});

// ...

lambda.invoke({
  FunctionName: "zipedit",
  Payload: JSON.stringify({
    src: "s3://bucket/path/to/archive.zip",
    changes: {
      "path/to/file1/inside/archive": "new content of file1",
      "path/to/file2/inside/archive": "new content of file2",
      // ...
    }
  })
}, (err, data) => {
  let result = JSON.parse(data.Payload);
  let totalChanges = result.added + result.modified;
  if (totalChanges === expected_no_of_files_from_changes_list) {
    // all izz well!
  } else {
    // too bad, we missed a spot :(
  }
});

Once QuickBuild has completed updating the artifact, it’s simply a matter of calling UpdateFunctionCode on the affected lambdas, with the S3 URL of the artifact:

lambda.updateFunctionCode({
  FunctionName: "original_function_name",
  S3Bucket: "bucket",
  S3Key: "path/to/archive.zip"
})
.promise()
.then(() => { /* done! */ })
.catch(err => { /* something went wrong :( */ });

(In our case the S3 URL remains unchanged (because our lambda simply overwrites the original file), but it still works because the Lambda service makes a copy of the code artifact when updating the target lambda.)

To speed up the QuickDeploy for multiple lambdas, we can even parallelize the UpdateFunctionCode calls:

Promise.all(
  lambdaNames.map(name =>
    lambda.updateFunctionCode({ /* params */ })
    .promise()
    .then(() => { /* done! */ }))

.then(() => { /* all good! */ })
.catch(err => { /* failures; handle them! */ });

And that’s how we gained an initial 4x improvement in our lambda deployment cycle, sometimes even faster than the native AWS Lambda console!

Sigma QuickBuild: Towards a Faster Serverless IDE was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.

Application deployment and orchestration is no exception: in just three decades we have gone from legacy monoliths to modular systems, P2P integration, middleware, SOA, microservices, and the latest, functions or FaaS. The deployment paradigm has shifted to comply, with in-house servers and data centers, enterprise networks, VMs, containers, and now, “serverless”.

Keeping up with things was easy so far, but the serverless paradigm demands quite a shift in the developer mindset (not to mention the programming paradigm). This, combined with the lack of intuitive tooling, has fairly hindered the adoption of serverless application development, even among the cutting-edge developers.

And (you guessed it), that’s where _____ comes into play.

Missing something?

Yup.

A way to glue stuff together.

A way to compose a serverless application care-free. Without having to worry — and to read tons of documentation, watch reels of tutorials, or trial-and-error till your head is on fire — about all the bells and whistles of the underlying framework and related services.

Essentially, a sum-up of all that is serverless.

Sum.

Sigma.

Σ.

What’s in a name?

As the name implies, (quoted from the official website)

The Sigma editor is a hybrid between the simplicity of drag-and-drop style development, and the full and unlimited power of raw code.

The drag-and-drop events generate sample or usage snippets to quickly get started, and introduce a powerful, uniform and intuitive library with auto-completion, which allow users to quickly become productive in developing Serverless applications that integrate with a myriad of AWS based services.

Making of…

Before Sigma, a bit of background of its origins.

As a first-time user of AWS Lambda, one of our team members brought up an impressive series of questions: if serverless is so cool, why is it so complicated to get an application up and running in Lambda?

And we ourselves started trying out the same thing. Guess what, we got the same questions as well.

So we set out to devise something that could bypass all those tedious steps: something where we could just write our code, save it, and deploy it as a working serverless application, without having to wander from dashboard to dashboard, or sift through heaps of documentation or reels of video tutorials.

And we ended up with Sigma!

Yet another IDE?

At first glance, Sigma looks like another cloud IDE that additionally supports deploying an application directly into a serverless provider environment (AWS so far).

However, there are a few not-to-be-missed distinctions:

• Unlike many of the existing cloud IDEs, Sigma itself is truly serverless; it runs completely inside your browser, using backend services only for user authentication and analytics, and requires no dedicated server/VM/container to be running in the background. Just fire up your browser, log in, and start coding your dream away.
• Sigma directly interacts with and configures the serverless platform on your behalf, using the credentials that you provide, saving hours of configuration and troubleshooting time. No more back-and-forth between overcomplicated dashboards and dizzying configurations.
• Sigma encapsulates the complexities of the serverless platform, such as service entities, access policies, invocation trigger configurations and associated permissions, and even some API invocation syntaxes, saving you the trouble of having to delve into piles of documentation.
• All of this comes in a fairly simple, intuitive environment, with easy, drag-and-drop composition combined with the full power of written code. Drag and drop a DynamoDB table into the UI, pick your operation and just write your logic, and Sigma will do the magic of automatically creating, configuring and managing the DynamoDB table on your AWS account.

Now, I won’t say that’s “just another IDE”; what say you?

A serverless platform?

Based on the extent of its capabilities, you may also be inclined to classify Sigma as a serverless platform. This is true to a great extent; after all, Sigma facilitates all of it — composing, building and deploying the application! However…

Hybrid! It’s a hybrid!

Yup, Sigma is a hybrid.

Fusion of a cloud IDE (which in itself is a hybrid of graphical composition and granular coding) and a serverless development framework (which automatically deploys and manages the resources, permissions, wiring and other bells and whistles of your serverless application).

One of a kind.

To be precise, the first of its kind.

A new beginning

With Sigma, we hope to redefine serverless development.

Yup. Seriously.

From here onwards, developers shall simply focus on what they need to achieve: workflow, business logic, algorithm, whatever.

Not about all the gears and crankshafts of the platform on which they would deploy the whole thing.

Not about the syntax of, or permissions required by, platform-specific API or service calls.

Not about the deployment, configurations and lifecycle of all the tables, buckets, streams, schedulers, REST endpoints, queues and so forth, that they want to use within their application.

Because Sigma will take care of it all.

And we believe our initiative would

• make it easy for newcomers to get started with serverless development,
• improve the productivity of devs that are already familiar with — or even experts of — serverless development,
• speed up the adoption of serverless development among the not-yet-serverless community,
• allow y’all to “think serverless”, and
• make serverless way much fun!

We have proof!

While developing Sigma, we also wanted to verify that we were doing the right thing, and doing it right. So we bestowed upon two of our fellows, the responsibility to develop two showcase applications using Sigma: a serverless accounting webapp, and a location-based mobile dating app.

To our great joy, both experiments were successful!

The accounting app SLAppBook is now live for public access. By default it runs against one of our test serverless backends, but you can always deploy the serverless backend project on your own AWS account via Sigma and point the frontend to your brand new backend, after which you can use it for your own personal use!

The dating app HotSpaces is currently undergoing some rad improvements (see, now it’s the frontend that takes time to develop!) and will be out pretty soon!

So, once again, we have proof that Sigma really rocks it!

Far from perfection, but getting there; fast!

Needless to say, Sigma is pretty much an infant. It needs quite a lot more — more built-in services, better code suggestions, smarter resource handling, faster builds and deployments, support for other cloud platforms, you name it — before it can be considered “mature”.

But we are getting there. And we will get there. Fast.

We will publish our roadmap pretty soon, which would include (among other things) adding more AWS services, supporting integration with external APIs/services and, most importantly, expanding to other cloud providers like GCP and MS Azure.

That’s where we need your help.

We need you!

Needless to say, you are most welcome to try out Sigma. Sign up here, if you haven’t already, and start playing around with our samples (once you are signed in to Sigma, you can directly open them via the projects page). Or, if you feel adventurous, start off with a clean slate, and start building your own serverless application.

We are continually smoothening the ride, but you may hit a few bumps here and there. Possibly even hard ones. Sometimes even impassable. Maybe none, if you are really lucky.

Either way, we are eagerly waiting for your feedback. Just write us about anything that came to your mind: a missing functionality, a popular AWS service that you really missed in Sigma (there are hundreds, no doubt!), the next cloud platform you would like Sigma to support; a failed build, a faulty deployment, a nasty error that hogged your browser; or even the slightest of improvements that you would like to see, like a misaligned button, a hard-to-scroll pop-up or a badly-named text label.

You can either use our official feedback form or the “Report an Issue” option on the IDE Help menu, post your feedback in our GitHub issue tracker, or send us a direct email at info@slappforge.com.

If you would like to join hands with us in our forward march, towards a “think serverless” future, drop us an email at info@slappforge.com right away.

Welcome to Sigma!

That’s it; time to start your journey with Sigma!

Sigma: The New Kid on the Serverless Block was originally published in Think Serverless on Medium, where people are continuing the conversation by highlighting and responding to this story.