There are 4 different threads, each one will invoke exactly once one of the four functions fizz, buzz, fizzbuzz and number and we want them to produce the output in the expected order. For more details please refer to the Leetcode problem description.

I have tried to solve the problem in different ways, using different python concurrency primitives (sorry I haven’t realised until today how bad the syntax highlighting is in Medium).

Solution 1: Busy waiting with a lock

We have 4 distinct loops, but because of the mutually exclusive if conditions, only one will increase the shared counter variable at any given time. They keep spinning all the time, until it’s “their turn” to increment the counter value and print an output.

You might think that given only one of the if conditions evaluates to true for any given counter value then we don’t need the lock at all. Unfortunately that is not the case: at the end of the loop you can have a value > n printed despite self.counter >= n, as the condition might be evaluated before the counter is incremented in a different thread.

from threading import Lock
class FizzBuzz:
    def __init__(self, n: int):
        self.n = n
        self.counter = 1
        self.lock = Lock()

    def fizz(self, printFizz):
     while self.counter <= self.n:
            with self.lock:
                if self.counter <= self.n and self.counter %3 == 0 and self.counter %5 != 0:
                    printFizz()
                    self.counter +=1

    def buzz(self, printBuzz):
        while self.counter <= self.n:
            with self.lock:
                if self.counter <= self.n and self.counter %3 != 0 and self.counter %5 == 0:
                    printBuzz()
                    self.counter +=1

    def fizzbuzz(self, printFizzBuzz):
        while self.counter <= self.n:
            with self.lock:
                if self.counter <= self.n and self.counter % 15 == 0:
                    printFizzBuzz()
                    self.counter +=1

    def number(self, printNumber):        
        while self.counter <= self.n:
            with self.lock:
                if self.counter <= self.n and self.counter %3 > 0 and self.counter %5 > 0:
                    printNumber(self.counter)
                    self.counter +=1

Solution 2: a single thread orchestrating the other 3 with three conditions

fizz() , buzz() and fizzbuzz() threads are waiting and activated in turn by the number() thread, which orchestrates the whole program logic by notifying to a thread using its dedicated condition and then waiting for a signal on the same condition once the unblocked thread has printed and increased the counter. After n has been reached, the number() threads must take care of unblocking all the waiting threads despite it’s not their turn, just to let them (and the program) terminate. Compared to the solution 1, this is way faster, as doesn’t involve busy waiting.

import threading
class FizzBuzz:
    def __init__(self, n: int):
        self.n = n
        self.counter = 1
        self.cond_fizz = threading.Condition()
        self.cond_buzz = threading.Condition()
        self.cond_fizzbuzz = threading.Condition()

    def fizz(self, printFizz):
        while True:
            with self.cond_fizz:
                self.cond_fizz.wait()
                
                if self.stop_condition():
                    break
                
                printFizz()
                self.counter +=1
                self.cond_fizz.notify()

    def buzz(self, printBuzz):
        while True:
            with self.cond_buzz:
                self.cond_buzz.wait()                
                
                if self.stop_condition():
                    break
                
                printBuzz()
                self.counter +=1     
                self.cond_buzz.notify()               

    def fizzbuzz(self, printFizzBuzz):
        while True:
            with self.cond_fizzbuzz:
                self.cond_fizzbuzz.wait()            
                if self.stop_condition():
                    break
                printFizzBuzz()
                self.counter +=1
                self.cond_fizzbuzz.notify()            
       
    def number(self, printNumber):        
        while not self.stop_condition():            
            
            if self.counter %3 > 0 and self.counter %5 > 0:
                printNumber(self.counter)
                self.counter +=1
            elif self.counter % 3 == 0 and self.counter % 5 > 0:
                with self.cond_fizz:
                    self.cond_fizz.notify()                    
                    self.cond_fizz.wait() # waiting to get back control
            elif self.counter % 3 > 0 and self.counter % 5 == 0:
                with self.cond_buzz:
                    self.cond_buzz.notify()
                    self.cond_buzz.wait()
            else: # fizzbuzz
                with self.cond_fizzbuzz:
                    self.cond_fizzbuzz.notify()
                    self.cond_fizzbuzz.wait()
                    
        self.terminate_all_threads()
                    
    def stop_condition(self):
        return self.counter > self.n# all threads are waiting
    def terminate_all_threads(self): 
        with self.cond_fizzbuzz:
            self.cond_fizzbuzz.notify()
        with self.cond_fizz:
            self.cond_fizz.notify()    
        with self.cond_buzz:
            self.cond_buzz.notify()

Solution 3: Having the four threads passing control each other

Unlike Solution 2, here we have some kind of “decentralised” choreography: it’s not anymore just the number() thread containing all the logic, but each thread, once in execution, establishes what thread will be unblocked next, and then proceeds to unblocks it. Again, once we reach the stop condition we need some termination-specific code to unblock all the threads and let the threads (and the program) to terminate.

import threading
class FizzBuzz:
    def __init__(self, n: int):
        self.n = n
        self.counter = 1
        self.fizz_event = threading.Event()
        self.buzz_event = threading.Event()
        self.fb_event = threading.Event()
        self.number_event = threading.Event()
        self.number_event.set()

    def fizz(self, printFizz):

        while True:
            self.fizz_event.wait()
                
            if self.stop_condition():
                self.stop_all()
                break
                
            printFizz()
            self.counter +=1
            
            self.fizz_event.clear()
            self.trigger_next_event()

    def buzz(self, printBuzz):
        while True:
            self.buzz_event.wait()
                
            if self.stop_condition():
                self.stop_all()
                break
                
            printBuzz()
            self.counter +=1     
            
            self.buzz_event.clear()
            self.trigger_next_event()
                
    def fizzbuzz(self, printFizzBuzz):
        while True:
            
            self.fb_event.wait()
            if self.stop_condition():
                self.stop_all()
                break

            printFizzBuzz()
            self.counter +=1
            
            self.fb_event.clear()
            self.trigger_next_event()
    def number(self, printNumber):        
        while True:            
            self.number_event.wait()
            
            if self.stop_condition():
                self.stop_all()
                break

            printNumber(self.counter)
            self.counter +=1
            
            self.number_event.clear()
            self.trigger_next_event()                            
        
                    
    def stop_condition(self):
        return self.counter > self.n    
            
    def stop_all(self):
        self.number_event.set()
        self.fizz_event.set()
        self.buzz_event.set()
        self.fb_event.set()
            
    def trigger_next_event(self):
        if self.counter %3 > 0 and self.counter %5 > 0:
            self.number_event.set()
        elif self.counter % 3 == 0 and self.counter % 5 > 0:
            self.fizz_event.set()
        elif self.counter % 3 > 0 and self.counter % 5 == 0:
            self.buzz_event.set()
        else: # fizzbuzz
            self.fb_event.set()

Solution 4: Using a single condition, with every threads checking on a different predicate whenever we send a signal to all the threads waiting.

Every threads wait on a condition for a specific predicate self.counter == <next_number_this_threads_must_print>. When the predicate is true and a thread is unblocked, it prints and then send a signal on the condition to all the other threads. We are guaranteed that one of the threads will have its predicate evaluating to true, so one thread will print something, increment the counter and then notify again to all the threads. This is the faster solution described in this post.

import threading

class FizzBuzz:
    def __init__(self, n: int):
        self.n = n
        self.counter = 1
        self.cond = threading.Condition()

    def is_fizz(self):
        return self.counter % 3 == 0 and self.counter % 5 != 0

    def fizz(self, printFizz):
        while self.n >= 3:
            with self.cond:
                self.cond.wait_for(self.is_fizz)

                printFizz()
                self.counter += 1

                self.cond.notify_all()

            if self.counter + 2 > self.n or ((self.counter + 2) % 5 == 0 and self.counter + 5 > self.n):
                break

    def is_buzz(self):
        return self.counter % 3 != 0 and self.counter % 5 == 0

    def buzz(self, printBuzz):
        while self.n >= 5:
            with self.cond:
                self.cond.wait_for(self.is_buzz)

                printBuzz()
                self.counter += 1

                self.cond.notify_all()

            if self.counter + 4 > self.n or ((self.counter + 4) % 3 == 0 and self.counter + 9 > self.n):
                break

    def is_fb(self):
        return self.counter % 3 == 0 and self.counter % 5 == 0

    def fizzbuzz(self, printFizzBuzz):
        while self.n >= 15:

            with self.cond:
                self.cond.wait_for(self.is_fb)

                printFizzBuzz()
                self.counter += 1

                self.cond.notify_all()

             if self.counter + 14 > self.n:
                 break

    def is_nothing(self):
        return self.counter % 3 > 0 and self.counter % 5 > 0

    def number(self, printNumber):
        while True:
            with self.cond:
                self.cond.wait_for(self.is_nothing)

            printNumber(self.counter)
                self.counter += 1

            self.cond.notify_all()

            # find next number call
            i = self.counter
            while i % 3 == 0 or i % 5 == 0:
                i += 1
            # this is the last number printed
            if i > self.n:
                break

Solution 5: Using barriers

In this solution each thread has its own loop 1 to n, and we make sure we advance the loops in sync by using a barrier, waiting for the 4 threads to reach the barrier before unblocking all the threads. At that point the loop will continue, the next number will be processed by all the 4 threads (only one thread will eventually print something), and the program will wait again for all the threads to reach the barrier, and so on.

class FizzBuzz:
    def __init__(self, n: int):
        self.n = n
        self.barrier = threading.Barrier(4)

   def fizz(self, printFizz):
        for i in range(1, self.n+1):
            if i % 3 == 0 and i % 5 != 0:
                printFizz()

            self.barrier.wait()

   def buzz(self, printBuzz):
        for i in range(1, self.n+1):
            if i % 3 != 0 and i % 5 == 0:
                printBuzz()

            self.barrier.wait()

    def fizzbuzz(self, printFizzBuzz):
        for i in range(1, self.n+1):
            if i % 15 == 0:
                printFizzBuzz()

            self.barrier.wait()

    def number(self, printNumber):
        for i in range(1, self.n+1):

            if i % 3 > 0 and i % 5 > 0:
                printNumber(i)

            self.barrier.wait()

I’ve stopped writing updates on my AWS Solution Architect Professional certification prep, but I kept studying... and managed to pass the exam two days ago 🎉.

Because of COVID-19 all the exams were taking place online, with some cumbersome software which skyrocketed the cpu usage on my old MacBook Air Mid 2011 to the point that I was afraid my laptop would have eventually shut down due to overheating 🤷‍♂️. The software was recording you via webcam, not sure if the cpu usage was caused by some sophisticated software detecting if I was doing something dodgy or if the software was just very poorly written and the my behaviour was checked by a human behind the scenes…

Anyway, in the online proctored exams you’re suggested to login 30 mins before the exam start time to get everything set up. Too bad the certification platform actually had an outage until 2 minutes before my exam start time 😬 (it was a multiple hours outage, I wasn’t sure I’d be able to take the exam until the very last minute).

Alessio Piergiacomi on Twitter

Quite ironic from someone delivering training/exams on how to build highly available architectures to have a 4+ hours complete outage

A relevant difference between the on-site and online exams (or at least between how the on-site exam was set up 3 years ago and how it is done online today) is a set of restrictions which make the exam even harder from a body-mind perspective, for example:

• you can’t drink water during the exam, or you get immediately disqualified
• you can’t stand up and stretch during the exam, or you get immediately disqualified

So the online candidate has to sit for 180 minutes straight (or 210 with the extra 30 minutes for non-native English speakers), without drinking anything, while 75 moderately long questions are shot at them.

I did close the window to avoid outside noise, but being my room exposed to the sun during the exam this caused the temperature to increase over time, as my dehydration, making focusing on the questions a bit harder.

I can’t really talk much of the contents of the exams: it was covering a lot topics (but nothing really surprising, having prepared following the course I described here) and often tricky (some questions had 2 completely reasonable multiple choice answers out of 4, and you had to find out which one was the “most correct”), but the hard part was really the not optimal conditions in which I took the exam.

I ended up passing with 86% (75% was required for passing), but I feel like the exam tested too much your ability to keep calm and your mental toughness (breathing techniques and checking the time definitely helped me there).

My final 2 cents: personally I would change the format of the exam by reducing the number of questions and the duration of the exam, and maybe balancing the exam toughness by increasing the % required to pass. I think such measures would make the exam more about AWS knowledge and less about your speed reading/time management/mental toughness game.

While useful, it requires me to proof read and “refactor” the text, and ends up subtracting quite a substantial amount of time from the study.

I’ll keep doing the daily check-ins on the exam preparation to keep me accountable to push with the study, but I won’t add any technical details on the topics of the day, I’ll leave the technical details to my private (unrevised) notes.

I would rather have a course which integrates the important bits from the Whitepapers rather than a course that says “go and read them yourself”. So I’ve looked for some opinions around and settled for the DolfinEd Udemy course, which is more expensive than the acloud.guru, but apparently more complete.

I’ve started the new course, which starts with a AWS Certified Solution Architect Associate exam simulation, to see where you are.

That took pretty much my whole slot of study for the day, the exam simulation alone took 2+ hours 😃.

What I learned from the CSAA exam simulation

I passed the exam simulation (77%, 75% was required), but I’ve realized I have some gaps I’ll need to fix before attempting the Pro Exam. In particular the gaps regarded:

• what’s the retrieval process from S3 Glacier (i.e. temporarily storing for X days on S3 Reduced Redundancy Storage)
• diffrent types of AD (AWS Directory Service > 5000 users with trust relationship with the on-premise AD, Simple AD < 5000 users, AD Connector with a gateway that redirects directory requests to your on-premise Microsoft AD without any caching)
• Redis now supports cluster mode to scale more with sharding over more primary nodes.
• Virtual Interfaces (VIF) for Direct Connect: private VIF for VPC entry points or public VIF for access to AWS services or transit VIF for AWS Transite Gateway hub
• ASG: can span multi-AZ (and auto re-balance), health check grace period after startup is 300 second by default, different auto-scaling policies

Different auto-scaling policies in particular was a topic I completely removed off my head after the certification I took 3 years ago (now I start remembering bits and pieces). There are 4 types of policies:

• Manual scaling: manually change ASC min/desired/max
• Cyclic (schedule based) scaling, for predictable load changes
• On-demand/Dynamic (event based) scaling: in response to a CW alarm (CPU utilization, memory, network I/O) or an event. You need a scale-out and scale-in policy configured. You can have a cool down timer to expire before responding to more alarms (with simple scaling) or a warm-up timer which sets the time it takes for an instance to be ready and contribute to the watched metric
• Predictive scaling (both on demand but also tries to predict future traffic based on daily and weekly trends, using Machine Learning).

DNS and Route53

DNS

Naming system for computers, services or other resources connected to the internet or a private network. It is hierarchical: when you try to resolve google.it you first go to your local resolving server, if it already knows the mapping it returns you the IP, otherwise it goes to the top-level domain name server for the .it top level domain, which gives you the address of the content server who actually contains the record for google.it, then query it and finally obtain the IP address.

The mappings are stored as set of resource records (RR), characterised by a type, a name, RDATA, TTL and other fields less relevant in this context.

Type defines the type of mapping name -> RDATAcontained in the record. Examples of type include:

• A: address record, maps a name to an IPv4 address
• AAAA: IPv6 address record
• CNAME: canonical name record, maps an hostname to another hostname
• MX: mail exchange record
• NS: name server record, they tell you what content name server(s) is authorattive to a particular zone or domain name

Route53

Route53 is AWS’s managed DNS service. DNS records are grouped into “hosted zones”, collection of records that can be managed together, belonging to a single parent domain (e.g. alessio.engineer) .

Route53 does not support all the DNS record types (for example it doesn’t supprot DNSSEC related records, where DNSSEC is a protocol to secure the DNS traffic). It supports a custom type of record called alias, which is a mapping from a name to an AWS resource such as ELB, CF, API Gateway, S3 bucket).

Route53 has several DNS routing policies:

• Simple routing policy – Use for a single resource that performs a given function for your domain, for example, a web server that serves content for the example.com website.
• Failover routing policy – Use when you want to configure active-passive failover.
• Geolocation routing policy – Use when you want to route traffic based on the location of your users.
• Geoproximity routing policy – Use when you want to route traffic based on the location of your resources and, optionally, shift traffic from resources in one location to resources in another (assigning some bias, e.g. be biased towards assigning more traffic from Spain to the Dublin region and not the Frankfurt one)
• Latency routing policy – Use when you have resources in multiple AWS Regions and you want to route traffic to the region that provides the best latency.
• Multivalue answer routing policy – Use when you want Route 53 to respond to DNS queries with up to eight healthy records selected at random (very raw load balancing)
• Weighted routing policy – Use to route traffic to multiple resources in proportions that you specify.

Load Balancers

Region specific, internet facing (you have hostname) or internal load balancers. They are defined within a subnet. Charged hourly + per traffic. All LB support logging API calls via AWS Cloud Trail.

Classic Load Balancer

Older version still around, works at both L4, and L7 (no HTTP/2 support, or HTTPS websocket support). Need at least two AZ and you need at least 8 available IP addresses (for scaling purposes). Default health check 30s (range 5–300). Must have a security group, controlling traffic that can reach ELB front end listener. Subnet ACLs must also allow traffic to/from ELB both ways.

Supports session stickiness, via a special cookie.

Does not support Server Name Indication (SNI): you can upload a certificate via ACM but you can’t add multiple certificates for the same IP/port pair.

Elastic Load Balancer

L4 Load Balancer, supports TCP/TLS listeners for client requests and UDP traffic. Scales more (but requires a pre-warm for spikes of traffic), and has lower latencies than other ELBs.

Supports client source IP address preservation via “proxy protocol”.

Both active and passive (networks) health checks. The passive are how the targets react to connections.

Does not support sticky sessions.

Application Load Balancer

L7 Load Balancer, supports HTTP/2 and websockets over HTTPs. Can serve multiple application on separate fleets (target groups, characterised by a protocool and a port, they can be EC2 instances, ECS containers, IP addresses; not public internet routable ip addresses though).

When a condition is met traffic is forwarded to the associated target group.
Content based routing: based on the content request, route to a certain path or host.

ALB supports WAF. It also supports SNI.

Basic VPC concepts from the associate curriculum

Virtual Private Clouds are “virtualized data centers”, each AWS account comes with one “Default VPC” per region, set up in order to make as easy as possible to start using EC2 instances connected to the internet; such VPC contains:

• 65531 IP addresses ( /16 CIDR block, minus 5 reserved addresses)
• one subnet per AZ (subnets can’t span multiple AZ)
• one Internet Gateway (more on this below)
• one route table mapping locally all the traffic within the VPC IP block and to the internet gateway all the other traffic
• one Network ACL allowing all the traffic (both inbound and outbound) associated to all the subnets (you can associate 1 ACL to many subnets, but one subnet to a single ACL)
• one Security Group allowing all traffic outbound and only inbound traffic coming from within the SG itself

The “default VPC” recipe is different from the one used to create additional new VPCs. New VPCs contain:

• a CIDR block of IP addresses (between /16 and /28). This will be the primary block and you won’t be able to modify it later. You can still add other blocks though.
• one route table mapping locally all the traffic within the VPC IP block (so no automatic mapping to an Internet Gateway, NAT Gateway or any other mean to reach the public internet — there aren’t any for this VPC)
• one new Network ACL allowing all the traffic (inbound and outbound), associated to no subnets — there aren’t any for this VPC
• one Security Group allowing all traffic outbound and only inbound traffic coming from within the Security Group itself

Security Groups and Network Access Control Lists

NACLs area stateless component that filters the traffic in and out from a subnet.

Security Groups are stateful componenst that filter traffic in and out from an EC2 instance or an Elastic Network Interface (virtual network interface you can attach/detach to EC2 instances).

Stateful vs stateless
In this context stateless means that the component doesn’t keep track of requests coming in and out.

For example if we allow traffic in for port 80, we’ll also have to open outbound traffic to the ephemeral ports range (1025–65536, pretty much every port except the well known 1–1024 range), because that’s where the HTTP client will expect the return traffic from a web server.

As opposed to this, a stateful Security Group can achieve the same result by only enabling traffic inbound at port 80: it will recognize that an outbound request to an ephemeral port corresponds to the return traffic from the previous request and it will let it through without the need of explicit rules.

Communicating with the internet

There are several ways to have a VPC communicating with the internet.

Internet Gateway is a fully managed (hence scalable, available, etc.) resource which you can use to access the internet in both direction from your VPC. It doesn’t sit in a specific subnet.

The NAT Gateway is another fully managed service, sitting within a public subnet (hence in a single AZ, if you want redundancy you need to have everything duplicated in a different AZ) which allows outbound traffic from a private subnet to reach the internet (or other AWS services), but prevents the internet from initiating a connection with the subnet.

A NAT instance is an EC2 instance running a special AMI which does what the NAT gateway does but it’s not managed. Not recommended.

Egress Only Internet Gateway is similar to the Internet Gateway but supports egress only and IPV6 only.

Routing

Each routing table must contain a route for the local CIDR block. The most specific route for an IP address is applied.

Border Gateway Protocol (BGP), propagates information about the network to allow for dynamic routing, used by Direct Connect and optionally by VPN. Each endpoint has an identifier called ASN (Autonomous System Number). Local routers have weights for outbound traffic, higher weight will be the preferred path.

How to place your EC2 instances within a VPC

Three type of placement groups of EC2 instances:

• Clustered: Physically close, co-located in the same rack for lwer network latencies. You should allocate all the capacity beforehand, there might not be space in the rack to add capacity later. Single AZ only. If the rack goes down, so does the whole placement group.
• Spread: higher availability as they sit on different hardware, can also span multiple AZ (max 7 instances per group per AZ).
• Partition: hybrid solution, instances in the placement group are spread into partitions across racks.

Flowlogs

Traffic logs in and out of a VPC. Some traffic is excluded (e.g. Amazon DNS, VPC Router, Amazon Time Sync Service, DHCP traffic, etc.)

Network to VPC Connectivity

DYI VPN

You can buy a VPN appliance or install aVPN software on an EC2 instance with access to the internet, this will give the max flexibility and control but unlike the the AWS managed VPN option you will come with updates, availability, etc. already sorted out.

AWS VPN

In order to define an AWS managed VPN you need to define three resources:

• a Customer Gateway (i.e. physical device or software which act as endpoint on the customer side of the VPN)
• a Virtual Private Gateway (i.e. endpoint of the VPN in your AWS VPC)
• a Site-To-Site VPN connection between the Customer Gateway and Virtual Private Gateway (uses the IPSec VPN protocols to manage auth and encryption of IP packets)

Bonus: you can use AWS VPN CloudHub to make your Virtual Private Gateway act as a hub to manage different customer gateways to communicate each other using an AWS VPN.

Direct Connect

Dedicated network over private lines straight into AWS backbone, not over the internet, from a “AWS Direct Connect Location” (e.g. third party data center). Provides more predictable network performances and potential cost reductions (data transfer to AWS is free — not from AWS though).

If you have particular security requirements you can use a VPN on top of a Direct Connection connection.

VPC to VPC Connectivity

VPC Peering

VPC Peering lets you connect two VPC networks together., possibly belonging to different AWS account. Transitive peering does not work, you need to explicitly define the links.

AWS Transit

Hub to manages traffic between different VPCs in a hub & spoke model. n VPCs talking each other require (n-1)! VPC peerings (as transitive peering does not work), this is not scalable. With AWS Transit you define only one connection (to the Transit Gateway hub), making easier to scale the network.

PrivateLink

Private network between VPCs, AWS service or on promise servers. Prevents traffic from passing through the outside internet, can save money.

VPC endpoint

A VPC endpoint enables you to privately connect your VPC to supported AWS services and VPC endpoint services powered by AWS PrivateLink without requiring an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection.

Interface Endpoints are ENI with special DNS entries to redirect traffic and avoid it going out to the internet. You can secure them with Security Groups.

Gateway Endpoints (fewer services supported e.g. S3, DDB) use prefix lists in the route table to redirect the traffic, again preventing it from going out in the internet. You can restrict access through custom policies (similar to IAM policies).

SaaS Storage strategies

This whitepaper is all about three models to design the storage of a multi-tenant SaaS application on AWS, and how the models apply to RDS, DynamoDB and Redshift .

Silo model

Every tenant has its own completely independent database.

Good for:
- Isolation
- Tenant level tuning
- Tenant level availability (smaller blast radius)
- Eventual compliance requirements

Bad for:
- increased costs
- increased complexity of pretty much every aspect (provisioning, deployments, maintenance)

Bridge/Hybrid model

One shared database instance/account, but each tenant has different “sub-databases” or schemas.

Pool model

Shared database, shared schema.

Good for:
- reduced costs
- easier to maintain, to deploy, to manage

Bad for:
- bigger blast radius
- requires good application level authorization policies (can’t rely on IAM, at least for some services)
- potential issues with uneven usage of resources between tenants (hot key problems*)
- scalability issues at some level (RDS databases have size limits)

*There is a nice suggestion using an additional DDB lookup table, keeping track of shards and their usage, and use it to “balance” out shards when deciding where to allocate a new key.

Elasticache whitepaper

I have found this whitepaper more interesting, these points stood out to me

Memcached

• Very simple but extremely scalable and fast, supporting multiple threads to take advantage of powerful multi-core machines (unlike Redis).
• Complexity is built in the client, implementing a consistent hashing algorithm (i.e. key hashes are mapped to the closest integer representing a node, much more efficient in scaling than the simplistic hash % n servers algorithm, which requires old_n_server/new_n_server key re-mappings)

Redis

• Fancier features than Memcached: lists, sets, sorted sets, pub/sub, persistency,
• Less scalable than Memcached: runs mostly on a single thread and scales only vertically (you can scale horizontally by creating multiple clusters and sharding in the client, but you’ll lose the fancier features, and at that point you’re probably better off just using Memcached, unless if you specifically need persistence).
• Can scale separately for reads (with replicas, also in different AZs) and writes (vertical scaling of the primary node, single AZ). Recommended deployments: multi-AZ with auto-failover.
• Data loss might occur in replication if you have rapidly changing data (i.e. go for another database offering if it’s a critical requirement)
• Examples of advanced Redis feature’s usage: leaderboard with sorted sets, chat messaging with ephemeral pub/sub, rapid queues
• Redis supports backup/restore of snapshots from S3, backups are suggested with a minimum of 7 days retention

Caches in general

• Beware of very frequently used cache keys, a lot of requests might hit the database at once when they cache items expire or get removed (thundering herd). Pre-warm caches if possible, use jitter on cache items TTL.
• Monitor CPU usage (EngineCPUUtilization for Redis, which tracks the usage of the core used by the engine, other cores are under-utilized), cache misses, evictions (too high means I’m running out of space).

There are a lot of different storage services on AWS, sometimes overlapping in features (i.e. here is more than one “correct way” to do something using different AWS services).

There is a good whitepaper on all the storage options available in 2016: it’s comprehensive but unfortunately outdated. I recommend reading it for the trade-offs tables, but you definitely need to integrate it with some other source: I don’t have any good recommendation on this. I can only suggest to read the AWS official documentation if you spot a knowledge gap on a specific service.

I have extensively used in the past EBS and S3, so I wasn’t really interested in studying these (but admittedly I’m not super up-to-date with the latest updates to them products, so I’ll play around with them in the next few weeks). I spent most of my time trying to fit into my head the AWS Storage Gateway offering.

AWS Storage Gateway

I have never used it, but I can see how it’s valuable if you still have on-premise servers. It’s deployed as a virtual machine that you can install in your on-premise hardware (or in an EC2 instance) and offers some kind of storage virtualisations.

It comes in different flavours:

• File gateway: store and retrieve files to S3 using a NFS (or SMB) mount in your on-premise servers. You know where files are mapped in S3 and you can access them individually from any other AWS service.
• Cached Volume for Volume Gateway: storage volume mountable as iSCSI device on-premise, but actually backed on the cloud (as EBS snapshot on S3, I believe). There is a local cache of the most frequently accessed files stored on-premise. Requires less local storage than the next option, but also provides optimal latency for the most frequently accessed files only.
• Stored Volume for Volume Gateway: storage volume mountable as iSCSI device on-premise, entirely stored locally on-premise but asynchronously backed on the cloud in the form of EBS snapshots to S3. Classic use case: offsite backup. Encryption happens in the Volume Gateway, so if you run it on-premise unencrypted data never leave the on-premise site.
• Tape gateway: virtual taped storage on the cloud, supposedly a drop-in replacement to on-premise tape archive, stored in AWS cheapest S3 class cold storages (Glacier or Deep Archive).

My knowledge on the Gateway Storage is super-shallow, but I’ll look for gaps in my knowledge and fill them later, at the stage when I will be ready for full exam simulations.

Trade-offs and evaluating solutions

~60% of the exam will be on “Designing new solution” and “Improving existing solutions” according to the official Exam Guide, so I would imagine the most impactful thing to study are the trade-offs and evaluation guidelines to pick a storage service given some requirement.

Requires extreme performances and does not strictly require data persistency?

EC2 Local Instance Store: it’s on the same physical machine as the EC2 instance where it is mounted (i.e. 0 network latency), see the appendix below for surprising performance comparison.

Data is not persisted if you stop an EC2 machine (even if you restart it later). If you only reboot the instance data doesn’t get lost.

Do you need to share the data between different EC2 machines?

Use EFS (you can mount an EFS volume on multiple EC2 instances) or S3.

Do you want to pay per usage or per provisioned space?

With EC2 instance store and EBS you pay per provisioned space. EFS, S3 and the Storage Gateway solution let you pay per usage.

How much do you care high durability and availability?

S3 boasts 11 nines durability, so it’s a clear winner on that aspect. EBS is the least “high available” of the lot, because it’s only replicated within an AZ, unlike EFS and S3 (which also has an One-Zone infrequent access option if you have specific cost requirements). EBS has a snapshot feature if you need to increase durability or have specific disaster recovery requirements. This is helpful for e.g. database file systems.

Does your data change very frequently?

EBS and EFS are better solution than S3 in this case, with the latter being eventually consistent and also offering higher latencies.

Do you need to access your data rarely?

The best option here is S3, picking an appropriate storage classes for your latency, availability and cost requirements. See Appendix 2 for all the storage classes currently available (I remember this being the subject on multiple questions on my Associate level certification, well worth a review).

Do you need to serve static data or web content?

The whitepaper suggests EFS or S3 in this case, but I would go with S3 here: setting a public static website with S3 it’s as easy as it gets. Providing access to static data it’s easy too, with granular permissions enforceable.

If you want improvements in performance IN and OUT from AWS data centers, you can use S3 Transfer Acceleration to make the clients communicate with the AWS edge location, which is usually closer to the client location (there are less than a couple dozens of AWS regions and hundreds of edge locations).

Similarly in concept, you can also use CloudFront which is also being served by the edge location data centers, but also caching the content in the process, so subsequent requests won’t just reach S3 faster, they’ll be entirely served by the CloudFront cache.

Do you need to do some analytics on your stored data?

S3 shines again here: AWS Athena and Redshift Spectrum let you run queries against your structured data stored on S3.

Athena was a bit immature when I played with it 3–4 years ago, but I’m sure it improved in the meantime. Redshift Spectrum proposition sounds interesting: keep the most frequently used data in your Redshift cluster, and the rest of the data on S3, combine them in your queries. I haven’t used it yet, but I can see how it makes sense as a proposition, considering I remember Redshift being very slow to scale up/down in my experience at Amazon.

Do you need to make sure files can’t be deleted (e.g. for compliance reasons)?

S3 is the right choice, you can define Vault Lock Policies for S3 Glacier which prevents any deletion, or add some multi factor authentication to the delete operation on S3 standard class buckets.

Conclusion

The “storage on AWS” subject is super broad and I feel like I haven’t even scratched the surface. I want to keep pushing with my study and get the certification as some kind of “result” out of this COVID quarantine period, so I’ll try to not get discouraged by this 😊.

Appendix: Impressive IOPS capabilities by I/O optimized instances:

Appendix 2 — S3 storage classes options:

From the official documentation:

Pricing for storage decreases going from left to right, while (mostly) API calls cost increases in the opposite direction (you’re supposed to do less frequent calls to Glacier or Infrequent Access Storage).

I currently possess the AWS Certified Developer Associate and AWS Certified Architect Associate certifications expiring in July 2020, which should provide me of two 50% off voucher for new certification exams of my choice. I’m not going to try and renew both of them, I would like to try something different and go for the AWS Certified Architect Professional certification, which will also renew my old AWS Certified Architect Associate for 3 more years (so I should still be able to say I own 2 certifications 😬).

The exam difficulty

The exam is supposed to be harder than the ones I took three years ago, but how much? In order to understand how much study is required for this, I had a quick peek at the sample questions on the AWS website.

My first impression is “this looks really hard”! According to this Japanese official file the correct answer is D. I am not sure why this is not C, in a production system I wouldn’t store logs on an ephemeral disk and I thought that one of the many “S3 server-side encryption” options was able to satisfy the “logs only decrypted by employee of your company” requirement, but it looks like that’s not the case.

I’ll have to study quite a bit before approaching the exam. I think I was in a slightly better position 3 years ago, at my current company (Monzo) we just deploy and operate open source software, such as Cassandra, Kafka, HashiCorp Vault, etc.) as opposed to use AWS out-of-the-box managed services as I did in my previous 2 roles, especially at Amazon Prime Video where paying an extra to AWS for managed services was unsurprisingly the recommended guideline.

Where to study?
For the past two AWS certification I remember buying acloud.guru courses. The associates exams didn’t go super deep and the courses were good enough that covered most of the ground. I decided to trust them once again, heading to their website.

Apparently their business model changed, now they don’t charge you X$ to buy a course, but they since then moved to everyone’s favourite (ironic) monthly subscription model. If your goal is a certification I’m not sure a yearly subscription makes any sense: I’m going to go for a monthly subscription for a single month and then cancel it, but to begin with, I have started with the free weekly trial.

A Cloud Guru Professional Architect Certification preparation course

I’m not super-impressed when I see the amount of videos available for this course, it’s ~12.5 hours, pretty much the same as the corresponding Associate course which has to cover less topics.

I liked the introduction: the instructor is careful to not over-promise, pointing out the course is just a guideline but you’ll need to read AWS whitepapers, watch AWS re:Invent videos, and have expertise on the field in order to ace the exam.

They also introduce the Bloom Taxonomy concept to point out the different expectations between the Associate level and the Professional level certifications, which I found really a really interesting concept more broadly, not limited to the AWS Certifications world.

A short introduction to how the exam works, and a reminder (I’ve already noticed in the sample questions), that exam questions are long, and have more than one answer technically correct, but only one also “contextually” correct according to all the specifics of the question.

The course then proceeds to storages, but I guess that’s the topic for the next blog post..

Reading post-mortem documents is highly educative (please read this tremendously good blog post about it), in my experience because a recent post-mortem is up-to-date and very detailed by definition, two characteristics unfortunately rare to find in internal documentation.

I casually stumbled upon the concept of pre-mortem, developed by research psychologist Gary Klein and published on Harvard Business Review in 2007.

This reminded me of Amazon’s approach of working backwards (explained here by AWS CTO Werner Wogel), but with a different starting point: a spectacular failure rather than a great success. I wonder how these two approaches compare in terms of effectiveness.

And I wonder even more if pre-mortem as a tool can be used in our industry by development teams to help product owners and management to understand the depth and implications of technical debt.