Introduction

The email sending function is a common feature found in almost every website. For developers, AWS SES (AWS Simple Email Service) is a popular choice, particularly when the website is hosted on the AWS cloud. Although AWS SES doesn’t directly support email activity tracking, it does offer the possibility to create custom tracking features by integrating with other AWS services. In this post, I will explain how to build an application that tracks email sending activities using SNS HTTPs endpoint.

SES Setup

Configure AWS SES

Implementing email communication with AWS SES is a straightforward task, thanks to the support of various programming languages provided by AWS developer tools. To send an email using SES, we can directly call the AWS SDK. Here’s an example using NodeJS:

const client = new SESClient(config);
AWS.config.loadFromPath('./credentials.json'); // for testing purpose
const command = new SendEmailCommand({
    Destination: {
      ToAddresses: ['testemail@gmail.com'],
    },
    Message: {
      Subject: { Data: 'Test Subject' },
      Body: {
        Text: {
          Charset: 'UTF-8',
          Data: 'This is a test email',
        }
      }
    },
    Source: 'noreply@mydomain.com',
  });
  
  const response = await client.send(command);

For production ready, I am using the NodeMailer package instead, which provides a slightly better interface:

const ses = new aws.SES({});
const message = {
    from: 'noreply@mydomain.com',
    to: 'testemail@gmail.com',
    subject: 'Test Subject',
    text: 'body in text',
    html: '<p>HTML content</p>',
  };
  const transport = nodemailer.createTransport({
    SES: {ses, aws},
  });
  await transport.sendMail(data);

Rather than directly calling the SES Client, NodeMailer redirects the API call to SES through the Transport object, making it easy to switch to different transportation options. This flexibility is particularly helpful for developers when running the code in development and test environments. We’ll explore this further in the Testing section later on.

SES Dashboard

In the code example, the app only needs to pass the necessary data, such as fromEmail, toEmail, subject, body (and attachments), to the function. AWS SES will then handle the entire process of sending out the email.

A pressing question arises: how can we determine whether the email has been successfully delivered to the user’s inbox or rejected by the user’s email server?

The SES Dashboard provides valuable insights into account usage, such as the total number of sent emails, historical sending data, rejected rate, bouncing rate, and complaint rate. However, these metrics do not provide specific details about each individual email activity. A complete email system requires this information to troubleshoot any potential bounces or rejections caused by external factors. Thankfully, the SES Notifications feature, integrated with AWS Simple Notification Service and AWS Simple Queue Service, helps bridge this gap and provides the necessary information for a complete email system.

Complete email system

Below is a draft design that outlines the components of an email system.

In our context, after observing how SES integration is set up in various applications, we have separated the Email sending system as a standalone application. This approach allows other applications to avoid managing their individual AWS configurations for email communication. Instead, they can simply make an API call to the email system, which will handle the entire lifecycle of the emails.

Here are some more details about each component:

• API Gateway: allows client applications, SNS Subscriber (HTTP/HTTPS) and dashboard interacts with the Email System.
• Email Sending API: provides API to trigger email, it handles the Emails Service that responses for communicate with the AWS SES, and the Recipients Service that creates/updates the recipients’s profiles.
• Activity API: provides public endpoints for SNS subscriber to push the notification about the email’s activities. The notification will be proceed later by a background job processor to the domain data.
• Dashboard API: provides APIs for the Frontend Dashboard page to connect and render the email activities data.

AWS SES notification config

In each SES Domain Identity, there is a Notifications config, where it allows to setup the SNS topic for each feedback type. Here is an example:

Before setting up the SES notification, it’s essential to create the required SNS Topics. In the example provided, separate topics are used for each notification type (bounce, complaint, and delivery), but a single topic can suffice for simpler use cases.

The subsequent step involves defining the subscription for the SNS Topic, for which we’ll be using the HTTPs (or HTTP) protocol in this blog post. AWS SNS supports various other subscription types, including SQS (event queue), email, and SMS subscriptions.

To create the HTTPs subscription, select the subscription type and provide the endpoint URL. Please note that the endpoint URL should be publicly accessible on the internet, as private IP addresses won’t work.

In the example screenshot, the website supports creating an email hook, wherein each URL is assigned a unique identifier to distinguish email notifications from different identities. However, for simpler use cases, a general URL such as https://mydomain.local/ses_notifications will suffice.

After creating the subscription, the SNS will make a request call with subscription_confirm message to the subscription URL. The application needs to store the request payload, which contains the Subscribe URL and the Subscription Token to confirm the subscription.

To confirm the subscription, we can open the SubscribeURL on the browser or implement an automated logic that calls the ConfirmSubscriptionCommand as follow:

const input = { 
    TopicArn: "STRING_VALUE",
    Token: "STRING_VALUE",
    AuthenticateOnUnsubscribe: "STRING_VALUE",
  };
const command = new ConfirmSubscriptionCommand(input);
const response = await client.send(command);

To avoid the message be formatted, SNS subscription allows to send the raw message instead of the formatted message. However, there could be some issue when reading the request body with the raw payload if the option “Enable raw message delivery” was enabled. For example, if the project uses ExpressJS (NodeJS), this raw-body package can be used to parse the request body, the default access to request.body doesn’t return any data.

Once the subscription is confirmed, whenever an email is sent using this SES domain identity, an HTTP request is triggered from SNS to the preconfigured endpoint. The email system captures the request payload and processes it to determine whether the email is delivered successfully.

The request payload may look like this:

{
   "notificationType": "Complaint",
   "complaint": {
      "complainedRecipients": [
         {
            "emailAddress": "richard@example.com"
         }
      ],
      "complaintFeedbackType": "abuse",
      "arrivalDate": "2016-01-27T14:59:38.237Z",
      "timestamp": "2016-01-27T14:59:38.237Z",
      "feedbackId": "000001378603177f-18c07c78-fa81-4a58-9dd1-fedc3cb8f49a-000000"
   },
   "mail": { // Content is trimmed }
}

In this example, the notificationType is “Complaint” which indicates that the email reached the recipient’s mailbox but was marked as spam.

Core components’ implementation

I hope that I have covered the main concepts and functionalities of the Email System which integrates with AWS SES. In this section, we will explore the implementation of the core components in this email system.

Email Sending Controller

The controller exposes an API that allows other applications to make requests to send out emails. gRPC is used as a default choice for microservice protocol, the gRPC proto file looks like this:

message TriggerEmailAttribute {
  string fieldName = 1;
  string value = 2;
}
message EmailAttachment {
  string name = 1;
  string remoteUrl = 2;
  optional string cid = 3;
}
message TriggerEmailRequest {
  string subject = 1;
  repeated string toEmails = 2;
  string mailBody = 3;
  repeated EmailAttachment attachments = 4;
  repeated TriggerEmailAttribute attributes = 5;
}
service EmailsServiceController {
  rpc triggerEmail(TriggerEmailRequest) returns (TriggerEmailResponse) {};
}

The triggerEmail call receives an input of type TriggerEmailRequest and returns a response with type TriggerEmailResponse. The TriggerEmailRequest definition includes 5 fields used to make the call to the AWS SES API. In the implementation, the controller interacts with two services: the EmailsService, responsible for handling the email submission to AWS SES, and the RecipientsService, which manages the creation of recipients based on the provided information. The notifications received from the SNS Subscription are later mapped to these recipients.

By providing the toEmails parameter in the request object, the service allows for specifying multiple email addresses to receive the same email in a single request.

Notification endpoints management

As the Email sending service is used by multiple applications, each application has a distinct notification endpoint for AWS SNS to publish the email notifications.

To demonstrate how to support separate endpoints for each application, the entity relationship diagram is as follows:

When an application makes an API call to the Email Sending Service, it includes the token in the meta fields to authorize the request, enabling the service to determine the email activity based on the hook id. Subsequently, when SNS subscription publishes the notification request to a hook, the system knows how to create the corresponding activity on the application’s usage.

Background Job Processor

Background jobs are crucial in modern software development for two main reasons. First, they handle long tasks without slowing down the main application, so users don’t have to wait. Second, they automatically try again if something goes wrong, minimize number of tasks gets lost or left undone.

The flow for the email sending and the hook notification processing looks like below:

• Send Email flow

• Process notification flow

Let’s clarify the difference between another task scheduler system, where the processor acts as the master node distributing tasks to worker nodes, and the current case. Here, the processor is a consumer class with methods that process jobs in the queue or listen to events on the queue. For more information on tasks queue documentation, you can refer to the bull documentation.

Data extraction from the notification can be done in the background as real-time updates are not required for analytic data. Parsing all the data from the payload may not be necessary, but we can store the notification for future data back-filling.

Testing strategy

Email testing can be expensive if actual emails need to be triggered. Fortunately, there are third-party services like MailCatcher or MailTrap that provide email sandbox solutions, rendering sent emails in a web interface. This eliminates the need to trigger actual emails to test purposes.

While third-party email sandbox services are often sufficient, I opted for a slightly improved solution with MailDev. Setting up MailDev for this project was straightforward, and I’ll document the configuration process here.

The maildev can be run in Docker, here is the Docker Composer service definition:

maildev:
    image: maildev/maildev
    expose:
      - "1025"
    ports:
      - "51080:1080"

Nothing is fancy here. The maildev service exposes the port 1025 for the Email App to connect, and allows the traffic to port 1080 (51080 from host) to access the dashboard.

Since we use NodeMailer package, it creates a Transport object to connect to AWS SES API. To make the app send emails to MailDev in Dev environment, we need to create another transport object and hook it during the initialization.

The CreateTransportService takes in a driver parameter, which indicates the type of transport, the code looks like below:

enum MailDriver {
  ses,
  maildev,
}

export class DummyTransportation extends StreamTransport {}
export class CreateTransportService {
  constructor(private readonly options: TransportServiceOptions) {}
  create() {
    switch (MailDriver[this.options.driver]) {
      case MailDriver.ses: {
        const ses = new aws.SES(this.options.sesConfig);
        return createTransport({
          SES: { ses: ses, aws: aws },
        }).transporter;
      }
      case MailDriver.maildev: {
        return createTransport({
          ...this.options.mailDev,
        }).transporter;
      }
      default: {
        return new DummyTransportation({
          streamTransport: true,
        });
      }
    }
  }
}

The mailDev options object is simple with port and host, which can be passed into the app through a configuration file:

maildev:
    port: 1025
    host: maildev

To speed up the test running time, we can defined the DummyTransportation to create mock Transportation when running test.

Capacity calculation

Understanding the traffic and data is important to determine the actual value of the platform we have developed. The application primarily focuses on handling email sending requests from various applications within the organization. While it also offers a dashboard to monitor email activities, its main functionality revolves around efficiently managing the email communication.

To demonstrate, I will give an example how to calculate the capacity of the email system.

• Sending rate: assume the app sends out 1 email per second, or 86k emails per day, 2.6M per month.
• For each email, the system stores the activities data and the recipients data. The recipient data could be like this:

// TypeORM syntax   
@PrimaryGeneratedColumn('increment')   
readonly id: number;   

@Column({ unique: true })   
email: string;   

@Column('json')   
attributes: RecipientAttribute[];   
readonly createdAt: Date;   

// other columns

• Each row recipient takes ~20bytes, we can assume if there are 10M of users in all managed applications, the total size of the recipients table is around 200MB, which is very affordable.
• In the another side, the activities table could be a little bigger as the data is keep increasing. Let’s assume each activity takes around 30bytes, the size of the activities table would be:
• in a month: 2.6M * 30bytes = 78MB
• in a year: 1GB
• in 3 year: 3GB

While storage is not a concern, the sequential scanning of the activities table may result in slow performance. On a 1GB/sec (SSD disk) speed, generating complex dashboard data queries could take up to 3 seconds without optimization. Fortunately, the use of database indexes can greatly improve query times.

Overall, the application remains well-suited for the next three years, considering its current sending rate and user base.

Conclusion

In conclusion, I hope you found this blog post insightful and informative. We’ve discussed some of the key aspects of the centralized Email service, though it’s important to acknowledge that there are many other considerations, such as integrating it with different app clients, data tracking, and storage usage, among others. As someone who has been involved in this project from its inception, I feel fortunate to have had the opportunity to design the application, define its features, and develop a comprehensive plan. Implementing a robust and efficient Email system can greatly enhance communication and efficiency. However, it’s important to acknowledge the challenges that arise with the complexity of email configuration.

I’m thrilled to share this approach with you, and I hope it inspires you to enhance your organization’s platform. Thank you for taking the time to read this!

Objective

The local domain are helpful to setup the connection between the Frontend and Backend on the development machine. It is also a common solution when the actual address and port is difficult to remember. In this post, I will explain how to set up the local domain and config the SSL certificate on the MacOS laptop.

Software required

In order to config the local domain name and issue the SSL cert, we’ll need these tools installed:

• Openssl
• MacOS Keychains app
• Nginx: for configuring the proxy
• Admin account
• vi: to edit the host and create the configuration files

The nginx and vi are just my preference, you can use VSCode, emacs for editor and Apache, HA proxy for the proxy.

Create the local domain

Setup the domain is the most simple task, you can open the hosts file with the admin privilege.

sudo vi /etc/hosts

Add a line for the new domain and map it to 127.0.0.1

127.0.0.1 mydomain.local

If you run the server inside a virtual network, for instance by running the proxy in the Virtual box or Docker, then use the IP of the virtual network interface instead.

After that, you may flush the DNS cache by using this command:

sudo dscacheutil -flushcache

Create the Root Certificate Authority (Root CA)

The SSL in the normal website are issued by a trusted Certificate Authority. However, for the local development, we don’t need to purchase a SSL (and we don’t own the domain), hence we have to setup the Root CA as well. During the handshake step, the browser checks whether the cert is valid by asking the Root CA about the cert, and if the Root CA doesn’t confirm with the browser, the browser marks the site as unsafe.

To generate the Root CA, we can use the openssl tool. The steps are:

1. Generate the root CA key
2. Generate the root CA cert

Personally, I suggest to create a directory to store all the files that are created during this process together, I usually create one in /tmp/ssl for this purpose. All the files we create after this are in the same directory.

To start with, we have to create the RootCA configuration file.

# fileName: config.conf
[req]
prompt = no
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
C = CA
ST = Kitchener
L = ON
OU = Org
CN = mydomain.local
emailAddress = dev@mydomain.local

[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = mydomain.local
DNS.2 = api.mydomain.local
DNS.2 = www.mydomain.local

Please modify the content of the file to match with your information and the local domain. The file references and examples can be found from IBM OpenSSL configuraton and OpenSSL document

After creating these 2 config files, we can generate the rootCA key and cert with these 2 commands

openssl genrsa -out rootCA.key 2048
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem -config config.conf

The values of each parameters can be altered by different values, such as the number of days can be change to 365, or the algorithm can be set to sha512. For the details of each parameters, checkout the OpenSSL manual.

Issue the Self-signed SSL cert

Create the ext file in the same directory with the content like below

# filename: mydomain.local.ext

authoritykeyIdentifier = keyid,issuer
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = mydomain.local
DNS.2 = api.mydomain.local
DNS.2 = www.mydomain.local

After this, we can generate the cert key file and the cert signing request file

openssl genrsa -out mydomain.local.key 2048
openssl req -new -key mydomain.local.key -out mydomain.local.csr -config config.conf

The last step is to issue the cert

openssl x509 \
        -in mydomain.local.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial \
        -out mydomain.local.crt -days 1024 -sha256 -extfile mydomain.local.ext

After that, you should have the the cert (and other files) in the temporary directory. We don’t need the config file as well as the ext file. A common practice I usually take is to move these files to a more permanent place, such as /etc/ssl, so the files aren’t gone after reboot. To import the cert to the Apple Keychains, we can either use the Keychains UI or by running this command

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /etc/ssl/rootCA.pem

Noted that we don’t have to import the cert for the local domain name.

Config the proxy

We can configure the SSL cert for the website from the proxy level instead of the app server. However, if you prefer to configure the cert from the app, please skip this section.

I prefer to use Nginx for the proxy because it is easy to setup and the document is excellent.

Here is how the Nginx config look like

#filename: /opt/homebrew/etc/nginx/servers/mydomain.local.conf
upstream frontend {
   server localhost:3000;
   keepalive 100;
}

upstream backend {
   server localhost:8888;
   keepalive 100;
}

server {
  listen 443 ssl;
  server_name mydomain.local www.mydomain.local;

  ssl_certificate     /etc/ssl/mydomain.local.crt;
  ssl_certificate_key /etc/ssl/mydomain.local.key;

  # locations config

  location / {
    proxy_http_version 1.1;
    proxy_pass http://frontend/;
  }

  # allows large request body for upload location

  location ~ /uploads/(file|image) {
    proxy_http_version 1.1;
    client_max_body_size 50M;
    proxy_pass http://frontend/;
  }
}

server {
  listen 443 ssl;
  server_name api.mydomain.local

  ssl_certificate     /etc/ssl/mydomain.local.crt;
  ssl_certificate_key /etc/ssl/mydomain.local.key;

  # location config
  # the backend use Apache Web server, the headers are required

  location / {    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Access-Control-Allow-Origin *;
    proxy_http_version 1.1;
    proxy_pass http://backend/;
  }
}

With Nginx, we can setup complicated routing rule to for the website traffic and configure the SSL without hassle.

A pro tips: If nginx is installed through brew, we can use the brew services to make it starting automatically when the laptop is boosted (brew services start nginx).

Recap

This short post is a summary for the steps to setup the local domain on the local machine. I hope you find it useful for your works. And if you have set up the same stuffs in the Linux machine, please share the steps with me.

The internet technology changes the communication completely. It takes milliseconds to deliver your message to someone, the receiver can see your message in less that a second. We receive a hundred (plus) of notifications everyday from different communication apps, news, social network feeds. To read and answer them (and reply) isn’t an easy job. This is my idea to survive in the notification flood.

9.30 AM, I arrive to the office. I go to my desk and turn on my laptop, checking Slack, Github Pull Requests and notifications, checking emails, writing some updates on Slack (just like post a FB status everyday).

If nothing’s special, I start a day with my daily todo list, prioritize important things to be done. It take a bit of time but the returned value is significant. If you are doubt about it, read the book Eat that frog.

Then I review my colleague Pull Requests. It usually takes a bit longer than 30 mins.

I take a break and start working on my tasks.

PING!

A beautiful TCP package is delivered to my machine. With a natural reaction, I acknowledge by saying Hi/Morning — means “Yes I received the package”. I don’t want the man-to-man connection takes too long to response, it makes people feel my communication system is unreliable.

The handshake protocol is successfully setup, the information exchanging process start happening in a TCP socket.

Some more TCP packages are broadcast in different channels.

Thanks for modern web browsers to show a little banner on my top right screen, I get notified when someone is talking somewhere. Just simply clicking on the banner, I can follow them to the place that conversation is happening, which saves me time to figure out.

I also use smart phone, it’s even more helpful. All the notifications are there, I can scan through each notification and knowing where are they from. Joining each conversation, making sure I don’t miss anything, provide useful information to my colleagues can’t be easier.

What is the time for me to focus on my own stuffs? Seem like not much.

Different with the TCP protocol, which is indefinitely waiting until the reply message arrive. A retry mechanism could be triggered if there is no response after a certain time (how long is depended on the patience configuration), another TCP package could arrived using the same system, sometimes can be in different systems, or back to the old slow Dial-up connection. And sometimes it can switch to UDP protocol.

I used a bit of metaphor in talking of TCP/UDP, I hope you don’t feel annoyed.

Having Slack open all day is truly death by a thousand paper cuts for your productivity. – I am Devloper

That hurts.

I don’t hate distraction, another way to say is I don’t hate people annoying me when I am working. I hate myself when I can’t finish my works as promised. I have to kill distraction to boost up my productivity.

I start re-reading the book “The Productive programmer” of Neal Ford. A good book for every developer who wants to improve productivity. It’s important to read many books as we can, and not necessary to remember everything from the book. When we face a problem, we know there is a book about it, we can always go back to the book. Now I am taught this lesson well.

Personally, I prefer the concept of the book rather than the specific technique and example. I take those concepts from the book and apply them to my current working flow.

One of the techniques mentioned in the book is the “Quiet time”. It’s the time span that we can focus on our works.

For my current job, I can find 2 quiet time spans from 11AM — 12PM (1 hour), and 1.30PM — 3PM (1.5 hour) and sometimes 4.30PM — 6PM (1.5 hours). Between these time spans, I can do different works that are less affects to the key metrics.

The technique works best when people can maximize their performance during the short period. I can drink a soft-drink or coffee to maximize my focus. Based on some researches, after drinking coffee, we may feel a bit sleepy during first 30 mins, which exactly the time needed to get into a flow

3 questions I have to answers before start my quiet time

1. What will be done?
2. How important it is?
3. Is it affected to the key metrics?

What can I get done in an hour and a half is a good question. Human attention doesn’t last long. If I can’t find out what I can get done during a quiet time, I am better to not enter the quiet time, but go back and breakdown the task, clarify the requirement and the scope again. It’s better to figure out them before working on the story.

The second question is how important my work is. Sometimes I can spend the whole day to fix a CSS. This is important to me, but not for the team. However, it doesn’t apply for all times. For an example, a CSS is written by you, and it doesn’t written well, and your colleagues start copying this CSS to many places. Now it’s bad, it start bothering you whole day and night. If I were in this situation (yes, I am), I wish I could write it properly from the beginning.

Don’t misunderstand the importance of the task with the value of the task to the key metrics. Writting document is valueable, but it’s hard to measure in term of key metrics. The key metrics are designed to show the quality of the product, that can be understood by anyone include people without technical background.

In an Agile company, burn down rate, velocity, rejection rate, … are the important metrics. Some people suggest to use customer satisfaction and revenue as the development team metrics as well. The customer satisfaction is count by the error rate, the reliability of the system, the time it takes to release new feature… However, those metrics depends on vary costs and it’s more about the overall company effectiveness.

I don’t disagree with the idea of using those key metrics, and encourage each person thinking them before putting a story to icebox, bottom of backlog and start working on it. Yeah, we have 3 times to think about those business metrics.

And of course, by talking about produce-wide metrics, I don’t mean to sacrify the development metrics. It’s about the maintenance cost, testing cost, training cost…

Any fool can write code that a computer can understand. Good programmers write code that humans can understand. — Martin Fowler, CodeWisdom

It tells us something. Software development is man to man work. The product of it brings values to both team and product. We may deliver a lot of feature with a ton of technical debts and neglect the maintainance cost.

Unless people tell us to neglect the cost of maintenance and we know it isn’t a lie, then we can go ahead to create as much of technical debt, hacky stuffs as we “have to”.

Most of the time customer doesn’t know what they want, we can’t use them to excuse for create a legacy system. Remember the story of scorpion and a frog. Be aware of. Don’t neglect our team metrics, our values on building maintainable software. And learn to say no when we have to.

To summarize, productivity is a result of technique, planning, and execution. Quiet time technique create quality time spans to produce the most important works in limited of time. Planning ahead helps us evaluate what is really important, it allows us to focus on an individual task at a time. The final step is execution, it’s the most important step to produce result. There is always enough time to raise the standard of code quality, not cutting the corner if we have discipline of following the first two steps to create the quiet time and planning, prioritize the works.

The opening argument isn’t the main focus of this post. There is nothing wrong with the internet, nothing is wrong with the communication system. If we know to take values from, it’ll help us improve our productivity. It’s more important to learn how to use those key metrics to measure our performances, and the distractions from the communication system is no longer a problem.

The first part is more than true. I Code for work, code for fun, code for free, and code cause I don’t know what else to do.

Imagine how short a day is. For developer, it’s extra short when we have a bug.

Some days we just want to get through. I meet some guys, who always count how many days til TGIF.

We may neither finish anything, nor learn anything in a bad day. These bad days won’t go away if we don’t know how to turn them off. They just come back again again everyday, until we switch the situation, or hope someone can help us turn if off.

I also have some bad days like this. But I think one day I live is one day I lost. And I know I can’t live too long (may be 5 of 5 years ahead), I want to enjoy every single day, learn something from it, even a bad day, instead of just give it away.

One way to treat your bad day is…

Reading the book “Don’t sweat the small stuff”. There are many good advices in this book, and I would to recommend you to check it out.

These techniques in the book are easy to use, but require us to practice again. Otherwise when we loose your motivation, or can’t control the emotion, we’ll forgot or ignore them, we only follow our emotion and end up worst.

Another thing that I learn is a bad day happens, I remind myself it’s opportunity for me to apply some techniques from the book, and evaluate myself if I get angry easily (and say to myself “Failed to apply. Not enough skill yet”). Be an audience of your life, not a actor/actress.

Last but not least. Be prepared.

My “adopted” father has a talk on preparation, included prepare for a good day. He said: Life is designed to not give us what we want, not give us what we need, but life is designed to give us what we deserved.

Here is his talk on this. Recommend you to watch, just cost you 5 mins

That’s all I want to share today. Hope you enjoy it. I wish we all learn something from this post and from the next day.

P.S. The original link to my post.

Hi everyone, it’s such a long time (10 months omg) I haven’t written anything.

I built this blog myself on Rails, and host it in Digital Ocean. It was a cool project, I am both developer and user, I understand how each feature values to users.

However, from time to time, it becomes difficult to maintain the project. Many times I want to write a post, and I found some bugs. Then I spent my 2 hours to fix bugs instead of doing what I tend to do. It is demotivated me a lot whenever I want to write.

I decide to stop maintain my blog and rebuild it using Github page and focus on content more than code. I spent a lot of times to develop my blog. It was tough to make this decision.

Now I have a new blog using Github page, Jekyll plugin. There is no excuse not writing blog post because there is a bug. There’re still bugs, but the cost of maintenance the site is much less than.

I think writing content is more valuable than maintain it. It was hard to make this decision, I love the project, and I was not sure I enjoy writing content in markdown syntax. Also this time I haven’t know how to config the website in Jekyll. After moving, I know that I was too conservative. I should have more changes in life and make it as a habit. And that’s all I want to share in this post.

We want to change our life, fix our bad habit to live better. We want to get out of our comfort zone. We want to have something we don’t have. So, we must do something that we haven’t done before.

Knowing this lession is important. However, there is one more thing to know, we don’t like changing. We know the benefit of changing, we don’t change.

For me, I am afraid of changing.

I love my comfortzone. As a developer, I love my programming language I am using, I love the framework I am using. I love the testing framework I am using. Ah, Just kidding, It is not I love or dont’ love, I am confident with them, I know I can get thing done by these things. I rarely use different languages, different frameworks, cause I am sure it supports well for my job.

I am not good at swimming. I learn swimming because I want to swim with my girlfriend. Sometimes I want to swim, but go to swimming complex and swim with many professional swimmers is quite stressful.

I am good a badminton. I may win or lost, but I am pretty sure I play well, cause I play badminton for almost 5–6 years. If choosing between badminton and swimming, definitely I go with badminton, safe choice.

Another example, I always call Ice Latte when I go to the coffee. I tried many coffee shops just to figure out the best Ice latte After having many ice lattes, it becomes a safe choice for me whenever I go to coffee shop. Just want to see a new coffee shop’s ice latte is better.

The last example may not be relevant, just because I am having an ice latte now, so I put it in my post ^^. I use these examples to prove a point, if we’re confident about one thing, we prefer to go with it instead of other options. It isn’t bad, it just limit us to try new things, and it make a habit not to try.

It is so hard to try new thing, even though I have methods, I have motivation, I have some reasons. I did hard to found them, but I still don’t take action, because I don’t have the habit of doing new thing. I feel not enough to take action yet.

My solution for this is challenge myself more often. I read this blog https://themission.co/a-list-of-30-day-challenges-you-can-start-today-532c28bc2585

At first, it sounds a bit weird. However, it’s exactly what I don’t have, that’s why I feel nervous to change, to challenge myself to do something I haven’t done before. I may not do anythings in this list, I am not able to do it for 30 days also, I just want to make it happened once

• Cycles 12kms nonstop
• Write a non-tech blog
• Talk to my ex-girlfriend
• Give free advice to some strangers
• Order something different with Ice Latte
• Give a talk about something that isn’t related to my experience or knowledge

And here is my list to do next

• Contribute to a Golang lib
• Pushup 100 times nonstop
• Writing a blog about Psychology or personal development
• Go some places alone
• More things to come

The point here is: feel more naturally to challenge yourself, to build a habit to try new thing, even it’s small thing. Then there is more opportunity for you, you can do anything with full confident and belief to create a better life.

P.S. This is my non-tech blog post.

Starting working with docker waas really amazing. I pulled several images and ran it in my machine, 100% of which worked correctly. I found this the way to go for my career: stick to Docker. I used a quote from the Polymer community: there will be a “container” for it. Definitely true, almost all services I use to run my website are available in Docker hub or Github. I had no problem to use it, a bright future opens in front of me. Until…

I am using mac to develop software, and docker only supports native Linux kernel, which means I need to install Docker in a Virtual machine. Fortunately I don’t need to do it manually because of docker-machine, they allow me to use a tiny linux with built-in docker, also to map the `home` directory in my mac to /Users inside the machine, which is perfect.

At first, it took a while to understand how to use the command, the concept of image vs. container, how to use docker-compose to automatically run all the containers, link them together and get the website running; but this wasn’t the hardest part. I got frustrated of the volume part for a long time. In this post, I don’t try to explain the volume part in docker. I just want to share with you how I understand about it and my mistakes when using volume in an incorrect way.

This is how I run the rails container:

docker run -d --rm -v `pwd`:/myapp -p3000:3000 -t my_web_image bundle exec rails s -b 0.0.0.0

Yeah, it works. Of course this command doesn’t link with database container, just a stand alone one. Believe it or not, it took me a year to write that :cry:. This command uses -v flag to set the volume from the current directory to /myapp directory in the container, so that I don’t need to restart the container every time I make a change. Of course, in the Dockerfile, I need to run my webapp in /myapp directory, and of course, don’t use ADD or COPY.

That’s the basic volume concept in docker. I think after you see my command, you can write it yourself for many containers. I myself use this command to run many containers for a long tims, and I forgot about the mapping /Users stuff in my VM in order to make docker work.

Several weeks ago, I had to use the mysql image to run a new project. Turned out this one didn’t work with the command I was using for a…a decade. It just said the data wasn’t accessible. So I checked the permission in my mac. Maybe something’s wrong, so I gave the directory full permission 766, but still didn’t work. After many days of continuously checking many documents, including github issues, blogs, and docker hub, I found out this was the problem with the running user in the docker images.

After I understood the problem, I thought it’d be easy fix. I was so naive to try to change the file ownership in my mac and hoped :)). But you know what? In my macbook, this directory is owned by `kevin:kevin`, but in docker-machine, it’s own by `docker:staff`, which is different. Even when I tried to create a new user and group in my mac, then assigned the new user+group to this folder, it still kept the same `docker:staff` in the docker-machine. You may ask why I didn’t change the permission inside docker-machine instead, the problem would be every time you stop the machine, things are gone. So I’m smart enough to not try to do it. Actually I did try to add some config file for another docker images in my docker-machine before, and it was just gone after restart. I know it doesn’t work, and if it works, it isn’t the most efficient way to do.

I got trouble with it in a couple of weeks. I almost gave up with docker, and even said to myself that docker was only for linux users man. Can’t use Docker in OSX, or I need to install Ubuntu and use Vagrant with docker, which makes thing harder than it should be. May be it’s just a nice toy, but not usable.

When I almost gave up on docker, I found an unofficial docker for mysql for macosx. It just works perfectly, so I was very curious to understand how the creator fixed it. Then I spent a few hours to understand his dockerfile, and realized that maybe this is the only way to do, and no one mentioned it in the docker document yet, so I want to share it with you how he does it, and what we can do to overcome this situation. Don’t be like me last time haha.

Remind of the situation:

In my mac

./project_x kevin kevin

In docker

./myapp docker(1000) staff(50)

And in the mysql image, they use user mysql (id 999) to run.

So what he did in his custom image is:

ENV BOOT2DOCKER_ID 1000

   ENV BOOT2DOCKER_GID 50
   RUN useradd -r mysql -u ${BOOT2DOCKER_ID} && \
    usermod -G staff mysql


   RUN groupmod -g $(($BOOT2DOCKER_GID + 10000)) $(getent group $BOOT2DOCKER_GID | cut -d: -f1)
RUN groupmod -g ${BOOT2DOCKER_GID} staff

What does it mean?

First, he created user mysql with id 1000, also added user to group staff, the same with the user id with the ownership in docker-machine. Also he added the user to group staff, which made the ownership in docker-machine and the docker-image consistent. Right now, the ownerships in both environment look like.

docker-machine

/myapp docker(1000) staff(50)

docker-image

/myapp mysql(1000) staff(xxx)

The user’s name does not matter, just the user id. The final thing we need to change is the group id, which is occurred by the last 2 commands. This guy tried to go extra step by finding the current group holding id 50, then changed this group first before assign id 50 for staff. I wonder why he didn’t do it for the user id as well :).

That’s it, I think you got the idea now. If you understand this code, you can do the same with other images, although it may not be easy. Also, since there is a different approach with ` — volumes-from`, you can also try to fix the issue inside the data volume. I can’t tell you it would work or not, so make sure your hands are dirty.

I hope my post will help you have a better look on docker, and give it a try for your current/next project. Keep calm and containerize it.

The Dockerfile I use in my blog is https://github.com/dgraziotin/osx-docker-mysql.

UPDATE: Docker has release the beta version Mac. Hopefully it can help to eliminate the problem I have. Check out this post for more information

https://medium.com/@nzoschke/docker-for-mac-beta-review-b91692289eb5#.pyn86ic7x

Docker mac/windows link: https://beta.docker.com/