freeStyleJob('build-some-app-master-image') {
  description 'Build some-app docker images from master'
  scm {
      git {
          remote {
              url(https://medium.com/'git@github.com:acme/some_app.git')
              credentials('xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx')
              branch('master')
          }
          triggers {
              githubPush()
          }
      }
  }
  steps {
    shell(readFileFromWorkspace('build_app_master_image.sh'))
  }
}

#!/bin/bash
set -e

export AWS_DEFAULT_REGION="us-west-1"
export DOCKER_IMAGE="some-app"
export ECR_REPO="111111111111.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"

docker build -t ${ECR_REPO}/${DOCKER_IMAGE}:latest .
docker push ${ECR_REPO}/${DOCKER_IMAGE}:latest

If the need arises to add an additional build step to this job, for example if there is a requirement to enable slack notifications for each build job failure, each DSL file for each application needs to be modified to support this change

A better way to create and maintain these files/jobs

The solution to this is to centrally maintain a single template which accepts parameters and creates multiple jobs according to the parameters this template receives.

Given below is the same build job that was shown above written in such a way to receive parameters so that it generates multiple build jobs.

An array is defined which contains the parameters for each application. This array is fed to a loop which traverses the array through a JSON parser.

When you run this DSL script on your Jenkins server it produces the build jobs that you defined in the array.

import groovy.json.JsonSlurper

def slurper = new groovy.json.JsonSlurper()

def application_list = slurper.parseText('[{"name":"app_one","organisation":"acme","app_repo":"app_one","branch":master","aws_account":123456789123","aws_region":ap-southeast-1","image_name":app_one"}, {"name":"app_two","organisation":"acme","app_repo":"app_two","branch":master","aws_account":123456789123","aws_region":ap-southeast-1","image_name":app_two"}]')

application_list.eachWithIndex { application, index ->
  freeStyleJob("build-master-image-${application.name}")
  {
    description 'Build ${application.name} docker image from master'
    scm {
      git {
        remote {          
          url(https://medium.com/'git@github.com:${application.organisation}/${application.app_repo}.git')
          credentials('xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx')
          branch('${application.branch}')
        }
        triggers {
          githubPush()
        }
      }
    }
    environmentVariables {
      envs(
        AWSACCOUNTID: "${application.aws_account}"
        AWSDEFAULTREGION: "${application.aws_region}"
        DOCKERIMAGE: "${application.image_name}"
      )
    }
    steps {
      shell(readFileFromWorkspace('build_app_master_image.sh'))
    }
  }
}

#!/bin/bash
set -e

export DOCKER_IMAGE="${DOCKERIMAGE}"
export ECR_REPO="${AWSACCOUNTID}.dkr.ecr.${AWSDEFAULTREGION}.amazonaws.com"

docker build -t ${ECR_REPO}/${DOCKER_IMAGE}:latest .
docker push ${ECR_REPO}/${DOCKER_IMAGE}:latest

This approach gives you a base to build increasingly complex templates by using the various capabilities and characteristics of the groovy language. To make it even more interesting this template can be configured to get the application configuration list from a remote api. In an environment which follows DevOps practices a pipeline can be created to automatically trigger the jobdsl build job when the application api is updated.

import groovy.json.JsonSlurper

def application_list = new JsonSlurper().parseText( new URL("https://application_list.acme.com/list").getText() )

application_list.eachWithIndex { application, index ->
 freeStyleJob("build-master-image-${application.name}")
 {
   description 'Build ${application.name} docker image from master'
   scm {
       git {
           remote {  
               url(https://medium.com/'git@github.com:${application.organisation}/${application.app_repo}.git')
               credentials('xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx')
               branch('${application.branch}')
           }
           triggers {
               githubPush()
           }
       }
   }
   environmentVariables {
     envs(
       AWSACCOUNTID: "${application.aws_account}"
       AWSDEFAULTREGION: "${application.aws_region}"
       DOCKERIMAGE: "${application.image_name}"
     )
   }
   steps {
     shell(readFileFromWorkspace('build_app_master_image.sh'))
   }
 }
}

In conclusion, as a company’s infrastructure grows, managing an increasing number of similar Jenkins jobs can become a significant challenge. To streamline this process and maintain efficiency, the solution proposed involves creating and maintaining a single template capable of generating multiple jobs based on parameters. This approach not only simplifies job management but also opens the door to automation, allowing job creation to align seamlessly with changes in application configurations through a remote API, in line with DevOps practices.

Centrally Managing Jenkins Jobs With a Jenkins Template was originally published in Property Finder Engineering and Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Based on my industry knowledge and experience, I have created this mind map that covers a wide range of essential testing tools and platforms. It includes IDEs, Performance and Functional Testing Tools, Reporting Solutions, Container Platforms, Database Testing Tools, and more.

👉 No more hassle searching for testing tools! Each component comes with a clickable link, making navigation a breeze. Share this testing mindmap with your fellow testers 🤝

If you want to navigate to that tool page, just click on the click from that box as shown below

Link —https://xmind.app/m/YfjDUt

Thanks for reading. Happy learning — AB!

Thanks, Naveen AutomationLabs for the guidance.

Anshita Bhasin
QA Chapter Lead

GitHub | LinkedIn | Twitter | Youtube

Software Testing Ecosystem was originally published in Property Finder Engineering and Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Are you tired of spending countless hours debugging your tests? Look no further than Cypress version 13’s new feature: Test Replay. This game-changing addition, in conjunction with the Cypress Dashboard, will take your testing workflow to the next level.

In this blog post, we’ll explore what Test Replay is and how the Cypress Dashboard enhances your testing workflow.

What is Cypress Dashboard?

The Cypress Dashboard is a cloud platform that provides a comprehensive view of your test runs across different environments and browsers.

Below are some of the main features of Cypress Dashboard:

• It provides a centralized platform to view detailed test results, including screenshots and videos, to quickly identify and analyze test failures.
• Using Cypress Dashboard, you can run your test cases in parallel across multiple virtual machines in CI and save a lot of time.
• It helps you gain a comprehensive view of your test suite’s health and performance over time.
• It helps you find the flaky test in your code based on their frequency, and severity.
• It provides you with a quick summary of errors occurring in your test. You can easily read the exact error message and understand how many tests are impacted by each error.

By default, It comes with 3 users and 500 runs for the free version but if you want to leverage more test runs, you can always switch to its paid version ( Here’s the pricing, you can choose based on the requirements)

You can configure it for multiple projects and record your test cases on Cypress cloud (Dashboard) to view the screenshots, and videos and even configure to check the flaky test cases.

It shows, which git branch you are using for running test cases in case you are running on CI/CD, just like shown below:

In addition, you also have the ability to configure it to monitor the slowest test runs in your project, as well as Flake Tests, Test Runs, and durations.

Simply navigate to the configuration option in the left side menu and customize it according to your specific requirements.

It also offers exceptional integration capabilities. You can easily integrate it with various tools like Github, Gitlab, BitBucket, etc.

Now that you have a clear understanding of what the Cypress Dashboard is, let’s move on to the next section and explore what Test Replay is.

Introducing Test Replay

Test Replay is a new game-changing feature introduced in Cypress version 13 that empowers you to conquer flaky and failing tests with ease during Continuous Integration (CI) runs.

• It allows you to re-run specific test cases or scenarios, providing valuable insights into what went wrong during the test execution on CI/CD.
• With Test Replay, you can pinpoint issues, debug failures, and make your test suite more robust and reliable.
• This feature simplifies the debugging process, making it faster and more efficient, saving you time and effort.

The best part is that there is no additional cost to use this feature if you are already using Cypress Dashboard. Simply update your Cypress version to version 13.0 in your package.json file, and you can start using Test Replay right away.

Test Replay is compatible with all three types of testing, including E2E, API, and Component Testing. To access Test Replay, navigate to your project and then proceed to your test results.

Simply hover over any failed test cases, and you will notice the appearance of the Test Replay button, as illustrated below:

Upon clicking on the Test Replay button, a new window will appear, displaying the test execution along with the following details:

Operating System on which it is running

Browser Name

Test Case Name

Network Calls

Console

Inspect element

You have the flexibility to adjust the speed of test execution, allowing for better debugging and viewing capabilities.

Benefits of Test Replay:

Test Replay is a powerful tool that enables automation engineers and developers to identify and debug issues in their test suite quickly and efficiently. By re-running specific test cases or scenarios, Test Replay provides valuable insights into what went wrong during the test execution on CI/CD.

Here are some of the key benefits of using Test Replay:

Collaboration:

Test Replay makes it easier for team members to collaborate on debugging efforts. Since the test execution and results are captured, team members can easily share and discuss the details with each other.

Faster Debugging:

Test Replay reduces the time required for debugging by providing a detailed view of the test execution. This helps automation engineers/developers identify the root cause of the failure faster and take corrective action.

Improved Test Coverage:

By identifying and fixing flaky tests, Test Replay helps improve overall test coverage. This ensures that your application is thoroughly tested and reduces the risk of bugs in production.

Customizable:

Test Replay is customizable, allowing you to select which parts of the test execution you want to replay. This is particularly useful when dealing with long test cases or when you only need to replay a specific portion of the test.

If you would like to learn more about how to utilize this feature, you can watch the below video for a more comprehensive understanding.

Conclusion:

In conclusion, Cypress Dashboard is a powerful tool that can help you streamline your testing workflow and with the new TestReplay feature, you can easily debug failing tests. So, whether you’re running E2E, API, or Component Testing, Cypress Dashboard has got you covered

Thanks for reading. Happy Learning! AB

Ref — https://docs.cypress.io/guides/cloud/debugging/test-replay

Anshita Bhasin
Sr. Automation Engineer

GitHub | LinkedIn | Twitter | Youtube

Cypress 13: Test Replay and Cypress Dashboard was originally published in Property Finder Engineering and Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In terms of infrastructure, we have discovered the most cost-effective solution available on the market — Fargate Spot, operating on an ECS cluster. This solution not only offers affordability but also seamlessly integrates with our existing environment.

By leveraging Fargate for Vault, we get simplified deployment, cost efficiency, scalability, enhanced security, high availability, and seamless AWS integration, creating a robust platform for secure secret management.

To learn more about the installation and configuration of Vault on our infrastructure using ECS, please feel free to contact me through LinkedIn. I would be happy to provide you with further details and insights.

Setup Vault login via OIDC

Before a client can interact with Vault, it must authenticate against an auth method to acquire a token. This token has policies attached so that the behavior of the client can be governed.

Auth methods perform authentication to verify the user or machine-supplied information. Some of the supported auth methods are targeted towards users while others are targeted toward machines or apps.

Onelogin Setup

1. Log in to your OneLogin admin portal.
2. Go to “Apps” and click “Add App” to add a new application.
3. Search for and select the “OpenID Connect (OIDC)” app.

4. Provide a name for the app and click “Save”. You will be able to filter the app by name you provided.

5. In the “Configuration” tab, enter the following information:

Login URL:
https://vault.example.com/ui/vault/auth?with=oidc

Redirect URI:
http://localhost:8250/oidc/callback,
https://vault.example.com/ui/vault/auth/oidc/oidc/callback

Post Logout Redirect URIs
https://vault.example.com/ui/vault/auth?with=oidc

Note: vault.example.com is just used as reference name, please replace it with your vault domain.

6. Under SSO Tab, Get the Client ID, Client Secret and Issuer URL

Due to security concerns, we have greyed out the client ID and issuer URL.

7. Next step is to create a new role, Go to OneLogin admin portal, Users > Roles, and click New Role.

8. Give new role a name (in our case we gave Dev Vault) and select the applications it should include, then click Save.

9. Assigning Roles to Users: From Users > Roles, open the role and go to Users. Enter one or more users into the Search for a user field and click Check. The user appears with a button Add to Role, this way you can add users manually.

Alternatively you can create mappings as well (for example: department ~ Engineering) to add all users under engineering department automatically.

10. Now Go to Applications > Vault Application that we created > Access > Roles, add new role we created above.

11. To verify, Go to Users tab and check users.

Vault Setup

1. Login to Vault via Root Token
2. Click on Access → Enable New Method → Select OIDC

3. OIDC Configurations: Check SSO Page of Onelogin and add Onelogin Issuer URL as OIDC discovery URL, Onelogin Client Key as OIDC Client Key, Onelogin Secret Key as OIDC Secret Key, Role as reader.

4. Create Vault Policies: Create the policy file named reader.hcl. This policy grants read and list capabilities to all paths in the KV secrets engine enabled at the secret path.

tee reader.hcl <<EOF
# Read permission on the k/v secrets
path "/secret/*" {
capabilities = ["read", "list"]
}
EOF

Select the Policies tab in Vault and then select Create ACL policy. Toggle Upload file sliding switch, and click Choose a file to select your reader.hcl file you authored, Click Create Policy to complete.

5. Create Vault Role: Click the Vault CLI shell icon (>_) to open a command shell, Within the terminal, create the readerrole.

export ONELOGIN_CLIENT_ID="CLIENT_ID_HERE"

vault write auth/oidc/role/reader \
      bound_audiences="$ONELOGIN_CLIENT_ID" \
      allowed_redirect_uris="https://vault.example.com/ui/vault/auth/oidc/oidc/callback" \
      allowed_redirect_uris="http://localhost:8250/oidc/callback" \
      user_claim="sub" \
      token_policies="reader"

Login with OIDC via UI

1. Sign out of the Vault UI if you are already logged in.
2. Select OIDC from the Method drop-down list.
3. In the Role, you can enter reader or since the default role is already set to “reader”, you can leave it blank.

4. Click Sign in with OIDC Proivder.

5. Now, you should be logged in.

Login with OIDC via CLI

1. Log in with the oidc method as role of a reader.

export VAULT_ADDR="https://vault.example.com"
vault login -method=oidc role="reader"

2. When prompted, accept and authorize the Vault access to Onelogin App.

3. Back to terminal, you should see log in successful.

Conclusion

The seamless integration of Vault with OneLogin via OIDC offers a comprehensive solution to the challenges of securing sensitive information. By replacing traditional login methods with OIDC, you enhance your organization’s security posture and mitigate the risk of unauthorized access to critical secrets. By following the steps outlined in this article, you can establish a robust platform for secure secret management, contributing to the overall security and success of your organisation.

Help and reference

• OIDC Auth Method
• Onelogin Openid Connect

Safeguard Your Entry: Seamlessly Secure Vault Login with OIDC was originally published in Property Finder Engineering and Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Problem

At Property Finder, we have been using AWS EKS unmanaged clusters. Recently, our engineering teams have noticed an alarming issue with unreasonably slow S3 file uploads from their applications.

To investigate further, they conducted a comparison by running the same application on their local computers. Surprisingly, they observed that uploading a 10KB file to S3 took approximately 5 seconds on our EKS cluster. However, on their local machines, the identical upload process took approximately 1 second. Consistent testing revealed an average slowness factor of 5 when compared to the local computer uploads.

This noticeable increase in latency raised significant concerns, urging us to uncover the underlying root cause to identify possible solutions.

Investigation

To identify the issue and locate the bottleneck, we came up with the following plan:

1. Create a S3 bucket in the region where our EKS cluster is running.

2. Deploy a debug pod in the EKS cluster using the latest AWS CLI docker image.

# Debug pod configurations
apiVersion: apps/v1
kind: Deployment
metadata:
name: debug
spec:
selector:
  matchLabels:
    app: debug
replicas: 1
template:
  metadata:
    labels:
      app: debug
  spec:
    containers:
    - name: debug
      image: amazon/aws-cli
      command: ["sleep", "infinity"]

3. Locate the EKS node where the debug pod is running.

kubectl get pods -l=app=debug -o wide

4. Create a 10KB file and upload it to the S3 bucket using AWS CLI from both the debug pod and EKS node where the debug pod is deployed. Measure the time taken for each upload.

# AWS CLI upload command
time aws s3 --debug cp 10kb.png s3://yasithab-debug-bucket/10kb.png

Identifying the Issue

Based on the test conducted, it was discovered that uploading the 10KB file from the EKS node takes around 1 second, which is similar to uploads from a local computer. However, uploading the same file from the pod running on the same EKS node takes approximately 5 seconds. This indicates that there is something in between causing this issue, but the exact cause is still unknown.

# AWS CLI upload results from the EKS node where the debug pod is running
real   0m1.460s
user   0m0.908s
sys    0m0.085s

# AWS CLI upload results from the debug pod
real   0m5.612s
user   0m0.956s
sys    0m0.073s

As we have AWS enterprise support, we reached out to them with the test results and debug logs. They recommended provisioning a new EKS cluster and conducting the same test to isolate any other dependencies. However, we decided against this approach as we consider it unnecessary to set up a new cluster solely for debugging this issue. Therefore, we are currently on our own in resolving this matter, unfortunately.

With no clear direction at this stage, a possible course of action is to compare the AWS CLI debug logs for both node and pod uploads. This can be achieved by utilizing a file comparison tool. Initially, the focus was on investigating potential network delays by comparing the duration of the upload process.

The upload process began when the CLI displayed the following lines.

MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
MainThread - botocore.utils - DEBUG - Registering S3 region redirector handler

The upload process ended when the CLI displayed the following lines.

upload: ./10kb.png to s3://yasithab-debug-bucket/10kb.png
Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

Then, the duration from the beginning of the upload to the end of the upload event was calculated.

• In the node upload process, the start time was recorded as 2023–07–13 11:03:14,047, and the end time was recorded as 2023–07–13 11:03:14,161. Hence, the upload process took 114 milliseconds.
• In the pod upload process, the start time was recorded as 2023–07–13 10:57:54,740, and the end time was recorded as 2023–07–13 10:57:54,828. Hence, the upload process took 88 milliseconds.

The almost identical upload times indicate that this issue is not caused by network latency.

Upon reviewing the logs further, it was observed that in the pod upload debug log, most of the time was spent on establishing the connection to the Instance Metadata Service (IMDS) endpoint. This process took around 1 second and was attempted three times, totaling approximately 4 seconds. On the other hand, the initialization of the connection to the IMDS endpoint from the EKS node where the debug pod is deployed, only took 190 milliseconds.

# From the EKS node where the debug pod is deployed - 190 milliseconds
2023-07-13 11:03:13,857 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-07-13 11:03:14,047 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)

# From the debug pod - 4 seconds and 259 milliseconds
2023-07-13 10:57:50,481 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-07-13 10:57:54,740 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)

When comparing the total time taken from the debug pod (5 seconds and 612 milliseconds) with the time taken for establishing the connection to the Instance Metadata Service (IMDS) endpoint from the debug pod (4 seconds and 259 milliseconds), a difference of 1 second and 353 milliseconds is observed. Remarkably, this difference closely matches the time taken from the EKS node upload results.

Hooray! The Instance Metadata Service (IMDS) has been identified as the bottleneck. However, the next step is to understand why this is happening and determine how to resolve it.

What is IMDS and What it does here?

The AWS Instance Metadata Service (IMDS) is a service that allows EC2 instances to access information about themselves. This information includes details like the instance ID, instance type, network configurations, identity credentials, security groups, and more. Applications running on EC2 instances can use IMDS to retrieve dynamic information about the instance. IMDS has two versions available.

• Instance Metadata Service Version 1 (IMDSv1) — a request/response method.
• Instance Metadata Service Version 2 (IMDSv2) — a session-oriented method.

By default, you have the option to use either IMDSv1 or IMDSv2, or both. The instance metadata service differentiates between IMDSv1 and IMDSv2 requests based on the presence of specific PUT or GET headers, which are exclusive to IMDSv2. If you configure IMDSv2 to be used in the MetadataOptions, IMDSv1 will no longer function.

To retrieve the MetadataOptions for an EKS node, use the following command. Replace <EKS_NODE_INSTANCE_ID> with the specific instance ID of the EKS node you wish to retrieve the MetadataOptions for.

aws ec2 describe-instances --instance-ids <EKS_NODE_INSTANCE_ID> --query "Reservations[].Instances[].MetadataOptions"

The above command will give an output similar to the following:

[
    {
        "State": "applied",
        "HttpTokens": "optional",
        "HttpPutResponseHopLimit": 1,
        "HttpEndpoint": "enabled",
        "HttpProtocolIpv6": "disabled",
        "InstanceMetadataTags": "disabled"
    }
]

In the output, it shows that the HttpTokens attribute is set to optional.

When the HttpTokens is set to optional, it sets the use of IMDSv2 to optional. You can retrieve instance metadata with or without a session token. Without a token, IMDSv1 role credentials are returned, and with a token, IMDSv2 role credentials are returned.

On the other hand, when HttpTokens is set to required, IMDSv2 becomes mandatory. You must include a session token in your instance metadata retrieval requests. In this case, only IMDSv2 credentials are available; IMDSv1 credentials cannot be accessed.

IMDSv2 is an improved version of instance metadata access that adds an extra layer of security against unauthorized access. To use IMDSv2, a PUT request is required to establish a session with the instance metadata service and obtain a token. By default, the response to PUT requests has a response hop limit of 1 at the IP protocol level.

However, this limit is not suitable for containerized applications on Kubernetes that operate in a separate network namespace from the instance. EKS managed node groups that are newly launched or updated will have a metadata token response hop limit of 2. However, in the case of self-managed nodes like ours, the default hop limit is set to 1. Therefore, setting the HttpPutResponseHopLimit to 2 in our EKS is mandatory for retrieving InstanceMetadata over IMDSv2.

How does the upload work with HttpPutResponseHopLimit set to 1 despite the slow upload speed?

When the HttpTokens attribute is set to optional, the AWS CLI/SDK automatically use IMDSv2 calls by default. If the initial IMDSv2 call does not receive a response, the CLI/SDK retries the call and, if still unsuccessful, fallback to using IMDSv1. This fallback process can introduce delays, especially in a container environment.

In a container environment, if the hop limit is set to 1, the IMDSv2 response does not return because accessing the container is considered an additional network hop. To avoid the fallback to IMDSv1 and the resulting delay, it is recommended to set the hop limit to 2 in a container environment.

Resolve the issue

According to AWS documentation, containerized applications running on EKS require the EKS node HttpPutResponseHopLimit in MetadataOptions to be set to 2. To configure this, you can use the following command, replacing <EKS_NODE_INSTANCE_ID> with the instance ID of the specific EKS node you want to retrieve MetadataOptions for.

aws ec2 modify-instance-metadata-options --instance-id <EKS_NODE_INSTANCE_ID> --http-endpoint enabled --http-put-response-hop-limit 1

In many cases, EKS nodes are part of AWS auto scaling groups, making manual modification of instance metadata difficult. To configure Instance Metadata Service (IMDS) for an auto scaling group, you need to modify its associated launch configurations.

To set the HttpPutResponseHopLimit for all instances associated with the launch configuration, go to the Additional configuration section under Advanced details of the template. Set the Metadata response hop limit to 2. If no value is specified, the default is set to 1.

When using Terraform to provision your infrastructure, you can utilize the aws_launch_template resource to modify instance metadata in the following manner.

resource "aws_launch_template" "foo" {
  name = "foo"

  metadata_options {
    http_endpoint               = "enabled"
    http_tokens                 = "optional"
    instance_metadata_tags      = "disabled"
    http_put_response_hop_limit = 2
  }
}

After implementing the HttpPutResponseHopLimit modifications, we conducted the same tests as in the investigation phase. The results of both the EKS node and debug pod uploads were nearly identical.

# AWS CLI upload results from the EKS node where the debug pod is running
real    0m1.310s
user    0m0.912s
sys     0m0.078s

# AWS CLI upload results from the debug pod
real    0m1.298s
user    0m0.931s
sys     0m0.082s

Conclusion

In conclusion, our investigation into the significantly slow S3 file uploads from applications running on AWS EKS clusters revealed that the Instance Metadata Service (IMDS) was the root cause of the issue. By comparing upload times from EKS nodes and pods, we identified a delay in establishing the connection to the IMDS endpoint from within the pods. This delay was not present when uploading directly from the EKS nodes.

To resolve this issue, it is necessary to set the HttpPutResponseHopLimit in the MetadataOptions to 2 for EKS nodes. This can be achieved by modifying the associated launch configurations for auto scaling groups or using the aws_launch_template resource in Terraform.

By setting up the HttpPutResponseHopLimit to 2, we can eliminate the latency and significantly improve the speed of file uploads to S3 from AWS EKS pods. This resolution will enhance the overall performance and efficiency of our applications on the EKS cluster.

References

1. Amazon EKS now supports EC2 Instance Metadata Service v2
2. Describe EC2 MetadataOptions
3. Retrieve instance metadata
4. Modify Instance Metadata
5. Issue with EC2 Instance Metadata running inside Container
6. Configure the instance metadata options for Auto Scaling Groups

Unveiling the Speed Mystery: Investigating Slow S3 Uploads from AWS EKS Pods was originally published in Property Finder Engineering and Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Detox — a peculiar name. A quick google search on Detox gives you this:

But that’s not what we are here to talk about.

Here, we talk about the Javascript End-to-End App Testing and Automation framework created and maintained by Wix. It can be built into the test application so that the test execution begins with the app launch.

In this article we’ll take a look at app automation with Detox, why it is a strong contender to traditional app automation frameworks like Appium and its pros and cons.

We’ll also have a brief look into the prerequisites needed and a sample test case.

Is this article relevant to you?

If you are a QA tester looking at options to automate your mobile app, or even an app developer looking to write automation scripts for the app you are building, this article is for you!

We have Appium, so why Detox?

Our search for an alternative app test automation framework began due to the flakiness of Appium like false failures and failure to locate page objects on time.

Appium is a black box testing tool with no access to the code it is testing because of which we need to add sleep statements. This makes it slow and flaky.

On the contrary, the Detox philosophy is “Gray Box Testing”, where it knows about the internals of the test product. This is what makes it a strong contender to Appium and the main reason we decided to explore it for the myCRM app automation.

What makes Gray box testing tools like Detox better than Appium?

• More stable scripts
  Gray box testing tools like Detox synchronises with the test app and can monitor the network requests, JS threads, timers and page loads. This eliminates the need for “sleep” statements and makes the test more stable.
• Faster scripts
  Gray box testing tools are 5–10 times faster than black box testing tools.
• Access to app code
  Detox can access the code and data of the app, eliminating the need for writing additional code to do so.

What are the highlights of Detox?

Sounds too good to be true, doesn’t it?

Detox is not without its limitations. Some of which are listed below:

• No real device testing support for IOS,
• Not a large online community for support which is so important,
• It does not support Web views and Hybrid,
• It’s capabilities are not as wide as Appium.

What are the prerequisites to run Detox?

1. Homebrew: Homebrew is an open source software package management system that simplifies the installation of software on the macOS.
2. Node.js 8.3.0 or above: Node.js is a JavaScript runtime environment Detox will run on.
3. Applesimutils: This is a collection of utils for the Apple simulator
4. Detox command line tools.

What does a Detox test look like?

This is a sample Login code snippet:

As seen above the syntax is easily readable. We can easily write synchronous tests using await/async statements that:

• Keep track of all network requests that are currently in-flight and wait until they complete,
• Keep track of pending animations and wait until they complete,
• Keep track of timers and wait until they expire or are cancelled,
• Keep track of the React Native operations.

Open the simulator and run the test script:

Summary

In summary, Detox is a handy tool for app test automation considering the speed and stability of the execution of tests. However, the lack of online support is currently the biggest detriment to its adoption.

We are still in the early stages of implementing it in our project and will have a better idea of its capabilities as we proceed.

You can read up further about Detox as follows: https://reactnativetesting.io/e2e/setup/

https://github.com/wix/Detox

LET’S DETOX was originally published in Property Finder Engineering and Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the most common problems you may encounter is misconfiguration. Through this blog post, I will explain basic concepts to better understand the cause of problems and provide you with a compilation of best practices in Go and MySQL interaction.

So I’ll share some advice about configuring connections to MySQL. We’ll go through the main points that you need to pay attention to when working with MySQL using Go language.

• We’ll cover the basics of client-server protocol MySQL, its basic structure and operation principle.
• Then we’ll move on to the Go part and will analyze the implementation of the connection pooling.
• We will go from configuring connections to making requests simultaneously looking at the driver code.

MySQL Client-server protocol

It is a half-duplex protocol, in other words, it is synchronous. It means you can either send a request or accept it.

After sending a command to the server the client should wait for a response before initiating another action . At the same time the message should be received entirely, the client cannot break it off abruptly, unless the connection is interrupted.

All data transmitted over the protocol is packed into packets. There are predefined packet types on the server and client sides and they can exchange these packets between each other.

Generally the client sends a request in one packet, while the server may respond with several packets. And even if the client needs only part of the packages, it still has to receive all of them.

In this protocol, TCP will be used as a transport that is used by the client and server to exchange these same packets in binary.

The MySQL protocol has two main phases:

Connection phase

Once TCP connection is established, the MySQL server sends a packet with a handshake to the client after which parameters are exchanged. For example, a client can request an SSL connection, and then send a response packet to the server for a handshake.
After a successful first phase the protocol enters the command phase mode.

Command phase

In this phase the client takes the initiative and sends command packets to the server. These include queries, prepared expressions, stored procedures, and replication commands. The server processes client requests and responds with a result.

We can look at the MySQL protocol. To do this the database has a special plugin that displays the trace. It is convenient to use for testing (not production solutions), so it is disabled by default. To enable it you need to create the database server using the WITH_TEST_TRACE_PLUGIN option.

You can watch the trace output in the terminal after connecting to the server. Here’s what it looks like.

Another way to look at the data being transmitted is to use the ngrep utility on the client:

sudo ngrep -x -q -d any port 3306

The output between the client and the server will look as follows.

Libraries for GO

The standard library contains only the main interface — database/sql for all sql or sql-like databases. The language maintainers left the implementation of protocols for specific databases to the third-party developers. That is why we will use its official go-mysql driver, which is a separate package itself.

The standard sql interface package uses a connection pooling for database connections. This is because MySQL protocol requests occur synchronously — each client can execute the next request only after it completes the previous one.

But are there any other options else?

One connection

The first thing that might come to mind is to use one global connection for the entire service. It is clear that as a result all new requests will wait until the connection is free. This will be a bottleneck and our service may hang for a long time.

Connection per each request

You can create a new connection on each request, closing it on completion.

While it solves the main issue of waiting for a connection, there are some other challenges related to its use.

First, we won’t be able to control the growth of connections to our database. The number of our connections will directly depend on the number of requests — i.e. how many connections the service will open will depend on how many requests will come. And it is not good for the database. In a normal mode, we may not have problems, but at times of peak loads you can reach the maximum possible number of connections from the database, and it will simply stop responding. There will be errors when connecting to the database both in our service and in all others that use it.

Second, each time we will be forced to go through the first phase of connecting to the MySQL server, in which TCP connection, handshake and authorization are established. Compared to the command phase, this one is more time consuming as we will spend extra time creating a new connection each time.

Thus, the connection pool solves all these problems to one degree or another.

Connection pooling

Pools help us control the total number of connections that a service can use to access the database. Thus we can secure both the database and minimize the number of passes of the first phase of the connection. Because as a rule in the pool we have connections prepared in the command phase, with the first phase of the connection already passed.

Pool initialization

Let’s start with initializing our pool and see what happens in the code directly from the documentation of the go-sql-driver package.

At the very beginning when importing the go-sql-driver the mysql protocol is registered.

Then when calling the Open method, we say that we want to open a pool that will use the protocol called mysql (first parameter) while connecting, that is implemented in the go-sql-driver package with the configurations passed using the second parameter. After that the methods for configuring our pool are called, which we will discuss at a later stage.

Pool view

This structure represents our connection, outlining various structure fields required for configurations. For example, the freeConn field, which is a slice of links to connections using our chosen protocol (in fact, this field represents the set of our free connections itself). Or the connRequests field which is used to request a new connection, etc.

Calling the Open method results in the creation of a new DB structure.

Creating an empty pool

Inside it the OpenDB method is called, in which the structure with the initial data is initialized.

As you can see from the code the pool is initialized empty. It means that at this stage we have not yet established any connection to the database. Therefore, if you have incorrect data in the configuration (for example an incorrect login or a password) when you call the Open method, you won’t know about it yet.

So, we have created a pool, and after that it must be configured.

Pool configuration

The following methods are mainly used to configure a pool.

• SetMaxOpenConns()
• SetMaxIdleConns()
• SetConnMaxLifetime()
• SetConnMaxIdleTime()

Let’s see them in more detail.

Maximum number of connections

Using the SetMaxOpenConns() method we set the maximum possible number of open connections to the database from our pool. Be careful — by default this value is 0, which means unlimited number of connections. This can lead to the situation we talked about above, when every request to your service will open a new connection.

What’s the time when the connections are created? Usually after the pool is initialized, the ping method is called, which creates a connection to the database. At this stage, you can assess whether our configuration is correct or not.

err = db.Ping()

Connection creation strategies

Before each request our pool tries to get a connection by calling -

conn(ctx context.Context, strategy connReuseStrategy)

The method implements 2 strategies for creating a connection. AlwaysNewConn always creates a new connection, and cachedOrNewConn first tries to take a free connection from the pool, if any.

If there are no connections in the pool, then the method will create a new one. If it is impossible to create a new connection due to the limits set on the number of open connections, then a request will be created to open a new connection and it will wait for its execution.

The connection received using this method immediately goes into use and is marked as busy, then we are ready to make our requests. After we release this connection it will try to return to the pool.

Pool size limitation

Using the SetMaxIdleConns() method we adjust the maximum number of free connections that can be stored in the pool and wait until they are used. By default you can put up to two connections in the pool.

The «query» method is one of the most frequently called methods for performing queries, in which the connection is released after use and an attempt is made to put it back into the pool of free connections. In the specified code the dc.releaseConn function is responsible for this. If the pool already has the maximum number of connections, then the freed connection will be closed.

However, if the connections are still closed on time, it can still be configured.

Connection lifetime

The SetConnMaxLifetime() method is responsible for determining the lifetime of the connection in the pool from the moment it was created.

Age check is performed every time you try to get a connection from the pool. connectionCleaner also monitors the age. It simply closes expired connections by looping through them and checking the lifetime.

But wait there’s more. Starting with Go version 1.15 you can specify a connection time-out.

Pool time-out

The SetConnMaxIdleTime() method sets the maximum amount of time a connection can be idle in the pool. This setting is very similar to the previous one, but the main difference is that the connection lifetime is counted from the date of its creation, and the time-out in the pool is counted from the moment when this connection was returned to the pool.

After this time has elapsed the connection will also be closed.

Returning a connection to the pool

As we have already said, when executing a request the connection is taken from the pool or created if there is nothing in the pool, after which it is marked as busy. After making requests we must release it.

When we receive data for our request we get back a structure with the results of Rows, which also contains our connection.

When the Close() method is called, the connection is released and an attempt is made to put it in the pool or close it if there is no free space in the pool. Therefore, writing code as in the example above is not good. Be sure to call Close at least in defer.

If you do not release connections, then they will simply hang with the «busy» status, consume memory and clog the number of open connections. As a result we can get an empty pool, since all connections will be considered busy, but at the same time we won’t be able to create a new connection, as we have reached the limit of maximum open connections. This will lead to a situation in which we can neither create new connections nor use existing ones.

But it is even better to do this as follows.

When we initiate Scan(), Go tries to convert the bytes that came from MySQL to the data types passed in the parameters. And note that the request execution method is tied to the context. If the context is reset during the execution of Scan() the data may be corrupted. This problem is described in considerable detail in this article, section «The race».

You can catch this error explicitly calling the Close method and checking for an error. In defer we leave Close to make sure that if some panic occurs, the connection will return to the pool.

Next, I will use synthetic examples to show how all this affects real requests. But before that, let’s look at one more interesting nuance that affects our requests.

Interpolation of parameters

Generally the placeholders are used to make requests with parameters to avoid possible sql injections.

db.QueryRow(“SELECT * FROM client WHERE id=?”, id)

But this way of making requests can work differently depending on the settings.

The parameter interpolateParams=true/false is also supported as a MySQL connection configuration. By default its value is false, which is the same as a non-passed parameter.

If it is false, to make the above request a command is sent to the database server to prepare the expression (Prepared Statements), after which a new command will be sent to execute the prepared expression.

If we have a large and complex request with numerous placeholders that will be executed several times in a method (for example, when creating batches of work), then the prepared expressions can speed up our work. Having performed the preparation once on the MySQL server, we use the prepared request without spending resources on parsing it each time.

If interpolation is enabled, i.e. set to true, the placeholders are replaced by values in the go library code. But if you do not enable interpolation of parameters for regular one-time requests, we will receive two TCP requests to the MySQL server instead of one (not counting the connection itself).

Request with the parameter interpolateParams=false

The same request with interpolateParams=true

Connection Pooling in a Real Sense

Let’s see some pool configuration examples in action. I will loop through the request “SELECT SLEEP(2);” to a MySQL database and see what happens with the connections. In the specified code there is a pause of one second after each request. This results in connections not having the time to be reused, so we can see them on an informational basis.

And this will lead us to the situation we talked about above. When each call of our method will block the connection on itself, the rest of the requests will wait until the connection is released. Please note that this takes 11 seconds.

Then I will make a change maxOpenConns := 0:

maxLifeTime := time.Second * 10

maxOpenConns := 0

maxIdleConns := 0

db := NewPool(maxLifeTime, maxOpenConns,maxIdleConns)

This will lead us to our second case, when each request will create a new connection to the base. As a result we will get an error when trying to connect due to the restriction on the number of connections on the database side.

Now we’ll set maxOpenConns=3, as we have some restrictions.

maxLifeTime := time.Second * 10

maxOpenConns := 3

maxIdleConns := 0

db := NewPool(maxLifeTime, maxOpenConns, maxIdleConns)

In this case we already have the opportunity to open three connections that can work in parallel. And the same five requests took 6.2 seconds instead of 11 seconds as in the first case. But at the same time we see that for each request, we go through the stage of connecting to the database again.

Let’s configure the pool to minimize connections. Let’s set maxIdleConns=3:

maxLifeTime := time.Second * 10

maxOpenConns := 3

maxIdleConns := 3

db := NewPool(maxLifeTime, maxOpenConns, maxIdleConns)

As a result we go through the connection stage only 3 times and put the prepared connections into the pool. On subsequent requests to the database, they are taken from the pool with the authorization phase already passed, and you just have to send the command. And in general requests are made faster.

Let’s sum it up

1. Never leave the connection pool settings at default, you should always customize it to suit your needs.
2. Set maxOpenConns below the configured connection limits on the database side
3. maxOpenConns = 0 means unlimited number of connections
4. If your MySQL server has timeouts configured for idle connections and they are closed on the database side, then there is no need to use maxLifeTime, as the go driver will detect the break and will make the connection again
5. Choose the value of maxLifeTime to minimize the creation of connections during peak loads on your service, and when the load drops, connections should not hang idle for a long time
6. Do not forget the method SetConnMaxIdleTime (with go 1.15). Using this method you can reduce the connection timeout in the pool by deleting it if no one is using it. Thus you can save on the number of open connections in the database and reduce memory consumption in the service.
7. Always release a connection after the request (rows.Close()). Using some methods it is done automatically, but it’s okay if you repeat the Close() call to be safe.
8. If you use multiple pools for the same database (for example, a read pool and a write pool), then the total number of maxOpenConns of these two pools must be less than the configured connection limits on the database side.
9. If you scale your service, don’t forget that open connections will also increase. If you add three instances, then make sure that the sum of maxOpenConns of all three is less than the configured connection limits on the database side.
10. Do not forget about various proxies that, including its timeouts and restrictions, can stand between you and your database. Consider intermediary settings too
11. Consider special aspects of your service. Take into account the nature of the load and adjust the pool settings.
12. Do not forget about monitoring, and also keep checking the connection pool.

Thank you for taking the time to read my article. If you have any questions, please leave a comment below and I will be happy to answer.

Go and MySQL: Setting up Connection Pooling was originally published in Property Finder Engineering and Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Golang (Go) is a general purpose programming language by Google. Many Developers have embraced Go because it’s fast, efficient, offers concurrency, garbage collection, memory safety, and a structural typing system.

According to the 2022 Stack Overflow Developer survey, Go is one of the best-paid technologies and the second most “demanded” backend programming language.

Popular Companies that have already embraced Go include the following:

Google, YouTube, Apple, Amazon, Facebook, Netflix, Dropbox, Docker, BBC, IBM, Twitter, Spotify, Soundcloud, Uber, SendGrid, and of course Property Finder…

Why Go frameworks?

Finding the right web framework is essential as it will save precious time and resources while developing an application. A framework also supports the development process allowing Developers to use their energy on building the features and functionality rather than having to write the base code.

Go frameworks provide many advantages to a development team:

• Eliminating code redundancy and duplication
• Better code security
• Helping to improve consistency in code
• Fewer bugs
• Improving reliability
• Easier testing and debugging processes
• Improving programming productivity
• Helping to establish better programming practices
• Stronger documentation

Are you planning to develop using Go and confused which framework to choose? Here is a list of popular Go Frameworks:

Now, let’s delve into each one of these frameworks individually looking at their main features, pros and cons:

1. Gin Gonic

Gin is considered a very minimalistic framework. It was built based on Martini. Gin Gonic focuses on significant features libraries, uses HTTP router, the quickest HTTP router of Go language which makes it 40x faster and quicker than Martini.Gin has made monitoring or debugging very simple. Gin is highly suitable for building extremely high-performing Rest Apis.

Documentation : Gin Gonic Documentation

Significant features:

• Speed : Gin is a fast framework because of its route grouping and small memory. It’s built on HttpRouter, a lightweight, high-performance HTTP request router. Gin has a low memory footprint and it provides faster performance.
• Great error management and crash recovery : Gin makes it easy to track errors occurring during an HTTP request.

When an error is encountered in a HTTP request Gin documents the errors:

• Easily Extendable : It’s also very easy to create middleware which can be plugged into the request pipeline:

Creating a router and adding middlewares:

• Built-in rendering:

Gin provides an easy to use API for JSON, XML and HTML rendering.

Build with JSON replacement:

Gin uses encoding/json as the default json package but you can change it by building from other tags:

Build without MsgPack rendering feature

Gin enables MsgPack rendering feature by default. But you can disable this feature by specifying nomsgpack build tag:

Pros

• Gin includes only essential features, libraries, and functionalities
• Extensive documentation available
• Well-tested and numerous middlewares
• Reduces boilerplate code
• Reasonable exception capture and error handling,
• Fast running speed, good packet routing.

Cons

• Gin is not suitable for applications that require a large backend or several complex server functions and features as it is suited for building REST API.

Some notable companies use Gin Gonic in their tech stacks, including AB180, SWORD Health, Teachable,SpartanGeek, Sezzle, and SEASON,Wevox, Aircloset, Screeb, Shippo etc.

2. Beego

Beego is the next framework up for review. Beego has great functionalities and features similar to django for python. It is mostly used to develop enterprise web applications. Beego has a fully-fledged Model-View-Controller (MVC) framework with its own libraries and built-in tools.

Example

Documentation : Beego Documentation

Significant Features:

• Bee Tool: BeeTool watches out for code changes and runs tasks when changes
• Fully fledged Built in modules: Beego has built in ORM system, built-in cache handler, faster processing and traffic routing, session management tools, automated testing and automated deployment.
• Great Routing and Monitoring: It’s also very easy to monitor your QPS, memory and CPU usages, and go routine status.
• High Performance: With native Go http package to handle the requests and the efficient concurrence of go routine, Beego applications can handle massive traffic.

Pros

• RESTful support,
• MVC Architecture.
• Modularity
• Auto API documents,
• Annotation router,
• Faster development,
• Powerful development tools, Full stack for Web & API.

Cons

• Caching: Beego caches previous versions of pages, which is fine for user experience, but it can lead to silent build failure.
• Less community support.

Companies using Beego include Getyoutubers, Danale Inc, Huawei Opera, 360, Youdao, Weico, Jd.com, Bmob, Ardan studios, Taobao.com, Tencent games etc.

3. Go Kit

The kit framework is a programming toolkit for building robust, reliable and maintainable microservices in the Golang.kit. This framework gives RPC safety, system observability and infrastructure integration. Kit provides separation of Concerns (transport layer, endpoint layer, service layer). Using go-kit results in clean, reliable and maintainable codebases

REST API:

Documentation : Kit Documentation

Significant Features

• Transport: Go kit ships with support for HTTP, gRPC, Thrift, and net/rpc, NATS, AMQP,Kafka
• Service Recovery: Go kit ships with support for Consul, etcd, ZooKeeper, and DNS SRV records.
• Observability: Go kit ships with support for modern monitoring systems like Prometheus and InfluxDB, as well as more traditional systems like statsd, Graphite, and expvar, and hosted systems like Datadog via DogStatsD and Circonus.
• Logging: Go kit’s package log has evolved through many design iterations, extensive benchmarking, and plenty of real-world use to arrive at its current state.

Pros

• Separation of concerns : Go- kit nicely separates the business logic, endpoint and transport,Low-level libraries,
• The Clean Architecture and Domain-Driven Design
• Middlewares to encode logic with single responsibility principle,
• Adapters for common infrastructure components,
• Makes Go attractive to the organization by providing solutions to common problems of distributed programming like distributed tracing, monitoring and instrumentation, clean architecture.
• Good for both micro services and monoliths

Cons

• Heavy use of interfaces
• Overhead of adding API : To create a simple api you need to add a lot of code Function in the interface, Implementation, Endpoint factory function, Transport function, Request encoder, Request decoder, Response encoder and Response decoder, Add the endpoint to the server, Add the endpoint to the client etc.

4 . Echo

Echo is created by Labstack. It is a micro framework consisting of a standard library and a router with very good documentation. Echo allows Developers to define their own middleware and has built-in middleware to use as well.

Example

Documentation : Echo Documentation

Significant Features

• Optimized Router:Highly optimized HTTP router with zero dynamic memory allocation which smartly prioritizes routes.
• Scalability: Build robust and scalable RESTful API, easily organized into groups.
• Automatic TLS: Automatically install TLS certificates from Let’s Encrypt.
• HTTP/2 support: HTTP/2 support improves speed and provides better user experience.
• Middlewares: Many built-in middleware to use, or define your own. Middleware can beset at root, group or route level.
• Data Binding : Data binding for HTTP request payload, including JSON, XML or form-data.
• Data Rendering: API to send a variety of HTTP responses, including JSON, XML, HTML, File, Attachment, Inline, Stream or Blob.
• Templating: Template rendering using any template engine.
• Extensibility: Customized central HTTP error handling. Easily extendable API.

Pros

• Easy to use
• Light weight
• Easily customizable and extensible
• Minimal
• High performance
• Documentation
• Nicer handler func signature (returning an error)
• A router without absurd pattern matching issues
• Centralized HTTP error handling

Cons

• Lack of support
• Infrequent code updates

some companies reportedly use Echo in their tech stacks, including Hepsiburada, tools, GAPO, Modanisa.com, Finema, The Lonero Foundation, FLUX,arivu. one, Nuxi, Agflow, Orangesys, The Lonero Foundation, Ant Media Server, AI and Robotics ventures etc.

Other Popular frameworks

5. Fiber

Fiber is one of the fastest Go frameworks. Fiber is an express inspired framework with low memory usage and rich routing. Built on top of the Fasthttp which is the fastest HTTP engine for Go, it has been created with the main focus of minimalism.

Example

Documentation : Fiber Documentation

Significant Features

• Built-in Rate Limiter:Used to limit repeated requests to public APIs and/or endpoints
• Serve Static Files: Static files, like style sheets, scripts, and images can be easily cached, resulting in less memory, and the content remains static upon every request.
• Extreme performance: JSON serialization, database access and server-side template composition.
• Robust routing
• Low memory footprint
• Middleware support
• Rapid server-side programming
• Template engines
• WebSocket support
• Server-Sent events
• Translated in 16 languages

Pros

• Rich routing
• Fast
• Low memory consumption
• Support most popular template engines

Cons

• No support for http2

Some of the companies reportedly using Go-Fiber in their tech stacks, including GAPO, Modanisa.com, and Cartesius, issac, bitkub.

6. Revel

A high-productivity fully featured web framework. Revel always ranks as one of the top frameworks on the list and it came up with innovative features and functionalities. It is a self-sufficient framework that does not require any third-party plugins or middleware.

Documentation : Revel Documentation

Significant Features

• Full Featured: Revel is fully featured out of the box, with packaged libraries and feature sets for everything from small to complex tasks.
• Hot code reload:Code reload tool compiles your code and templates.
• Comprehensive:Revel provides routing, parameter parsing, validation, session/flash, templating, caching, job running, a testing framework and even internationalization.
• Reusable MVC components
• Flow control:Revel is built around composable middleware called filters, which implement nearly all request-processing functionality.
• Stateless:Revel provides primitives that keep the web tier stateless for predictable scaling.
• Go mod ready

Pros

• Fully fledged
• Efficiency
• Ease to Learn
• Popularity
• Community support
• No third-party middleware, configuration, plugin and setup]

Cons

• Large codebase: not a lean framework
• No MongoDB support : mongo support can be added using third party implementations
• Non idiomatic code

companies reportedly use Revel in their tech stacks, including Atcoder.jp, backend, and Simon G,Jackpot Rising, Adwisely

7. Gorilla

Gorilla is one of the top Golang frameworks. Gorilla is popular in the application development communities. Gorilla provides http library’s reusable elements and components. Its’ design is modular, scalable and supports web sockets.

Example

Documentation : Gorilla Documentation

Significant Features

• It implements the http.Handler interface so it is compatible with the standard http.ServeMux.
• Requests can be matched based on URL host, path, path prefix, schemes, header and query values, HTTP methods or using custom matchers.
• URL hosts, paths and query values can have variables with an optional regular expression.
• Registered URLs can be built, or “reversed”, which helps maintain references to resources.
• Routes can be used as sub routers: nested routes are only tested if the parent route matches. This is useful to define groups of routes that share common conditions like a host, a path prefix or other repeated attributes. As a bonus, this optimizes request matching.
• It has Bolster WebSockets
• Gorilla is also the largest web framework which caters to Net/HTTP library’s reusable components

Pros

• Routing: it’s packed full of features, including support for method-based routing, dynamic URLs, regex route patterns, and host-based routing. Importantly, it’s the only router on this shortlist which supports custom routing rules and route ‘reversing’ . It also allows setting custom handlers for 404 and 405 responses.
• Gorilla can be easily scaled can add/remove packages, extensions, etc,
• Reusable elements and components.

Cons

• Configuration effort: Gorilla requires more time and effort to configure components,
• Performance: There’s a decrease in performance between Gorilla and other “lean” frameworks.

8. Goji

Goji is a lightweight, minimalistic and flexible web framework. The features of Goji include re-configurable middleware stack, shutdown, and URL pattern. Goji is a HTTP request multiplexer, similar to net/http.ServeMux .

Example

Documentation : Goji Documentation

Significant features

• Compatible with net/httpURL patterns
• Reconfigurable middleware stack
• Context/environment object threaded through middleware and handlers
• Automatic support for Einhorn
• Graceful shutdown, and zero-downtime graceful reload when combined with Einhorn.
• supports Web-socket.

Pros

• Thin: an abstraction layer — very minimal ,lightweight and efficient
• Einhorn: Integrates with Einhorn, a language agnostic socket manager that provides out of the box web socket supportWeb socket assistance to the developers for support
• Goji has a minimalistic HTTP request multiplexer
• Stability
• Not suitable for large enterprise projects.

Other Mentions:

• Buffalo
• Mango
• Martini
• Gocraft

Final Thoughts

Each one of these frameworks have an array of capabilities and functionalities that can be used and adopted. Before you choose the one that is right for your requirements, ensure you fully assess the needs of the project and then recommend the right framework to match those requirements.

Top Golang Frameworks for Gophers was originally published in Property Finder Engineering and Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.