This article explores AWS resource creation (using SQS as an example) via the Backstage Software Templates plugin and how it helps application and platform developers deploy code faster, cleaner, and safer.

Introduction

As a company grows in size, number of collaborators, and system complexity it’s good (or dare I say, essential) as an engineering platform team to look at the Developer Experience as a whole.

One of the aspects of DevEx that flourished over the past couple of years is the implementation of Internal Developer Platforms (IDPs) to make application developers deliver software cleaner, safer, and faster than before. This is done by decreasing the cognitive load on those teams by providing tools like a software catalog, service quality tiers, and software templates.

At Inventa we built our IDP using Backstage as our central piece. In this article, we will explore how we leverage Backstage’s Software Templates plugin (plus some extensions 👀) and Terraform to enable application developers to create AWS resources on our infrastructure easier and faster.

Quick Recap

We already have an awesome article that goes into detail on our Terraform repo file distribution and deployment, but the TLDR is that we have a monorepo where each one of our services declares the necessary resources in the following way:

"product-catalog" (* Project main path *)
├─ "catalog-info.yaml" (* Backstage configuration file *)
├─ "ecr.tf" (* Elastic Container Register definitions *)
├─ "env"
│   ├─ "backends"
│   │   ├─ "dev.tfvars" (* Terraform backend file definitions *)
│   │   └─ "prod.tfvars" (* Terraform backend file definitions *)
│   ├─ "dev.tfvars" (* Terraform variables with values for development environment *)
│   └─ "prod.tfvars" (* Terraform variables with values for production environment *)
├─ "locals.tf" (* Terraform file to create expressions, so you can use the parameters multiple times within a module instead of repeating the expression. *)
├─ "route53.tf" (* Route53 definitions *)
├─ "secrets.tf" (* Secrets Manager definitions *)
├─ "sqs.tf" (* SQS Queue definitions *)
└─ "vars.tf" (* Terraform file to manage all variables *

In the example above we have a product-catalog service that creates ECR, Route53, Secrets Manager, and SQS resources through the files above (ecr.tf, route53.tf, secrets.tf and sqs.tf respectively). Beyond that, specific variables are set either on env/dev.tfvars or env/prod.tfvars.

An important point of this architecture is that the recipe files are rarely (preferably never) changed, the changes are made using env/dev.tfvars or env/prod.tfvars variables, depending on the environment.

For example, the sqs.tf looks something like this:

module "list_queue_deadletter" {
  # Module reference
  source = "..."
  for_each = { for r in var.sqs_list_of_queues : r.name => r }
  name                      = "${each.value.name}-dlq"
  ...
}
module "list_encrypted" {
  # Module reference
  source = "..."
  for_each = { for r in var.sqs_list_of_queues : r.name => r }
  name                       = each.value.name
  sns_sub                    = each.value.sns_sub
  event_sub                  = each.value.event_sub
  s3_sub                     = each.value.s3_sub
  ...
  depends_on = [
    module.list_queue_deadletter
  ]
}

And the dev.tfvars/prod.tfvars looks like this:

...
# SQS vars
sqs_list_of_queues = [
 {
   "name": "product-catalog-created-event",
   "sns_sub": true,
   "event_sub": false,
   "s3_sub": null
 }
]
...

The being that every time we want to create a new SQS queue for this service we only need to add a new entry to the sqs_list_of_queues array.

Before the IDP…

Now, say that we have a new hire that wants to create a queue for a service that still does not have any queues. The creation flow would be as follows:

1. Clone the monorepo where we declare the resources for each service;
2. Find the service folder and create a sqs.tf file (probably copying from other services);
3. On the env/dev.tfvars and/or env/prod.tfvars create the sqs_list_of_queues array;
4. Add the queue with the necessary information to the array;
5. Open a PR so the platform team can review the changes;
6. If everything in the step above is correct, merge the PR. The resources will be created automatically through the pipeline.

The flow works, but it has its flaws:

1. Application developers should not necessarily know the ins and outs of how the infrastructure is created. The extra cognitive load that the process above brings takes away valuable time (and brain power) that could be focused on delivering value to the product.
2. The code quality can not be guaranteed at any level when the PR is opened, leading to more time used by the platform developers on the review process.
3. The process above has a lot of boilerplate and copy-and-paste, which leads to a more time-consuming and error-prone flow.

To address both points we decided to use Backstage’s Software Templates feature to help us automate all the steps necessary to create an infrastructure resource, but before tackling that we had a roadblock in the way that we needed to address…

Terraform, Backstage, and Updating Files

The Software Templates is a great feature from Backstage, but it had a limitation for our specific use case. You see, it’s all well and good when we want to create new files or delete existing files during the template steps, using built-in actions like fetch:template, fetch:plain and fs:delete, the problem comes when we want to update an existing file surgically.

Using our SQS example, if we wanted to add a new product-catalog-deleted-event we should change the *.tfvars files to something like:

...
# SQS vars
sqs_list_of_queues = [
 {
   "name": "product-catalog-created-event",
   "sns_sub": true,
   "event_sub": false,
   "s3_sub": null
+ },
+ {
+   "name": "product-catalog-deleted-event",
+   "sns_sub": true,
+   "event_sub": false,
+   "s3_sub": null
 }
]
...

Sadly, these changes are not easy to do using just the actions that come with Backstage. Luckily, extensibility is a topic that the Backstage team takes very seriously, and the Software Templates plugin is no exception.

Introducing Roadie’s scaffolder-backend-module-utils actions package which, as the name implies, is an extension of the Software Templates plugin with actions that help work with files, JSON, YAML, etc. For our use case, we used the merge JSON action in conjunction with the fact that Terraform supports JSON files with Terraform variables.

So, for this to work, we migrated our dev.tfvars and prod.tfvars files to dev.tfvars.json and prod.tfvars.json, looking now something like this:

{
...
"sqs_list_of_queues": [
 {
   "name": "product-catalog-created-event",
   "sns_sub": true,
   "event_sub": false,
   "s3_sub": null
 }
],
...
}

No other changes were made in the structure and now we are ready to implement the template on Backstage :)

Template Implementation

With all that we discussed above the idea is to create a template that automatically configures the necessary files depending on what is needed:

• If the service already has the SQS configuration it just updates the sqs_list_of_queues array with the new queue.
• If not, it creates the array, and creates the sqs.tf file.

An example of how this template YAML looks like is:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template

# some metadata about the template itself
metadata:
  name: sqs-queue-template
  title: SQS Queue Template
  description: Template for creating a new SQS queue
spec:
  owner: engineering-foundation
  type: sqs-queue

  # these are the steps which are rendered in the frontend with the form input
  parameters:
    - title: Service information
      required:
        - entity
      properties:
        entity:
          title: Service
          description: Choose a service to create the queue for
          type: string
          ui:field: EntityPicker
          ui:options:
            allowArbitraryValues: false
            catalogFilter:
              - kind: Component
                spec.type: service
            defaultKind: Component
    - title: SQS queue general information
      required:
        - queueName
        - envs
      properties:
        queueName:
          title: Queue name
          type: string
          description: Name of the new queue
        envs:
          title: Envrioments
          description: Select environments to create the queue
          type: array
          minItems: 1
          items:
            type: string
            enum:
              - production
              - development
          uniqueItems: true
          ui:widget: checkboxes
        createConfiguration:
          title: Create initial SQS configuration
          description: Should the initial SQS configuration be created for the service?
          type: boolean
          default: false
    - title: SQS queue specific information
      properties:
        snsSub:
          title: SNS subscription
          description: Will the queue subrcibe to a SNS topic?
          type: boolean
          default: false
        eventSub:
          title: Eventbridge subscription
          description: Will the queue subrcibe to a Eventbridge event?
          type: boolean
          default: false
        s3Sub:
          title: S3 subscription
          description: Set the ARN of the S3 bucket (leave empty if the queue will not subscribe to an S3 bucket)
          type: string

  # here's the steps that are executed in series in the scaffolder backend
  steps:
    - id: fetch-vars
      name: Fetch service terraform variables
      action: fetch:plain
      input:
        url: <https://github.com/inventa-shop/aws-resources/blob/main/terraform/$>{{ parameters.entity | parseEntityRef | pick('name') }}/env
        targetPath: ./output/env
    - id: add-queue-dev
      name: Add queue to list of queues (development)
      if: ${{ parameters.envs.includes('development') }}
      action: roadiehq:utils:json:merge
      input:
        path: ./output/env/dev.tfvars.json
        mergeArrays: true
        content:
          sqs_list_of_queues:
            - name: ${{ parameters.queueName }}
              sns_sub: ${{ parameters.snsSub }}
              event_sub: ${{ parameters.eventSub }}
              s3_sub: ${{ parameters.s3Sub if parameters.s3Sub else null }}
    - id: add-queue-prod
      name: Add queue to list of queues (production)
      if: ${{ parameters.envs.includes('production') }}
      action: roadiehq:utils:json:merge
      input:
        path: ./output/env/prod.tfvars.json
        mergeArrays: true
        content:
          sqs_list_of_queues:
            - name: ${{ parameters.queueName }}
              sns_sub: ${{ parameters.snsSub }}
              event_sub: ${{ parameters.eventSub }}
              s3_sub: ${{ parameters.s3Sub if parameters.s3Sub else null }}
    - id: add-empty-queue-list-dev
      name: Add empty queue list (development)
      if: ${{ parameters.envs.includes('development') === false and parameters.createConfiguration }}
      action: roadiehq:utils:json:merge
      input:
        path: ./output/env/dev.tfvars.json
        mergeArrays: true
        content:
          sqs_list_of_queues: []
    - id: add-empty-queue-list-prod
      name: Add empty queue list (production)
      if: ${{ parameters.envs.includes('production') === false and parameters.createConfiguration }}
      action: roadiehq:utils:json:merge
      input:
        path: ./output/env/prod.tfvars.json
        mergeArrays: true
        content:
          sqs_list_of_queues: []
    - id: fetch-skeleton
      name: Fetch new SQS configuration files
      if: ${{ parameters.createConfiguration }}
      action: fetch:plain
      input:
        url: ./skeleton
        targetPath: ./output
    - id: publish-pr
      action: publish:github:pull-request
      name: Create pull request
      input:
        repoUrl: github.com?repo=<REPO>&owner=<OWNER>
        branchName: feature/add-${{ parameters.queueName }}-queue
        targetBranchName: main
        title: Create ${{ parameters.queueName }} SQS queue
        description: |
          This PR adds a new ${{ parameters.queueName }} SQS queue for the ${{ parameters.entity | parseEntityRef | pick('name') }} service.
          Author: @${{ user.entity.metadata.name }}
          _Created automatically via Backstage_ 🚀
        sourcePath: ./output
        targetPath: terraform/${{ parameters.entity | parseEntityRef | pick('name') }}
        commitMessage: (feature) add ${{ parameters.queueName }} SQS queue
        forceEmptyGitAuthor: true

  # some outputs which are saved along with the job for use in the frontend
  output:
    links:
      - title: Pull Request
        url: ${{ steps['publish-pr'].output.remoteUrl }}

In this example, we get the service that needs the queue via the EntityPicker and, after that, the queue information and which environment it will be created. On the steps, we can see that we use the createConfiguration variable to determine if we are going to use the skeleton or not (where the sqs.tf file resides).

Below we have some screenshots of the actual template that we are using (a little more complex than the one we presented above but the basics are the same):

After the PR is opened the platform team can review it as usual, with the difference that only what is changed by the user on the template needs to be checked since the rest is pre-configured on the template.

This logic can be extended for all kinds of resources in the company and this strategy solves the pain points mentioned at the beginning of the article, making it possible for application teams to easily create resources and focus time and effort on the business while platform teams have to focus only on code that is generated via user input.

Infrastructure Resource Creation with Backstage was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.

Context

In the world of product development, building a Minimum Viable Product (MVP) is a common approach to validate market fit without investing too much time and resources upfront. Once the MVP is validated, it’s time to migrate to a more scalable product base to support the entire flow.

This was the situation when I joined Inventa, a startup that had been running for a year with an MVP constructed based on Shopify and Zoho customizations and a series of 1000-line lambdas. The market-fit results were impressive, but Inventa faced problems in product evolution and handling a lot of incidents. Our main goal was to change this scenario! We needed to prioritize a full product migration to a solid foundation for this promising startup.

The migration prioritization

We realized the need to internally establish a scalable foundation for our product in our infrastructure, which we referred to as “Shopify / Zoho Out”.

Creating a new marketplace involves more than just the front end. We had to reconstruct the entire workflow, from user sign-up to order management, finance operations, and internal back offices. Understanding and maintaining everything was challenging, so we recognized the need for an “engineering rebranding”. To ensure a successful product, our focus was on creating a solid and flexible foundation that would allow for easy and quick product evolution. We were able to do that because we shared the main limitations and incidents that caused a lot of instability and limitations to our business.

However, we also acknowledged the potential for quick wins and improvements in the current product. Therefore, we aimed to strike a balance between enhancing the current product and developing the new marketplace in-house, considering all trade-offs. This approach helped us understand the ROI (return on investment) for each promising new change.

Before proceeding with coding and developing the entire platform with 8 teams, we needed to create a detailed plan for executing this game-changing project. The following topics will outline this plan.

Mapping AS-IS e TO-BE

To ensure a smooth and successful migration, we conducted a thorough analysis of the system’s current state (AS-IS). Since understanding the current product was challenging, we created a service blueprint of the product as it is, which includes all its business rules and interactions with internal and external users.

The Engineering leadership led this process because no one truly understood the product, and we were still in the process of building a Product and Design Team at the beginning of 2022 and 2023.

Through this process, we gathered existing knowledge about the product and made several discoveries to understand and map the current developments. This helped us identify end-to-end rules that could be developed or improved.

We created a template for the expected service blueprint we needed. Although we did not strictly adhere to precise service blueprint principles, we created a representation of customer actions and backstage actions that could be taken by internal users or our system. You can see the template below.

With this template and through a rigorous and multidisciplinary effort led by the Engineering team, we have mapped the entire journey of Inventa. This includes the user journeys from their initial interaction to the final interactions, covering the perspectives of sellers, buyers, and internal operations.

This allows us to envision the desired future state (TO-BE). In this step, we need to:

• Address the issues identified in the current state (AS-IS)
• Incorporate new desired flows
• Consider potential possibilities

As a result, we have created a TO-BE blueprint using the same template:

This process was collaboratively built. We brought together different disciplines, stakeholders, and C-level individuals in the same room. Our goal was not to dictate every detail of how things should be done but rather to establish a guiding principle to follow.

“If you do not know where you want to go, it doesn’t matter which path you take.” — Lewis Carroll

This approach allowed us to identify areas that needed improvement and establish a clear vision for the migration process. Now, with all of this in mind, we need to create a plan to put it into action.

“A goal without a plan is just a wish.” — Antoine de Saint-Exupéry

Developing a migration plan

Putting it all together, we had a clear understanding of both the current product and our north star based on the available information. Now, it was time to develop a strategy for migrating our product. While many companies opt for the big bang approach, we were determined to avoid that.

When discussing a phased migration, the first step is to take a step back and consider all the potential options. This could involve exploring different user migrations, product flow migrations, or internal system migrations. There is no right or wrong approach, but it is important to determine the best option for your specific case.

When examining Inventa's scenario, I thoroughly considered the following aspects:

• Viability: Is it feasible to migrate in this manner? Are there any dependencies that could make it difficult or even impossible to change one part without affecting the other?
• Pros: What are the potential positive impacts of this migration? How would it benefit the team, maintenance, users, and the company?
• Cons: What are the potential negative impacts of this migration? What challenges might arise during the development process?

After carefully analyzing all the possibilities, it became evident that using the legacy product based on Shopify and lambdas was not a viable option for either the product flow migration or the system migration. The most suitable approach was to migrate by user groups.

This means that I needed to understand the personas of our users and the business requirements for each type of persona. This understanding would help me determine if there was a clear boundary that I could use to create delivery phases. The first types of personas were the Individual User (CPF) and the Business User (CNPJ).

However, having thousands of users for each type was not sufficient. We needed to determine the minimum business requirements for users to comprehend how we could promptly deliver the initial version and gradually enhance it to accommodate more users.

Consequently, we identified the following initial features that could assist us: Historical Info (for new users or those already registered with historical information) and Credit for Business Users (pre-approved credit that they could utilize in the marketplace).

Based on that vision, we drafted an initial rollout version with 6 phases, taking into account the business requirements and engineering complexities of the product development. We then held meetings with various teams and stakeholders. During these interactions, we evaluated and improved our versions of the phases to be delivered. The revised versions are outlined below:

Then we arrived at the final version (2.1) after successfully completing 4 phases. The final result was:

V0 — Alpha: This phase focused on signing up individual users. Since these users were unfamiliar with our site and we had minimal features, signing them up was the first step.

V0.1 — Beta: In this phase, we targeted all individual users who didn’t require the complexities associated with businesses, such as taxes, shipping, and enrichment by government providers.

V0.2 — Gamma: We focused on current business users in this phase. These users were already on our platform and had undergone credit analysis and business profile enrichment by the government. As a result, we were able to introduce these complexities in the final phase while delivering it to our existing users.

V0.3 — Delta: This phase involved signing up business users. It represented the last and most comprehensive stage of product development, catering to all users, including new ones, with all the necessary complexities.

The Beta and Gamma phases underwent internal rollouts, which were collaboratively constructed by the engineering and data teams. This allowed us to select the most active users for the rollout while excluding inactive ones.

Migrating Users

There are two important points to understand about the process of migrating all users: Users data migration and User group rollout.

For the user’s data migration, we started using the legacy platform to migrate users and their access to the new platform. When users log in to Shopify, we began the migration process 2/3 months in advance. This allowed us to migrate the data of all active users. When the migration time arrived, we constructed the structure and performed the data migration with the help of the data and engineering teams.

For the rollout of the user group, we created an isolated table in our database to control the migrated users. For each rollout, we brought together the product, engineering, and data teams. We defined the strategy for selecting users based on our business strategy, promotional campaigns, and what we wanted to test and monitor on the new platform. Next, we created a list of users and the engineering team added all of these users to our system through an endpoint. When a user did visit our homepage inventa.shop, they are redirected to the correct platform based on the information in this control table.

Mapping Dependencies and Delivery Dates

While creating the release phases, we considered the scope of each team and identified areas with higher demand. To align with the migration plan, it was crucial to understand how to execute it. This involved planning the game with all teams and mapping out the dependencies between them. Thus, the team planning puzzle emerged.

Delivery planning plays a vital role in ensuring a smooth and successful product migration. By understanding the relationships between different tasks, identifying dependencies, and estimating timeframes, teams can effectively coordinate their efforts. This is exactly what we did.

Initially, we faced the worst-case scenario where critical dependencies were estimated to be completed after the development tasks that required integration with them. To address this, we realigned the scopes and adjusted the delivery order for all teams and tribes. Recognizing the need for additional assistance, we focused our team efforts accordingly. With this plan in action, we successfully achieved the first Alpha release, marking a significant milestone.

Moving on to the next phase, we conducted a similar dependencies mapping exercise. As you can see, we successfully aligned all the necessary components for the upcoming versions. This accomplishment is illustrated in the image below.

Follow-up Process

As mentioned before, this project was the most important one for the company at that time. It was crucial for us to ensure that we stayed on track, monitored the progress, managed dependencies, addressed cross-discussions, resolved pending definitions, and handled any potential issues. To achieve this, we established a weekly meeting with the following structure:

Discussion Points

We began with an open agenda, providing an overview of the project’s progress. Then, all meeting attendees had the opportunity to add topics they wanted to discuss in the meeting notes. This approach worked effectively for everyone involved, as it allowed us to address cross-tribe themes that were important to all stakeholders.

Pending Definition

We also need to align follow-up pending definitions for each phase. This will help us keep track of progression, definition, and development blockers.

Intruders

One major challenge when prioritizing a structural task like migration is that the company’s operations continue, which means new bugs and requests may arise. Depending on how many of these we prioritize, it can increasingly delay our migration progress. It is important for us to be able to report this to the company leadership. Therefore, we make a note of all the issues that arise during development. This has been very helpful, especially when we needed to update the proposed timeline and get approval from the leadership.

This approach provides a clear and easy way to communicate what is happening and keep everyone informed.

Progression

Each tribe was responsible for conducting their own follow-up and updating the meeting table with the progression of each macro delivery before each meeting. They also indicated their level of confidence. This process greatly helped us understand which teams required more attention or assistance, as well as when we needed to rethink the plan or the defined dates. The discussions that followed were beneficial for everyone involved, particularly for the company and our users.

Results

I wrote this article one year after the migration, and I can confidently say the results were excellent. Not only during the migration process but also in terms of the solid foundation we built for our product. This foundation enabled us to conduct various tests, experiments, and developments for the growth and profitability of the company, all while keeping maintenance costs low. The engineering team has a clear and concise understanding of our system architecture and takes ownership, resulting in fewer recurring and preventive maintenance tasks. Incidents are now rare, which is a significant improvement compared to when we started this article.

It is not easy for any company to stop delivering new features or fixes to prioritize structural changes. However, the results that follow have a huge impact on everyone involved, especially our users. They now have a stable product that allows for faster delivery of new features and changes.

Learnings

Of course, it wasn’t a smooth journey. We have learned a lot from things we did right and from the challenges that gave us headaches.

Collaborative work between product, data, and engineering in the rollout

This was the best choice we made. We executed it well, and the results greatly contributed to the successful completion of the migration.

Corner Cases

One corner case that we overlooked was the creation of Personal and Business users with the same email on different platforms (legacy and the new marketplace). Although we had accounted for some of these cases, one slipped through. As a result, we had to perform additional migrations to fix it promptly. The main lesson here is to give more emphasis to corner cases and test them thoroughly before releasing them to the product. However, some cases will inevitably arise that we were unable to anticipate. This is part of the process, and the key is to monitor and respond quickly.

Data migration errors due to low-quality legacy data

We encountered challenges with low-quality legacy data that needed to be migrated to our new system for accurate take rate calculations. As we progressed, we discovered more and more corner cases, which led to the creation of complex rules to understand the migrated data. The key takeaway here is to simplify as much as possible and test the rules with a diverse group of people who may not have the same level of context as you do. This will help validate the rules being created.

In conclusion, the journey of migrating Inventa’s product from its initial MVP constructed on Shopify and Zoho customizations to a more scalable and stable platform was a complex yet rewarding process. The team recognized the need for a solid foundation and undertook a meticulous engineering rebranding. The results one-year post-migration were deemed excellent, with a stable product, reduced incidents, and enhanced understanding of the system architecture. The article highlights the significance of collaborative work, addressing corner cases, and the importance of data quality in ensuring a successful and impactful product migration.

If you want to know more about us, add me on LinkedIn or apply to one of our open positions! Let’s learn together and build something bigger and better than we could do individually!

Thanks for reading! I’m always open to receiving feedback, recommendations, or questions, feel free to contact me!!

Pamela Peixinho, Engineering Lead @ Inventa
LinkedIn: https://www.linkedin.com/in/pamepeixinho

My last name is LittleFish, “sea” you later! 😉

Product Migration: Moving from an MVP to a Scalable Product Base was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.

We have observed that our applications often consume a lot of memory, even with only 2 pods. As a result, the pods frequently terminate with the “OOMKilled” exit code. This could be due to various reasons since we employ a microservices architecture.

During our search for a Kubernetes-native solution, we conducted tests using HPA (Horizontal Pod Autoscaler). However, after learning more about its operation, we realized that it performs better with CPU metrics whereas we needed a solution that focused on memory. Our goal was to find a solution that allowed application developers to choose which metrics would trigger automatic pod count scaling. As a result, we explored Keda and its capabilities.

KEDA

Keda is an abbreviation for Kubernetes Event-Driven Autoscaling. Simply put, it is a tool that enables automatic scaling of Kubernetes applications based on events. According to their website:

With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed.

KEDA is a single-purpose and lightweight component that can be added into any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication. With KEDA you can explicitly map the apps you want to use event-driven scale, with other apps continuing to function. This makes KEDA a flexible and safe option to run alongside any number of any other Kubernetes applications or frameworks.

KEDA scalers can both detect if a deployment should be activated or deactivated, and feed custom metrics for a specific event source.

Components

KEDA performs two key roles within Kubernetes:

1. Agent — KEDA activates and deactivates Kubernetes Deployments to scale to and from zero on no events. This is one of the primary roles of the keda-operator container that runs when you install KEDA.

2. Metrics — KEDA acts as a Kubernetes metrics server that exposes rich event data like queue length or stream lag to the Horizontal Pod Autoscaler to drive scale out. It is up to the Deployment to consume the events directly from the source. This preserves rich event integration and enables gestures like completing or abandoning queue messages to work out of the box. The metric serving is the primary role of the keda-operator-metrics-apiserver container that runs when you install KEDA.

Architecture

The diagram below shows how KEDA works in conjunction with the Kubernetes Horizontal Pod Autoscaler, external event sources, and Kubernetes’ etcd data store:

‎As you can see in the architectural diagram, KEDA leverages the Kubernetes Horizontal Pod Autoscaling (HPA) engine to scale your deployments. A Keda internal Kubernetes Controller is responsible for examining external events (so-called scalers) and instructing the metrics adapter to feed the HPA with corresponding scaling requests for the individual deployments.‎

To begin with, we are looking for a tool that can be seamlessly integrated with Prometheus, as we currently use it for other functions. This tool caters perfectly to our requirements, enabling us to leverage the various metrics that applications transmit to Prometheus. It provides engineers with the flexibility to choose the best possible trigger for their applications.

Configuring and Installing Keda

For easy installation and maintenance, we chose to install Keda through Helm 3. See the commands below:

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda --create-namespace

This way, the components were installed within the Keda namespace:

The following parameters are used to integrate Keda and Prometheus:

• serverAddress - Address of Prometheus server.
• metricName - Name of the metric.
• threshold - Value to start scaling for. (This value can be a float).
• query - Query to run.

And this is the result of the YAML file:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kotlin-spring-template-deployment-memory
  namespace: engineering-foundation
spec:
  scaleTargetRef:
    name: kotlin-deployment
  pollingInterval: 30 # Must be seconds
  minReplicaCount: 2
  maxReplicaCount: 3
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-server-prometheus.prometheus.svc.cluster.local:9090
      metricName: pod_memory_percentage_usage_kotlin
      threshold: '80'
      query: sum(round(max by (pod) (max_over_time(container_memory_usage_bytes{namespace="engineering-foundation",pod=~"kotlin-spring-template-deployment.*",container="kotlin-spring-template"}[1m])) / on (pod)(max by (pod) (kube_pod_container_resource_limits{namespace="engineering-foundation",pod=~"kotlin-spring-template-deployment.*",container="kotlin-spring-template",resource="memory"})) * 100,0.01))

This query is designed to find the average memory usage of the pods. When the usage reaches 80%, a new pod is created. Similarly, when the usage drops to 40%, a pod is deleted. It is worth noting that to determine the value required to reduce the number of cans, the current usage percentage should be divided by the threshold value until the result is 0.5. This was a particularly interesting piece of information that took us some time to fully understand.

When viewed from within the cluster, it will look like this:

In this case, the example given is used as a standard in the absence of a defined metric by the engineer responsible for the application. However, it is important to note that the engineer still has the freedom to change the metric if necessary. There are a variety of metrics available in Prometheus, such as the number of requests HTTP or the resources used by the JVM, that can be used as triggers.

Conclusion

It’s easy to set up KEDA on an EKS cluster, using Keda autoscaler provides an efficient and flexible solution for automatically scaling workloads in containerized environments, improving the efficiency and scalability of applications in Kubernetes. With its ability to adjust resources based on real-time demand, Keda helps optimize performance and save resources, making it a valuable tool for DevOps teams and developers.

References

• https://keda.sh/

Scaling Pods based on Prometheus Metrics using Keda was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.

Scaling your dbt project is no different than scaling your data team. They share a lot of similarities. Scaling effectively is a challenge that every team must face. As it matures, the best practices need to mature too.

We started using dbt at Inventa in May 2022. As our project quickly grew, so did the need for a common ground of best practices. Initially, we selected our linter, set up CI, and embraced code reviews. As the project continued to expand, we faced new challenges. Running the full pipeline became time-consuming and the local development workflow became resource-intensive. Ensuring documentation consistency was non-trivial. The DAG grew to a point that we needed to assign ownership over different portions to different people. We could not afford the whole team looking at the entire DAG and this made focusing impossible. We realized we needed an on-call rotation to decentralize this responsibility.

By this point we had around 1,400 models and over 3,000 tests with 14 contributors.

This is when I had the opportunity to join the Advanced dbt Team at CoRise to build this course. Motivated by what we had learned during the last year, I decided to join the team. My goal was to organize our knowledge in such a way that could be useful for other data teams. The dbt community is growing at such a pace, that many teams are now struggling with the same things that we did at our early stage.

I was excited to join Lindsay Murphy (Head of Data at Secoda) and Steve Haraguchi (Learning Design & Operations at CoRise). As a development partner, my role was to contribute to what Lindsay was building (kudos to Lindsay! 🥳). Not an easy task. But I was confident that the experience of our Data Team could provide a valuable case study for the course material.

The first line of the course outline was: “Learn: Best practices for scaling dbt projects, workflows, and documentation”. I would recommend any dbt practitioner to enroll based on this statement alone! But the similarities kept coming in Weeks 2, 3 and 4.

Soon we realized that our teams operate in very similar ways. We seek out feedback and improve upon it. These iterations result in many lessons and insightful projects.

The first step that we took at Inventa with our modern data stack was to embrace best practices. We defined our coding conventions, enforced them within the project and doubled down on documentation. This is what Week 1 of the course is all about: how do we enforce best practices within the project? How do you scale documentation? How do you enforce coding conventions?

Week 2 of Advanced dbt brings more lessons we learned the hard way at Inventa. As our codebase grew, we were applying the DRY (Don’t repeat yourself) principle through the use of Jinja and macros. We looked for existing packages before writing custom code. For example, dbt_expectations helped us increase test coverage quickly with minimal lines of code.

Managing data quality at scale was our next challenge. With expectations of continued growth, we wrote tests often and early. But our test suite started to become too expensive and slow. We needed to optimize. We also complemented our testing stack with data quality tools like Datafold. These challenges are now the focus of Week 3.

Testing was not the only part of the project that needed optimization as we matured. Our runs started taking longer, slowing down the team. To move fast in the beginning, our models were materialized as tables or views. But many were taking too long and our Snowflake bill kept climbing. We decided it was time to standardize our use of incremental models. At the same time, we optimized our Slim CI (learn more about how we did this in How we slimmed down Slim CI for dbt Cloud). In Week 4 Lindsay gets into the weeds about the importance of these practices and more!

Advanced dbt was born out of the desire of creating a game-changer course to scale data teams effectively. Leveraging our experiences at Inventa and collaborating with Lindsay and CoRise, we crafted a course that is more than dbt. It offers insights into building and scaling advanced data teams and best practices. It was so much fun to work on with Lindsay and Steve. We hope you enjoy learning from the course half as much as we enjoyed making it!

Link to the course: Advanced dbt Course at CoRise

Unlocking Data Team Potential: The Advanced dbt Journey was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the current landscape, it is crucial to prioritize the delivery of exceptional products and services to our customers. To do so, we must learn quickly from our users and do our job as an engineering team. The quality responsibility is, in most cases, being addressed by some people from the team, the leader, some more senior engineers, or those with more time at the company, but, this could result in a lack of knowledge sharing and opportunities for people to grow. So, by focusing our energy on the core business and collaborating as a team, we can achieve the finest craft of engineering.

In order to achieve that, rotating team members through different roles and responsibilities (inside the engineering universe) ensure that every aspect of a project is tested and examined.

According to the book, Staff Engineer: Leadership beyond the management track, is the responsibility of the leadership/engineering team to advocate for a high-quality product. In addition, it brings some steps over in how we can manage technical quality:

1. fix the hot spots that are causing immediate problems
2. adopt best practices that are known to improve quality
3. prioritize leverage points that preserve quality as your software changes
4. align technical vectors in how your organization changes software
5. measure technical quality to guide deeper investment
6. spin up a technical quality team to create systems and tools for quality
7. run a quality program to measure, track and create accountability

It is evident that having a clear understanding of the issues pertaining to domains/organizations is imperative to establish a process that fosters technical quality outcomes. The diagram presented below outlines a proposed procedure for managing technical quality, starting with an engineering quality guardian assuming responsibility for the process.

To support it, the main pillars are bug reports, code quality, operability, tech debts, and developer experience. Below we can understand each one of them and the complete process overview.

High-Quality Engineering Pillars

Bug Reports

The first pillar is bug reports. The team’s priority is to listen and learn from users (stakeholders, final clients). At Inventa, we have some dashboards that the engineering team could access in order to see the major problems. This helps to ensure that every team member is familiar with the product and its functionalities, making it easier to identify and fix any issues that arise.

Bug checking can be time-consuming and require a lot of effort, but it is a crucial step to ensure that the product works as intended. So, a good alignment with the product team/stakeholders is important to understand the criticality of the report to address the root cause. However, be aware that some quick fixes could help the team to move forward.

Code Quality

To improve our code readability and maintainability, we should use some tools in order to help us with this process. Sonarqube is a tool used to detect technical debts in code. It helps identify issues that could potentially cause problems down the line, such as code duplication, security vulnerabilities, and coding standards violations.

With the help of coding standards like Klint, DeteKt, and the use of Sonarqube we can deliver clean code, but, this tool is only useful if we keep track of issues and keep enforcing patterns along the way. Considering this a pillar of software quality, engineering should focus on checking it regularly for projects with ownership. Not only code quality tools + CI/CD pipelines could help in this process.

The image below is an example static code analysis pipeline here at Inventa, using Github actions.

So, to keep your code with a clear architecture and code writing those tools helps. Rotating leadership to check these tools helps to deliver a better product.

Operability

Operability in software engineering refers to the ease and effectiveness with which software can be operated and maintained, in a simple way it means to look at how things are running. The dashboard below contains some metrics related to a Spring Boot application at Inventa.

The Google SRE book introduces a concise set of key performance indicators known as golden signals. These signals, including latency, traffic, errors, and saturation, provide valuable insights into the health and performance of your application; if your team is looking where to start the reference link can guide you there. Keeping track of those numbers aims to deliver a better user experience, retention, and cost implications. To sustain long-term success your team/company should involve this as a pillar of quality.

Other important mechanisms to smooth the operation and success of a business are service level objectives, error tracking, and incident management.

SLO defines the level of reliability and performance that a business aims to provide to its customers. Establishing targets and meeting them builds customer trust. So, having a close look at those numbers is extremely relevant to build better products.

Curious why we have adopted the following tools in our operational engineering? Check the following article written by Marcos Gomes.

Tech Debts

According to Philippe Kruchten, in Managing Technical Debt: Reducing Friction in Software Development, a simple analogy of technical debt could be related to mechanical devices:

the more friction a device experiences due to wear and tear, lack of lubrication, or bad design, the harder it is to move the device, and the more energy you have to apply to get the original effect.

Bring this to a software engineering world, Technical debts can result in the accumulation of outdated software, inefficient coding, and outdated development practices and as mechanical devices, if we don’t manage it things tends to fail. A simple way to keep track of it inside your organization is to write issues inside the repository project. Since those debts are technically related, is very important that the engineering team give visibility to stakeholders of the immediate impact and backlog created.

Developer Experience

Investing in a positive developer experience has many benefits for organizations. It improves productivity, helps attract and keep talented developers, encourages teamwork, makes onboarding easier, and boosts developer satisfaction and morale. When organizations prioritize the well-being and success of developers, they create an environment that promotes innovation, speeds up software development, and leads to long-term success.

At Inventa, we have adopted Backstage as our tool to address developer experience. Currently, we are using the service catalog, API Docs, and Tech Docs features that come from the main backstage functionalities. To get this process running, we should not limit this work inside platform teams, all the engineers should participate in searching for solutions to get our work more effective and funny. So, during the prioritization process of selecting a work to be done, we must consider DevEx prioritization. To start simply, the team can think of solutions related to this[source]:

• Create — Improve project setup and infrastructure definition
• Build — Verify that changes are free of syntax errors and compile.
• Test — Verify that all automated tests pass.
• Start — Launch end-to-end to simulate execution in a deployed environment.
• Debug — Attach debugger to start solution, set breakpoints, step through code, and inspect variables.

Which implies:

If effort is invested to make these activities as easy as possible, the returns on that effort will increase the longer the project runs, and the larger the team is. [source]

So, thinking about the long-term success of high-performing teams the developer experience is a must, considering it on a daily basis certainly changes the business impact.

Prioritization is the key

So far, we have talked about some pillars when searching for technical quality. If your team decides to dive deep into each topic, the team duties and backlog will multiply ad infinitum 🤯. Therefore, the art of prioritization comes in handy.

To effectively prioritize product issues, several key steps should be followed. Firstly, it is essential to define and align product goals, establishing a framework for evaluating issues that align with the product’s strategic direction. Next, assess the impact and urgency of each issue, considering factors such as customer satisfaction, revenue impact, and technical feasibility. Collecting stakeholder input from different company areas ensures a broader understanding of challenges and effective prioritization. Also, it is crucial to estimate the effort and resources required to address each issue, considering development time, dependencies, and available capacity. Lastly, prioritize is an iterative process, requiring continuous re-evaluation and adaptation based on changing circumstances, user feedback, and new insights gained during development. Being able to adjust and adapt helps you focus on important problems and create successful products.

A simple overview of each step over prioritization:

1. Define and Align on Product Goals
2. Assess Impact and Urgency, and find a useful framework 🙂
3. Gather Stakeholder Input
4. Estimate Effort and Resources
5. Continuously Re-evaluate and Adapt

Getting the Job Done and Assessing Impact

Throughout our discussion, we have explored the advantages of effectively managing team quality and identified key pillars that contribute to the development of a high-performing product. It is essential to emphasize that these pillars should be tailored to meet the specific needs of your team and company. Each organization will find inspiration in these pillars as they adapt them to their unique context.

Once your backlog is populated, the next crucial step is prioritization. Given the fast-paced nature of our work in product development, there may be instances where our focus is solely on addressing bug reports or resolving technical debts. This is a normal occurrence. However, it is imperative to remain prepared for the moment when the opportunity arises to enhance the architecture, refine the code style, and improve the overall developer experience.

By continually evolving these aspects, we can ensure that our products are built on a solid foundation and foster an environment conducive to growth and innovation.

As part of the process, it is important to assess the impact of measuring the results, some metrics are:

Bug Resolution/Tech Debts

• The number of bugs reported; Bug resolution rate; Mean time to resolve (MTTR)

Code Quality

• Code coverage; Code complexity, use Sonarqube for it
• Architecture: https://medium.com/quintoandar-tech-blog/software-architecture-analysis-25bd0274d837
• Explore more metrics in [source]

Developer Experience

• Read Measuring an Engineering Organization [link]

Operability

• To start simple: latency, traffic, errors, and saturation
• Latter: Mean time to detect (MTTD) and mean time to resolve (MTTR), Availability, and uptime
• Advanced: SLOs

Finale

To achieve a five-star product, as a team we must be extremely connected and communicate in a high standard way. From the perspective of engineering, it is important to give visibility to issues and solutions that may not work long-term /or are scalable. With this visibility, the product, designer, data, and engineer can come together to decide the next steps. Listing, acting, and measuring results with consistency will lead to great results. A diligent team achieves more.

If you want to know more about us, add me on LinkedIn or apply to one of our open positions! Let’s learn together and build something bigger and better than we could do individually!

Managing Technical Quality To Ensure a Five-Star Product was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unleashing the power of code. A step-by-step strategy for building a scalable and efficient infrastructure with IaC.

Building a robust infrastructure is a crucial aspect of any company's success. As the business grows and evolves, the need for a scalable and resilient infrastructure that can adapt to changing requirements becomes inevitable. However, manually managing infrastructure can be time-consuming, error-prone, and inefficient. That’s where Infrastructure as Code (IaC) comes in. IaC allows you to define the infrastructure using code, making it more manageable, repeatable, and scalable.

In this article, we’ll explore how we defined an IaC strategy that enables us to build and manage infrastructure efficiently.

IaC

Infrastructure as Code is a method of defining and managing infrastructure using code. By treating infrastructure as code, IaC allows for greater efficiency, scalability, and repeatability than traditional manual infrastructure management methods. With IaC, infrastructure resources such as servers, storage, and networks are defined and deployed using code that can be versioned, tested, and automated. This approach provides several benefits, including the ability to quickly spin up and scale infrastructure resources, reduce human error, and increase the speed of infrastructure deployment. In the next section, we will explore the basics of IaC and how it can help engineers efficiently manage their infrastructure.

Choosing the Right IaC

Choosing the right IaC tool is a decision that can greatly impact the effectiveness and efficiency of your infrastructure management. Several popular IaC tools and frameworks are available, each with its strengths, capabilities, and limitations. When selecting an IaC, we consider the following factors:

1. Evaluate the tool’s capabilities and ensure it supports the infrastructure providers we use or plan to use, such as AWS, Azure, or Google Cloud.
2. Consider the learning curve associated with it. Evaluate its documentation, community support, and available learning resources such as tutorials or courses.
3. Determine if the tool allows us to customize and extend its functionality to meet your requirements. Look for features such as support for custom plugins or the ability to define your resource types.
4. Assess the size and activity level of the tool’s community. A vibrant community can provide valuable resources, plugins, and community-contributed modules, making it easier to leverage best practices and benefit from shared knowledge.
5. How the IaC tool integrates with our existing development and deployment workflows. Look for features like integration with popular CI/CD pipelines, source control systems, or infrastructure monitoring tools.
6. Evaluate the tool’s roadmap, release cycle, and vendor support. Choose a tool that is actively maintained, regularly updated, and has a solid user base to ensure ongoing support and updates.

Evaluate different IaC tools and frameworks to find the best fit for their unique requirements. Terraform, with its extensive provider ecosystem, offers a versatile and cloud-agnostic solution. Ansible, on the other hand, excels in configuration management and automation. CloudFormation provides native integration with AWS services, making it an excellent choice for organizations heavily reliant on AWS.

Embracing Terraform for Streamlined Infrastructure Management

With its extensive feature set and widespread adoption, Terraform offers different benefits that align with the needs of seeking efficient infrastructure management and that is why it was chosen by us as our IaC provider. Let’s delve into some of the advantages of choosing Terraform as your IaC tool:

• It supports multiple cloud providers, and integrates with different cloud services and platforms, allowing organizations to avoid vendor lock-in, simplify the management of complex infrastructures, and adopt a multi-cloud or hybrid cloud strategy.
• Enables defining infrastructure state declaratively, automating provisioning and management, and ensuring scalability and repeatability.
• Promotes a modular approach, creating reusable building blocks for accelerated development and consistent environments.
• A thriving community, contributed modules, best practices, and documentation make Terraform a reliable and well-supported choice for infrastructure management.

Designing Your Infrastructure as Code

Designing an effective IaC solution involves careful planning. We started by defining infrastructure components and their interdependencies — breaking down infrastructure into reusable modules fosters code reusability and maintainability. We employed best practices, such as utilizing variables for flexibility, keeping code modular, and implementing testing and validation to ensure the reliability of the infrastructure.

Here are some reusable modules that were implemented and used by all the engineering teams:

Implementing the IaC Solution

Once the design is in place, it’s time to implement the IaC solution. This involves deploying code, managing configuration, and monitoring infrastructure. Engineers can reduce manual effort, minimize human errors, and maintain consistency across environments by automating these processes. Additionally, adopting practices like change management ensures smooth updates and enhancements to the infrastructure codebase.

When developing Terraform modules, it is essential to follow best practices to ensure reusability, ease of use, and maintainability. Here are some insights that we use to implement a Terraform module that adheres to these principles:

• Start by organizing our Terraform module into a logical directory structure, making it easier to navigate and understand.
• Split the modules into smaller, reusable components to promote modularity.
• Leverage input and output variables to allow users to customize the module’s behavior.
• Provide comprehensive and clear documentation for your module explaining the module’s purpose and usage examples.
• Implement variable validation to ensure inputs conform to expected formats.
• Foster a community around your Terraform module by encouraging user feedback, contributions, and issue reporting.
• Actively address user concerns, feature requests, and bug reports to improve the module’s quality and user experience.

Managing Our IaC Infrastructure

To effectively manage infrastructure as IaC, we consolidated all Terraform projects into a mono repository. This speeds up code management, simplifies version control, promotes collaboration, and enables consistent CI/CD pipelines with GitHub Actions. Treating infrastructure code like a software project, the mono repository facilitates efficient tracking, branch management, and pull request handling, also boosts collaboration, automates testing and deployments, and encourages code reuse. This approach ensures stable and more reliable infrastructure management.

Let’s go a little deep to understand our pipeline workflow:

The above workflow runs using a Terraform project with files structure like:

Imagine we have the “Product Catalog” project, where we manage all the necessary cloud resources for this application. Let’s take a look at the files you’ll find in this project:

"product-catalog" (* Project main path *)
├─ "catalog-info.yaml" (* Backstage configuration file *)
├─ "ecr.tf" (* Elastic Container Register definitions *)
├─ "env"
│   ├─ "backends"
│   │   ├─ "dev.tfvars" (* Terraform backend file definitions *)
│   │   └─ "prod.tfvars" (* Terraform backend file definitions *)
│   ├─ "dev.tfvars" (* Terraform variables with values for development environment *)
│   └─ "prod.tfvars" (* Terraform variables with values for production environment *)
├─ "locals.tf" (* Terraform file to create expressions, so you can use the parameters multiple times within a module instead of repeating the expression. *)
├─ "route53.tf" (* Route53 definitions *)
├─ "secrets.tf" (* Secrets Manager definitions *)
├─ "sqs.tf" (* SQS Queue definitions *)
└─ "vars.tf" (* Terraform file to manage all variables *

The ECR, Route 53, Secrets Manager and SQS definition files are using our Terraform Modules repository as a source.

Building a solid infrastructure is vital for engineering success, and Infrastructure as Code offers a powerful solution. Organizations can establish a strong foundation by understanding the concept, selecting the right tools, designing a scalable architecture, implementing best practices, and adopting robust management strategies. Embracing Infrastructure as Code empowers to iterate quickly, scale effortlessly, and focus on what matters most: innovation and growth in an ever-evolving business landscape.

If you’re seeking a dynamic and cutting-edge environment, we have some exciting opportunities for you. At Inventa, we pride ourselves on leveraging state-of-the-art platforms and pushing the boundaries of technology development. We’re on a mission to build a remarkably reliable, highly observable, and scalable product that will make waves in the industry.

Ready to embark on a thrilling journey with me? Take the first step by applying simply click here to explore our open positions. If you have any questions or want to learn more about our team, don’t hesitate to reach out to us on LinkedIn.

Defining an Effective Infrastructure as Code (IaC) Strategy: Building for Scalability and… was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.

Kotlin is a versatile programming language that is widely used in developing mobile and web applications. At Inventa, we’re using Kotlin extensively and having really nice results in terms of maintainability, reliability, scalability, and also in developer experience. One of the most prominent features of Kotlin is observable, which enables developers to create reactive and event-driven programs. In this article, we will explore what Kotlin’s observable feature is and how it can be used in practical applications.

Here on the order-management squad at Inventa, we are responsible for managing the lifecycle of an order from its payment at checkout until it is delivered to the retailer. During the order process, there are various steps that can result in changes to the order amounts. Some scenarios include:

• Insufficient product quantity: If the supplier does not have the purchased quantity of a product in stock, they can communicate with the retailer and reduce the quantity accordingly. This reduction in quantity will lead to a decrease in the order's total value.
• Recalculating service fees: When the order’s total changes, it may be necessary to recalculate the service fee for the order. The service fee calculation can be adjusted based on the updated order's total value.
• Adjusting vendor's receivable amounts: In addition to service fees, changes in the order total may also impact the vendor’s receivable amount. If there are modifications to the service fee, the vendor’s receivable amount should be recalculated accordingly.

This is just one of the scenarios where order amounts ​​can change and cascade changes to other fields.

Initially, we implemented the logic for recalculating the values ​​in Hibernate’s listeners @PrePersist and @PreUpdate, with this we were able to guarantee that regardless of the use case that changed some value of the order, the persisted data would always have the correct value. The example below shows how it was implemented:

@PreUpdate
@PrePersist
fun updateReceivableAmount() {
   this.receivableAmount = this.subtotal
       .plus(this.taxes)
       .plus(this.freight)
       .minus(this.serviceFee)
       .minus(this.discounts)
}

The orders table was getting too big with too many fields, so we split that table and created a new one with the shipping-related information. Among these moved fields, there is a field containing the value of the freight, so every time this field changes, we need to recalculate the vendor’s receivable amount. As they are now different entities, even if we change the shipping value using the order entity as a base, hibernate’s @PreUpdate will not be triggered, since the change was not the Order entity, but the Shipping entity. This solution presented above no longer works for us.

How to solve the problem

Looking for alternatives on how to figure this thing out, we found Kotlin’s Observable feature.

Observable properties are variables that can be monitored for any changes in their value. When a change is detected, the property notifies any listeners that have subscribed to it. This allows other parts of the application to react to changes in the property’s value.

To create an observable property in Kotlin, we can use the ‘observable’ function provided by the Kotlin standard library. Here’s an example:

import kotlin.properties.Delegates

class KotlinObservableExample {
    var total: BigDecimal by Delegates.observable(BigDecimal.ZERO) { 
        prop, old, new -> 
        println("$old -> $new")
    }
}

In this example, we create a KotlinObservableExample object with an observable property called total. The property is initialized with the value BigDecimal.ZERO. We use the ‘observable’ function to specify a lambda expression that will be called whenever the property’s value changes. The lambda expression takes three arguments: the property being changed, the old value, and the new value.

Applying this to our use case, we have something like this:

class Order(
  val id: Long,
  val uuid: UUID
  ... // other propeties
) {
  var subtotal: BigDecimal by Delegates.observable(BigDecimal.ZERO, computeTotals())
  var serviceFee: BigDecimal by Delegates.observable(BigDecimal.ZERO, computeTotals())
  var discounts: BigDecimal by Delegates.observable(BigDecimal.ZERO, computeTotals())
  var taxes: BigDecimal by Delegates.observable(BigDecimal.ZERO, computeTotals())
  var shipping: Shipping? by Delegates.observable(null, computeTotals())
  
  private fun computeTotals(): (
     property: KProperty<*>,
     oldValue: Any?,
     newValue: Any?
  ) -> Unit = { _, _, _ ->
     this.computeServiceFee()
     this.computeTotalPrice()
     this.computeReceivableAmount()
  }
}

In the property declaration, instead of just declaring the type, we declare a delegate of the observable type, informing the default value of the property and the callback method to be triggered when the value changes. In our use case, for any value that changes we want to recalculate the service fee, the total price, and the receivable amount. We use the same callback for all properties, but you can create many different callbacks depending on your use case.

The callback method has three parameters with the following information:

1. The first parameter contains information about the changed property itself. It provides details such as whether the property is final or open, its type, and other relevant information that you can check here.

2. The second parameter holds the value of the property before the change occurs. It allows you to access the previous value and perform any necessary comparisons or operations.

3. The third parameter represents the value of the property after the change. It provides the updated value, allowing you to track the modification that took place.

With this solution, in addition to being able to solve the problem we have:

• Loose coupling: we removed the dependency of the database layer for updating the properties value.
• Testability: we can test this logic with unit tests. When using Hibernate's listener it was necessary to create integration tests, loading the application to insert and update an entity to verify if the business logic was right.
• Maintainability: we don’t need someone to know how Hibernate’s listeners work to understand that value will only be updated when the entity is saved. We also can change our entity without having side effects on the business logic, and if it does, the unit tests will fail.

I hope to have helped you to get to know a little about this Kotlin feature and how can it be useful in a real-world problem. If you’ve used this feature in any other way that you find interesting to share, add your comment below to enrich the article. Thanks for reading.

If you want to know more about us, add me on LinkedIn or apply to one of our open positions! Let’s learn together!

Supercharge your code with Kotlin Observable: Watch your properties recompute in real time! was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.

Upon arriving at Inventa, one of my primary responsibilities was monitoring our applications to better understand their behavior and our infrastructure.

At this point, Inventa was about a year old and we were transitioning from a serverless Lambda architecture to a microservices architecture with Kubernetes (EKS), where we didn't have a strategy for observability, alarms, or logs. More here How we developed our deployment model.

So, we started investigating something that would allow us to scale, that was modern, and that the price wouldn’t be a pain for the business as we scaled.

Based on these requirements, an RFC was structured and the result came up with three possible options.

Zabbix: Simple and easy to implement, with an agent on the instances, but the tool was created to monitor networks and evolved to keep up with the evolution of highly dynamic systems in a structure of containers and microservices. Since we were starting from scratch, we understood that something more robust and modern could meet our needs in the short, medium, and long term. We immediately ruled it out.

And then we went into a more in-depth comparison between Prometheus and Elastic Search, where we analyzed the following aspects:

Purpose:

• Prometheus: is a real-time monitoring tool for distributed systems, designed to collect, store, and query metrics in real-time.
• Elasticsearch: is a search and data analysis engine that can be used to store, search, and analyze structured and unstructured data at scale.

Storage of Data:

• Prometheus: stores data in a time series format.
• Elasticsearch: uses an inverted index engine to store data.

This means that Prometheus is optimized for storing and querying time series data, while Elasticsearch is more suitable for search and unstructured data.

Scalability:

• Prometheus: is highly scalable and designed to perform well in distributed environments.
• Elasticsearch: is also scalable, but requires more configuration and tuning to perform well in distributed environments.

Queries and Analysis:

• Prometheus: optimized for querying and analyzing real-time metrics.
• Elasticsearch: more suitable for analyzing search data in unstructured data.

With this, we understood that the ideal strategy for our scenario, in the medium and long term, would be to use both, where application and infrastructure metrics would be collected and stored using Prometheus, and for querying event logs and application logs, we would use Elastic Search.

However, we faced a significant challenge: we only had one engineer available to do the entire implementation. It was a large project where we wanted to cover the basic pillars of observability, which are logs, metrics, and tracing.

In the presentation of the RFC, one of our lead engineers blurts out in the virtual room:

Egberto: Do you guys know about Grafana Cloud?

Although no immediate decisions were made, an action point was added to explore the potential benefits of this solution.

In the study, we identified:

Roughly speaking, when we create an account on Grafana Cloud, which is a SaaS observability tool, the structure of the following services is automatically provisioned:

• Grafana: Customizable data visualization platform that allows users to create, explore, and share dashboards and panels in real time. It supports multiple data sources, including databases, cloud services, and APIs, and offers a range of features for monitoring and analyzing systems like Kubernetes, Instances, Synthetic monitoring, Usage insights, Network usage, and more….
• Grafana Loki: A service used for log ingestion and querying. Configuration is done through an agent provided by Grafana Cloud upon account creation.
• Grafana Prometheus: A powerful open-source time series database, that can be a monitoring and alerting toolkit that enables users to collect and analyze metrics from various systems in real time, like our apps in Kubernetes.
• Grafana Tempo: On the server side, it is an easy-to-use, highly scalable open-source distributed tracing backend. It is deeply integrated with Grafana, Prometheus, and Loki. It can ingest common open-source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry, which is our case. On the application side, an auto-instrumentation agent for Java based on OpenTelemetry was integrated to generate tracing spans.
• Grafana Alerts: It is a customized version of AlertManager highly integrated with the above-mentioned services.
• Grafana Incidents: also integrated with the services mentioned above, for incident management, in the same format as PagerDuty, but all these services are already available and configured when creating an account on Grafana Cloud.
• Synthetic Monitoring: We monitor how a specific endpoint is performing, with the possibility of configuring logins, headers, tokens, and more. For example, we use it to monitor our blog hosted outside our engineering stack.
• Grafana OnCall: Also integrated with the services above, when we have an alert with an incident, it is possible to escalate it to the responsible team to act on the incident, and at the same time we have a unified location where all engineers can track the status of the entire incident cycle.
  In the free version we can register up to 5 incidents.
• Grafana Data sources: Grafana has plugins to integrate with virtually any possible data source, from Cloudwatch, Elastic Search, Kubernetes, Databases, Sentry, Windows, GitHub, and more.

Amazing, isn’t it?

Based on this new information, we decided to adopt Grafana Cloud as our observability stack, and then we got down to work:

By leveraging Grafana Cloud’s pre-existing infrastructure as a SaaS model, we were able to streamline our configuration process and focus on creating processes and automation to effectively utilize their dashboard and visualization service, alerting and notifications service, and data source and integrations service.

But we know that not everything is perfect, right? Let’s not forget that only one engineer is working on this project. At the beginning of any new tool that we start using, we have a learning curve, which is normal and during this journey, we noticed an incredible team in the CS department, with amazing and delightful customer service always helping us!

If you’ve made it this far, you’re likely curious about the cost of using Grafana Cloud. The pricing for Grafana Cloud is determined by four different variants:

• Active metrics: the number of metrics being generated by the clusters and applications and consumed by Grafana Prometheus.
• Logs: the number of logs in GB consumed by Grafana Loki.
• Tracing: the amount of tracing in GB consumed by Grafana Tempo.
• As it is a SAAS with a customized UI and authentication, we also pay per active user per month.

After the applications were created and deployed in the new infrastructure, the costs grew as expected, but we identified an opportunity to improve our costs related to large-scale consumption of custom metrics, given each application sends thousands of metrics by default and this would significantly increase the costs of Grafana Cloud. Therefore, we adopted a strategy to consume these custom metrics from our Kotlin applications using a local Prometheus that we deploy and manage ourselves, and later, we extended this strategy to infrastructure metrics as well.

Today, an overview of our observability architecture would look like this:

As we were migrating from Lambda to Kubernetes, we still needed to see the logs from the Lambdas. In Grafana Cloud this is easy because we can integrate a bunch of Datasource providers as a plugin by Grafana.

With Grafana Cloud working in our favor to abstract all the infrastructure parts (creating, configuring, and maintaining), we only had to configure integrations and create processes and automation. It took us a little over 2 months to create scalable and applicable ingestion processes for logs, metrics, tracing, incidents, and on-call in our organization.

This is how our days are here at Inventa. How about joining us!? We are looking for Engineers who enjoy the state-of-the-art platform and application development, and want to help build a very reliable, observable, and scalable product in Inventa! Hit here to apply and, if you have any questions, do not hesitate to ping us on LinkedIn.

Defining Inventa's Observability Strategy was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is the most crucial thing the People team has to have to work with technology teams?

I was asked by Inventa’s Head of Engineering on my interview for the role I’m currently in. I’ve been working in the HR area for over a decade now, and this simple and tricky question made me realize how much things have changed for the People area — including its name.

In the past, Human Resources was primarily viewed as a service and support function, focused on meeting basic business needs and legal requirements such as hiring and payment. The human factor was often treated separately from the goal of delivering productivity, with employees being seen as mere resources. However, over time, this perspective has changed to view human needs and well-being as critical to business success (this is a very recent event, but… finally!). This new approach considers factors such as behavior and purpose, recognizing that engaged employees perform better and lead healthier lives.

Despite this shift in perspective, one critical aspect of Human Resources has remained largely unchanged: its mindset. Many companies still view HR as a service-oriented department tasked with creating processes and projects that are thought to be best for everyone, without truly understanding how these initiatives impact employees. HR teams often lack direct experience with the routine challenges and experiences that employees face, making it difficult to develop effective solutions. This disconnect is particularly problematic when it comes to product development and squad management, where a deep understanding of employee needs and experiences is not only crucial for achieving success but also for creating solutions that have genuine meaning for people.

I want to make it more tangible with an experience I had in the past. As a People Partner within a squad that was developing a new product for a retail tech company. The developers’ coding skills were awesome, they were really good at making code scalable and clean, but they weren’t a proactive team. The team tended to get stuck on minor issues or obstacles, relying heavily on the leader to resolve every single problem, they also depended on him to stretch to other squads to understand the core part of the product. As a result, the leader became very overwhelmed, the team lost time, and progress stalled. It’s easy in a situation like that to think as a People area person: “They don’t have the culture of autonomy yet, we could do a group dynamics, or a culture talk to solve this”, etc. Since I had the opportunity to participate in retrospectives, daily meetings, and planning sessions, as well as hold regular 1-on-1s with the team, I had the right proximity to stay in the loop. In addition, my ongoing learning and willingness to ask even the most basic questions (no questions are silly here) meant that I had a good understanding of software development.

I observed that the team had not been actively involved in refining our backlog and that the items on the list were too narrowly defined as tasks. The squad leader took charge of organizing all the information and always provided direction. While it’s certainly helpful for a leader to be objective and support the team’s progress, this approach had the unintended consequence of diminishing the team’s sense of ownership over building solutions alone. So, we decided to make a change: distribute more responsibilities, making them owners of agile rites and refinement, asking them more questions on development directions than providing the answers (and a lot of changes more on routine, planning meetings, etc). One month later, we saw a significant shift in our team’s dynamic, they became the protagonists. Through our 1-on-1s, we reinforced our culture of autonomy and empowerment, using recent actions as examples and providing valuable feedback.

People can play a crucial role in any technology team by not only helping to bridge the gap between the team and the wider organization but boosting the area by applying psychology knowledge and techniques in an assertive way. By actively participating in the team’s activities, gaining deep knowledge of their area, and building close relationships with team members, People can offer valuable insights and support on issues related to people, leadership, structure, group relationship, and organizational culture. The gap between People solutions and what the tech team really needs will become smaller and smaller.

Moreover, by working closely with specialists in areas such as design, data, engineering, and product, People can help to ensure that the team is fully aligned with the wider goals of the business. Through these efforts, a skilled People person can make a significant difference in driving team success and fostering a culture of innovation and collaboration.

This is the kind of boost that the People team should strive to provide. Instead of running group dynamics that don’t resonate with employees, we should aim to be a true Partner to the team, working alongside specialists in design, data, engineering, product, and business. To be effective we’ve gotta be there, we’ve gotta know tech and how things work.

That was my answer to Adilson.

Processes and organization alone may define my role as a department, but it’s my proximity, deep understanding of the area, and active participation that enable me to become a strategic partner… a true partner to people — therefore — to business.

You can apply this mindset to any business area and people’s specialty. Did you identify with our mindset and way of thinking? Feel free to get in touch with me on LinkedIn or visit our website to learn more about careers and job openings!

Unlocking the Secret to Engineering Team Success: How a People Partner Can Make All the Difference! was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.

As a software-reliant company, we understand the importance of selecting the right tools and technologies for our projects. With so many options available, we’ve carefully chosen our frontend stack based on their unique benefits and ability to help us build robust and reliable frontend applications quickly and efficiently. This article aims to present the challenges, specific requirements, and trade-offs that we considered when defining our frontend stack.

Past:

At the moment I’ve joined the company (August 2022), we had some different ways of doing/deploying frontend applications:

• React + AWS S3
• Vanilla JS + AWS S3
• NextJS + AWS Amplify

This fragmentation made every team a universe in itself, each doing things its own way and solving the same problems many times over.

As a first investigation, I’ve attempted to identify the cons from our current usage of NextJS (with AWS Amplify), mainly because I believe that NextJS is a good tool to solving the kind of frontend problems that we have here at Inventa.

Some of the issues, at the time, that this approach involving NextJS and AWS Amplify presented:

• Next.js major version at the time (12) wasn’t supported so we were locked into 11.x.x
• Amplify creates all the AWS resources required for the deploy in a automatic fashion, the problem with that is managing all this resources was not trivial, some of it wasn’t ‘visible’ on AWS Console, neither clearly specified on billing section.
• At the time we’re doing some restructuring of our AWS management principles in order to optimize the billing and provide more uniformity to our resources and its configurations. This context also pushed us away from keeping Amplify as a solution internally.
• Also regarding the automatic management of AWS resources, we’ve had problems when trying to fine-tune the Cloudfront distribution invalidations upon deploying ( resulting in a wasteful deploy process from the perspective of billing).
• Required a lot of webpack-specific customizations

After taking the decision to abandon AWS Amplify as the default “deploy stack” we started reviewing which components of our current stack made sense to be kept and formalized and which should be, at least, not recommended.

Engineering Design Review Process:

At the company we have a design review process to solicit feedback from peers and domain experts on your proposed changes to the codebase in pursuit of some objective.

When is a design review necessary?

• A new project, service, or component is being created
• Anytime a data contract change is proposed for an established service
• Any significant change to an existing design or deployment
• Services/Patterns that could be used as cross-company approach

The result of this process is a series of documents and discussions structured in a way to enable all engineers to quickly grasps what’s being discussed and, if applicable, a high-level of implementation details.

Given that we were considering to change the default frontend stack for the company, this process was required to make sure that the solution was appropriated.

Throughout this article we’ll use the material from this process to present a what was researched.

Present/Future:

We’re in a continuous process of improvement so what follows is a description of one of those iteration processes.

Stack Core

One of the kept frontend technologies is Next.js, a React framework that offers complex features and provides ‘missing’ React features in a well-defined API. Next.js comes with a built-in router that is easy to use and helps maintain uniformity in the project’s folder structure. Additionally, Next.js offers multiple rendering strategies, including Static Generation, Server-Side Rendering, Incremental Static Regeneration, and Client-Side Rendering.

This flexibility allows us to choose the best rendering strategy for each project and ensures our code is well-documented and widely used. Next.js also allows for the easy creation of BFFs (using the `/api` route) which reduces the necessity of extra logic in a different project to ensure that we’ll be able to interact with a specific service.

(We also support static deployed web applications but they aren’t the focus of this article).

Internationalization

To streamline our internationalization efforts and ensure our applications are ready to be accessible to users that speak other languages, we use react-intl, a library related to internationalization. It helps us build applications that can be easily translated into different languages. It does that by using messages.

Simple message definition:

const presentationMessage = {
  description: 'A simple presentation message', // Description should be a string literal
  defaultMessage: 'My name is {name}', // Message should be a string literal
}

Simple message usage in React:

// Hook
import { useIntl } from 'react-intl'

const Component = () => {
  const intl = useIntl();

 return (<div>{intl.formatMessage(presentationMessage)}</div>)

}

// Component
import { FormattedMessage } from 'react-intl'

const Component = () => {
  const intl = useIntl();

 return (<FormattedMessage {...presentationMessage} />)
}

Styling

For building responsive and accessible user interfaces, we rely on ChakraUI, a flexible and easy-to-use component library. ChakraUI offers a range of pre-built components that can be easily customized to fit our specific needs, making it a valuable tool for building complex interfaces quickly and efficiently. When deciding we’ve compared it with Material UI, Chakra UI offers developers more flexibility in modifying CSS classes of exported components and layouts, which can save time and effort in customizing interfaces. On the other hand, Material UI provides a wider range of pre-styled UI components and may be more suitable for projects that do not require extensive custom styling. For our context as a company, the ability to do more custom styling but still keeping some sort of theme structure is really valuable.

Another option considered was Tailwind CSS (we have projects that used tailwind and still do); Chakra UI provides a more comprehensive solution, offering not only styling benefits but also a library of carefully created React components that takes care of all the details, such as accessibility, keyboard navigation, and component composition. While Tailwind CSS provides atomic CSS classes to help style components, Chakra UI comes with an intuitive, prop-based model of styling components, making it easier to learn and understand. We’re still maintaining the internal projects that use Tailwind and constantly learning its limitations and advantages, but for now the decision is to avoid it in new projects.

In a goldilocks-like situation, it seemed like MaterialUI was too much (in various senses but mainly: bundle size, opinionated components), and Tailwind CSS was too little (in the sense of requiring too much time/code to create simple React components that matched our current design). ChakraUI provided a more direct approach to do the necessary modifications without defining too hardly the component style.

To style and manage our React components, we use Emotion (@emotion/styled, @emotion/react) — a library designed for writing CSS styles with JavaScript. Emotion provides powerful and predictable style composition, it uses the same API as `styled-components` (pure CSS inside a template string syntax that gets parsed by a utility function). Another point for `emotion` was the fact that ChakraUI uses it under the hood.

import styled from '@emotion/styled'
 
const Button = styled.button`
  padding: 32px;
  background-color: hotpink;
  font-size: 24px;
  border-radius: 4px;
  color: black;
  font-weight: bold;

  &:hover {
    color: white;
  }
`

We also use Typescript to add type safety to our projects, which helps us catch errors earlier in the development process and improve our overall development experience. With Typescript, we get better IDE support, making it easier to work with complex codebases. What is currently ongoing is a complete transition to TS only, as we have a legacy codebase where the majority of code is in JavaScript, there is the necessity to support less strict linter rules to enable this hybrid environment.

Testing

And when it comes to testing, we’ve chosen Jest and React Testing Library as our preferred testing frameworks, one more focused on unit tests and the other on testing the interaction with React components. Jest offers a range of features that make it easy to write and maintain tests, such as a simple and intuitive API and code coverage analysis.

Given the need for robust and reliable frontend applications, it is important to use a tool that can ensure that the applications are working as expected, for that we are using Cypress as our end-to-end testing tool in order to write tests that simulate user interactions and provide features that can help us to deliver fast and reliable tests such as:

• Intercepting requests and mocking endpoints helping us reducing flakiness, speeding up test execution doing by api things that we don’t need to validate in each test and simplifying test setup
• Adding plugins to extend Cypress capability like cypress-grep that allow us to tag our tests and run a suite of tests based in tags like smoke and regression, Axe plugin for accessibility tests, Lighthouse Plugin to runs performance, accessibility, best practices, SEO and other useful plugins from a rich ecosystem.
• Cypress allows us to define custom commands that can be reused across our tests. This helps us simplify our test code and make it easier to maintain.
• Since our cypress tests live in the same repository that our application source codes we can reusing locators from the source code. With Cypress we can easily reuse the same locators from our applications to write tests. This means that we don’t have to duplicate the element locators in our tests, which can save a lot of time and reduce the chances of errors.

Data Fetching

Finally, we use Axios for handling HTTP requests in our projects. We’ve considered using the new `fetch` API available for Node 18 (the majority of our projects were using it), but the fact that it was experimental (Stability 1) made us choose a more consolidated solution.

Axios offers a unified API that can be used across both Node.js and browser environments, allowing us to easily reuse code regardless of the JS runtime. It’s a powerful tool that abstracts some of the more tedious aspects of other implementations, such as `await data.json()` from XMLHttpRequests. This makes it easy for us to extract parts of the logic to its own packages using the instance mode, creating “SDK-like” structures.

During the research about how to architect isolated(or at least easy to extract) solutions using Axios some options were considered:

Option 1

Overview

• Create a function for every endpoint that will be consumed in the application.
• Avoid using SSR to simplify the flow and implement the applications on the client side only.

Pros

• Simpler approach.
• Avoids having to know anything about SSR.
• Provides more flexibility for the engineer.

Cons and Limitations

• It does not scale so well if we have a bigger quantity of endpoints.
• Each endpoint is an independent part of the code.
• Specific logic that is shared cross API needs to be repeated, request timeout, auth logic, for example.
• Doesn’t leverage SSR.

Option 2

Overview

• Map the APIs used on the application using axios’ instance mode
• This way is like we have mini SDKs for the APIs and later on, is easy to extract this logic to a separated package and re-use it across projects.
• This approach can be used in conjunction with Next’s SSR for more simple pages.

Pros

• Easy to re-use.
• Enables us to create sensible defaults for all the APIs, such as error reporting, cache rules, custom headers.
• Easy to share API configurations across endpoints, such as timeout, authentication, custom redirect logic, status code validation.

Cons and Limitations

• More overhead when compared with Option 1
• Makes more sense when we have multiple endpoints to implement at once, implementing a single endpoint with this logic would be overkill.

Option 3

Overview

• This approach builds upon the second option, with a slight change.
• Use react-query as a wrapper around axios, providing cache manager behavior and enabling SSR data fetching to be simpler and consistent across client and server.

Pros

• leverages Next’s SSR.
• added cache layer that is SSR aware helps to fine-tune cache behavior.
• prevents unnecessary refetches, react-query manages fetch calls and tracks information staleness.

Cons and Limitations

• adds an overhead that can be confusing or misused introducing bugs.
• usage is not trivial
• as we would be promoting react-query to a core dependency, there would need to be extensive internal documentation to support the teams’ ramp-up when using this dependency.

As a default option we’ve chose the second option, keeping the third as a special case to be studied and formalized later on. The main reason behind this decision was a trade-off, we’ve recognized the value of the 3rd option but also knew it to be more complex to implement and maintain at our current stage.

Conclusion

At the end of the day, our frontend stack choices are critical to the success of our projects. By carefully selecting the right tools for the job, we’re able to deliver high-quality software that meets the needs of our clients and end-users alike. We’re proud of our frontend stack choices, and we’re confident that they will continue to help us build robust and reliable applications for years to come.

Did you like the challenge? How about joining us!? We are looking for software engineers who enjoy the state-of-the-art platform and application development, and want to help build a very reliable, observable, and scalable product in Inventa! Hit here to apply and, if you have any questions, don’t hesitate to call me on LinkedIn.

Building Inventa: Frontend Stack was originally published in Building Inventa on Medium, where people are continuing the conversation by highlighting and responding to this story.