Handling Remote Write: The Trade-offs Between Cost and HA

A few months ago, we decided to make a significant change in how we handle our metrics. Up until then, we had been collecting infrastructure and application metrics. We stored them in Prometheus local storage on Kubernetes PVCs, with a 7-day retention. For long-term storage, we kept metrics in an S3 bucket for 1 year and queried the data using Thanos. We wanted a managed solution for storing and querying metrics. So we decided to send all our metrics to Coralogix using the Prometheus remote write feature, aiming to deprecate our Thanos stack in the future.

It sounded simple enough. But, as we rolled out the change in our HA setup, we hit a challenge with our two redundant Prometheus replicas. We realized that we had two options. We could send the metrics from both Prometheus replicas. This would maintain our HA but double our costs in Coralogix and network traffic. Or, we could scale down to a single replica to save costs. This would mean losing our HA and its reliability. Since neither of the options worked for us, we searched for a different solution.

Prometheus-Elector: Maintaining HA While Sending Metrics from a Single Pod

We searched for a way to keep high availability (HA) while only sending metrics from a single pod. We then found a great open-source project, Prometheus-Elector, created by Julien Levesy. It uses a sidecar container for Kubernetes leader election. It applies different configurations based on the pod role: leader or follower. You can read more about it in this Medium blog post.

As we explored the project, we realized that it didn’t support the Prometheus Operator out of the box. So we decided to add that support ourselves. You can find the code in this GitHub repository and the Docker image here. In the next section, we’ll dive into the details of how this solution works with the Prometheus Operator.

Configuration Process: Generating Leader and Follower Configurations in a Prometheus Operator Environment

The Prometheus Operator monitors the Kubernetes API server for changes to specific objects, like ServiceMonitor and PrometheusRule. When it detects changes, it regenerates the Prometheus config and stores the config in a Kubernetes Secret, which is mounted to the Prometheus pods. To support the leader election mechanism, we created a separate secret. It contains only the leader’s config, which contains the remote write config.

The Prometheus Elector now watches for changes in both the Prometheus config that Prometheus Operator generates and the leader config Secret, which the user can update. In the follower pods, it will use only the config generated by the Prometheus Operator. For the leader, it merges the leader config with the base config generated by the Prometheus Operator.

This diagram summarizes the configuration process for Prometheus and Prometheus Elector.

Step 1: Prometheus Operator — Generate Configuration
The Prometheus Operator generates the config and stores it in a Kubernetes Secret.

Step 2: Secret Mounted to Prometheus Pods
The generated Secret is mounted to the Prometheus pods at /etc/prometheus/config/prometheus.yaml.gz.

Step 3: Leader Configuration Secret
A separate Kubernetes Secret stores the remote write config for the leader only. This Secret is also mounted inside the Prometheus pods at /etc/config_leader/leader.yaml.

Step 4: Config Reloader
The config-reload container watches for changes to the mounted Prometheus config file at /etc/prometheus/config/prometheus.yaml.gz. When a change is detected, it automatically generates a new config file at /etc/prometheus/config_out/prometheus_config.yaml It then reloads the Prometheus config to ensure it’s up-to-date.

Step 5: Prometheus Elector
The Prometheus-Elector container uses Kubernetes leader election. It determines if the pod is a leader or a follower. The pod’s role determines its action. It generates a new Prometheus config file and stores it in /etc/prometheus/config_out/prometheus.env.yaml.

Step 6: Prometheus Pod
On startup, Prometheus uses the Elector container’s config file to set its config.

So now that it’s clear how it works, it’s time to configure and deploy it!

Configuration Walkthrough: Integrating Prometheus Leader Election Sidecar into Your Deployment

In this section, we will guide you through the key steps to integrate Prometheus Elector into your Prometheus deployment using the Kube-Prometheus-Stack Helm chart. The examples are based on chart version 51.2.0.

You can find all the examples and the full Helm values file in this GitHub repository.

Step 1: Create a Kubernetes Secret for Remote Write configuration.
Create a Kubernetes Secret to hold the leader’s remote write config.

---
apiVersion: v1
kind: Secret
metadata:
  name: prometheus-leader-config-secret
  namespace: infra
type: Opaque
stringData:
  leader.yaml: |-
    remote_write:
      - url: https://your-remote-write-endpoint.com
        remote_timeout: 120s
        name: coralogix
        tls_config:
          insecure_skip_verify: true
        authorization:
          type: Bearer
          credentials: credentials
        follow_redirects: true
        enable_http2: true
        queue_config:
          capacity: 10000
          max_shards: 50
          min_shards: 1
          max_samples_per_send: 2000
          batch_send_deadline: 5s
          min_backoff: 30ms
          max_backoff: 5s
        metadata_config:
          send: true
          send_interval: 1m
          max_samples_per_send: 2000

Step 2: Mount the Secret in Prometheus Pods.
To do this, add the Secret to the volumes section under prometheusSpec in the kube-prometheus-stack Helm values file.

volumes:
  - name: leader-volume-secret
    secret:
      secretName: prometheus-leader-config-secret

Step 3: Create a Role and RoleBinding.
Next, create a Kubernetes Role and RoleBinding. This will grant Prometheus permission to access lease resources. In this setup, the ServiceAccount name for Prometheus is prometheus-elector-prometheus.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus-elector-role
  namespace: infra
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - get
      - list
      - watch
      - create
      - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus-elector-rolebinding
  namespace: infra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus-elector-role
subjects:
  - kind: ServiceAccount
    name: prometheus-elector-prometheus
    namespace: infra
  - kind: ServiceAccount
    name: prometheus-operator-prometheus
    namespace: infra

Step 4: Configure Additional ServiceMonitor.
To scrape metrics from the Prometheus Elector container, add this config under the prometheusSpec section of the Helm values file.

additionalServiceMonitors:
  - name: "prometheus-elector"
    endpoints:
      - path: /_elector/metrics
        port: http-elector
    namespaceSelector:
      matchNames:
        - infra
    selector:
      matchLabels:
        app: kube-prometheus-stack-prometheus
        release: prometheus-operator-elector
        self-monitor: "true"

Step 5: Add Additional Port for Prometheus Service.
To expose the leader elector metrics, configure an additional port in the Prometheus service section of the Helm values file.

additionalPorts:
  - name: http-elector
    port: 9095
    targetPort: http-elector

Step 6: Add the Prometheus Elector Sidecar Container.
Add the Prometheus Elector container as a sidecar. To do this, add it under the containers section of the prometheusSpec in the Helm values file.

containers:
  - name: prometheus-elector
    image: yakirlevi/prometheus-elector:1.0.0
    imagePullPolicy: Always
    args:
      - -lease-name=prometheus-elector-lease-remote
      - -leader-config=/etc/config_leader/leader.yaml
      - -lease-namespace=infra
      - -config=/etc/prometheus/config_out/prometheus_config.yaml
      - -output=/etc/prometheus/config_out/prometheus.env.yaml
      - -notify-http-url=http://127.0.0.1:9090/-/reload
      - -readiness-http-url=http://127.0.0.1:9090/-/ready
      - -healthcheck-http-url=http://127.0.0.1:9090/-/healthy
      - -api-listen-address=:9095
    command:
      - ./elector-cmd
    ports:
      - name: http-elector
        containerPort: 9095
        protocol: TCP
    securityContext:
      capabilities:
        drop: [ "ALL" ]
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
    volumeMounts:
      - mountPath: /etc/prometheus/config_out
        name: config-out
      - mountPath: /etc/config_leader
        name: leader-volume-secret
      - mountPath: /etc/prometheus/config
        name: config

VolumeMounts section defines two mount paths:

• config-out: This path will have the leader election config from the secret we created.
• config: All containers in the Prometheus pod share this volume.

Args section:

• lease-name: You can set this to any name you prefer for the Kubernetes lease resource.
• lease-config: This argument points to the leader election file we mounted to the container.
• lease-namespace: Set this to the current namespace of your Kubernetes cluster.
• config: This is the name of the configuration file generated by the config-reloader container.
• output: This specifies the configuration file that the Prometheus service will use.

Step 7: Add Prometheus Elector Init Container.
In the Helm values file, add the Prometheus Elector as an init container, under initContainers in the prometheusSpecsection. The init container must create the config file for the Prometheus container before it starts. Without this step, Prometheus would fail at startup. It wouldn’t find the config file.

initContainers:
  - name: init-prometheus-elector
    image: yakirlevi/prometheus-elector:1.0.0
    imagePullPolicy: Always
    args:
      - -config=/etc/prometheus/config_out/prometheus_config.yaml
      - -output=/etc/prometheus/config_out/prometheus.env.yaml
      - -leader-config=/etc/config_leader/leader.yaml
      - -init
    command:
      - ./elector-cmd
    securityContext:
      capabilities:
        drop: [ "ALL" ]
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
    volumeMounts:
      - mountPath: /etc/prometheus/config_out
        name: config-out
      - mountPath: /etc/config_leader
        name: leader-volume-secret
      - mountPath: /etc/prometheus/config
        name: config

Step 8: Override Config Reloader.
Due to the limitation of overriding the Prometheus configuration file using a simple flag in the Helm values file, we implemented a solution to change the output config file parameter used by the config-reloader. This enables us to pass the updated configuration to the Prometheus Elector.

To get the current settings for the config-reloader and init-config-reloader in your setup, extract the YAML of the StatefulSet (STS) created by the Prometheus Operator. You can do this using the following command:

kubectl get sts <statefulset-name> -n <namespace> -o yaml

This command will display the complete configuration of the StatefulSet resource. In the output, look for the init-config-reloader and config-reloader arguments.

Copy the args config frominit-config-reloader to the initContainers section. Then copy the args fromconfig-reloader to the containers section.

Here’s an example of what the configuration might look like:

containers:
  - name: config-reloader
    args:
      - --listen-address=:8080
      - --reload-url=http://127.0.0.1:9090/-/reload
      - --config-file=/etc/prometheus/config/prometheus.yaml.gz
      - --config-envsubst-file=/etc/prometheus/config_out/prometheus_config.yaml
      - --watched-dir=/etc/prometheus/rules/prometheus-prometheus-elector-prometheus-rulefiles-0
      - --log-format=json

initContainers:
  - name: init-config-reloader
    args:
      - --watch-interval=0
      - --listen-address=:8080
      - --config-file=/etc/prometheus/config/prometheus.yaml.gz
      - --config-envsubst-file=/etc/prometheus/config_out/prometheus_config.yaml
      - --watched-dir=/etc/prometheus/rules/prometheus-prometheus-elector-prometheus-rulefiles-0
      - --log-format=json

Step 9: Deploy and Verify
At this point, all configurations are ready. You can deploy the updated Helm values file and all the Kubernetes manifest resources.

To verify everything, connect to your Kubernetes cluster. Check that the Prometheus pod has all the required containers, as shown in the example below:

Then, run the following command to verify that an active leader is present.

kubectl describe leases <lease-name> -n <namespace>

In the output, you should see the current active leader, for example, prometheus-prometheus-elector-prometheus-1.

Finally, use the pod IP and port 9090 to connect to the web UI of both Prometheus instances (leader and follower). Verify that only the leader contains the remote write configuration.

Closing Thoughts

Integrating Prometheus Elector into our Prometheus deployment has been valuable. It solved our issue of sending duplicate metrics while keeping high availability.

But, we encountered a significant challenge. We could not override the Prometheus arguments through the Helm values. This limitation forced us to override the config-reloader args. It’s a workaround, but it may not be efficient long-term. This shows a need for improvement. A more efficient process could significantly reduce configuration complexity.

Prometheus Operator with Leader Election: Solving Duplicate Remote Write Metrics in HA Setup was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unlocking Cloud Savings: Yotpo’s Secret Sauce Slashed Spot Prices. Ready to Dive In?

The ‘Era of Spots’ is over, or so they say. If you’re already using spot instances, you may have noticed your savings start to dry up. Spots are becoming scarcer, the prices of spots are increasing and it seems like AWS is pushing customers toward switching to savings plans/reserved instances. But what can we do about it? Let’s explore the solution!

At Yotpo, we are proud to have 80% of our prod workloads on EC2 spot instances, the pinnacle of cost-saving excellence, but from February to May 2023, we witnessed an astonishing spike of nearly 30% (yes, you read that right, !!!) in our EC2 spot prices. Suddenly, the most cost-effective option became an expensive one.

In these times where reigning in costs reigns supreme, how did we tackle this predicament, you ask? The conventional wisdom points towards the usual suspects: fine-tuning services, optimizing resource utilization per service, and, of course, getting nifty with Kubernetes to tweak requests and limits. We gave it a shot, and it did help, but it was like trying to fit a square peg into a round hole. We needed something grander, something bolder.

Turns out, the bulk of the cost surge was due to the spot instance type itself. As described in “Graph A” above, the same spot instance on a different AZ came at a ~30% discount (!!!). Not only that, but we discovered we could embrace stronger instance types at a lower cost(?!?). Our optimization journey began, but it quickly became evident that manual adjustments made little sense in this ever-shifting landscape where prices fluctuate with demand. We craved automation.

The quest for an automatic solution led us down a rabbit hole. Should we craft our custom tool? or could we rely on one of the many cloud cost-reduction solutions available on the market? We embarked on an expedition, exploring tools like AWS cluster auto scaler, Karpenter, Spot.io, Granulate, Perfect Scale, Cast.ai, and many more (there are far too many). After rigorous digging, we found the solution that meets our needs.

Cast.ai proved to be much more than just a solution; it made a significant impact. The learning curve was steep, and it meant rethinking our fundamental assumptions, but it was worth it! With Cast.ai in our arsenal, we slashed our spot prices by a whopping 30%! Even in the face of Amazon’s price hikes, we now have an automatic mechanism that always keeps us on the pinnacle of instance optimization.

There’s more to the Cast.ai story. Alongside providing the tools we need for automatically selecting the best, most cost-effective instance and AZ, Cast.ai comes armed with a remarkable trick up its sleeve: a sophisticated bin-packing algorithm. In simple terms, this algorithm works wonders for your containers in Kubernetes. Think of it as an expert puzzle solver. It takes your pods and neatly fits them onto a single EC2 instance, making sure every ounce of computing power is put to good use. What’s fascinating is that this approach contrasts Kubernetes’ default behavior, which often spreads your pods across multiple EC2 instances, prioritizing availability over cost.

Cast.ai’s algorithm is optimized for larger machines, where spot prices are more budget-friendly. In the example below you can see workloads that have moved to c5d.18xlarge instance type, which provides lower cost with higher utilization ( we wouldn’t have thought to make this move) — it’s a smart move that ensures efficient utilization while keeping your expenses in check.

And let’s not forget the incredible Cast.ai team. They didn’t just understand our needs; they made it their mission to exceed them.

In times of economic slowdown, I assume we are not the only company out there that had to deal with this challenge, and I wonder:

Have you faced a similar challenge?

Did you find a different solution? A Better solution?

If you did, please share it in the comments! We would love to learn from you, and maybe even reduce the cost even more.🚀💰

How Could You Save 30% on Spot Prices in K8S? was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to the first blog post in our exciting new series, “Unleashing the Power of Module Federation.” In this series, we will dive deep into the journey we embarked upon to overcome two critical challenges in modern front-end development: sharing dependencies efficiently and automatically consuming changes made to “remote” components. Whether you’re a front-end developer, architect, or part of a development team seeking to enhance collaboration and accelerate development velocity, this series is designed to provide valuable insights and solutions. So, fasten your seatbelts and get ready to explore the requirements we identified, the innovative architecture we devised, and the obstacles we encountered along the way. Prepare to be amazed by the incredible possibilities of module federation!

A Challenging Path to Modern Front-End Development

As a member of the Front-End Infrastructure team at Yotpo, our mission is to deliver comprehensive front-end solutions to support the entire R&D organization. With a primary focus on empowering developers and enhancing development velocity, we strive to provide self-service front-end capabilities.

Yotpo’s journey in front-end development has been a fascinating one. We started with a single AngularJS / Angular monolith and gradually expanded into a multi-app, multi-framework company, adding React to our tech stack. This evolution highlighted the need for a robust cross-framework infrastructure to ensure seamless collaboration and increased development velocity.

To meet these challenges, we recognized the importance of creating tools, frameworks, and architectural patterns that enable our teams to develop and deploy applications with greater efficiency and agility. Accelerating the development process and enabling faster iterations became paramount, allowing our teams to iterate faster and deliver high-quality experiences to our users.

The Introduction of Cross-Framework Features Shared Dependencies and Web Components

To facilitate the development of cross-framework features, we made a significant advancement by introducing an innovative web-components-based design system. Web components are a standardized set of web platform APIs that allow developers to create reusable components encapsulating their functionality and styling. These components can then be used across different frameworks, providing a seamless and consistent experience.

By adopting web components, we unlocked a new level of flexibility in our front-end development. Developers no longer need to be constrained by the limitations of a single framework but could leverage the power of web components to build features that seamlessly integrate into applications regardless of the underlying framework. This approach promotes code reuse, reduces duplication, and streamlines development efforts when working with multiple frameworks within a single organization.

However, embracing new changes introduced to our design system posed significant challenges, especially in a multi-app architecture. Traditionally, each new version of an npm package, including our design system, required a time-consuming development cycle for each application using the library. This process involved opening pull requests, code reviews, quality assurance, and deployment through the continuous integration (CI) pipeline. With multiple applications relying on the design system, even a minor change in a shared component could result in a substantial development effort across the entire ecosystem.

This manual and time-intensive process hindered our ability to iterate quickly and delayed the deployment of critical updates. It became evident that we needed a more efficient approach to automatically consume changes made to components in our design system, reducing the burden on development teams and accelerating the delivery of updates.

When converting an Angular (or any other framework) component into a web component, it becomes bundled alongside its dependencies, including peer dependencies. This bundling strategy, while initially promising, can lead to unforeseen complications. Web components are registered in the browser as soon as they are imported, allowing them to be used across different frameworks. However, conflicts arise when multiple components attempt to register under the same name, resulting in the “DOMException: Failed to execute ‘define’ on ‘CustomElementRegistry’” error.

Consider a component from our design system, such as a “button.” This button component is used in both an Angular component and a React application. During the rendering process of the converted Angular component, an error known as “DOMException: Failed to execute ‘define’ on ‘CustomElementRegistry’” occurs. This error occurs when the Angular component attempts to import the button component file from its own bundle.

To provide a clearer picture, here’s a code snippet demonstrating the typical registration of a web component:

// Register the "button" web component with a custom tag name
customElements.define('button', class ButtonComponent extends HTMLElement {
  constructor() {
    super();
    // Component logic and behavior implementation
  }
});

In this example, we register the “button” web component using the customElements.define method. We assign the custom tag name 'button' to this component, which can be used in our HTML markup. The class definition for the web component includes only a constructor for simplicity.

Conflicts can arise when web components with the same name, such as the “button” component, are imported and registered in different parts of an application. This can result in the “DOMException: Failed to execute ‘define’ on ‘CustomElementRegistry’” error, highlighting the challenges of double registration in the web component ecosystem.

Seeking a Real and Permanent Solution: The Birth of Module Federation

Recognizing the urgency of finding a comprehensive solution, we established two key requirements:

1. Shared Dependencies: We needed a mechanism to share dependencies among components to avoid the double registration issue. This would not only resolve the immediate problem but also improve performance by leveraging shared resources.
2. Automatic Consumption of Component Changes: We desired a streamlined process that would automatically propagate any changes made to components built using our design system. Manual updates and npm package management across multiple projects were cumbersome and error-prone. We sought a more efficient and automated approach.

Enter Module Federation, an Intriguing Webpack Feature

Our quest for a real and permanent solution led us to module federation, an intriguing feature of Webpack. Module federation offered the perfect answer to our challenges: efficient dependency sharing and automatic consumption of component changes. We were particularly captivated by two capabilities: the ability to share dependencies and dynamically load modules that exist outside the bundle. However, we were fully aware that module federation came with its own set of challenges, requiring a deep understanding and significant effort for developers to master.

In the upcoming articles, we have an exciting lineup of topics that dive deeper into our module federation journey. One crucial aspect we’ll explore is ensuring seamless integration between the host and remote modules by tackling the challenges of sharing Angular’s injection tokens in the module federation ecosystem. We’ll discuss the adjustments and strategies required to ensure smooth compatibility and effective communication between the different modules. Additionally, we’ll dive into the world of RxJS within the module federation environment and uncover various challenges that arise when utilizing RxJS in this context. From managing observables across boundaries to handling asynchronous operations and communication between federated modules, we’ll provide valuable insights and practical solutions. Get ready to unravel the secrets of quality assurance and discover how we tackle these complex problems in our module federation adventure. Stay tuned for an enlightening exploration of these topics in the upcoming articles!

Introducing the Series: Unleashing the Power of Module Federation was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Let’s first start by stating the most important point:

Programming is not dead.

With the rise of low-code and no-code platforms, some have speculated that programming skills will become less important in the future. However, as software systems become more complex and the demand for custom solutions grows, programming skills remain a fundamental requirement for developers.

The challenge facing developers today is how to keep up with the pace of change and maintain high productivity levels while ensuring code quality. Writing boilerplate code, performing routine tasks, and dealing with unexpected errors can be time-consuming and frustrating, leaving little time for innovation and creative problem-solving.

This is where AI-powered tools come in. By automating repetitive tasks, improving code quality, and streamlining day-to-day tasks, these tools can help developers work more efficiently and effectively. In this blog, we will explore how AI-powered tools are transforming the development process and unlocking developer productivity.

Automating Repetitive Tasks

One of the biggest challenges developers face is the need to write repetitive code for every new project. This code includes setting up basic project structures, defining classes and functions, and handling common tasks like input validation and error handling.

AI-powered tools can help automate these repetitive tasks and free up developers to focus on more important work. For example, code generation tools can create project templates and generate code for common tasks like database connections, web services, and user interfaces.

In addition to generating code, AI-powered code editors can suggest code completion options and detect common errors. By automating these routine tasks, developers can work more efficiently and with fewer distractions.

Improving Code Quality

AI-powered tools can also help improve code quality and reduce errors. Code analysis tools can identify issues like code smells, anti-patterns, and potential security vulnerabilities, allowing developers to identify and fix them before they become a problem.

In addition, AI-powered testing tools can automate the testing process and generate test cases based on the code. This ensures that the code is thoroughly tested and reduces the likelihood of bugs in the final product.

By improving code quality and reducing errors, AI-powered tools can save developers time and help them produce higher-quality code.

Mindset Shift To Conversation-Based Searching

One of the ways that AI-powered tools are transforming the development process is through a mindset shift towards conversation-based searching. Traditionally, developers have relied on keyword-based searches to find the information they need. However, with the rise of AI-powered tools, developers can now use natural language queries to find the information they need quickly and easily. This not only saves time but also allows developers to focus on more important work. Additionally, conversation-based searching can help developers find answers to complex questions that may not be easily searchable with traditional keyword-based searches. As the technology continues to evolve, conversation-based searching is likely to become an increasingly important tool for developers looking to improve productivity and stay competitive in the industry.

The Human Touch: Reviewing AI Suggestions in Code Development

As AI-powered tools become more prevalent in the development process, it is important for developers to remember their responsibility to review the suggestions made by these tools. While these tools can automate repetitive tasks and improve code quality, they are not infallible and may make mistakes or suggest suboptimal solutions. Therefore, it is crucial for developers to review and validate the suggestions made by these tools before implementing them in their code. This review process not only helps ensure the quality of the code but also allows developers to improve their own skills and understanding of the code. Ultimately, developers are responsible for the code they produce, and by taking the time to review the suggestions made by AI-powered tools, they can ensure that they are producing high-quality, reliable code.

Wrapping Up

AI-powered tools are transforming the software development process by automating repetitive tasks, improving code quality, and streamlining day-to-day tasks. By freeing up developers to focus on more important work, these tools are unlocking productivity and enabling developers to deliver high-quality code more efficiently. As the technology continues to evolve, developers can expect to see even greater benefits from AI-powered tools in the years to come. From conversation-based searching to automated testing and code analysis, AI is becoming an increasingly valuable tool for developers looking to stay competitive in an ever-evolving industry.

Unlocking Developer Productivity: How AI-Powered Tools Are Enhancing the Development Process was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR

Are you tired of chaos? Unplanned work in your DevOps role? Do you want to increase your R&D organization’s velocity? At Yotpo, the data-driven approach helped us dramatically reduce unplanned work and increase development velocity.

We have managed to reduce the number of requests to DevOps in Half! How did we achieve it? By creating visibility and conducting lots of experiments until we achieved clarity.

Want to know more? Just scroll down…

1st Stage: Chaos

At first, like lots of organizations — We had a Slack channel where any person from R&D could ask any questions or make any requests to DevOps. This is the only channel we got requests in (no emails or Direct Messaging). Most of the questions/requests were not even in the DevOps domain, and there was no visibility. We asked ourselves:

• How many requests are we getting per day? Per week?
• How many requests did we manage to resolve per day? Per week?
• What was the area people were asking the most questions about?
• In which areas there are the most requests/questions?
• How long does it take to resolve a request?

We were in the dark. We had so many questions — but no way to answer them…

2nd Stage: Visibility

We started by tackling visibility with a cool tool called “halp.” It turned every conversation in our DevOps Slack channel into Jira tickets, allowing us to see how many requests we were getting and in which areas. We used this visibility to experiment with solutions. We gathered our DevOps leaders and looked at the data every week, and from this forum — lots of ideas were generated.

3rd Stage: Solutions

We started to experiment — every week, we defined an experiment and defined a KPI to see if the experiment worked or not. Some of them worked incredibly well, and some less so. Here are a few of the experiment’s examples:

Categorization of Requests

The first thing we wanted to have visibility on — is in which areas we get the most requests. In other words — where should we focus?

We started by using the “component” field in Jira, and manually went over all our requests to categorize them— this gave us a first clear picture of what is going on!

It was great and exciting! But it was not enough. We needed to see the trends, to see the impact of each change we made, and if the impact was positive or negative.

One of the challenges we faced is to have a mechanism that will make sure that every Jira we closed is categorized.

So the challenge was how to categorize every new request. At first, we asked the DevOps members to categorize items every week — it did not work. We thought about asking the developers to do it — but that did not work either. The only thing that worked is making the component field mandatory when closing a Jira. Once that was done, we could easily measure the impact of each change in future experiments.

Auto Answers Experiment

We saw that a lot of the questions repeated themselves, and the answers were also the same. So we composed auto answers based on keywords, and in this way, the response was sent immediately, without our involvement.

The KPI for this experiment was how many questions can be auto-answered in a given timeframe (week). The impact was less than what we imagined — 5.6% of the questions were auto-answered, but we were looking for something that would make an even bigger impact.

Roles & Responsibilities

Many of the requests we got were not in our domain, so we took the time to map our roles and responsibilities, and for every request that was not in our domain, we composed an automated answer directing the requester to the correct owner of the domain. Here are a few examples:

This reduced the load and provided a faster response for the R&D since they knew exactly who to contact to get a quicker response.

Priority Queue

In our Slack channel — it was FIFO, or better yet — the one making the most noise would be the first serve. Following the move to Jira, we defined the level of priorities and changed our queue from FIFO to a priority-based queue. Items were handled first by priority, and then by their age. We ensured that any requestor would have visibility into the requests queue and where they stood, any user can open our tasks queue and see exactly where s/he stands — see the picture below.

This enabled us to change our approach and stop with the “everything is always urgent” way of work that our organization was used to. Now the R&D developers know where they are in the Queue and have an ETA and SLA according to priority.

Self-Services

In any area that we saw repeating requests for, we thought about what self-services we can provide so that developers could do it themselves. We decided to use Backstage as our hub of self-service generators. Backstage enabled us to create multiple self-services in various areas such as S3, RDS, GitHub Actions’ connection to AWS, and more. With it, we enabled people from our organization to do what they needed to without waiting for us.

4th Stage: Clarity

Our efforts paid off! Over six months, we were able to cut the number of weekly requests in half, speeding the R&D cycle time & lead time by reducing dependency on the DevOps team, reducing context switching, and allowing the DevOps teams to up the focus on infrastructure development.

Next Stages

We are not done. Using the data-driven approach, we continuously examine the areas our developers need help with and make sure that they will have self-services for them.

Furthermore, we are now starting another initiative called “DevOpsClub” to give more freedom to the R&D teams. The DevOpsClub is a team of developers who are trained on DevOps subjects and given DevOps Permissions. They are granted 20% time allocation to do DevOps tasks and further increase their R&D group velocity — we will tell you more about it in another blog…

So, if you want to reduce chaos and unplanned work in your DevOps role, take a data-driven approach as we did at Yotpo!

I’m very curious to know:
Did you’ve faced similar challenges?
What solutions have you found to help you stay on track and focused?

Happy to hear your feedback in the comments!

From Chaos to Clarity was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Io 🌒

The year is 2399, and the human species has finished its colonization of Io, the 3rd largest of Jupiter’s 79 moons.

These colonizers’ teams combined humans, robots, and humanoid robots.

They handle scheduled and incoming tasks almost immediately, and due to enhanced brains, also simultaneously.

Today, as simplified as we’ve got in 2022, multi-tasking is a feature we’ve yet to develop completely.

Who is brave enough for this challenge?

Is your team working on high-priority features?

Are tasks piling up?

Is your backlog expanding and attention at its maximum?

If yes, keep reading this post and I’ll add some logic and balance to those important daily decisions.

Eventually bringing you closer to completing your tasks, while staying attentive to those other topics and channels.

Who’s in charge of being in charge?

Deciding which developer should be assigned to which incoming tasks is something we can’t assess the effect of on the Sprint.

Some questions take time to answer, meaning that some answers need research and former knowledge.

In the same way, some bugs take time to debug and fix.

A developer’s attention is distracted all of the time, juggling between reviewing code, answering questions, learning, and meetings.

The core reason we deliver or fail our schedules is derived from our time management and task assigning methods.

Without being overly dramatic, teams that keep the delivery keep delivering, and the roadmaps keep roadmapping. The method is fine, but can we try and change it a little bit?

I mean that most of us are kind of good, even without a plan to manage the unmanaged work.

So why have I asked you to consider changing the system?

Because our goal (as a human code machine), should always be to improve our delivery and raise questions about our working methods.

The Agile Sprint concept is okay, but it can be more than that.

Handling incoming and important work can still be relaxed and fun. Inspecting suspicious behavior, debugging failed CD tests, and helping another team’s developers / PMs / Support can, and should be, included in the Sprint’s workload. And we can do so without knowing when they’ll arrive or preparing extra time in advance.

Introducing the Santa role: who deserves a present and who deserves coal?

Say Hello’ ho’ ho’ to jolly Santa, he’s here to help us handle alerts and questions, making everyone else much more concentrated on their tasks.

Santa’s job is to:

• Provide peer support on questions from the Slack channels
• Ask questions and get answers in standard time, preventing time-consuming meetings
• Assist with incoming support questions and queries, which (when not answered) can sometimes be translated into unnecessary bugs
• Get a first look at incoming production bugs
• If possible, do the work, and not necessarily the one who it’s assigned to eventually
• Respond to CI/CD failed sanity tests
• Keep the team reachable and attentive

What’s the price?

Santa will take less work on the Sprint, many times he can work on less urgent developments, but all those features he’s providing make him very valuable — even when not assigned to the regular Sprint’s tasks.

What impact do we gain from this role?

Better handling of incoming calls from Slack channels, making our CS quieter, and not opening unnecessary bugs we’ll need to close (after investigating) later.

We all know each of our products, and how to raise their environment. Our CI/CD is much more stable — failed tests rerun and are checked almost immediately.

My team develops customer reviews widgets, injected into ~600k eCommerce shops, handling ~48,000,000 reviews daily. In a microservices world, we’re first in the chain.

I once heard from my uncle that with great traffic comes great responsibility…

He’s mistaken! He probably meant bugs, many questions on Slack channels, multiple dashboards, CD Monitoring, PM requests here, and designers’ requests there.

So we’ve been able to become an integral part of Yotpo’s R&D while being attentive and helpful to other factors. We’ve created new backend services and rewritten our widgets in new advanced frameworks.

Our development has made an impact, enhancing the user experience through performance improvements and new designs.

All that has been done with only 3 developers, with one of us on production support every Sprint.

Initiating the Santa role is a key milestone in every new developer’s learning process.

While a new employee is answering questions, overlooking the CD, and handling Santa issues hands-on, they’ll boost their learning process and gain confidence faster.

My personal impression

This role is not a completely new concept for me, as I’ve worked for companies with similar ideas.

But, IDK, maybe being raised on a kibbutz has made me a socialist. And now, when this responsibility passes between everyone in the team, it makes the working environment something I prefer even more.

As mentioned before, this was a key milestone in my professional advancement at Yotpo.

Conclusion

We didn’t come any closer to colonizing Io, but we did find a way of preparing for those alerts that we weren’t ready for.

The Santa role turns all those unplanned incoming issues into something we can adjust our time to (if needed) and develop. It enables the entire team’s better understanding of legacy code and how to debug it, as well as how to solve production issues in general.

There is no doubt that our industry will require us, developers, to handle immediate requests, and there is no way to plan 100% ahead.

But there are methods that can make developers work calmly, keep delivery at high volume and backlog stack small (as much as possible), and still leave us with enough time to grab drinks together and plan fun days.

Seems to be much better than moving to a forsaken moon 🌒

Let me know how your team handles these issues…

Santa role — how and why to introduce a bug-focused developer was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Are you worried about your team’s efficiency?

Do you miss your goals week after week?

Even if you make good use of your team’s time, this post is for you. I’ll share my experience on how and where I improve performance with a few practical steps and a unique mix of common time management approaches.

As an R&D Manager at Yotpo with over a decade of experience in leadership positions within different domains, I find that one of our main challenges as leaders is to identify, manage, and reduce non-efficient time — the most expensive and limited resource we have.

Let’s take a look at the main sources of “time leaks” that I’ve identified in my day to day, and what works for me when tackling them.

DISCLAIMER: It took me a while to discover which approaches work best, and I can promise that the list of solutions that didn’t work is much longer.

So, where does my time actually go?

When ranking the top three reasons for a “time leak” in my environment — “Integrations” is the leader:

1. Management tools
   The time we spend reporting progress and status on each of our tools separately
2. Services and teams
   Integration with services and\or teams, which is a mandatory condition for specific features or projects. For example, development of new API\functionality, a new flow between two services, and using infrastructure tools for team development
3. Support multiple R&D environments
   Dealing and maintaining multiple environments to release a cycle pipeline, such as Local, Development, Staging, User Acceptance Tests, Production, etc.

The second “time leak” factor is the Company Culture:

1. Meetings

A lot of long meetings, with too many participants and unclear goals, are in place.

The price we pay as a company for meetings is expensive relative to the price of mistakes or failures without all of those discussions.

We should be focusing on the need and culture for lean and focused meetings.

Stop here and ask yourself: How many repeatable meetings do you have every week? Are all participants in your meetings mandatory? How many status meetings do you have with the same people in the same week?

2. Emails

My inbox fills up on an hourly basis.

Am I the mandatory recipient for all of those emails? What is the percentage of emails that really have relevant data for me? How much time do I spend reading those emails during the day?

The last “time leak” factor is the Context Switches:

R&D has different stakeholders and consumers like Product Managers, Support, Business Development, Project Management, and more, they are all looking for outputs.

As leaders, we have to set priorities in a proper way, by refocusing our teams on the most important tasks. Every moment we are dealing with context switches and being out of focus is expensive time that we won’t get back.

What changes can we make?

The way I tackle and solve “time leaks” in my groups is a mix of some common time management concepts that I find work well for me:

Integration — management tools

I constantly encourage efforts for strong integration between existing management tools, in order to spend less time syncing between all of them manually.

For example, occasionally attending a team’s daily meetings has shown me that a lot of time is lost on maintaining two different tools, GIT and JIRA, opening new branches, code status, finding commits, and so on.

With a few webhooks, we were able to create a strong and efficient integration that gives high visibility without updating each tool separately — you can find more about it here.

We did the same for our Jira-Product Board and Jira-Slack that have native synchronization without specific maintenance.

Integration — services and teams

Set the integration part in the first phase of the feature, even in the POC.

Does your new feature require integration with a different team, new service, or new platform? Don’t leave those integrations for the last part of the feature, or I can promise you’ll be late with the delivery.

Rather, strive to implement and test the integration part from the beginning. So, approve the integration before you start developing the whole feature.

Integration — multiple environments

For many years, I was a prisoner of the concept of several running environments for a safe release pipeline. At Yotpo, we don’t use all those environments, we use production — one environment.

We develop locally, followed by an all test pyramid (unit, component, integration, contract). Sometimes we use feature flags to complete the tests in the production environment, which is the only environment we need to monitor, maintain, and support.

Meetings

An effective meeting agenda clearly states the meeting goals and discussion topics, and it only needs to be 30 minutes, and no more.

A clear definition of the agenda and goals helps team members get on the same page before, during, and after the meeting. It provides all the necessary information to set the team up for meeting success.

Here the slope is very steep; I constantly avoid sets or being part of repeated meetings. Instead I set an ad-hoc meeting when I need it, and ask the same from my colleagues.

I limit my meeting participants to 4–6 people and 30–40 minutes maximum, and I also challenge my colleagues when I get invitations for meetings that are over that time limit or number of participants.

In most cases, after this challenge, the organizer gives more thought to the most valuable stakeholders for the meeting.

The same approach applies to meetings without a clear agenda and goals. When the meeting doesn’t have those, the discussion will probably be out of the relevant topics’ focus.

The same goes for meeting minutes. I finish each meeting with an understanding of who is responsible for summarizing and setting the meeting minutes and monitoring the action items till done.

Context Switching

Be flexible, but do not hurry to change the plan.

I limit my time for catching up on new tasks and requests to only 30 minutes every morning. By heavily considering my responsibility to finish commitments, I manage to avoid context switching.

I encourage the teams to define golden tasks for the members, these are tasks that must be completed on the same day. This helps to set the focus on the task, but not on the working hours and daily noise.

For unplanned/new important requests, I do my best to find tactical and short-term solutions to keep the planning as is. The price of context switching between tasks is expensive and drags on after delaying our commitments.

Spam Email

Use emails to broadcast a message, don’t abuse it. Keep your inbox clean.

We spend hours every day reading and responding to emails, and to change this we need to take personal responsibility and have the right company culture.

I use Jira to issue new tasks, and Slack for direct messages to one or multiple people for a specific request — I don’t need to email them. I only use email to broadcast messages, taking care not to spam others with unimportant content.

Every email requires action on the same business day, and to avoid backlog I plan time for it:

• Important and urgent — I stop to consider it, then reply with thoughts or action items I’ve taken
• Urgent, but not important — I find a solution and reply with acknowledgement
• Important, but not urgent — I set a slot in my calendar to dive into it
• Unimportant and not urgent — delete, spam, notify sender

I use email rules and colors for dedicated subjects, senders, and roles to be more focused on priority.

Here’s a tip: Use one unique email color when you’re on the main distribution list, and a different one when you’re on ‘CC’.

Let’s sum it up

Time leaks are here to stay, so we need to look them straight in the eye and respond with an active approach to reduce the impact on our working environments.

The recipe is simple: be aware of the leak, measure it, and take action to reduce it.

Colorful calendars, discussions with team members and sprint retrospectives help me to measure where the time is actually spent versus original plans.

Now you should be ready to take an active approach to making those areas more efficient.

Be ready for efficient and high-velocity teams of happy and focused employees who understand your goals, priorities, and how much you appreciate their time.

Time leak — Kill it! was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Cypress has enormous capabilities, one of which is creating custom commands.

In this article, I’ll demonstrate how custom commands can simplify your test, ensure its readability, and show that there are clear boundaries between business and test logic.

If you, like me, are eager to understand Cypress and write less flaky tests, I am sure that you will find this post interesting. And if you agree with me that testing your test utilities is a must, keep reading to learn how to test your custom commands.

Cypress bits and bytes

Cypress has a great custom commands feature, which lets you create your test helpers and integrate them smoothly into the test structure. You can chain custom commands to increase the readability of your test.

Everyone can write custom commands. But not everyone can do it perfectly.

The mediocre approach

Imagine that you are creating a custom command which does something with an actual HTML element.

What’s wrong with that, you might ask?

• It makes the test less readable and less maintainable. The test maintainer must be familiar with complicated coding techniques (callbacks) rather than just writing simple assertions.
• Mix and match test logic and technical aspects of doing something with an HTML element. While assertions represent product requirements, our custom command might have nothing to do with a requirement. It might be related to test logging and therefore not really test logic related.

Don’t get me wrong. This would work just fine. But there is a better way of achieving the same outcome.

The PRO approach

Let’s define some requirements from our custom command:

• Should extract the HTML element by itself: By that, we would make using our custom command much more straightforward. Coding knowledge is not a must and tests could be written by others than developers.
• Must NOT change the subject: Well, Cypress is all about chains. We can combine most assertions to create a firm and readable test without needing to reacquire the same element multiple times. Even more so if assertions or commands are not changing the test subject after execution.

For more information, visit Cypress's introduction tutorial.

Now let’s see what our test would look like:

Without a doubt, the test would be much more readable.

Bonus — unit testing

If you got this far in the article, you probably agree that tests are essential. So why not test your tests?

Let’s see what a unit test of a custom command might look like:

What are we doing here?

1. We are using a static HTML page as our fixture. This is much faster than launching an entire application.
2. Using good old jQuery, we store a reference ($element) to an element with a selector (‘.element’).
3. We execute doSomthing on an element with selector (‘.element’) and then get a closer look at the HTML element ($el) returned from doSomthing.
4. We make sure that $el and $element are identical.

Conclusion

Cypress came to the world for several reasons — one of them is to make our tests more readable. That’s why we should do our best to write the tests that way.

On a personal note, I learned a lot just by looking at the Cypress code base. If you are looking for inspiration or knowledge about an open source you are using, then you should definitely investigate it. You will be amazed by the number of things you will find out.

Writing Cypress custom commands like a PRO was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Have you ever wondered why we spend so much time dreaming of the perfect love, while the majority of our waking time is spent in a job that we don’t even like? Looking for a job should be just as selective as finding love, since we’ll spend so much time there we should feel that it is our dream job.

When I started my product design career, I wanted to mainly work for small startup companies. I was confident that working for one would boost my career and my knowledge. And so it happened — I was hired by a small startup and was the only product designer there.

As with first love, I was eager to get started — I had butterflies in my stomach when I realized how much responsibility it came with. I imagined how my whole life would change — how as a single product designer I would be expanding my skill set, influencing the organization, applying UX methodologies and setting the tone of the product.

After the butterflies passed and the workload increased, I encountered the stress and lack of many skill sets that came with being a junior designer.
The work was agile and everything was happening so fast. There wasn’t enough time for UX research and resources, and I needed someone to consult or work with. The dream faded, and it made me pause and think about where I saw myself in the near future and how I could find my dream job.

Sometimes first love brings you insights, and then it allows you to pinpoint your next choice.

My checklist criteria

Just as you do before starting to date, I made a list of things that I wished to have in my dream job. After the start of the pandemic, there were a lot of open positions in the market, so I had to get focused on what I wanted.

I narrowed my list down to three things that were must-haves:

Mentorship
Because I lacked a product design team, I was looking for a company that had a design director level and a product design team. I craved collaboration with other designers, as well as a manager who was going to mentor and guide me through my role.

UX research
As someone junior, it was crucial for me to work for an organization with resources for various UX research methods, such as user interviews, data usage, usability testing, and A/B testing, in order to enhance my skills.

Progressive product
Since I’d already worked on a product from scratch, I wanted to work on a product that was more developed, had paying users, kept growing, and had extensive areas to deal with.

My first date with Yotpo

Although we have a lot of mutual friends, we first saw each other on a modern dating app — Linkedin. It was pretty clear to me that I wanted to apply for a job at Yotpo, and I felt their open role was the answer to my dreams.

I prepared for the interview and learned about all the products Yotpo has to offer, just like social media research before an exciting first date. I wrote down notes and questions that interested me, as it was important for me to understand the company’s approach to its products.

During the interview I asked many questions — I had to know that we were looking for the same type of relationship. It was a match — Yotpo hired me!

But with the idea of getting into a serious relationship, I still had quite a few concerns before I agreed to join the company. What if teamwork wouldn’t suit me after working as a single designer? Would I be able to achieve my career goals? And what if I didn’t have enough experience?

I got the job! Now what challenges would I need to face and how did I deal with them?

Just like when you’re starting to date someone and getting to know them, the company has a past and things that happened before I arrived. Little by little I started piecing things together and getting to know them better.

As at the beginning of any relationship, there were difficulties and challenges I had to deal with:

My manager, who mentored me, left the role
As a new designer on the team this was a real challenge — especially when mentorship was one of my goals. I had to learn about the different areas in the product, push myself in projects, participate in meetings more and set myself initial goals, all while being new to the team.

Sometimes I felt the Product Managers called me to join the process too late
Feeling like I was missing out on being a significant part of UX problem-solving forced me to step out of my comfort zone and improve my communication skills. So I communicated it to the Product Managers I work with and we started building a mutual language and processes that worked better for all of us. It was important to me to show them that they could trust me with more responsibility.

A mature startup has more people to persuade
Unlike in a small startup where there are fewer opinions, here I sometimes get objections to solutions I propose for projects. In order to improve my work and justify my position, I focus on strengthening my capabilities. I learn more about UX methodologies, I share more with my team to get feedback, I back up my solutions by using data, and more.

Although Yotpo has a design system, it is still in the early stages
It was initially challenging to work with and communicate the benefits of a design system to the dev team.

The first reason was that there was no awareness among the developers that there is such a library that they can use. The second reason was the library wasn’t fully developed, and sometimes it added a lot of work instead of making it easier.

To conquer this challenge, I started by explaining what the design system is to the development teams I work with. I focused on how to use it efficiently and effectively, and explained how important it is for a product company to maintain consistency with a mutual language among their various products. I also learned how to plan my work better. What’s coming up for development? When and how, do I incorporate/promote the components I require for the projects I am working on?

Since Yotpo is a relatively large company with several products, while working on different projects to get to know the product better, I am still looking for my area of ​​expertise and what my passion is.

Is Yotpo the one? Did Yotpo match my expectations?

When I look back at the beginning of my journey, I can see how much I’ve been through.

I realize that the perfect job doesn’t exist and that what makes it a dream job is dealing with every obstacle in the way and maintaining this relationship. I’m happy to announce that we are a couple now!

Today I am more independent in my role, I have confidence and I learn a lot from my teammates, which was also one of my goals. I have more clarity about what I am dealing with. I feel my opinion is heard, by better navigating gaps in communication. I know everything is a part of my learning curve, and I try to get better and learn something new every day.

I am glad to be in a place that looks broadly at the products and is not afraid to change focus when needed. I also get to see how Yotpo empowers and invests in its people with different kinds of workshops and various career development programs.

Both of us invest in each other, are aware of each other’s needs, and keep working on improving our communication. Now I can tell that all of these are the keys to a good relationship.

Swipe right for your dream job — from heartbreak to a perfect match was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Yotpo, we handle a lot of different data sets, like reviews, browser analytics, campaign statistics, and others, all originating from different sources like:

• user-generated content
• 3rd-party services, like CRM
• application data
• application metadata

We specialize in transforming this pool of information into valuable insights. We have various ways to do this: training machine learning models, using decision support systems, and sometimes simply putting the data back in the hands of the customer.

Because our company that is so data-driven, our role as Yotpo’s Data Infrastructure Group is to improve the developer experience when using big data.

We focus on reducing the cognitive load of adopting a hoard of new tools, democratizing access to the data, and making things as self-service as possible.

We see self-service as a force multiplier that allows us to focus on the fun stuff we engineers love, by delegating the day-to-day work back to the people who can do it best: the owners of the data.

In this post, I’d like to share some examples of how we went about taking a data infrastructure process and making it easier to adopt.

Batch data pipelines

One of the most common tasks of data engineering is to create and maintain batch data pipelines. We use those to run all sorts of back-office flows: from aggregating data for reporting, through importing data from external services (like CRM, HR, and others), to training ML models with newly collected data.

At Yotpo, like in many other big data companies, we chose to implement them using Apache Spark as the execution engine, and Apache Airflow as the orchestrator and scheduler — all running on top of Kubernetes.

This means that in order to add data into our data mesh, a developer/analyst will have to learn three new complex tools that are not part of their day-to-day toolbox. And that’s before talking about the transformation process, or the different storage targets available.

Let’s lower the bar

Instead of adding a new stack of tools, we decided to use defaults, except for the bare minimum:

• The data source(s)
• The data destination(s)
• The transformation to apply, defined in SQL
• When to run the pipeline
• How wide the pipe is, in terms of unit-of-scale

All that’s needed is a couple of YAML files to express these settings, and our custom wrappers take care of the rest. Once we got the basics covered, we improved the integration with Yotpo’s stack — throwing Slack, Grafana, Prometheus, and even cost monitoring into the mix.

Now, instead of our group fixing broken pipelines — we can focus on improving utilization and extending our infrastructure capabilities (and there are always new requirements :)

Here’s what it looks like:

Change data capture processes

At Yotpo, our data store of choice is mainly MySQL. While all is fine and dandy for running the application, it’s a whole different ball game when it comes to analytics.

This gave rise to a requirement of synching our applicative MySQL (and other databases) to the data lake, where it can be consumed and analyzed without affecting the user experience or the application performance. Once again we chose a commonly used solution: we use Debezium to convert the database’s binary log to Kafka events, that in turn are persisted to the data lake — providing a near-real-time replica of the application’s data.

To address database updates and deletes, we chose Upsolver’s service to complete our pipeline. It now looks like this:

Got a pipeline — now what?

Appetite comes with eating, and so a basic requirement with every new feature was to have its data synced to the data lake. As you can see, we have a lot of moving parts in this pipeline. Adding a new table to the data lake requires updating all of them. So how do we make things easier and less cumbersome? First, start with a well-documented procedure. Then — automate.

As mentioned earlier, we’d like to stick with our current tooling as much as possible. In this case, it means leveraging our automation server (Jenkins is our tool of choice). With the power of its DSL plugin, it’s quite easy to generate simple forms that in turn can run complex pipelines.

Over several iterations we automated away the different steps, starting from the most out-of-the-daily-routine tool (Upsolver) to the least (git). As the level of automation grew — so did the velocity of the teams using it.

Like with batch processing, this process is not perfect either. We would like to have a better way to track schema changes, and frankly, our monitoring could be better. But since our main focus is to get out of the way of the developer and make it less of a yak shaving — we consider it mission accomplished.

User tracking events

Like every other data-driven company that operates over the Internet, we’re really interested in what Yotpo’s users are doing. Which features are they using? Where are they spending their time? Why didn’t they complete that flow?

We use Segment.IO’s platform to collect data points to answer these questions, and then store them in our data lake for further analysis. It looks something like this:

As a data team, one of the features we found appealing is the ‘Tracking Plan.’ It enables you to describe and validate the schemas of the collected events. Unfortunately, having a UI that allows you to customize everything means there is a lot of room for error. Also, introducing another tool to a large user base has a training cost that we’d like to avoid.

Putting on blinders

Luckily, Segment has great API support and they love JSON as much as we do. Therefore, we decided that instead of using Segment’s UI we’d create a CI-based (Continuous Integration) process:

• We use Segment’s tooling to generate the events’ schemas
• We store all the schemas in a git repository
• Whenever you change or add a new schema, our CI runs and validates that the change matches our guidelines. For example:
  - properties must be camel-cased
  - they must contain a description
  - new types must be nullable
• We ‘Release’ the new schemas’ version by posting them to Segment’s API.

This way we can have more control over the changes being made, making them less error-prone. Not to mention, controlling the process through git enabled us to take advantage of all the tooling already connected to it.

This is a great example of how even a tool that was built for a data-intensive task sometimes needs to be customized and altered to fit an organization. Things that make sense in a development organization of a few dozens become a real problem when you scale to hundreds and more.

Wrap up

As you can see from these examples, there’s a lot to do in terms of improving the developer experience when it comes to data. Even the most polished SaaS tools can benefit from making them fit your current stack instead of the other way around.

While we’re all waiting for the data scene to mature and offer more ubiquitous and integrated ways to work with data, we’ll keep focusing on bridging that gap. We see that when we succeed, it gives a sense of ownership to the individual teams in the organization, removes friction and enables them to deliver value to Yotpo’s customers faster. And isn’t that what it’s all about?

DataOps Self Service: Yotpo’s war on grunt work was originally published in Yotpo Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.