I started hearing the term cyclomatic complexity more often recently, especially with AI tools writing more and more of the code around us. I came across a joke from Gunnar Morling about Claude writing code like it doesn’t care much about cyclomatic complexity, and it made me realize something: AI might be good at producing code, but not always good at keeping complexity under control. And we probably need to pay attention to the signals that show when the code is getting harder to understand.

Cyclomatic complexity is one of those signals. It’s a simple metric that tries to capture how complicated a piece of code is by looking at how many different paths it can take. A higher score means more execution paths, which, as you can imagine, makes the code harder to test. A lower score means fewer paths, which results in better understandability, and makes testing and maintenance easier.

If you want to get a bit more concrete, cyclomatic complexity comes from graph theory. The lines of code can be visualized as the following control flow graph:

The cyclomatic complexity is calculated as E − N + 2, where E is the number of edges and N is the number of nodes in the control flow graph. So, for the piece of code I shared, the cyclomatic complexity is 10 (19 edges minus 11 nodes plus 2). For quick mental math, you can simply think of it as the number of decision points plus one (the +1 represents the base path, which exists even though there are no branches in the code).

Although high cyclomatic complexity doesn’t necessarily mean the code is bad, sometimes it’s just the right solution for the problem. I went looking for a really complex piece of code in some open-source projects and found a method with a complexity of 40 in Apache Solr. My first thought was, “this must be messy,” but when I actually looked at it, it was pretty clean, just a binary serializer routing for different data types. I honestly couldn’t improve it without making it slower or harder to read. Some complexity comes from the problem you’re solving (you can’t avoid it), and some comes from bad design choices (you definitely should avoid it). Knowing the difference is far more useful than just blindly following a “keep it under 10” rule.

So what can we, as software engineers, do to potentially reduce cyclomatic complexity? Essentially, every if-else, for, while, or switch statement introduces a new path in your code’s flow. You can reduce complexity by breaking large methods into smaller, focused functions, using polymorphism or strategy patterns instead of long conditional chains, and leveraging lookup tables or maps where appropriate. It’s also useful to know the cyclomatic complexity when planning tests, because it helps you estimate how many test cases are needed to cover all possible paths through a method. By understanding the complexity, you can prioritize testing the most critical or risky paths first, ensuring better reliability and maintainability, though I still sometimes overlook a few cases on the first pass.

But here’s where cyclomatic complexity falls short: it treats all decision points equally. A deeply nested if statement inside a loop inside another conditional is much harder to reason about than three sequential if statements, even though they might have the same cyclomatic complexity score. This is where cognitive complexity comes in, a metric that tries to capture the actual mental burden of understanding code. We’ve all seen a method that technically wasn’t huge, but still took way too long to understand.

Cognitive complexity cares more about nested code, because that’s usually where things start to get harder to follow. A flat switch statement with 10 cases? Not that hard. Three levels of nested conditionals? Much harder to keep in your head. This is why that Apache Solr method with complexity 40 felt readable, it was essentially a flat structure routing different types, not a nested maze of conditions. The cyclomatic number was high, but the cognitive load was manageable.

Remember that joke about Claude not caring about cyclomatic complexity? There’s some truth there. AI tools are great at generating working code quickly, but they tend to write long, procedural functions that handle everything in one place.

Let me know if you’ve run into tricky high-complexity methods and what your approach was, and I’ll see you on the next one!

References

https://dzone.com/articles/what-exactly-is-mccabe-cyclomatic-complexity

https://seeinglogic.com/posts/cyclomatic-complexity-illustrated/

A recent task at work got me diving into Kubernetes, KEDA, Prometheus, and Helm, and it didn’t take long to realize I had a lot to learn about how these tools work together.

So, I built a simple app that takes daily activity data, saves it to a MySQL database, exposes a REST API, and generates summary reports. I deployed it on Kubernetes, used Helm to manage the deployment, Prometheus for monitoring, and KEDA for autoscaling based on metrics.

It turned out to be a fun project, and I learned a lot. In this post, I’ll walk you through how I built and deployed the app, and what I learned along the way.

Data

First, I had to access the data that had been recorded over the last couple of months, and came across this GitHub repository: https://github.com/tcgoetz/GarminDB/. This repository contains Python scripts for parsing health data and manipulating it within an SQLite database.

I only used JSON files containing my activity data for this sample project, but you could also use FIT files (think of FIT files as the raw, detailed logs that your Garmin watch or fitness device records during your workouts) or implement a streaming approach. To get started, follow these steps:

1. Clone the repository:

git clone https://github.com/tcgoetz/GarminDB.git

2. Install the required package:

pip install garmindb

3. Set up the configuration file by creating the config directory and file:

mkdir -p /Users/inceb/.GarminDb && touch '~/.GarminDb/GarminConnectConfig.json'

Then edit this file to include your Garmin Connect username, password, and the start date of the data you want to download and copy it inside the json that you created.

4. Run the following command to download your Garmin data and build your local database:

garmindb_cli.py --all --download --import --analyze 

This will store everything under ~/HealthData.

In the FitFiles directory, you’ll find both the original FIT files and their corresponding JSON versions.

Application

What I wanted this app to do:

• Use Micronaut framework for lightweight, fast startup and built-in support for dependency injection, HTTP routing, and metrics.
• Upload and ingest activity data through the /upload endpoint using JSON files.
• Persist the uploaded data into a database in a normalized format.
• Generate simple summary reports over a selected date range, showing total steps, distance, and calories.
• Collect Prometheus metrics such as the number of uploads, which are later used for monitoring and autoscaling with KEDA.

You can find the full application code on my GitHub: https://github.com/betulince/garmin-scaler

Exposing Metrics with Micronaut and Using Them with KEDA

To enable autoscaling based on custom metrics, I needed to expose application metrics in a way Prometheus could scrape them. Micronaut makes this pretty straightforward.

I enabled Micronaut’s built-in metrics and Prometheus exporter by adding these configurations in application.yml:

micronaut:
  application:
    name: activity-analyzer
  metrics:
    enabled: true
    export:
      prometheus:
        enabled: true
endpoints:
  prometheus:
    enabled: true
    sensitive: false

With this setup, the app exposes a /prometheus endpoint that outputs metrics in the Prometheus format.

In the application code, I used Micrometer to create a gauge metric tracking the number of uploaded activities currently being processed. This metric is updated whenever activities are uploaded.

@Singleton
public class ActivityMetrics {

    private final AtomicInteger uploadedActivitiesGauge = new AtomicInteger(0);

    public ActivityMetrics(MeterRegistry meterRegistry) {
        Gauge.builder("activities.uploaded.gauge", uploadedActivitiesGauge, AtomicInteger::get)
                .description("Current count of uploaded activities")
                .register(meterRegistry);
    }

    public void incrementGauge() {
        uploadedActivitiesGauge.incrementAndGet();
    }
    
}

And in the controller method:

@Post(value = "/upload", consumes = MediaType.APPLICATION_JSON)
public HttpResponse<Map<String, Object>> uploadActivityJson(@Body DailyActivityDto activityDto) {
    try {
        DailyActivity saved = activityService.saveOrUpdateActivity(activityDto);
        activityMetrics.incrementGauge();
        return HttpResponse.ok(Map.of(
                "status", "success",
                "activityId", saved.getActivityId()
        ));
    } catch (Exception e) {
        return HttpResponse.serverError(Map.of(
                "error", e.getMessage()
        ));
    }

}

KEDA connects to Prometheus, scrapes this metric from the /prometheus endpoint, and uses it to decide how many replicas of the app to run. When the number of uploaded activities exceeds a configured threshold, KEDA scales up the pods to handle the increased load. When the metric goes down, KEDA scales back down.

Deployment

1. I created a Dockerfile that builds the Java app into a container image, ensuring all dependencies and configuration are bundled for consistent deployment.

2. I wrote Kubernetes deployment and service YAML templates to define how the app should run and be exposed inside the cluster. Then, I wrapped these templates inside a Helm chart to manage configuration and versioning more easily.

The Helm chart includes two important files:

• Chart.yaml: This file contains metadata about the chart such as its name, version, description, and dependencies.
• values.yaml: This file holds the default configuration values for the chart, like image repository, replica count, service port, and custom settings for Prometheus and KEDA.

3. To monitor the application, I included Prometheus as a subchart dependency in the Helm chart and configured it to disable unnecessary components like node-exporter, and alertmanager for a leaner deployment.

4. I defined a KEDA ScaledObject resource that uses Prometheus metrics to scale the app pods dynamically based on the number of uploaded activities.

5. I used helm install command to deploy the entire stack.

• Deploy Prometheus on Kubernetes:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus

• Set up KEDA for autoscaling:

helm repo add kedacore https://kedacore.github.io/charts
helm repo update

kubectl create namespace keda
helm install keda kedacore/keda --namespace keda --version 2.17.2

I deployed KEDA in its own namespace to isolate its components (webhook, operator, metrics server) from the app workload.

• Package and deploy the app using its Helm chart. Make sure to provide the path to the chart directory:

• Make sure that the backend is deployed to the cluster:

To see all pods in all namespaces across the Kubernetes cluster:

Customizing Prometheus Deployment

By default, the Prometheus Helm chart deploys several additional components including Node Exporter, Kube State Metrics, Alertmanager, and Pushgateway.

If you want to disable these unnecessary components, you can add the following configuration under the prometheus: section in your values.yaml file:

prometheus:
  alertmanager:
    enabled: false
  kube-state-metrics:
    enabled: false
  prometheus-node-exporter:
    enabled: false
  prometheus-pushgateway:
    enabled: false

And to simplify deployment, you can include Prometheus as a dependency inside your own Helm chart rather than installing it separately.

apiVersion: v2
name: garmin-scaler
description: A Micronaut Garmin activity scaler app
type: application
version: 0.1.0
appVersion: "1.0.0"
dependencies:
  - name: prometheus
    version: 27.29.0
    repository: https://prometheus-community.github.io/helm-charts

After adding this, run:

helm dependency update ./helm/garmin-scaler

This downloads the Prometheus chart into the charts/ directory. Now, when you install your chart with:

helm install garmin-scaler ./helm/garmin-scaler

Helm will automatically install Prometheus as a subchart, applying the customized values (such as disabling the extra components). This allows us to manage both the app and Prometheus together with a single Helm command, without needing to install Prometheus separately.

Trigger rollout updates when needed

When I made changes to the deployment (e.g., updated image or config), I used kubectl rollout restart deployment <name> to restart the pods gracefully and apply the changes without tearing everything down.

Monitoring

Before anything else, I used the following command to make my backend accessible from localhost:8080:

kubectl port-forward svc/garmin-scaler 8080:8080

This lets me access the /upload endpoint to send activity data for testing.

After deploying everything cleanly, I checked the Horizontal Pod Autoscaler (HPA):

kubectl get hpa -w

Once I uploaded Garmin activity data, things started moving. The metric hit the target value (3), so KEDA spun up 1 replica. As more metrics were reported (because more activities were still being processed), the metric target increased. The current metric (6) is now twice the target (3), which indicates the load is growing, and HPA prepares to scale further.

And indeed, after a short delay, replica count increased to 2, matching the need.

KEDA works by observing custom/external metrics (in my case, a Prometheus gauge based on the number of unprocessed activities). The HPA doesn’t instantly spike to max replicas. Instead, it scales gradually based on the metric-to-target ratio.

Conclusion

Finally,

• Prometheus scrapes the app’s metrics, and KEDA uses them to trigger scaling events based on workload.
• By uploading activity data, I was able to observe KEDA scale the app up and down in real time.
• This project helped me understand how all these tools fit together to make autoscaling possible.

The shenanigans will continue in the next one. 🧙‍♀️

Serialization and deserialization are concepts I used for years before I truly understood how they work under the hood. Eventually, I took some time to dig deeper and make sense of it all. This article won’t go too deep into the internals, but if you often use serialization and still find parts of it a bit unclear, we’ll go through a few key questions to help make it easier to understand.

• What is serialization, and what problem does it solve?
• What are the common use cases?
• How does it work under the hood?
• How can you implement it safely and effectively?
• And finally, what does it look like in a real-world example?

What is serialization, and what problem does it solve?

Think of it like turning your favorite dish into a recipe.

• The pancake is the real, delicious object built from its ingredients (data).
• The recipe is the serialized version: a transferrable format that someone else can read.
• When another cook reads the recipe and follows the steps, they deserialize it into a new pancake (object) that looks just like the original.

But why do we need this?

When you’re working with data in a program, it lives in memory, inside data structures like lists, trees, hash maps, or one of many other data structures. These structures are built in a way that makes sense only inside the program that’s running. The object exists as bytes, and your program accesses those bytes through memory addresses (pointers 🤡).

Here’s the problem:

These pointers are only meaningful within the same process and machine. If you want to send this object over the network, save it to a file, or pass it between services or systems, those internal references won’t make sense outside your program. Basically, the object needs to be translated into a format that’s independent of its in-memory representation.

That’s where serialization comes in.

Serialization is the process of converting an in-memory object into a byte sequence or structured format (like JSON, Protobuf, or Avro) that can be stored or transmitted. The reverse process is called deserialization.

What are the common use cases?

Serialization is more than just converting objects into stream of bytes. It’s an essential part of your application especially when different components need to talk to each other, or store data, or exchange messages reliably. It’s basically everywhere.

Here are some common scenarios:

• When services talk to each other over HTTP or messaging systems, they send data in serialized formats like JSON or Protocol Buffers. For example, a REST API sends JSON responses created by converting internal objects.
• Data stored in memory caches or saved to databases and files also needs to be serialized so it can be saved and loaded later. For example, Redis and Memcached store data in a serialized form to reduce the load on databases.
• Events, which represent changes in state or commands, are sent as serialized messages through message brokers. For example, Kafka events are often serialized using Avro or JSON.
• Remote Procedure Calls (RPC) send method inputs and outputs between machines by serializing them. For example, gRPC uses Protocol Buffers to serialize these messages.

How does it work under the hood?

Runtime vs compile-time serialization

• Runtime serialization: This approach uses reflection (we’ll talk about reflection in another article), or metadata (data that describes other data) inspection at runtime to dynamically convert objects to and from a serialized form. For example, Java’s built-in serialization or Python’s pickle operate this way. They inspect the object’s structure on the fly and serialize or deserialize accordingly without requiring pre-generated code.

Metadata inspection means that the program at runtime looks at the metadata associated with an object to determine how to convert that object into a serialized form.

• Compile-time serialization: This method involves generating serialization and deserialization code ahead of time, usually during the build process. For example, in Protocol Buffers, the .proto definition file is compiled to generate code that handles serialization for the defined message classes.

Schema and compatibility

Schema-based formats define the structure of data explicitly, allowing forward and backward compatibility by tolerating missing or extra fields during deserialization. This schema acts as a contract between the producer and consumer of the data.

• Forward and backward compatibility: One thing that confused me was: why does adding a field to a class break deserialization in some systems but not others?
  When an object is serialized, its structure is captured, and expected to match what the receiving code knows during deserialization. So, if you add a new field to a class, and try to deserialize older data (which doesn’t have this field), you cause a schema mismatch.
  Now the schema-based formats come into play. Schema-based formats are designed to handle schema evolution gracefully.
  If your new code can read data written by an older version, your system is backward compatible. If the older code can read data written by the newer version, it’s forward compatible. For reliable systems, we usually want both.
• Tolerance to missing or extra fields: During deserialization, if a field is missing in the serialized data but expected by the schema, the deserializer can use a default value or ignore it, or if extra fields are present in the data but not recognized by the schema, they can be safely ignored without breaking the deserialization process.
• Efficiency and type safety: Because the schema is known ahead of time, schema-based formats like Protocol Buffers, Apache Avro, and FlatBuffers can serialize data more compactly and efficiently than text-based formats, and provide type safety, ensuring that the data conforms to the expected types defined in the schema.

How can you implement it safely and effectively

Available tools

While most programming languages offer built-in serialization tools (like Java’s Serializable or Python’s pickle), it’s usually not a good idea to rely on them, especially when working with data from untrusted sources. This is because these native serialization mechanisms can introduce serious security risks.

For example, in Java, when you deserialize an object using ObjectInputStream.readObject(), the system can create instances of any class on the classpath that implements Serializable. This process bypasses constructors, which normally run important setup and validation code.

But what exactly does that mean?

Skipping constructors means objects can be created in unexpected or unsafe states. Even worse, hackers can create fake serialized data that tricks your system into running harmful code. This opens up many security risks and is a common way attackers cause problems.

Because of these risks, it’s safer to use formats like JSON, Protobuf, or Avro. They store data in a clear, standard way that’s easier to check and harder to misuse. Plus, they work across different languages and make updating data easier without causing errors.

With native serialization, even small changes to your classes (like adding or removing fields) can cause deserialization to fail or behave incorrectly.

Here is a very basic implementation of JSON Serialization with Jackson:

Serialize to JSON:

Deserialize from JSON:

Performance trade-offs between formats

I’m not going to compare the performance of different serialization formats here just yet, and here’s the reason:

Performance really depends on what you’re doing.

Different formats perform better or worse depending on the size, shape, and usage of your data. For example, if your data is simple and flat, most formats will work just fine. But if your data is large or has a lot of nested structures (like lists inside objects), some formats like Protobuf or Avro will handle it much faster and with smaller size than something like JSON.

Performance also depends on the programming language you’re using and how often you’re reading or writing the data. Some formats work really fast in languages like Java, but not as well in others. Also, if you’re sending data over a network many times per second, speed really matters, and binary formats are better for that. But if you’re just saving data to a file now and then, formats that are easier to read (like JSON) might be totally fine.

Lastly, other things like compression, network speed, and how often the data is used can make a big difference.

There’s no one “best” format. It really depends on your specific situation, so the best thing you can do is try a few and see what works best for your case.

We will look at an example of this in the next chapter.

And finally, what does it look like in a real-world example?

To understand where serialization fits into the bigger picture, I’d like to consider a company like LinkedIn, which handles billions of requests daily, such as viewing a company page, checking network connections, or sending messages, across web and mobile platforms. If you’re a LinkedIn user, you’ve likely noticed how quickly these actions are processed, with minimal latency.

So how do they achieve that level of performance?

This brings us to serialization.

At LinkedIn, their internal framework, Rest.li, originally used JSON as the default serialization format. JSON is widely supported and human-readable, which makes debugging easier. But in high performance systems, these advantages are not enough.

JSON is a textual format, which makes it quite verbose and that means larger payload sizes, more network usage, and longer processing times. Even with compression tools like gzip, the compression and decompression steps add extra CPU overhead, especially in environments like LinkedIn’s where services are written in garbage-collected languages like Java. JSON also struggles with serialization and deserialization throughput, which becomes a serious bottleneck at LinkedIn’s scale.

To improve this, they needed something:

• Compact (to reduce payload size and network latency)
• Fast (for both serialization and deserialization)
• Widely supported across many programming languages
• Easy to plug into their existing Rest.li infrastructure

They explored several options but Protobuf (Protocol Buffers) came out as the best choice. Since Protobuf is binary-based, the serialized data is much smaller and faster to process than JSON.

In the end, the switch to Protobuf brought these benefits:

• Up to 60% latency reduction for large payloads
• 6.25% increase in average throughput per host for responses
• No noticeable downsides when compared to JSON

This example shows how improving serialization boosts performance, scalability, and user experience, especially in microservices. You can check out this article for implementation details.

Conclusion

We’ve explored the fundamentals of serialization, and I hope it gave you a clearer understanding of how it works and why it matters.

Thank you for sticking around this far, and I’ll see you in the next one. Cheers! 🧚‍♀️

References

• LinkedIn Integrates Protocol Buffers With Rest.li for Improved Microservices Performance
• Overview
• Efficient JSON serialization with Jackson and Java

This blog post is for you if you’re interested in setting up Kafka on your local system from its source code. Whether you’re considering contributing to Apache Kafka or simply want to understand how to build and run such a significant project locally, this guide will help you get started.

I ran into some issues while building it myself, so I’ve included solutions for common Gradle and dependency problems as well.

In the end, we’ll check if Kafka works by creating a topic and trying out sending and getting messages.

But before we get started, let’s briefly explore what Apache Kafka is.

Apache Kafka is an open-source distributed event streaming platform that is used to build real-time streaming applications and data pipelines.

When we say ‘distributed’, we mean that Kafka can manage a lot of data by sharing the work across many servers. If the amount of data or the need for faster processing grows, more servers can be added to the Kafka cluster. This way, Kafka can handle the extra load without slowing down. This distributed setup makes it highly available and reliable. It copies data to several servers, so if one server fails, the data can still be accessed from another server.

Kafka can be used as a messaging system to handle communication between different components of a distributed system. Imagine you have a big system made up of many different parts, like servers, databases, and applications, all spread out across different computers. These parts need to talk to each other to share information and work together smoothly. Now, think of Kafka as the messenger that helps these parts communicate.

Steps to Set Up Apache Kafka on Your Local Machine

• Install necessary software and tools:

1. JDK: Kafka is written in Java and Scala, so you’ll need the JDK installed. Kafka requires JDK 8 or later. You can download it from Oracle.

2. Gradle: Gradle is the build automation tool used by Kafka. You can download it from Gradle’s website or use a package manager like Homebrew on macOS:

brew install gradle

3. Git: Git is used to clone the Kafka source code repository. You can download it from the Git website.

4. Apache Zookeeper: Zookeeper is required by Kafka for coordination between nodes. Kafka includes a Zookeeper server, so you can use the bundled version for simplicity, but you can also download it separately from the Zookeeper website.

• Create a copy of the Apache Kafka project by forking the repository.

After this, the URL of the project will change to: https://github.com/<YourUserName>/kafka.git

• Clone your forked project in your local machine:

git clone https://github.com/<YourUserName>/kafka.git

• In your git bash, change the path to point to your repository directory:

cd kafka

• Build a jar and run it:

./gradlew jar

After the build is successful, remember that we need to run three main components in the following order to get Kafka up and running: first, Zookeeper; second, the Kafka server; and lastly, either the standalone or distributed server.

• Start the Zookeeper service:

./bin/zookeeper-server-start.sh config/zookeeper.properties

Hmm, but what is Zookeeper?

Zookeeper is like a manager for distributed systems (like Kafka). Let’s say you have a group of workers (servers) that need to work together on a big project. Zookeeper helps them stay organized and ensures they can communicate with each other efficiently.

But before successfully starting zookeeper service, I encountered two problems that needed to be addressed:

1. Classpath is empty.

Classpath is empty. Please build the project first e.g. by running './gradlew jar -PscalaVersion=2.13.14' from source code

Well, the build is already successful, isn’t it?

You might face this problem because of different reasons. If you look around online, you’ll see that some people resolved the issue by trying different approaches. However, in my case, the issue was with the folder name. The folder structure where you put the Kafka project shouldn’t contain any spaces in their names. After renaming the directory from “/open source” to /open-source” I could start the Zookeeper service successfully.

(Yeah quite foolish of me, I know)

2. SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”.

The link provided states that:

This message is a warning and not an error. It is reported when no SLF4J providers could be found on the class path. SLF4J requires a logging provider because it is a logging API and not an logging implementation. Placing one (and only one) of the many available providers such as slf4j-nop.jar slf4j-simple.jar, slf4j-reload4j.jar, slf4j-jdk14.jar or logback-classic.jar on the class path will solve the problem.

In the absence of a provider, SLF4J will default to a no-operation (NOP) logger provider.

So, when I checked the “dependant-libs-2.13.14” folder within my generated build folder after the build process, I noticed that only the slf4j-api library was present, but not its implementation. Therefore, I downloaded the implementation (I chose slf4j-simple.jar) from Maven Central and placed it in the same folder as the API.

This resolved the issue. It’s important to note that the versions must match. At first, I didn’t realize that the versions needed to match, so it didn’t work.

Now the Zookeeper server is running on port 2182.

• Start the Kafka server (run it in a new terminal):

./bin/kafka-server-start.sh config/server.properties

This step starts the Kafka server, which facilitates communication between producers (who send messages) and consumers (who receive messages). This server operates on port 9092.

Now the server is running on port 9092.

• Run Kafka standalone server:

./bin/connect-standalone.sh config/connect-standalone.properties

This step involves starting a server that can operate either in standalone mode or as part of a distributed system.

Standalone server: This means the server runs by itself on a single computer. It’s good for development, testing, or small-scale projects. The server handles everything on its own without talking to other servers.

Distributed server: In distributed mode, the server is part of a larger cluster of servers that work together. This setup is used for big projects and large amounts of data. The servers share the work and help each other, making sure the system runs smoothly and can handle failures.

• Create a topic:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 --topic local.kafka.setup.tutorial

• Run the producer in the terminal:

bin/kafka-console-producer.sh --topic local.kafka.setup.tutorial --bootstrap-server localhost:9092

This command starts the Kafka console producer, specifying the topic ‘local.kafka.setup.tutorial’ and the Kafka server’s address ‘localhost:9092’. This producer allows you to input messages.

• Run the consumer in the terminal:

bin/kafka-console-consumer.sh --topic  local.kafka.setup.tutorial --bootstrap-server localhost:9092

This consumer listens for the messages on the specified topic.

Now when you write a message in the producer, you should notice it showing up in the consumer:

This demonstrates the real-time communication between the producer and the consumer, facilitated by Apache Kafka.

• Use CTRL+C to stop running the application.

Conclusion

Whether you’re new to Kafka or want to explore it further, building it yourself is a great way to learn. Thanks for sticking until the end, and see you on the next one! 🌱

References

• Apache Kafka
• Getting started with contributing to Apache Kafka (Part 1): Build and run Kafka from source code

Hi all,

At the outset, if you want to read the explanation of the banner you can check it out from here. I bet you’ll associate yourself with the situation of houseguest who has high tech-savviness. Actually this is the reason why I wrote this blog post. It was a nice Friday morning and I was just trying to connect to my institution’s WiFi in the library to be able to study. I am using Fedora Linux 35 (Workstation Edition) as an OS and what I go through with Linux distros can sometimes be a real pain in the neck. And it was one of those times.

Joking apart, Fedora wasn’t the problem. The main problem was that I had no idea of some terms and I found it beneficial to learn and explain them here.

Let’s kick off with what you encounter when you type ifconfig on your terminal.

Here, br-918f101aafe9, docker0, enp3s0f1, lo, wlp2s0 are the names of the active interfaces on our system. Running ifconfig (interface configuration) command shows information about all network interfaces currently in operation. It allows you to configure your network interfaces.

Okay but what does “network interface” actually mean? It is a software interface to networking hardware. At this point, it can be distinguished into two main types, physical and virtual. While physical network interface stands for an actual networking hardware, virtual network interface doesn’t need to represent any physical device.

The physical network interface appears as eth0 which represents Ethernet Network Card whilst we can recognize the virtual network interface when we see lo which represents Loopback. There are different kinds of virtual interfaces other than Loopback like bridges, VLANs, tunnel interfaces and so on. Note that loopback is a special network interface that the system uses to communicate with itself.

enp3s0f1

Now let’s go back to the output of ifconfig. As I said above, eth0 stands for ethernet network interface but why my ethernet interface is named enp3s0f1 instead of eth0? That’s because of a change in how now udevd (systemd-udevd listens to kernel uevents. For every event, it executes matching instructions specified in udev rules) assigns names to ethernet devices. Now our devices use the “Predictable Interface Names”, which are based on:

• firmware/bios-provided index numbers for on-board devices for example eno1
• firmware-provided pci-express hotplug slot index number for example ens1
• physical/geographical location of the hardware for example enp2s0
• the interface’s MAC address for example enx78e7d1ea46da
• classic, unpredictable kernel-native ethX naming for example eth0

I explained this because my first trouble connecting to eduroam was when there was a command I had to type using the name of my ethernet card. And I was expecting to see something like eth0. Without further ado, eth0 or in my case enp3s0f1, is used for communication with other computers on the network and on the Internet.

lo

There’s no place like ::1

lo is a special virtual network interface called loopback device. Loopback address specifies the local computer. Your computer uses lo in some cases like:

• communicating with itself
• diagnostics and troubleshooting
• connecting to servers running on the local machine

When a network interface is disconnected (for example, when an Ethernet port is unplugged or Wi-Fi is turned off or not associated with an access point) no communication on that interface is possible, not even communication between your computer and itself. The loopback interface doesn’t indicate any actual hardware, but it exists so applications running on your computer can always connect to servers on the same machine.

If you didn’t get the joke at the beginning, ::1 is IPv6 loopback address which relates directly to the localhost host name meaning this computer.

wlp2s0

The relationship between eth0 and enp3s0f1, also exists between wlan0 and wlp2s0. wlan0 is the name of the wireless network interface on the system. It refers to a wireless local area network WiFi card. The name is broken down into two sections. The first is WLAN and the second is 0. WLAN is a wireless LAN, and 0 represents your card number. Because of the new naming scheme, we can encounter different kind of namings like wlp2s0.

Basically, the first two letter describe the type of interface. ‘wl’ for wlan, ‘en’ for ethernet. The following code is often a description of the physical placement of the device in your computer — p2 is likely PCI bus 2, and s0 is likely slot 0.

We can reveal if an interface is virtual or physical by examining the result of ls -la command run under the /sys/class/net directory:

You can see that wlp2s0 is not a virtual interface.

docker0
docker0 is a virtual bridge interface created by Docker. This bridge creates a separate network for docker containers and allows them to communicate with each other. It randomly chooses an address and subnet from a private defined range. All the Docker containers are connected to this bridge and use the NAT rules created by docker to communicate with the outside world. Docker will be the topic of another article.

br-918f101aafe9

As you can decipher from the first two letters, br-918f101aafe9 is a bridge networking interface.

A Linux bridge behaves like a network switch. It forwards packets between interfaces that are connected to it. It’s usually used for forwarding packets on routers, on gateways, or between VMs and network namespaces on a host as you can see from the picture above.

Bridging your network connection is a handy method for sharing your internet connection between two or more computers. It’s useful if you can’t buy a router with more than one ethernet port, or if you’re an university student in a dorm room with limited ethernet jacks and no router.

Now let’s dive deep into interface details.

Here you can see,

UP: flag causes the interface to be activated.

BROADCAST: indicates that interface is configured to handle broadcast packets, which is required for obtaining IP address via DHCP.

MULTICAST: sets the multicast flag on the interface. It indicates that this interface can handle multicasting.

mtu: maximum transmission unit. IP datagrams larger than MTU bytes will be fragmented into multiple Ethernet frames.

As an example 70:4d:7b:45:fd:30 is hardware address of the ethernet interface (also known as MAC address).

inet: is IPv4 address assigned to the interface.

netmask: indicates the TCP/IP address where two devices can tell if they are on the same network.

txqueuelen length: sets the length of the transmit queue of the device.

RX bytes: is a total number of bytes received over interface.

TX bytes: is a total number of bytes transmitted over interface.

RX packets: is a total number of packets received.

TX packets: is a total number of packets transmitted.

Besides all the advantages we gained from ifconfig, ip command substituted its place by dint of capability to do a wide range of tasks. For example:

• Displaying/modifying interface properties.
• Adding/removing ARP Cache entries along creating new static ARP entry for a host.
• Displaying MAC addresses associated with all the interfaces.
• Displaying and modifying kernel routing tables.

While ifconfig mostly displays or modifies the interfaces of a system.

I hope this post will be a fresh start for my future posts on the topic of computer networks. As always, thank you for reading so far!

References

• Linux ifconfig command help and examples
• BridgeNetworkConnections
• Demystifying ifconfig and network interfaces in Linux

In this blog post, I will explain how I started to make new contributions after the article I published with the name of “My Apache Solr Contribution Journey from UI Improvement to Bug Solving” 3 months ago. I will explain how I added the screen where you can add parameter sets to Solr on the Admin UI and the section where you can use these sets in your queries on the query screen.

In order not to get lost in this pretty long blog, where I will make a lot of explanations about the improvements, I can divide them into the following headings:

• Where on earth did this paramsets feature idea come from?
• How did I start?
• What did I do during the development process?
• Final results

I want this article to cover the resources I used and the path I followed in the whole process. I know that if you proceed with a proper plan, you will contribute a lot to both yourself and the project.

Here you can access:

• The opened issue in JIRA for the development:
  https://issues.apache.org/jira/browse/SOLR-15853
• README.md file including the example shown in this article:
  https://github.com/apache/solr/tree/main/solr/example/films
• Pull request:
  https://github.com/apache/solr/pull/594
• The Request Parameters API documentation:
  https://solr.apache.org/guide/8_11/request-parameters-api.html

Where on earth did this paramsets feature idea come from?

It all started with this message from one of the Solr contributors, Eric Pugh, who did a review on my previous commits:

“If you feel ambitious, I’d love to see a basic UI for editing your ParamSets added to the Admin.
https://nightlies.apache.org/solr/draft-guides/solr-reference-guide-main/request-parameters-api.html#the-request-parameters-endpoint
I’d be super happy to review that PR.”

I had never used parameter sets before, I had never even heard of it. After examining Solr's documentation, I discovered that it is a super-powerful feature. And of course, I wanted to contribute and told Eric about it. He started by opening an issue in JIRA and explained to me in general terms what the parameter set does.

Here is how we can describe it briefly:

ParamSets are a great way of organizing your parameters into an algorithm and then referring to it by name. I’ve used it to drive A/B testing, or just to simplify my experimentation.

How did I start?

If you check you will see that there are not many sources other than Solr's documentation and films example under the /apache/solr/tree/main/solr/example file. So firstly I started by reading the Request Parameters API documentation and trying the set, delete, and update commands on the collection I created.

Before moving on, let’s summarize what a paramset is and what the main thing is to do with this feature.

You can do everything you see below with paramsets:

• To avoid frequently editing your solrconfig.xml to update request parameters that change often.
• To reuse parameters across various request handlers.
• To mix and match parameter sets at request time.
• To avoid a reload of your collection for small parameter changes.

The request to set or update request parameters is sent as a set of maps with names.

"set":{
    "myFacets":{
      "facet":"true",
      "facet.limit":5
    }
}

In the following example, we are setting 2 sets of parameters named ‘myFacets’ and ‘myQueries’ by using Postman.

These objects can be directly used in a request or a request handler definition. When making a request, the useParams parameter applies the request parameters sent to the request. This is translated at request time to the actual parameters. For example:

http://localhost:8983/solr/techproducts/select?useParams=myQueries

And I should indicate that it is possible to pass more than one parameter set in the same request. This information will be useful in the next sections. For example:

http://localhost:8983/solr/techproducts/select?useParams=myFacets,myQueries

So far we have seen what a parameter set is and how it is used. The lack of a UI in the admin tool makes it a very opaque feature to use. You have to use Postman or a tool like that to manage this. This is exactly how the paramsets feature that we will add to the Admin panel will work. So if I need to divide the feature into two parts:

1. A screen will be added to the admin panel where paramsets can be added, updated, deleted and all paramsets belonging to this collection can be seen. (This was the desired feature at first, then my mentor Ahmet Can Kepenek said it would be cool if we could just use these paramsets in the query, then I suggested this idea and it was approved, so the second item below was created.)
2. The section where these parameters can be listed and used while sending queries will be added to the query screen.

After spending some time understanding how paramset works and brainstorming what I can add more, I decided that I could start by adding the “paramsets” section to the Collection-Specific Tools on the Admin UI.

And now it looks like this:

To be honest, I found and created the icon next to the name of “Paramsets” from the first site that came up, just so I could see something there and be happy, but then I liked it very much and I didn’t change it! 😸

What did I do during the development process?

I will not talk about the whole process and the technical details under this title because I have already opened my final PR by making many changes as a result of the first review and the feedback. If you want to view the commits, you can check the GitHub link I shared at the top.

If I need to give an example, in the PR I opened before the first review, the paramsets were displayed as a list on the query screen and only one paramset could be selected, but since more than one paramset could be selected for the query; I was asked to turn this list into a multiple-selection drop-down menu, which I provided later.

For the Paramsets screen, I created these three new files: paramsets.html, paramsets.css, and paramsets.js. In the index.html file I added HTML and CSS files and the “Paramsets” as a menu item:

• For paramsets.css file:

• For paramsets.js file:

• Inside the core-menu section:

• Inside the collection-menu section:

In paramsets.js file I wrote the controller like this:

Then I added the following code snippet in app.js file:

In order to have a get and submit object that I can use wherever I want in the application, I wrote a factory method in the services.js file to stick with the rule of make a more robust application in parallel with the purpose of the factory method, and I defined the get and submit that I used in the paramsets.js file here:

An example of how I used “submit” in paramsets.js:

In general, these are the old-timer code snippets in the PR I opened. And the final screen looks like this:

With the provided convenience, if you want to see a sample json for the set command and copy-paste it to modify&save you need to click the “Show sample json!” button, and then you will see the following on the screen:

Let’s originate from the films example and add algo_a and algo_b as parameter sets. From this point, I suggest you read the films example because I’ll not mention all the detail here. At the end of this blog post, you will witness the use of this example on the Admin UI with the addition of the paramsets feature.

After clicking Submit Paramset button you can see that our parameter sets are added in params.json successfully.

⚠️ The parameters are stored in a file named params.json.

On the right side of the screen, you can see the added paramsets:

This was the screen where we played with paramsets. Now let’s move on to the section where we will use these paramsets, namely the query screen.

On this screen, you can see all the parameter sets you added on the Paramsets screen in the drop-down menu due to the changes made in the query.js file.

First of all, I used the controllerAs syntax in the query controller:

To provide multiple choices of the paramsets in the UI, I used a ui-select drop-down that supports search, filter, and multiple select features:

In this section you can search for your paramsets, you can choose them or delete them by clicking the cross given next to your chosen param which corresponds to the delete action:

In the code snippet below, I provided the join action for multi-chosen paramsets.

After clicking the Execute Query button, you can see the successfully assigned paramsets to the useParams parameter in the URL.

Final Results

As a result, you can see that we reached the final result of the films example. If I have to list the things to do in order with the parts that are not the subject of this blog:

1. Create a new collection. Add field types and fields. Then feed the collection with the data given from the films sample.
2. Add the algo_a and algo_b parameter sets with the special query parameters they contain using the set command.
3. Use these parameters in queries one by one, and then multiple choice.
4. See that it works correctly!

If you look at the screenshot below, you can see that we can use the paramsets as we want and it gives us the same results as in the example of the films.

Before I finish my article, I would like to thank my mentor Ahmet Can Kepenek, who supported me and helped me at every stage, and whom I could always ask my questions without hesitation. And I would like to thank Eric Pugh, who never hesitated to give me credit during the review and gave me ideas for this new feature.

I look forward to seeing you in the next contribution story! 🚀

References

• Request Parameters API
• solr/solr/example/films at main · apache/solr
• chorus/007_organize_algorithms_using_paramsets.md at main · querqy/chorus

“Without a transition, a change is just a rearrangement of the furniture. Unless transition happens, the change won’t work.” — William Bridges, Transitions: Making Sense of Life’s Changes

In this article, I will talk about what it feels like to do an internship with a company as a senior student and start working there as a part-time developer and step into the corporate world before finishing school. I will tell you how my experience has caused a paradigm shift in my world.

I consider my last six months of experience as a transition. All the essential information given in the school and all the technical words I heard during my education, such as agile environments, software development processes, meetings, teamwork, pairwork, know-hows, big projects, deployments, applications, million, billion, quadrillion lines of code, stress management, product management, problems related to self-confidence, etc. This experience has been highly enriching for me regarding technical and soft skills.

It is a great blessing to have the chance to be directly involved in the team’s daily tasks. For example, they could’ve easily given me tasks that had nothing to do with the team’s primary mission and said they were necessary to improve myself. And yes, technically, I would have improved in the end. However, I wouldn’t be able to get the company culture.

Prior to this internship at Segmentify, I wasn’t used to teamwork and always preferred to work alone. However, now I can clearly see that this experience has changed me for the better. Over time, constantly talking to others, asking questions, examining our shortcomings, and solving problems together led me to see how cool teamwork is!

On the technical side, during my internship, I contributed to the research and analyses made during the development process by performing various tasks within the Search Team, which works to develop Segmentify’s Searchandising product. With the help of the tasks regarding this project and my mentor, I have gained a comprehensive understanding of Solr — an open-source, highly flexible search engine that is a part of the Apache Lucene project.

My Solr journey that started here encouraged me to contribute to this open-source Solr project after becoming a Junior Developer. As a result, the code I committed last week for UI development was merged! You never know what a 20-day internship experience will bring.

In addition to the assignments given to me during the internship, I also wrote daily reports to my mentor about what I did, learned, and researched. Each week I was given a new research topic: I’d do my research and write an essay with what I learned. This way, I gained extensive knowledge about DNS, git and GitHub, Continuous Integration, Jenkins, fuzzy logic, etc.

This is how my journey of blogging started. The benefits of regularly writing are innumerable to me. First and foremost, the knowledge ı’ve gained has flown away and will always be a record of it somewhere on the Internet. Secondly, I’m the one who wrote those articles that people were curious about or interested in!

Of course, there were times when I got stuck, couldn’t make progress, and lost my motivation, but now I realise that these times can be overcome by communication, asking for help, and doing the proper planning, not by pushing myself down even further.

As a result of my software engineering internship at Segmentify, I took part as an observer at every stage, from analysing the points that need to be developed in software to the time it was developed, tested, and deployed. As a result, I have gained experience figuring out the problem, the exact cause, how to solve it, the steps to be taken and how to plan while following this path. For example, I learned how to do indexing in a search engine by meeting with Solr, creating configset and collection that must be done before loading data. I also learned how to add a field-type, field, data, define a synonym, etc. Afterwards, I have performed an analysis by calculating how much time-saving the idea, which provides efficiency in terms of time during product feeding from Ignite, will save time if implemented. Additionally, I’ve observed and gained experience about the discipline and culture in business life on office days.

And that’s how my internship went. And when they asked me if I wanted to continue working together, of course, I wouldn’t have said no! So now I’ve been working at Segmentify for four months. I’m currently learning how to solve issues, improve, and examine the connections between Segmentify’s different products. My job also involves sending queries from Solr, learning about applications in the search team and preparing documents. In addition to that, I sometimes need to do something about Linux servers. Still, not only that, but I also get the opportunity to listen to what other teams are dealing with. I believe this is a very effective method to keep the connection and communication between teams alive.

And every day, even if I don’t have crucial and urgent tasks at hand, I feel that I am preparing myself for life as an engineer in the business and corporate spheres after school. I see this when my teammates talk about their daily tasks or when I attend their presentations, or when I spend time debugging codes just for the sake of trying, or when I prepare the documentation for an improvement that has been made after I test it.

I have seen that the knowledge gained at school comes to life in a company environment by fulfilling the given tasks and participating in projects. Therefore, even if there is a change, if there is no transition, the incoming information stays where it is, indicated by the quote in the introduction.

The most valuable thing I’ve learned on this journey at Segmentify is that you should join the dance if you want to learn.

Thank you for taking the time to read this post!

My Segmentify Journey: From Intern to Part-Time Developer was originally published in Segmentify Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Contributing” to an Apache project is about more than just writing code — it’s about doing what you can to make the project better.

This article shows you how cool it is to contribute to open source by explaining the process until my code is merged in a detailed and straightforward way.

I improved Query Screen handling of raw query parameters. I changed the appearance of the “Raw Query Parameters” section in the Query Screen, added +/- buttons to enter query parameters one by one, and fixed one of the Solr 9.0.0 version’s bug, which is the query form was getting submitted two times each time you click the Execute Query button.

You can access the opened issue for this improvement from JIRA.

You can check out the pull request from GitHub.

First of all, let me explain where this improvement idea came from. We are constantly using Solr at Segmentify, where I am currently working as a junior developer in the search team. During the developments, to learn the source of the problem, we perform analyses by repeating the request sent to Solr from the website using the search product. When I say we repeat the request, I mean adding the values of the parameters in the outgoing request to fields of query screen such as q, fq, sort, fl, raw query parameters and then manipulating them, for example, adding, deleting or editing the values there.

But we had a problem while doing this; sometimes, we were entering very long queries in the raw query parameters, and due to the nature of this section, it was necessary to separate each value using either “&” or “\n”.

group.field=category_orig_TR&group=true&group.limit=16

And, for example, when we wanted to change the first value, we had to go back one by one since this raw query parameters field is not expandable. So we couldn’t see all the parameters together as a whole.

On Query Screen, the raw query parameters section was looking like this:

As you can see, the “q” field is expandable, which allows us to see all the query parameters written in the q field together. Therefore, I first created an issue for improvement in JIRA, thinking that if the raw query parameters field is made into an extensible structure like the q field, it will provide ease of use. I added a description like this:

“The current Query Screen doesn’t have a structure that allows to enlarge the text box area and see all of the query parameters when long parameters are entered, for example, in the “raw query parameters” or “fq” field. In addition, if the line separating between the form and result regions in the div tag in index.html were resizable, it would be an easier way to use Query Screen.”

Then I added raw query parameters inside textarea instead of input tag in the query.html file, and I set it vertical from query.css, committed it and opened a pull request with the issue number I opened.

First, I did the same for the fq field, too, but after the following feedbacks, I decided that there was no need for such an improvement because there were +/- buttons in the fq field and the parameters were entered one by one. So I reverted the change. And the following comment came for the raw query parameters:

“I think it would be nice to have a + / — list for raw params like we have for fq … perhaps with a toggle between a list of textboxes or the textarea added to this PR? The textarea is an improvement for sure, but with a long list, it might still be better to separate each parameter in its own line.”

It made a lot of sense to me, and I added +/- buttons for the raw query parameters, similar to how it was done for fq, and I checked until the request was made and the result was received, and after I made sure that everything was working correctly, I pushed the code. The changes I made on query.html:

Since the parameters to be entered in the raw parameters will be added to different boxes one after the other with the + button, not with “&” side by side; I defined a variable named rawParams that takes each parameter and value-added to separate boxes in the raw parameters section and keeps it inside.

Each rawP represents the value inside the raw query parameters box.

The insertToRawParams function takes two variables named argKey and argValue. ArgKey is the parameter we entered in raw parameters; argValue takes the value of this parameter. I changed the insertToRawParams function, as you can see below:

So we can check the raw query parameters boxes, take the value in each of them, and keep them in the rawParams that we have created.

I edited the addFilters function as follows so that it can handle the parameters entered in the raw query:

In addFilters, in the code snippet above, the boxes that meet this rawP value are checked one by one to see if the parameters have been entered side by side, then they are split, and the value is assigned. Because there were no +/- buttons before, the parameters were being entered side by side or one under the other with “&”. Although there are +/- buttons in this new feature, we still want the ability to enter more than one parameter with their values in a single box.

And finally, of course, I added functions for +/- buttons to create a new raw parameters box when clicking the “+” button and delete it by clicking “-”:

Voilà! I sent my second commit.

But, I missed a point in the commit I sent for the previous raw parameters improvement. The problem was; with our new improvement about the raw query parameters field, if we add more than one parameter with their values ​​one after the other with the + button and we want to make more than one request with the same parameters, the page would not be refreshed because the URL would not change (because all query parameters were kept in urlParams and added to the URL).

This is due to the working logic of $location.search(). This function only refreshes the page if there is a change in the URL. Therefore, if we execute it with the same parameter a second time, it will be renewed once.

Another problem was that if you add the same parameter more than once, it goes to the server in a comma-separated form. At the same time, since the page was refreshed in the query form, the values ​​in different boxes were sent to the raw query parameters in a single box, separated by commas:

Suppose you supply the same parameter more than once, such as stats.field (to get stats for multiple fields), then the values get concatenated into a single comma-delimited list sent to the server like "stats.field":"pages_i,price".

Because of what I mentioned above, as seen in the screenshot, after the first successful query was sent, when the query parameters that would give an error were entered, and the execute query button was clicked, the query form was sent twice.

Now let’s observe the commits I sent to solve all these problems I mentioned, and let’s get one step closer to merging the code.

I defined the $route parameter to refresh the page manually later.

I changed the insertToRawParams function again:

It is checked whether the type of the argValue value is Array because the problem was that when we wanted to use the same parameter more than once, it went to the server separated by commas.

$location.search() runs every time query.js runs; it adds the parameters given in the query form to the URL.

The solution is as follows: As I explained above, $location.search only reloads the page when there is a change in the URL, so when Execute Query is called, it will no longer run the query and print the result. First, it will add the parameters to the URL, then refresh the page, and after the page is refreshed, the doQuery will run. Therefore, it will run the query and print the result at that stage. I provided this with the isPageReload control.

In the following if-condition, it is checking if the page is reloaded. isPageReload goes into doQuery as true if the page is reloaded and undefined if it isn’t. In the if(isPageReload) block inside the doQuery function, the action is taken according to the value of this variable:

The Execute Query button executes doQuery. The code goes inside the doQuery function, so I edited the doQuery function as follows so that it can handle the parameters entered in the raw query.

I created a control block with if (isPageReload) { inside the doQuery function:

Here, thanks to the for( var key in adminParams ){ loop, the values of all parameters are added to adminParams one by one. Although a field is used twice since it is an Array type, no errors occur, and values do not get concatenated into a single comma-delimited list.

Now it goes to the URL correctly even if the same parameter is used multiple times.

When isPageReload is undefined in the above if block, that is, the page is not refreshed, the previous URL and the current URL are found, and it is checked whether these two URLs are equal to each other because when the URL is the same, the page is not reloaded. By controlling this, we try to reload the page manually. If it is the same, we reload it to run the query after the page is refreshed and the response is displayed on the screen.

As seen in the screenshot, even if the same parameter is given different values in different raw parameters boxes and called Execute Query, it is sent to the server correctly.

And the request is not sent twice, only once:

And after this improvement, the raw query parameters field looks like this without any errors:

Now that everything is solved, you can enter your raw query parameters one by one with the +/- buttons and enjoy your simplified query screen!

Lastly, I would like to thank my mentor Ahmet Can Kepenek, for his unwavering support and helpful feedbacks, and Solr developers for encouraging me and giving me cool suggestions. 🚀

Thank you for reading!

My Apache Solr Contribution Journey from UI Improvement to Bug Solving was originally published in Segmentify Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In A Nutshell

As you know, Linux is a command line based operating system. If you know a few basic commands, you can save too much time while working on your projects. In this post, I will try to show you some lifesaver commands that can help you a lot during your journey to become a Linux master.

1. Use grep if you need to search for a particular pattern

g/re/p stands for; globally search for a regular expression and print matching lines.

With grep command you can search text and strings that you want in a given file. Syntax is as follows:

grep [options] patterns [file]

options : command qualifiers

patterns : required search query

Some useful options:

• -i : The search will not be case sensitive. So if you want to search for the word “computer” then “COMPUTER” will also give the same result.
• -n: Showing matching line numbers.

In the example below, at first we didn’t give -i option and grep couldn’t find the pattern “The”. Because we were searching for “the” with lower case “t”.

• -r : Enables recursive search in current directory.
• -c: Counting the number of matches.
• -v : Displays the rows that don’t match the searched pattern.

If you combine grep commands, you can find patterns hiding in thousands of files. For example you can pipe the output of the “cat” command to grep for a particular pattern:

2. What about awk?

awk syntax is as follows:

awk 'program' [file]

Let’s say that you have a list of data sorted one after the other and you want to print them side by side separated by comma, then you can use awk:

awk ORS is a built-in variable. With the delimiter you give to ORS, each record in the output will be printed with this delimiter.

Built-in Variables:

• NF - gives you the total number of fields in a record.
• NR- gives you the total number of records being processed or line number.
• FILENAME - gives the name of the file being read.
• FS - Field separator.
• RS- Record separator.
• OFS - Output field separator.
• - Output record separator.

awk NR example:

As you can see NR gave the total number of records being processed.

With awk, you can control to process data. You can define variables, use string and arithmetic operations, use control flow and loops or generate formatted reports. You can process files with millions of lines to output a readable report that you can benefit from.

As you can see in the example above maybe we don’t need all the information that will come from a command and we just want to see the first field. We can pipe the output from who into awk and then tell awk to print only the first field.

• $0: Represents the entire line of text.
• $1: Represents the first field.
• $5: Represents the fifth field.

A BEGIN rule used to perform actions before records are processed. Actually, it’s executed before even reads any text. And END rule performs actions after records are processed. You can have multiple BEGIN and END rules, and they’ll execute in order. For example, maybe you want to print the sum of the numbers in a field:

This command calculates the total and print the sum in the end of the execution.

To print 10 random numbers from 0 to 9, you can use awk command. As you can see, the program statement tells awk what operation to do.

3. You can do too many things with sed

sed stands for stream editor for filtering and transforming text

sed is used for text substitution, find and replace operations, text manipulations like search, insertions deletion. And the good thing is with sed, we can perform all these operations on a file without opening it.

Syntax is as follows

sed [options] {script} [file]

— We can replace or substitute the string in a file. The example below shows simple sed command replaces the word “Unix” with “Linux” in the file.

— We can do the same thing for a particular occurence of a pattern in a line. So if we use /1, /2, /4 etc. flags, we can replace the first, second or fourth occurence of a pattern in a line. Let’s change the third occurence of a word “Unix” with “Linux” in a line:

— /g flag (stands for global replacement) replace all the occurences of the string in a line.

$sed 's/Unix/Linux/g' text.txt

— /1, /2, /4 etc + /g replace all the patterns from nth occurence of a pattern in a line.

— We can delete particular pattern:

sed '/pattern/d' text.txt

— We can delete lines, delete line from range x to y:

sed 'nd' text.txt
sed 'x,yd' text.txt

— We can replace string on a specific line number or replace string on a range of files. For example if we have a file like this:

[Unix]
[Unix]
[Unix]
[Unix]
[Unix]

And if we want to replace the “[“ with space, between the first and third line or only the third line:

Or we can combine cut and sed commands to delete ‘[ ]’ next to pattern:

Also sed can act like grep by combining the print operator(p) to function on all lines that match a particular pattern:

Conclusion

The above commands and examples are just the tip of the iceberg. For someone who knows how to use them effectively, these commands can really come in handy. We can combine them in different ways to get what we need.

Thank you for reading!

References

• How To Use grep Command In Linux / UNIX With Practical Examples
• Pipe, Grep and Sort Command in Linux/Unix with Examples
• The Grymoire's tutorial on SED

Last week I was in trouble with Solr’s synonyms. I was trying to understand what synonym means in Solr, how to define one for some collection, how to test whether this synonym I defined works or not and as if that wasn’t enough I was looking for a way to see the relevant data even though I made a typo while I was searching for the synonym I defined for a word in my data.

With this article, we will explore the use of synonyms and fuzzy together. But first let’s go through some basics and give three examples so you can fully understand.

Why we use synonyms in search ?

Imagine you speak Turkish as your mother tongue and you’re searching for skirts on your favorite shopping site which is locally in New York. You’re going into search box, typing “etek” which is the equivalent of the word skirt in English and expecting to see all “skirt” relevant data.

Synonyms are used to fix some relevancy problems. As you can see in the picture above, you searched for “etek” but the site replied with some irrelevant products.

Defining synonyms allows you to extend some terms based on their equivalence. Thus, users can reach the desired result when they search, even with a word that might not have been used in the original data.

To give an example, it allows users to find computers by searching for “Computer” and also by searching for “PC”, although the original data contains only “Computer”. This means PC and Computer are two different terms with the same meaning. So user will find all the products containing either PC or Computer by using any of those terms in the query.

We need to have the synonyms file which is a part of the Solr configuration in order to apply our needs. This file contains all the synonyms for the data in it. As I mentioned above, more than one synonym can be defined, not just one. For example; data might contain the words, “Blue Jean, Skinny Jeans, Slim-Fit Jeans, Regular Jeans, Cigarette Jean, Mom Jeans” for the term Blue Jean. When someone searches, they search for only one of these terms, but expects all these results to come up. To meet this expectation, it makes sense to create a file to contain similar terms specific to my data. You can also use language mixes that will be useful in some contexts, as in the language example I gave above.

Now i will show you how you can define it assuming you understand synonyms and know a little bit about Solr.

Define Synonyms on Solr

In order to define and use synonyms, it is necessary to go through several stages. I will explain all of them one by one in the following parts. I should also give the detail that I am working on Solr Cloud.

• In my previous post, I gave a brief introduction about collections, field-types and fields. To define a synonym and search for it, as you can imagine, you first need to create a configset and a collection. Then you need to define a filter that states that you will define synonym in this collection. You need to define this filter:

{
"class":"org.apache.solr.rest.schema.analysis.ManagedSynonymGraphFilterFactory$SynonymManager"
}

I did this by sending a request to my collection with Python like this:

import requests
import json

url = "http://localhost:7574/solr/my_collection_name/schema/analysis/synonyms/en"

payload = json.dumps({
  "class": "org.apache.solr.rest.schema.analysis.ManagedSynonymGraphFilterFactory$SynonymManager"
})

headers = {'Content-Type': 'application/json'}
response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)

At the end of the URL, you can see “en” which indicates that I will use the English synonym definition.

• Then you need to send request specifying what will be the synonym in this collection’s data. I defined a Turkish synonym for “midi skirt” as “kisa etek”:

import requests
import json

url = "http://localhost:7574/solr/my_collection_name/schema/analysis/synonyms/en"

payload = json.dumps({
    "Midi Skirt": [
        "Kisa Etek"
    ]
})

headers = {'Content-type': 'application/json'}
response = requests.request("PUT", url, headers=headers, data=payload)
print(response.text)

After this a file with the name of “_schema_analysis_synonyms_en.json” will be created in the Solr panel’s files and its content will contains this:

"managedMap":{     
"Midi Skirt":["kisa etek"],     
"class:["org.apache.solr.rest.schema.analysis.ManagedSynonymGraphFilterFactory$SynonymManager"]}

• It’s time to prepare the place where data will come, I mean adding field-types and fields. But now I would like to take a break from explaining how to define synonyms and talk about the thing that I was working on for two weeks.

During my internship, I realized that in the company, they are defining filters on the query analyzer while adding field-types. There is a reason for this. You can define synonym graph filter on the index time or on the query time. The difference is important because it’s changing everything.

My mentor asked from me to find a solution for some problem. When I defined a synonym for a word in my data and searched for that synonym, I was able to get all the relevant results. But if this synonym word that I defined doesn’t appear anywhere in my data and I made a typo during the search phase, I couldn’t reach the results I wanted. For example, if I defined the synonym of “kisa etek” for “midi skirt” and searched for “kisa etek”, I could find all “midi skirt”s, but when I made a typo in the form of “kisd etl”, I was getting very irrelevant results even though I defined fuzzy which allows us to make typos and gives us the right to make mistakes according to the fuzzy value we give. Of course, this happens when you define the filter in the query analysis.

While researching, I learned that If I define the filter in index analysis, this problem disappears and I can reach the results I want even if I make a typo.

In the next steps, I will continue as I defined the synonym in the index analysis. But you should know that this creates new problems:

— The index might get bigger, because all synonyms must be indexed.

— Changes in the synonym rules require reindexing of documents and reindexing is a very time consuming, inefficient way. I mean synonym rules can’t be changed for existing documents without reindexing.

But for now I don’t care about it because my main goal was to use fuzzy search and synonym together. Now let’s continue:

• In schema.xml add a field-type that indicates the synonym for the field. You need to use the synonym filter and indicate the language which is managed=”en”, and besides that it’s required to add FlattenGraphFilterFactory filter on index analyzers after graph filters.

{
  "name": "sgm_text_en",
  "class": "solr.TextField",
  "multiValued": false,
  "indexAnalyzer": {
    "filters": [
      {
        "class": "solr.ManagedSynonymGraphFilterFactory",
        "managed": "en"
      },
      {
        "class": "solr.FlattenGraphFilterFactory"
      },
      {
        "class": "solr.TrimFilterFactory"
      },
      {
        "class": "solr.LowerCaseFilterFactory"
      },

      {
        "class": "solr.ASCIIFoldingFilterFactory",
        "preserveOriginal": "true"
      },
      {
        "class": "solr.RemoveDuplicatesTokenFilterFactory"
      }
    ],
    "tokenizer": {
      "class": "solr.StandardTokenizerFactory"
    }
  },
  "queryAnalyzer": {
    "filters": [
      {
        "class": "solr.TrimFilterFactory"
      },
      {
        "class": "solr.LowerCaseFilterFactory"
      },
      {
        "class": "solr.ASCIIFoldingFilterFactory",
        "preserveOriginal": "true"
      },
      {
        "class": "solr.RemoveDuplicatesTokenFilterFactory"
      }
    ],
    "tokenizer": {
      "class": "solr.StandardTokenizerFactory"
    }
  }
}

• Once the filed type is added, we can add field that maps to the field type by using the name.

{
  "name": "name_en",
  "type": "sgm_text_en",
  "stored": "true",
  "indexed": "true",
  "indexType": "field",
  "multiValued": "false",
},

• And after adding our documents, we can search for “kisa etek” and get the results with full of “midi skirt”s:

And here even if I made a typo while searching, i got all the relevant data:

Thank you for reading!

Reference

• Multi-term synonyms plus fuzzy search
• Managed Resources
• Using synonyms in Solr