The LLM arena is an interesting and fascinating space… fundamentally transformer and self-attention models changed the world. Language processing, contextual understanding: all took a big leap forward, and there is no going back from that.

The release of the likes of Jev is a spanner in the works for frontier labs, and I foresee similar niche specific, distilled models being released in future. AI researchers might only grow in number, looking for different ways to create new models. Their interaction with traditional Machine Learning Models will be something of interest to watch and observe e.g. because Decision Trees still dominate bounded non-hard coded multi-parameter business rules (>50), which need (say it with me) DETERMINISM.

Coding is still an applicable space for Frontier LLMs, but that depends on the coding approach favoured. Cheaper long context models like DeepSeek v4 flash (not even mention newly released 4.1) are more preferable to top class Astra & Fable if you are trying to keep your hand on the throttle — if you want to track logic of what is happening in your system. They are cheap enough to write boring code for you, while allowing you to go step by step, with verification. There are entire philosophies to how this works.

We are in the period of transition. People are making bets. Some are betting their entire house. It remains to be seen who wins and who loses their shirt in the end. Some have taken a more careful approach: not building systems that will turn out to be unmaintainable if unsubsidized pricing turns out to be $8000 per subscription, instead of $20 or $200.

There is a tendency to feel like you are being left behind right now. A kind reminder: is that the world doesn’t run at the pace of infinity. You can ship code tomorrow, but are your users ready to adopt to every blinding new feature? Are the systems up/downstream ready to adopt to your new speed? You don’t need to deliver features every other day.

While the anxiety is real, what people look for is value. That has always been what we bring to the table. And the fundamentals remain, people still need to eat, sleep under a roof, and possibly, if the Lord allows, thrive in other spaces. Take a breathe, and re-evaluate.

It comes highly recommended because it focusses on building more than just hacking away.

I have tried Hackerrank, ProjectEuler, LeetCode, Codewars and other such tools, all in a bid to improve my coding chops. Each has its own strengths and areas that it focusses on. Leetcode, for example, when not being used to prepare for job interviews, can be very useful in learning algorithms.

Surprisingly, grinding Leetcode can teach you a lot about yourself:

• When subjected to pressure, do you underperform or rise up to the challenge? Do you overthink and suffer from analysis paralysis?
• Does your mind rush to complicated solutions at first, ignoring simpler solutions in a case of premature optimisation?
• Do you tend to address edge cases before establishing that base functionality works?
• When trying to debug loops, do you walk over the case of i=0, then skip to the last case? Do you skip over some code sections and assume they work? Is there a pattern in how you overlook certain aspects?

Codecrafters

https://app.codecrafters.io

I heard about it while watching Theo on Youtube(his sponsor segments are really useful most of the time). It promised the ability to rebuild, from the ground up, Redis, a shell, a HTTP Server, your own Kafka, and many more, with more to be added in future. The annual subscription appeared to be well worth the price, and even better, they offer regional pricing, with Africans getting a significant discount!

I’d recommend Codecrafters if you like to:

• approach things from 1st principles. Knowing how they work deep-down.
• want to challenge yourself and grow your skills.
• know that you have met the standard. They run your code against tests in a container to make sure it passes required specs. So it’s not you just saying that you wrote a HTTP server. They actually confirm that your server runs according to the specification guidelines they provide.

Here are some of the projects that I’ve embarked on:

1 Building a HTTP Server

See https://github.com/geonyoro/codecrafters-http-server-python

This is the first module I started with and I recommend it for its simplicity. Having chosen python, this helped me understand:

• http versions: 1.0 vs 1.1 vs 2.0 vs 3.0. Somewhere in there, you get to implement long-lived connections, which are a HTTP/2.0 thing. It now makes sense when you see HTTP/1.1 vs HTTP/2.0 and know what you can expect the server to do.
• The http spec: You get to know how requests are formatted, raw.
• how to handle multiple connections simultaneously. (Choose threads?)
• Why choosing threads for multiple connections doesn’t scale well to high loads (threads are resource intensive as opposed to green threads and other lightweight alternatives)
• Coding patterns and refactoring: There are a lot of things that are common to each request that may need to be handled. E.g. setting a content-length header for each response, handling encoding that has been specified by the client. How do you centralise all these?

2 Build out a shell

GitHub - geonyoro/codecrafters-shell-python

I’m still currently in the middle of this one and am really enjoying it. The goal is to implement a shell based on bash that supports some of the features.

It has helped me:

• implement the strategy pattern with built-in commands that we write (exit, echo)
• write a parser (actually 2 parsers, one for the shell commands with their escapes, and one for the redirect operators to stdout and stderr). This was really fun and brain-wracking.
• call external commands using the output of the parser and verify that I’m on the right track. (the cat command is the true test of whether things are okay)
• testing: Instead of using the codecrafters’ testing suites that run on a git push, I wrote my own internal tests that I can use to iterate over faster. The added benefit of modularity in code is not lost on me.

A word to the Wise

Don’t cheat yourself: In these challenges, there are libraries that have already been built out for handling some of these things. In the Shell example, I was tempted to (and indeed saw many people) go around the problem of building a parser for the shell by either:

• using the shlex module to parse arguments and clean the same safely and easily
• using the string.split() method to handle splitting the arguments

The problem with the above approach is that it robs you of the joys of knowing how to parse commands, and the different approaches. I hope to learn how to write a full compiler so this part really helps me in that. In writing a raw parser that consumes characters, you need to:

• reach back into some of the skills built during Leetcode
• analyse different approaches (implement look-ahead for characters while considering not getting index errors when you run out of characters vs implementing look-behind)
• analyse how to handle different types of parsing (escape sequences vs re-directions) and perhaps create different categories of parsers for each of the 2.

Because of the above, I’ve found myself moving slower than most, but getting value for money, and skills.

Once done with a section, I have learned to read the code that others have implemented and see their different approaches. There is a world to learn. This has, in turn, taught me humility. I thought I was hot stuff and implement the best code, only to read what someone else wrote and have my mind blown.

Check out my LinkedIn.

This refers to the set command, not the data structure.

In my previous post, Using Redis for Producer/Consumer Queues: Part 1, we explored using Redis lists as queues, and combined that with sets to ensure uniqueness. If the Redis list contains keys, then each key must first be confirmed to not exist in the set before pushing onto the list(queue).

A sorted set is a sort of hash-map of keys and values that are used to score the keys. Part 2 (unpublished as of the time of this publication) covers handling errors by using sorted sets (and setting the value for each key as an expiry time = current_time + expiry_timeout ). We would then iterate over the sorted set to find the values that have expired (value < current_time) and then remove those from the sorted set, and re-add them to the list.

In my recent discoveries, this last part can be optimised, combining the 2 data structures of the set and the sorted set into one. Redis’s set command allows us to set the value of a key. It can take optional parameters (ex for seconds, px for milliseconds) to declare time validity of the the value before it is unset automatically by Redis. This means setting a key with ex=1 means that after 1 second, the value shall no longer be found.

The set command additionally can take a NX parameter, which instructs Redis to only set the key if it does not already exist. It returns a 1 or 0, depending on whether the value was set or not. In this way, we can combine these 2 data structures or not.

This method provides the advantage of being atomic (in one command, we both handle setting the value, uniqueness, and the expiry), whereas the previous use of 2 data structures was not not atomic AND may require the use of a TRANSACTION to ensure atomicity.

Previously, I wasn’t keen on using this method because I thought it would end up polluting the redis key space with many keys. If I have 1000 items to enqueue, that means I have 1000 keys in redis, where as with a set, I have 1 more data structure. However, on further examination, tools like celery (when used with redis backing) work with the key setting procedure.

Locks

In distributed systems, acquiring a lock isn’t a simple thing. Solutions such as Apache Zookeeper exist, but they aren’t trivial. When running a single instance of Redis(as opposed to Redis clusters), one can use the set command with NX and EX to set up a lock.

Tips

Use a common prefix for all keys common to a certain domain, to then allow querying with the keys command. The below uses domain.com:new_user: as the prefix.

SET domain.com:new_user:x 1 NX EX 60 // returns 1
SET domain.com:new_user:y 1 NX EX 60 // returns 1
SET domain.com:new_user:x 1 NX EX 60 // returns 0 as domain.com:x was already set
KEYS domain.com:new_user:* // query all keys with the domain.com: prefix.

Happy Redising! Or is it val-key?

Github: https://github.com/geonyoro

LinkTree: https://linktr.ee/nyoropossum

TLDR: Python falls short when it comes to runtime safety guarantees and performance

How Python Excels

Python has come a long way from its first days in the 90s. Back when I started off in Python 2.7 in 2010 as someone new to programming, it felt extremely easy; extremely smooth. With little code, one felt productive; no public static void main(argc…..). Additionally, it supports libraries that allow a wide range of things to be accomplished:

• Desktop UI? Use the inbuilt tkinter library to generate GUIs
• On linux for UI? Run with GTK (pretty much Ubuntu apps are written with Python and PyGTK)
• Requests have never been simpler with requests
• FastApi, Flask, Django for really easy servers
• Data science flourishes in Python under various libraries

With Python 3 and the later versions, typing hinting and the use of dataclasses were introduced to . The language itself supports various programming paradigms:

• Functional programming
• Composition
• Full OOP with inheritance

Python Falls Short: Errors

While the adage “If it can be written in javascript, it will be" exists, I have for a long time adapted that mantra for Python. Core, business components are written in Java, but other more scripted systems, and things like browser automation are written in Selenium.

This sucks because it means eventually you hit errors in production that weren’t handled. Redis ran out of memory and disconnected? Well, oops, you find that out in production when something isn’t working. Mysql can disconnect, a query can fail to execute because of a bug, Json can fail to parse correctly.

Other languages such as Java (and more recently Golang, Rust, Ziglang) force one to think critically about the errors that can come up with each function call. They each go about this differently (throwing Exceptions vs errors as values), however, the general idea is that you will know that this function can fail with x or y errors and you will have to think about handling the failure upfront. Can the program continue to work after this error or do we need to exit? In this particular error, perhaps we should send an email. This reads to very stable programs in production.

In order to catch surprise exceptions in production in Python, the closest you could get to fool-proofing is wrapping an entire loop iteration in a try/except clause, then logging that failure, and the looking at errors later. This makes for a sucky language.

Performance

Python is just plain slow. We can chalk up the lack of speed to it being an interpreted language, but it’s so much slower than the likes of Javascript, which is also an interpreted language.

To achieve optimisations in Python, one has to reach into sections written in C or use libraries that have optimisations in C such as numpy.

Conclusion

While I do appreciate Python for it’s ability to work with a vast number of libraries, and it being my first language that brought me into the world of progamming years ago, it has hindered my ability to grow past a certain threshold.

I’d like to begin to use Golang as my go-to language for almost any and all things. It offers a nice mix of speed, simple syntax, and safety.

When it comes to orchestration, almost everyone uses Kubernetes, which is quite complicated. For someone new to this world, Docker Swarm offers a simple-to-run alternative as it extends the compose file with a few more keys and arrays.

With docker swarm in production, there are 2 main errors that one encounters, and they both relate to space issues:

• /var/lib/docker/containers folder is full.
• /var/lib/docker/overlay2 folder is full

This post covers the 2nd item.

Overlay2 is full

While running over 100 containers in production, I noticed a degradation in service performance and a quick overview showed that one of the servers was offline.

Investigations began. I logged into the server. When servers run out of space, one of the first things to go is the auto-completion of commands on the command line under bash. I used the df -h command to check for space and confirmed that usage was at 100%. Using du -sh * starting from the root, I was able to zero in on the /var/ folder and finally the overlay2 folder in /var/lib/docker as the culprit. There was one folder there that was big. I stopped all docker services, removed the folder, started the services again, and the problem was gone. For now.

Overlay2 Explained

Why couldn’t I find the service related to the big folder and simply fix that? Unlike the containers folder, the naming of folders under overlay2 doesn’t correlate to the service or the container name. Therefore, it is harder to zero in on the cause of the problem. (At least that’s what I thought at the time. A way to zero in is presented later.)

How does overlay2 get full? When using images in containerization, the runtime assumes that no process will write to the container’s filesystem directly. Instead, it assumes that all writes will be made to externally mounted systems or external services:

• For a database, we write to external Postgres/MySQL/MariaDB or a MOUNTED SQLITE file.
• Logs are written to a MOUNTED directory in the file system.
• Temporary files are generated to external hosts/S3 or mounted.

Container images are read-only by default, in order to save space. If one wants to write to a container image, docker generates a new, writable image in the background. All writes can then proceed. This is because docker uses a copy-on-write(COW) strategy.

Path to solution

Because I love pain and suffering, I used a self-written custom solution:

• Install Elasticsearch
• Install Metricbeat with mode: global in docker swarm(meaning a single instance per host/node on the swarm) to ship all logs to ElasticSearch
• Write a monitor script in python to query ElasticSearch for space above 80%
• When values exceed the threshold of 80%, send an email alert to a provided email.

A few days later, I woke up to emails indicating a server was past 95%. This meant I had <5% to fix this issue. I quickly narrowed down the issue to the overlay2 folder. One of the folders under this was at 245GB (typical size is about <2GB).

Isolating the faulty container

While it’s hard to correlate the containers/service name with the overlay2 image name, inspecting a running container on the erroneous host will show the DIRs section that many have names related to what we see in overlay2:

We can therefore iterate over the running containers and look for the big folder in the output using the command below.

for i in $(docker ps -q); do 
  echo $i; docker inspect $i | grep <big_folder>; 
done

I used exec to get into the container to see why it is so big. I found many temp files in the /tmp folder, because typically an external container is meant to be mounted and without the mount, it defaults to /tmp, as per the code below.

import os
TEMP_DIR = os.environ.get("TEMP_DIR", "/tmp") 

I rerun the service, mounting the necessary folders as needed, and with that, this never occurred again! :)

LEARN. GROW. PROSPER.

I, George “Possum” Nyoro, am a lead full-stack engineer. Checkout my linktree! https://linktr.ee/nyoropossum

[Source]: Pub-Sub is a design pattern that allows producers and consumers to communicate with each other. A broker/middleman/middle-ware is used to send messages from a producer that are then consumed by consumers.

[Edit by mcflyhalf] The beauty of this is that the publisher doesn’t need to know who subscribes and the subscriber doesn’t need to know who publishes. Each party simply needs to know the topic(s) that they are interested in (sort of like Medium wink).

Traditional SQL-Based Queues

If you come from traditional relational databases like MySQL, PostgreSQL, SQLite, you are used to using a table as a queue. You put items in the table according to a schema and can delete the items in the table each time you are done or leave them there, having a field indicating that they are processed.

create table myqueue (
  id integer, 
  item_id varchar(50),
  value integer,
  dt_added timestamp null,
  dt_updated timestamp default current_timestamp on update current_timestamp,
  priority int default 0
)

With the above schema, we can select the following:

- id: 1
  item_id: "item1"
  value: 100
  dt_added: 2023-01-01T00:00:00
  dt_updated: 2023-01-02T10:02:04
  priority: 1

- id: 2
  item_id: "item20"
  value: 120
  dt_added: 2023-01-03T00:00:00
  dt_updated: 2023-01-04T10:02:04
  priority: 2

This allows us to select items from the table, using it as queue with the following ordering:

• use the date_updated
• use the priority field (either in ascending or descending order, each has it’s caveat)

SELECT * FROM myqueue ORDER BY priority DESC, dt_updated ASC; 

Limitations of the above

• The SQL-based queue is okay if you are running one process that has both consumer/producer within it. In fact, it might even scale by using threading if IO blocking was the biggest bottleneck. This doesn’t work for multiple processes.
• If the process of selecting from the database is expensive by time or some other metric, e.g. selection takes 2 minutes, but you can only work on 2 items at a time.
• You want to reduce the load on the SQL server by not using it as a queue. Additional support for things such as no item can be processed by more than 1 worker at the same time, recovery from failure during processing requires that YET another database table would be added, or more fields would be added to the database table, which begins to complicate its design.

V1: Basic queue in Redis

Redis is an in-memory database that takes the form of a key-value store. Multiple types of values are supported e.g. bytes, lists, sets, sorted sets, and hash maps.

One of its most appealing qualities is that it doesn’t require data structure initialization meaning you don’t have to create the data structure before you use it. You can work on something that doesn’t exist, and Redis will just create it on the fly. Redis itself is also very easy to install.

For a basic queue, we use a Redis list. Values are pushed in on the right(using the RPUSH operation) and retrieved from the left(using an LPOP operation). Once you LPOP an item from the list, it is no longer there.

Here’s some REPL code in Python.

>>> import redis
>>> rcon = redis.Redis()
>>> rcon.rpush("mylist", 1)
>>> rcon.lpop("mylist")
1

If you need to put more complicated data structures in there, you can JSON encode the values (or pickle.dumps() in Python ) and save the resulting bytes in Redis. (NB. It’s possible to pass strings in python, but they will be encoded into bytes before storage.)

>>> import json
>>> rcon.rpush("mylist", json.dumps({"item_id": 30, "value": 230}))

V2: Uniqueness: Don’t re-add values

The problem with the basic queue presented above is that it doesn’t support UNIQUENESS. One can use an external data structure where the membership of elements can be tested.

To guarantee uniqueness in Redis, we use a SET. It doesn’t allow duplication of values.

Use:

SADD myset mykey

This command will return a 1 if the item was added to the set, otherwise returning a 0. If the item did not exist previously in the set, you can then add it to the list.

In Python, it becomes wise to wrap these 2 steps in a function:

from typing import Optional
def add_to_queue(redis_conn, val: int) -> bool:
    added = redis_conn.sadd("myset", val)
    if added:
        redis_conn.rpush("mylist", val)
    return bool(added)

def pop_from_queue(redis_conn) -> Optional[int]:
    val = redis_conn.lpop("mylist")
    if not val:
        return None
    redis_conn.srem("myset", val)
    return int(val)

"""
>>> add_to_queue(rcon, 1)
True
>>> add_to_queue(rcon, 1)
False
>>> pop_from_queue(rcon)
1
>>> pop_from_queue(rcon)
-1
"""

Because Redis only works with bytes, we can’t put whole objects or dictionaries/lists into sets easily.

V2.5 Complex Data Structures

While the above code works well for primitives, more complex types, such as objects/maps/dictionaries can’t be added to queues because queues only support strings/bytes/integers. We can therefore serialize them into strings/bytes before adding them.

import pickle
import json

def serialize_obj(obj):
  return pickle.dumps(obj)

def serialize_dict(mydict):
   return json.dumps(mydict)

A common problem one may encounter is that not ALL attributes/keys of these data structures are meant to be used for uniqueness. For example, a financial transaction may have an identifier which is meant to guarantee uniqueness/idempotence, but may also have a timestamp of when it happened. Sending the serialized version of this twice will result in it being added twice as well. The solution is to have a specific function for handling the computing of the key to be used in the set.

import json
from typing import Optional

import redis
RCON = redis.Redis()
def get_key(mydict):
    return mydict["uuid"]

def add_to_queue(redis_conn, mydict: dict) -> bool:
    added = redis_conn.sadd("myset", get_key(mydict))
    if added:
        redis_conn.rpush("mylist", json.dumps(mydict))
    return added

def pop_from_queue(redis_conn) -> Optional[dict]:
    raw_val = redis_conn.lpop("mylist")
    if not raw_val:
        return None
    mydict = json.loads(raw_val)
    redis_conn.srem("myset", get_key(mydict))
    return mydict
"""
example_dict = {
    "uuid": "1",
    "timestamp": "12312313231",
}
>>> add_to_queue(rcon, example_dict)
True
>>> pop_from_queue(rcon) == example_dict
True
""

Part 2

In the next part, coming within the next 2 weeks, we will handle more complex data structures, as well as error processing.

From valued readers who help me with proof-reading :

What happens if the Redis service restarts or the entire server restarts?

Redis can be configured to dump the in-memory values to disk based on a timeout or if a certain number of keys have been added. When starting, it will load from this particular dump.

[CAVEAT by mcflyhalf] It is true that Redis does do a db dump, but if it does it every 5 seconds, it is possible for a crash to happen and for you to lose 4.99 seconds of data.

LEARN. GROW. PROSPER.

George “Possum” Nyoro is a lead full-stack engineer in a remote team and a local team. He also moonlights as a pianist/keyboardist in Nairobi. Catch him at Geco Restaurant on Fridays from 7pm.

Github: https://github.com/geonyoro

What are you even saying?

In writing the previous article, at some point, when talking about a function that ships orders and takes in a list/array of orders, I had initially used the word “items” instead of orders, which I promptly corrected. Yes, this entire article is based on this correction.

You may say, that’s a small thing, it can be overlooked. Just the use of the word order? But I promise you it is not. It can be a small beginning to a very painful story.

In domain modelling, a core tenet Eric Evans establishes is the use of correct domain language. This is where we establish ubiquitous language to refer to different things and concepts.

Why? Well, to take a step back, software is normally written to address real-world problems (if it is not, then more power to you). Normally, that means interaction with people outside of the programming group. One or more developers are hired by a person(s) to write software. This person is a domain expert in their subject domain (very knowledgeable on the thing they want you to solve).

There are multiple domains and people communicate across domains all the time:

1. A business owner communicates requirements to the project manager ( OR team lead OR lead programmer). They will seldom use software language to describe the problem.
2. The project manager communicates with the other programmers but may use different terms that are more relevant to the developers. They can REPLACE some of the terms they used when talking to the business owner.
3. The business owner talks to a supplier about this new platform they are developing. Perhaps there are supplier requirements that are to be fulfilled in this software development.
4. The business owner talks to customers.

When people talk across domains (business domain, software domain, customers domains), the terminology used is different depending on who is talking. The term order may mean an order made by a customer, but may also mean an order made by the business from a supplier. This small-term mix-up can lead to large future problems. It makes future maintenance of source code very hard. When you see the word order in the code, which of the 2 types does it refer to? It’s a problem if in each context you have to clarify which is being referred to.

When anyone involved in the process, from the business owner to the programmer, uses a certain term, it’s important that we are all on board with what it means without having to clarify a single thing.

Business Level Understanding and Auditability

I shared my previous post in excitement with friends all over but some complained that they could not understand it.

Now, I‘m not sure if this is a principle, or if I have simply intuited it, but when writing business logic, it should be written in a way as is fairly readable even by those not as technically adept. If it is not readable, then it might be a sign of bad design. And the design is what domain modelling is here to solve.

Take the example of the ship order function from the previous article, and the requirement that some countries need phone numbers. Independent of language, a layperson should be able to understand once you tell them, for example, in Java, that the term null means empty. The logic here is easily readable.

In Java:

if (order.getCustomer().getCountry().needsPhone) {
    if (order.getCustomer().getPhone() == null){
        throw Exception();
    }
}

In Python:

if order.customer.country.needs_phone:
  if not order.customer.phone:
      msg = f"Customer phone is needed for country {order.customer.country}"
      raise ValueError(msg)

We get the country from the order’s customer and see if it requires a phone. and then we access the phone from the customer and see if it is empty.
Without being a fully-fledged developer, someone should be able to look at the code and generally get the flow of ideas. This introduces the idea of audibility. Here, the code being written isn’t code, but more like business logic with a light touch of language dependency.
The added advantage of this is that maintaining further down the road is simplified. In 5 years, you don’t have to remember what was done in this method. YOU CAN SIMPLY READ IT, and it shall make sense!

Processes

In my work as a back-end developer on an API at one of the local companies, a front-end developer sometimes asks me to add a certain field to a model. For example, we need to identify the supplier who will fulfill the supply for an order.

In the database, how can this be added? Well, some would add a field to hold a string, and put in the supplier name.

In the database, how can this be added? Well, some would add a field to hold a string, and put in the supplier name.

class Order:
  supplier: str

Others would add a field pointing to another table.

class Supplier:
  name: str

class Order:
  supplier: Supplier

The above doesn’t quite matter. What is important is PROCESS!

When the front-end developer asks to add this feature, how did they know about it? Who did they talk to? What questions did they ask? How well do they understand the business process?

These questions are important because most people aren’t modellers. It may turn out that the relationship wasn’t as straightforward. It may turn out that between an order and a supplier, there exists a CONSIGNMENT that sits between the 2.

class Supplier:
  name: str

class Consignment:
  id: int

class Order:
  consignment: Consignment

Conclusion

If you are not in the habit of asking deeper questions on EXACTLY how the business process is achieved, you WILL be surprised later down the road. Clients love to summarize; to skip steps. They don’t think like a developer who cares about the nitty gritty. It may turn out that the relation wasn’t as straight forward. It may turn out that between an order and a supplier, there exists a CONSIGNMENT that sits between the two, thus the place of process.

Yes, this entire article is based on a correction; a very necessary one.

More content at PlainEnglish.io.

Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.

Domain Modelling 2 : Use of Terminology and Processes was originally published in Python in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my previous post on this matter, we introduced and discussed the idea of timezone support especially in applications that are meant to be used in/from different parts of the world. If all your users are in one timezone, then everyone means the same time when they say 8pm. In Kenya, 8pm means 5pm GMT+3.

We also said, in JavaScript, for example, you can automatically append the timezone to each datetime string by getting the offset in minutes, converting that to hours and minutes, and appending that to the datetime string. For Kenya, the offset will come back as -180 minutes, which is 3 hours 0 minutes.

2021–01–01T0800(timezone naive) becomes 2021–01–01T0800+0300 (timezone aware).

Case Study

A user in Kenya(GMT+3) normally schedules a clinical appointment for his mother every Tuesday at (12 pm) noon local time. One Friday, he travels to London (GMT+1) for a week.

When he goes to schedule the weekly Tuesday appointment in the app for his mother, what is he meant to put in while in London? 12pm? However, if the app auto fills in the timezone as suggested previously in JavaScript, that puts us in a conundrum. He normally means 12pm+0300 but now he will put in 12pm+0100, which means 2pm locally. His mother will show up 2 hours early for her weekly appointment.

In this case, only a savvy user might end up putting in 10am+0100 GMT to represent the exact same time, if the app were auto-filling in the timezone information.

Is the end user supposed to keep track of time zones like this when he crosses regions? No, this complicates things for a simple user who wanted to make a 12pm booking in Kenya local time.

Solution

My friend(and colleagu)e and I were discussing this problem and thought to check what the likes of Google does with its calendar. There is a brilliant field that I’ve overlooked despite using the calendar for years.

Instead of doing a pure automatic manual timezone calculation, a hybrid is used. Just make it obvious to the end user what timezone is being used and/or allow a selector for time zones.

Providing a drop down/selector to users makes it a bit obvious which time zones your app applies across. With this solution, the user in the case study can simply switch his timezone from GMT+1 to GMT+3 and then make his booking as usual.

Perhaps even a warning/alert could pop up to indicate that the user is in a different timezone from the event planner that he is using. A user in London could be alerted that he is currently in a different timezone from the one the app uses.

LEARN. GROW. PROSPER.

George Nyoro is a lead full stack engineer in a remote team, and also moonlights as a pianist/keyboardist in Nairobi. Nyoro started dabbling in python in 2010,and went full stack in 2016, and got into docker in 2020 with swarm. He is a graduate of Mechatronic Engineering at JKUAT.

I have a scheduling app that I’ve been working on and off for a few years(gaining and losing momentum, sigh, such is life). DATE and TIME is at the core of this business idea.

PS. I highly recommend using projects to grow. It is based on my private development of this project that I have expanded my growth immensely. Every new thing I have wanted to learn I have learned in this private project.

In a django project, you can choose whether to use time zones or not, and specify the default timezone that the app will use. In a bid to conform to industry standards, I turned on timezone support and specified UTC as my timezone. (Red Flag: Warning, Danger ahead.)

When a user wanted to create an input in the schedule, they would send the start time and end time of the event to the back end. However, most of the datetimes sent would be timezone naive strings.

A timezone naive datetime is one that doesn’t specify what timezone it is in. The inverse datetime would be said to be timezone aware. You say it is Tuesday 3pm, but while it is 3pm in East Africa(EAT), it is 12 Noon in Britain(BST) and 5am in California, US(PST). If you hope to take your web based app internationally, timezone support IS A MUST!

All my targets in this alpha are in Kenya(+3 GMT). Having set up my app as UTC, when I would send in a string saying Tuesday 3pm, without timezone support, django would save the item in the backend as 3pm UTC, which is 6pm EAT.

Problems

• When a user thus creates an event and then goes to view it, it would appear to have been shifted 3 hours ahead.
• When doing any other checks either pre- or post- creation of the event which require computations with timing, they would fail, but not consistently.
• When testing on localhost, you might likely not be aware of any problems, but the moment a user from another timezone wants to schedule something, it starts failing.

Booking.com, airline scheduling, AirBnB and the like must have discovered these issues long ago. Sigh, I’m only discovering this right now.

Some apps will face these issues to varying degrees. Timing of checking credit card payment/mobile payment expiry may also start to matter. If you place an fast-food order, the exact timing of the placement of the order may or may not not matter. However, the timing of when it is going to be delivered might be an issue. Imagine placing an order knowing delivery takes 30 minutes and immediately being told it was delivered 5 hours ago or will be delivered 8 hours from now, only for then to hear the doorbell ring in 20 minutes.

Solution

Send/save timezone-aware strings.

When sending a datetime string, 2021–01–01T06:00 means 6am in UTC, 9am in +3UTC and 1am in -5UTC. If a user is in London, then that’s no problem, but users in other parts of the world should be aware.

That means, instead of

2021–01–01T06:00

Append the timezone information and send it in in ISO8601 format as

2021–01–01T06:00+0000 for UTC

or

2021–01–01T06:00+0300 for +3 UTC

This will help clarify.

To obtain this information for front-end use, JavaScript has a function to get the timezone offset in minutes for the user’s current region.

new Date().getTimezoneOffset()

This value is normally the negative value of what it’s supposed to be. .e.g +3 UTC will be given as -180 minutes, therefore one needs to subtract the value from 0.

LEARN. GROW. PROSPER.

George Nyoro is a lead software developer/devops in a small remote team, and also moonlights as a pianist/keyboardist in Nairobi. Nyoro started dabbling in python in 2010,and went full stack in 2016, and got into docker in 2020 with swarm. He is a graduate of Mechatronic Engineering at JKUAT(2015).

More content at PlainEnglish.io. Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, and Discord.

Timezone Support: Normally overlooked was originally published in Python in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Many developers are familiar with the term Big-O notation, which indicates how code scales with regard to size of input. If a function takes a list or array, it may run fast with 1 item in the array, but what about 20 million items in the array?

Is there something similar that tells us how code scales with regards to requirements? Is this even a problem? Domain Modelling is a term that refers to a frame of thinking that helps us with this. I first encountered this formal term in the book, Domain Driven Design by Eric Evans.

Example

Let’s have a look at a sample situation. This will help us see how to arrive at a crisis later.

Week 1: Our company ships orders world-wide. Each order is associated with a customer, and each customer has various fields. We can model the below as follows(in Python, though language barely matters):

With the above, you start shipping orders far and wide, worldwide. (It is assumed the ship_order function contains logic to ship the order as needed.)

Week 2: Later, we are informed, that, well, we need to support a new shipping_service of shipping orders (examples of shipping_services in the real world are FedEx, DHL, US Post etc). We have been using shipping_service SSA to ship our orders, but now we need to support shipping_service SSB. The ship_order function will now use attribute shipping_service on Order.

Week 3: After some time, it turns out that we need to support shipping service SSC, but in addition to that, when shipping to Germany using any service, we need to make sure that the phone number is filled in before shipping the order out. Below is a simple way of solving this by altering the ship_order function.

Week 4: We are later informed that, due to some changes(always expect changes, the real world is never constant), if we are shipping to Kenya using service SSB, we need to make sure that the postal_code is filled in. Again, we can extend the ship_order function.

Week 5: We are introduced to shipping_service SSD, and if an order is to be shipped using SSD, it should be moved to the top of the list of orders to be shipped. We will introduce a function sort_orders that is used to sort them orders before shipping them.

Week 6: Service SSC should include a receipt in the box of the order.

Week 7: Service SSE introduced that is also a priority along with SSD, and also requires a receipt.

So far, this is what we shall have:

Week 50: With weekly changing requirements, after 50 weeks, the ship_order function will have grown to have become a mammoth function that is unmaintainable, supporting about 50 different if statements for the various shipping services and country situations and fields.

When a new shipping_service is added and it should be a priority higher than SSE, and should include a receipt in the box:

1. How easy is it to extend any of the parts?
2. How many places in the code will we have to change things when a change is required? It would require changing the sort_single_order function to include the new function as a priority, and we should change the ship_order function in regards to the lines that handle priority. These are 2 places so far, but in the case of 50 weeks, maybe its about 10 places we need to change.

Clearly:

1. This is not sustainable
2. We cannot live in fear of changing requirements just because our code is becoming too long to read. Every time you are told to add a new service that does x a bit differently, you shrug as you imagine how inherently harder it is about to become to maintain this code.
3. Any time there are multiple if statements to handle wildly varying situations, it should be a clear sign that there might be something wrong in the modelling.

Definition

This is a weird place to put a definition, but I might as well. My friend, a proficient educator, insisted that it was a good idea. What is domain modelling?

The aforementioned friend said: I am currently thinking about the objects in your code map, how well they map to objects in real life, and how easy they are to move around when their real world counterparts move around.

A definition from study.com: It is a model that shows the real-world flow, relationships between entities, and concepts.

I say: Yes, all the above, but additionally, something that is able to rise above reality and conquer it.

In the cases of complex and convoluted requirements, in government and such, we have layers upon layers of bureaucracy to address situations where a problem is complex.

“This particular document is normally approved by this department, but because your hair is blue, it’s approved by that other department, which opens on Sundays, between 4am and 5am. You also need 2 signatories because we see here you like to write with pink markers. ”

Maybe, in this case, the real world should adhere to a software model, and not the reverse. LOL.

Refactoring

There are various ways of solving the challenge of scaling the maintenance. This is how I’d solve it. Feel free to suggest your own in the comments.

First of all, when did the rain start to beat us? When should we have caught wind of the issue? I posit that this was when we started writing the many “if” statements. Refactoring is a never ending process. By Week 4, we should have started having alarm bells ringing in our head as we wrote the requirements for Kenya and SSB.

Shipping Service

There are some aspects that “belong” to the shipping service. Whether a receipt is to be included in the box is related to the shipping service, not the order. The same applies to priority of the order.

Therefore shipping service becomes its own class.

Country

As per now, inclusion of a phone number is exclusively based on a country. So we can move the phone requirments there as well.

With the above changes, I can create a shipping service, and a country using:

And the ship_order function is this simple.

As a true test, adding a new shipping service SSF with a mix of requirements becomes as easy as:

Conclusion

In cleaning up the ship_order function, we have removed ALMOST ALL references to “literal” countries and shipping_services, and instead are running the model using the objects. This has made it extensible and easy to read.

However, we have not solved the issue of Kenya and shipping using shipping_service SSB. That is something that we shall solve in a subsequent article.

PS: Depending on how the requirements grow or don’t grow, you may not need to remodel your domain that much. A bunch of if statements thrown all over the place may suffice.

PS of the PS: Feel free to correct, contribute, suggest changes to any of the ideas provided above. This life is all about learning. :)

The writer is a lead software developer/devops in a small remote team, and also moonlights as a pianist/keyboardist in Nairobi. Nyoro has over 10 years experience with Python, 6 years in full stack and SRE, and 2 years in docker with swarm. The writer is a graduate of Mechatronic Engineering at JKUAT.

More content at PlainEnglish.io. Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, and Discord.

Domain Modelling 1: A problem you didn’t know you have was originally published in Python in Plain English on Medium, where people are continuing the conversation by highlighting and responding to this story.