Essential Python for Data Engineers

Photo by Kevin Ku on Unsplash

As a data professional, you will at one point come across a complex problem and writing code that is not optimal will only make the problem harder to solve. You might also come across an existing complex code base and using some of the python techniques we will talk about in this article might come in handy when it comes to optimizing the existing code base and making it faster and even more robust.

We will briefly and comprehensively talk about multiple python techniques, from the foundational topics to the more complex ones, some of which may require some knowledge of basic python syntax for easier understanding. These will include:

i) Using conditional statements and loops

ii) Standard built-in python functions, functions and functional programming

iii) File I/O and operations

All the code samples implementing the techniques can be found on my github for further reference. Let’s dive into the fun technical aspects.

Conditional Statements and Loops

Conditional statements are a powerful way which enable you to run different code blocks based on specific conditions. For and while loops are used to iterate over a set of values or to perform a repetietive task. A for loop for example can be used to iterate over list elements and perform calculations on each elements. More often than not, statements and loops are used together, but they can also be used independently. Some of the examples we will look at use both conditional statements and loops. Conditional statements can vary in type, with multiple types existing such as: simple conditional statements, list conditional statements, boolean conditional statements, ternary conditional statements and nested conditional statements.
Simple conditional statements are used to check for one or more conditions and execute the code in which the satifactory condition is met. A simple example is code that is used to grade students based on their scores.

score = 85

if score >= 90:
    grade = "A"
elif score >= 80:
    grade = "B"
elif score >= 70:
    grade = "C"
else:
    grade = "D"

print(f"Your grade is {grade}.")

Consider the code above, here the student has a score of 85 and the conditional statement executes the second code block which satisfies the requirement of the marks being greater than or equal to 80 but at the same time less than 90. Our score therefore is a B. Below is the output of the code above.

Your grade is B.

List conditional statements are the next topic we will look at where the code iterates over a list till the specified condition is met and only then does it execute. Consider a conditional statement that prints a statement based on a name being present on a list.

names = ["Bob", "Alice", "Charlie"]

if "Alice" in names:
    print("Alice is in the list")
else:
    print("Alice is not in the list")

The above code loops through the list elements till it finds the name “Alice” after which it returns a statement and if the condition isn’t met throught the whole list, an alternative statement is returned. In our case, we have the output below.

Alice is in the list

Boolean conditional statements are the next thing we will look at, where code is executed based on a boolean value, which could either be true or false. In my country, Kenya, you have to be a citizen aged 18 years and above to vote, which I will be doing in the next elections to try and advocate for better governance. The code below checks that a person is aged 18 or more and that they are a citizen which is in boolean in order to allow them to vote.

age = 25
citizen = True

if age >= 18 and citizen:
    print("You are eligible to vote")
else:
    print("You must be 18 years or older and a citien to vote")

Our candidate is aged 25 and is a citizen according to the boolean, therefore eligible to vote. The output of our code proves that.

You are eligible to vote

Ternary conditional statements are what we will look at next. It is a shorthand way of writing statements that writes them in one line making complex conditional statements look more readable and concise. We will use the example of eligibility to vote above.

age = 15
status = "eligible" if age >= 18 else "not eligible"
print(f"You are {status} to vote")

What the code above does is quite similar to the preceding example, just in half the lines of code which offers more readability and conciseness. The output is now different since the age is below 18 which is a mandatory requirement to be able to vote.

You are not eligible to vote

Nested conditional statements are the final type of conditional statements we will look at. Nested conditional statements are conditional statements within conditional statements. An example explains this better, and we will still use the example of eligibility to vote.

age = 25
citizen = True

if age >= 18:
    if citizen:
        print("You are eligible to vote")
    else:
        print("You must be a citizen to vote")
else:
    print("You are not eligible to vote yet")

The code above first checks whether the citizen is aged 18 and above and if that’s the case, it then checks to confirm that they are a citizen after which it returns an output based on the citizenship. If the citizen is not aged 18, the code just executes the else statement directly and doesn’t check the second condition. Below is the outcome of the code above.

You are eligible to vote

That is a summary of conditional statements and is quite brief but exhaustive. Be sure to try them out for practicality and better understanding. Next, we will look at functions.

Python Functions and Functional Programming

Python functions and functional programming are two different things which may sound similar but we will look at both of them and will be able to identify the difference. Functions are used to organize blocks of code into reusable blocks. This facilitates the organization and modularization of your code. For example, you could create a function to perform a specific data transformation task and call it in multiple places in your code instead of writing it out in every single place the transformation is needed.
Functional programming on the other hand is a programming paradigm that avoids changing state or mutable data and treats computation as the evaluation of mathematical functions. Functional programming can be benefitial for data engineering as it makes testing, debugging and parallelization simpler. Python is not a purely functional language, compared to other functional languages such as scala, but it offers modules such as functools that provides higher-order functions which are functions that operate on other functions. This can mean returning a function, taking a function as an argument or both. They also offer sophasticated functional programming tools such as decorators for function composition or memorization or caching.

We will first look at the python standard built-in functions.

1. print()

It is used to display the output to the console. Consider the example below.

def display_passed_info(product, sales, revenue):
    print("Product", product, sep=':')
    print("Total Sales", sales, sep=':')
    print("Revenue", revenue, sep=':')

display_passed_info("Laptop", 100, "300,000 USD")

The code above outputs the following result to the console.

Product:Laptop
Total Sales:100
Revenue:300,000 USD

2. len()

Used to determine the length of a string or list.

names = ["Alice", "Tom", "Rhoda", "Dennis", "Steve", "Trevor"]

print(len(names))

The code above prints the number of list elements of the list passed to it.

6

3. range()

Used to generate a sequence of numbers.

def generate_sales_days(start_day, end_day):
    return list(range(start_day, end_day + 1))

print(generate_sales_days(1,7))

The code above returns a list containing numbers between 1 and 8, which is the range given to it.

[1, 2, 3, 4, 5, 6, 7]

4. help()

This returns the documentation of the function passed to it, which is pretty straight forward. Consider the code below and the example.

def calculate_discounted_price(price, discount):
    """
    This functions takes the price and calculates the discount based on the discount percentage specified
    Returns: float
    """

    return price * (1-discount / 100)

help(calculate_discounted_price)

Help on function calculate_discounted_price in module __main__:

calculate_discounted_price(price, discount)
    This functions takes the price and calculates the discount based on the discount percentage specified
    Returns: float

5. sum()

This returns the sum of list elements passed to it.

def calculate_total_sales(sales):
    return sum(sales)

print(calculate_total_sales([150, 200, 300]))

The above code sums up all the sales passed to it in the list returning 650 .

Get Guchu Dennis’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

We will now look at functional programming as a programming paradigm. We will first look at some higher order functions which enable us to implement functional programming and we can later look at the module functools and how it enables us to implement functional programming. Another feature of python’s functional programming is the lambda function. It is used to define short, inline functions which are used inside higher order functions to perform quick, straightforward operations. We will get to see how they are used.

1. sorted()

def sort_sales_data_by_sales(data):
    return sorted(data, key = lambda x: x['Sales'], reverse=True)

sales_data = [
    {'Product' : 'Laptop', 'Sales' : 450},
    {'Product' : 'Mouse', 'Sales' : 200},
    {'Product' : 'Keyboard', 'Sales' : 300},
]

print(sort_sales_data_by_sales(sales_data))

In the code above, the function sort_sales_data_by_sales(data) is used to sort the sales data based on the amount of sales in descending order using the sorted() function. That means the product with the highest sales appears first on the list as a key-value pair all the way to the product with the least sales.

[{'Product': 'Laptop', 'Sales': 450}, {'Product': 'Keyboard', 'Sales': 300}, {'Product': 'Mouse', 'Sales': 200}]

2. map()

def apply_discount(prices, discount):
    return list(map(lambda x: x * (1-discount / 100), prices))

print(apply_discount([100, 200, 300, 400, 500], 10))

In the above code, the function apply_discount(prices, discount) takes two arguments, the price and discount as a percentage and calculates the discount amount based on the percentage and the price of the item, then goes on to deduct the discount amount from the normal price. All this is done using the map and lambda functions. The code then returns the prices of the goods after the discount.

[90.0, 180.0, 270.0, 360.0, 450.0]

3. filter()

def filter_high_sales(sales, threshold):
    return list(filter(lambda x: x > threshold, sales))

print(filter_high_sales([100, 150, 90, 200, 300], 150))

The code above has a function filter_high_sales(sales, threshold) which filters the sales that meet the threshold set. In this example, it filters sales which are greater than the threshold which is set at 150 . This then returns a list of all the sales that meet the threshold.

[200, 300]

4. zip()

def combine_sales_data(products, sales):
    return list(zip(products, sales))

print(combine_sales_data(['Laptop', 'Mouse', 'Keyboard'], [150, 200, 300]))

The code above contains a funtion combine_sales_data(products, sales) which then returns a list of tuples whereby each element in the products list is matched to the corresponding element in the sales list. That means it returns a list containing tuples with all the products with their corresponding sales.

[('Laptop', 150), ('Mouse', 200), ('Keyboard', 300)]

5. enumerate()

def list_products_with_index(products):
    for index, product in enumerate(products):
        print(f"{index + 1}. {product}")

list_products_with_index(['Laptop', 'Mouse', 'Keyboard'])

In the code above, the function list_products_with_index(products) is used to match the products to their index using the enumerate() function. This then returns each product with an index assigned to it, but in form of a number preceding it.

1. Laptop
2. Mouse
3. Keyboard

We will now move on to the module functools and how its order functions are used to operate on other functions.

1. reduce()

As the name suggests, it reduces the elements in a list to a single value by applying a binary function cummulatively to the items of list or any other iterable.

#reduce() applies a binary function cummulatively to the items of an iterable, reducing it to a single value
from functools import reduce

#Calculating the product of a list of numbers
numbers = [1, 2, 3, 4, 5]
product = reduce(lambda x, y: x * y, numbers)
print(product)

The code above reduces the list numbers by applying the lambda function which takes in two numbers and multiplies them by each other, which results in a single value, a product of all the elements in the list, 120 . Use reduce() when you want to perform cumulative operations such as summing or multiplying elements in a list. It is more elegand compared to a loop but can be less readable for complex operations.

2. partial()

Partial allows you to set a certain number of parameters to a fixed value and therefore create a new function with less parameters

#partial() allows you to fix a certain number of arguments of the function and create a new function with fewer parameters
from functools import partial

#Fixing the first argument of a multiplication function
def multiply(x, y, z):
    return x * y * z

#Creating a new function with x fixed at 2
double = partial(multiply, 2)
print(double(3, 4))

Consider the function multiply(x, y, z) which multiplies the three numbers passed to it. In the case that one of the numbers changes and is set to be constant, we can use partial to set on of the parameters to 2 and therefore come up with a function with one less parameter. The output of the code is therefore, 24 . Use partial() when you repeatedly pass the same argument to a function, which saves time and makes the code more elegant at the same time.

3. wraps()

wraps() is a decorator that helps keep the metadata of the original function intact when wrapping it with another function.

#Using functools.wraps to preserve the function's metadata
from functools import wraps
def add_exclamation(func):
    @wraps(func)
    #Decorating the wrapper with @wraps(func) forces the functions metadata to be passed onto the wrapper
    def wrapper(name):
        return func(name) + ' ' + '!'
    return wrapper

@add_exclamation 
def greet(name):
    """Greets someone"""
    return 'Hello' + ' ' + name

print(greet('Tom')) #Prints "Hello Tom !"
print(greet.__name__) #Prints "greet"
print(greet.__doc__) #Prints "Greets someone"

Use wraps() whenever you write decorators to ensure the original function’s metadata is preserved. This makes it easier to understand and debug your code. For better understanding, decorators are used to modify the code of the decoratee without necessarily having to change the source code. The code below will help you better understand decorators better and hence also understand wrappers better. Be sure to check it out and try it out to see the results.

#A decorator is a function returning another function usually applied as a function transformation using the @wrapper syntax
#The decorator modifies the behavior of its decoratee without having to change the source code
def add_exclamation(func):
    def wrapper(name):
        return func(name) + '!'
    return wrapper

@add_exclamation
def greet(name):
    return 'Hello' + ' ' + name

print(greet('Tom')) #Prints "Hello Tom!"


#Advance decorators with two inner functions
def add_symbol(symbol):
    def decorator(func):
        def wrapper(name):
            return func(name) + symbol
        return wrapper
    return decorator

@add_symbol('!')
def greet(name):
    return 'Hello' + ' ' + name

print(greet('Tom')) #Prints "Hello Tom!"

@add_symbol('??')
def greet(name):
    return 'Hello' + ' ' + name

print(greet('Tom')) #Prints "Hello Tom??"


#Using classes as decorators
#We need to implement the __call__ method which defines behaviour when we call an object like we call functions
class add_symbol:
    def __init__(self, symbol):
        self.symbol = symbol

    def __call__(self, func):
        def wrapper(name):
            return func(name) + self.symbol
        return wrapper

@add_symbol('!')
def greet(name):
    return 'Hello' + ' ' + name

print(greet('Tom')) #Prints "Hello Tom!"

#Using functools.wraps to preserve the function's metadata
from functools import wraps
def add_exclamation(func):
    @wraps(func)
    #Decorating the wrapper with @wraps(func) forces the functions metadata to be passed onto the wrapper
    def wrapper(name):
        return func(name) + ' ' + '!'
    return wrapper

@add_exclamation 
def greet(name):
    """Greets someone"""
    return 'Hello' + ' ' + name

print(greet('Tom')) #Prints "Hello Tom !"
print(greet.__name__) #Prints "greet"
print(greet.__doc__) #Prints "Greets someone"


#Using __wrapped__ to undecorate
#To be able to undecorate, we need to: 1. Use functools.wraps to decorate our wrapper function in our decorator 
#And 2. Use __wrapped__ to access the original undecorated function
from functools import wraps
def add_exclamation(func):
    @wraps(func)
    #Decorating the wrapper with @wraps(func) forces the functions metadata to be passed onto the wrapper
    def wrapper(name):
        return func(name) + ' ' + '!'
    return wrapper

@add_exclamation 
def greet(name):
    """Greets someone"""
    return 'Hello' + ' ' + name

print(greet('Tom')) #Prints "Hello Tom !"
print(greet.__wrapped__('Tom')) #Prints "Hello Tom"


#Decorators to decorate classes
#Here we create a class decorator that adds a common function hello to every single class it decorates
def add_hello(cls):
    cls.hello = lambda self: print('Hello')
    return cls

@add_hello
class Dog: pass

@add_hello
class Cat: pass

Dog().hello() #Prints "Hello"
Cat().hello() #Prints "Hello"

The code above explores everything decorators, including using classes as decorators and decorating classes.

4. lru_cache()

lru_cache() is a decorator that caches the resullts of function calls, using a Last Recently Used(LRU) cache.

#lru_cache() is a decorator that caches the result of function calls using a Least Recently Used(LRU) cache
from functools import lru_cache

@lru_cache(maxsize=32)
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n-1) + fibonacci(n-2)

print(fibonacci(10)) #Output 55
print(fibonacci(10)) #Still outputs 55 but at a faster speed

#Use lru_cache() to optimize functions that are called frequently with the same arguments reducing computational overhead by caching results
#Use this function if you are calling the function with the same parameter

Use lru_cache() to optimize functions that are called frequently with the same arguments, which reduces computational overhead by caching results of a previous similar function call.

5. cache()

cache() is a shorthand for lru_cache(maxsize=None) .The main difference between lru_cache() and cache() is that when lru_cache() reaches the maxsize it will remove the least used recent cache item.

#cache() is a decorator that caches the result of function calls. It's a shorthand for lru_cache(maxsize=None)
from functools import cache

@cache
def factorial(n):
    if n == 0:
        return 1
    return n * factorial(n-1)

print(factorial(5))
print(factorial(5))

#Use cache() for functions where results should be cached without a size limit, esp when calls with the same arguments occur frequently
#The difference between cache and lru_chache is that when lru_cache reaches the maxsize, lru_cache will remove the least-recent used cache item

Use cache() for functions where results should be cached without a size limit.

6. cached_property()

cached_property() is a decorator that caches the result of the method storing it for future access without need for recomputation.

#cached_property() is a decorator for class properties that caches the result of the method, storing it for future access without recomputation
from functools import cached_property

class Square:
    def __init__(self, side):
        self.side = side
    @cached_property
    def area(self):
        print("Calculating area")
        return self.side ** 2

square = Square(4)
print(square.area)
print(square.area)

#Use cached_property() for expensive-to-compute properties that don't change often, or if recalculating them each time is inefficient

Use cached_property() for expensive to compute properties that don’t change often, or if recalculating them isn’t efficient.

7. singledispatch()

singledispatch() is a decorator that transforms a function into a single dispatch generic function, allowing different implementations based on the first argument type.

#singledispatch() is a decorator that transforms a function into a single-dispatch generic function, allowing different implementations based on the first argument type
from functools import singledispatch

@singledispatch
def process(value):
    print("Default processing")

@process.register(int)
def _(value):
    print("Processing integer", value, sep=":")

@process.register(str)
def _(value):
    print("Processing string", value, sep=":")

process(10) #Prints out "10"
process("hello") #Prints out "hello"

#Use singledispatch() when you need to handle multiple types within a function, providing a clear and organized way to define type-specific behavior
#In a class, we need to use singlemethoddispatch
#This we can use when we want to restrict the type of variable

Using singledispatch() in the above example enables us to implement both an integer and a string from the first function. Use singledispatch() when you need to handle multiple types within a function. In a class we use singlemethoddispatch when we want to restrict a type of variable.

8. cmp_to_key()

cmp_to_key() converts an old-style comparison function into a key function for use with sorting.

#cmp_to_key() converts an old-style comparison function into a key function for use with sorting
from functools import cmp_to_key

def compare(a, b):
    return (a > b) - (a < b)
sorted_list = sorted([3, 2, 1], key=cmp_to_key(compare))

print(sorted_list)

#Use cmp_to_key when you have legacy comparison functions that you want to reuse with python's sorting functions

Use cmp_to_key() when you have legacy comparison functions that you want to reuse with python’s sorting function.

File I/O

You will often work with data stored in files and therefore being able to work with large datasets stored in files is essential. Python offers some simple built-in functions that enable us to do read and write operations on files. When reading from a file, the file contents are stored in memory and made available for manipulation. When writing to a file, the data is written to a file for storage or later retrieval.

1. open()

To open a file, we use the open() which takes two arguments, the file name and the mode you want to open it in, whether read or write.

def write_sales_to_file(filename, content):
    "Writes content into a file"
    with open(filename, "w") as file:
        file.write(content)

2. write()

We use this when we want to write to a file, which writes a string to the file. You can also use writelines() to write multiple lines to a file.

def write_sales_to_file(filename, content):
    "Writes content into a file"
    with open(filename, "w") as file:
        file.write(content)

You can also append data to a file using the same write() method, just that we will use the "a" tag in open()

def append_sales_to_file(filename, content):
    "Appends new content to an exisitng file"
    with open(filename, "a") as file:
        file.write(content)

3. read()

To read the contents of a file, you use theread() function which reads file contents as a string. To read a single line of the file, you can use the readline() function.

4. close()

After reading or writing to files, it is also important to close the files. For this, we use the close() function. In our case, we use with in our open statements which automatically closes the files. This is more convenient and easier.

def write_sales_to_file(filename, content):
    "Writes content into a file"
    with open(filename, "w") as file:
        file.write(content)

def read_sales_file(filename):
    "Reads file contents"
    with open(filename, "r") as file:
        return file.read()

def append_sales_to_file(filename, content):
    "Appends new content to an exisitng file"
    with open(filename, "a") as file:
        file.write(content)

write_sales_to_file("sales.txt", "Product: Laptop\nSales: 150")
print(read_sales_file("sales.txt"))

append_sales_to_file("sales.txt", "\nProduct: Mouse\nSales: 300")
print(read_sales_file("sales.txt"))

Using the above practices when writing your code will help you write code that is more elegant, more readable and faster. Be sure to try them out and try incoroporating them in your code. Thank you for reading this article and if you need further reading, the link to the references can be found below.

References

Cracking the Data Engineering Interview

Python Documentation

Functools: Python Package Every Developer Should Know

I am a data engineer and data scientist and would be happy to connect with you on LinkedIn for any discussions, feedback, gigs or anything data. Be sure to reach out. Thank you.