But I want to start with the problem, not the money.

Creators build the room. Someone else collects the rent.

Every day, creators bring people together. They spark conversations, answer questions and turn followers into real communities. Yet most of that value flows to the platforms hosting them. Creators get reach, sometimes. They rarely get a fair share of what their community is worth.

We started Klavicle because we believe the people who build communities should be the ones who earn from them.

What Klavicle is

Klavicle gives creators a home for their community that combines two things that usually live apart: the depth of forum-style discussions and the energy of chat. Members can dig into long threads or jump into fast conversation, all in one place.

Then we built monetization in from day one. And they don’t wait 30 or 60 days to see it. Payouts are instant.

What the funding is for

This round lets us move faster on two priorities:

1. Product. Better tools for discussions, moderation and community discovery, plus more ways to earn beyond ads. [Edit to match roadmap.]
2. Creator growth. A dedicated team to help creators launch, grow and earn on Klavicle.

Thank you

To the creators who trusted an early product and told us exactly what was broken: you shaped everything we’ve built. To our investors: thank you for believing that community, not just content, is the future of the creator economy. To our team: this is just the start.

If you run a community, or want to start one, come build it on Klavicle. Your community. Your earnings. Paid instantly.

— Alexander Obidiegwu

Digest 3 months’ content in 18 minutes

The Google Data Analytics certificate is said to be worth $60,000 because it prepares you for entry-level jobs as a data analyst. Whether that’s true is up to debate but regardless, I decided to take it because I have a thing for data and exploration. And who else to learn about data from other than Google?

The course is divided into steps a data analyst should take when performing analysis.

These steps are

• Foundation — 18hrs
• Ask Data-Driven Questions — 21hrs
• Prepare Data for Exploration — 25hrs
• Process Data from Dirty to Clean — 26hrs
• Analyze Data to Answer Questions — 32hrs
• Share Data Through The Art of Visualization — 25hrs
• Data Analysis with R-programming — 34hrs
• Google Data Analytics Capstone: Complete a case study — 11hrs

We would only be going through the first 4 modules. Why? Well because those are the only ones I have finished lol. Might update this article or create a new one once I complete the remaining 4.

If you ever need help grasping any of these steps, leave a comment or message me on LinkedIn.

Course 1: Foundation

This module lays the groundwork for the necessary skills, tools, and fundamental knowledge a data analyst would need to function properly in the job market.

The module is less technical and more theoretical than others. The main objective is to give you a bird’s eye view of the landscape of a Data Analyst.

What is Analytical Thinking?

Analytical Thinking involves identifying and breaking down a problem in a step-by-step manner in order to find patterns or insights. It involves being problem-oriented, strategic, seeing the forest instead of trees, detail-oriented, and identifying correlations and patterns.

How to think analytically

Most problems often stem from a root cause. By asking and finding out the root cause of a problem, we have solved half of the problem. You can use the 5 whys Principle to help extract the root cause of any problem.

We can also look for gaps in our processes. This is usually used when we want to move from one state to a better one.

We should also consider what wasn’t considered before. By reviewing our steps and analysis, we can properly identify blind spots that weren’t visible initially to us.

Skills Needed as a Data Analyst

Curiosity: As a Data Analyst, curiosity would take you far. Being able to dig through your data till you uncover every aspect of it is a highly desirable skill.

Understanding Context: Context is crucial when it comes to any form of analysis. It enables you to be able to see a given set of data holistically. A random date provides us with no meaningful information except that it’s on the time continuum. But when you specify that it’s when you were born, we can now derive more meaningful information from that and be able to ask further relevant questions.

Technical Mindset: This skill is synonymous with thinking analytically. It requires breaking down complex problems into smaller and simpler parts. It requires us to think logically and creatively when approaching problems.

Data Design: This deals with how you organize your information. For instance, alphabetically arranging contact list information is an example of data design.

Data Strategy: This skill often involves the management of people, tools, and processes in data analysis. You want to ensure you have acquired the right people with the right skill sets, knowledge of the required tools, and a well-planned process to execute properly.

Your data strategy should depend on the dataset and the deliverables of the project. If for instance, your client wants a simple dashboard, Google sheets can be used. But a dashboard that continuously updates itself, then tableau would be the right choice.

The Six Phases of Data Analysis

There are six phases of data analysis, but we’ll only be focusing on three because those are the phases covered in courses 1–4.

Ask Phase: In the ask phase, you ask fundamental questions to understand a certain problem and how to approach it.

It usually involves meeting with stakeholders in order to gain clarity on the objectives of the project. This phase allows you to understand the context in which to approach the problem or derive a solution.

Prepare Phase: This phase involves data collection/gathering. We have to worry about what kind of data we need to gather, the source of the data, how relevant is this data to the business objectives, how much data to collect, etc.

The Ask phase prepares us to attack this phase head-on. Using the answers we gathered in the ask phase, we can properly execute this phase. If we encounter any hiccup, it probably means we probably didn’t ask the right question.

Process Phase: The process phase involves cleaning the data we have gathered. A critical part of this phase is data integrity. We need to ensure the data we are cleaning is reliable and accurate.

Once we ensure data integrity, we can then begin to clean data. This involves handling duplicates, misspellings, invalid data types, null values, and misrepresented values.

Once the data is shiny and ready to use, we can then begin the analyze phase.

Data Lifecycle

Data has its own lifecycle just like all living things. This lifecycle usually begins with planning and ends with archiving or deleting.

Different companies sometimes have different data lifecycles unique to their requirements. For instance, the U.S. Fish and Wildlife Service does not have a delete cycle because historical data is important to them.

Generally though, there are key lifecycles that most companies follow:

• Plan — What kind of data is important to us? What tools are required to get that data? Who is responsible for that data? What are the barriers stopping us from getting the data? How long would it take to get the data? Where can the data be found?
• Capture — We capture the data using tools available to us
• Manage — We maintain and store the data using tools we deem appropriate.
• Analyze — We derive insights from the data and use it to make decisions and solve problems.
• Archive — We archive the data for later use.
• Destroy — We wipe the data from memory and delete any shared copies of the data.

When I was learning this section, it was difficult to separate the six phases of data analysis from the data lifecycle. In order to separate them, let’s look at it this way.

First, the data lifecycle phase is to be adapted and modified based on the company’s needs. It’s just a guiding framework you can use to manage your data.

The sixth phase of data analysis is a process every data analysis project must go through. There’s no skipping the line. Every step of the way is as critical as the first. You are allowed to add some more phases to your liking but the entire six phase stated must be included in your adapted framework.

The data lifecycle is all about the data. What state the data is in, and how it can get to the next state.

The sixth phase of data analysis is all about the problem/objective. How is the state of our data relevant to the problem we are trying to solve and how can it become more relevant?

Course 2 — Ask Questions Make Decisions

At every step of the way in the data analysis process, you are going to need to ask questions. These questions are going to determine the quality of your analysis.

In the Ask phase, questions like “What are the problems I need to solve? How do I go about solving it?” are needed to be asked.

In the Prepare phase, questions like “What kind of data do I need to solve this problem? What are the right tools to get this data? What research do I need to do to understand this problem and it’s context.” would prove invaluable.

In the Process Phase, questions like “What data inaccuracies am I likely to encounter? What tool would help me clean this data efficiently?” would prove invaluable.

In the Analyze Phase, questions like “What is my data telling me? Am I inferring any kind of bias on this data? Who needs my company’s product or services and what type of person would most likely use it?”

In the Share Phase, questions like “How can I visualize this to stakeholders so they can understand it? Who are my target audiences? What would help me understand it if I were the listener? How can I make this engaging?”

In the Act Phase, questions like “How can we use the analysis to improve products or meet stakeholders’ expectations? What and who are needed for the analysis to be acted upon?”

SMART QUESTIONS

A lot about asking good questions is making them SMART. The SMART framework has been used since 1981. It was initially used as a means of setting goals but have now been adopted in different branches of self-development.

The ability to ask and communicate those questions effectively is a very valuable skill in 2024. In a field like Data Science, asking good questions must become second nature to stand out.

• S — Specific — They are significant, and very closely bound to our ideas.
• M — Measurable — They can be quantified.
• A — Action-oriented — They inspire action or change. So rather than asking, “How can we get customers to recycle our product packaging?” You could ask, “What design features will make our packaging easier to recycle?” This brings you answers you can act on.
• R — Relevant — They matter to the problem being tackled.
• T — Time Bound — They have specified timelines or deadlines in which we would get our answers.

You must not always validate your questions in the default chronological order (SMART). For example, I prefer to use relevant to validate my question before anything else. Besides, what does it matter if your question isn’t relevant to the problem being solved? I take this step Relevance, Action-oriented, Time-Bound, Measurable, and Specific. So RATMS.

Most likely if you get the first four, you have a specific enough question. If you get the first 3, it can be easily measurable, and so on. But this is mine. Do you have your own customized framework?

Questions you should avoid at all costs are:

• Leading questions: The weather is cold, isn’t it?
• Close-ended questions: Do you like ice cream?
• Vague Questions: How’s life?

Data-Driven Decisions vs Data-Inspired Decisions

A data analyst professional is critical to decision-making in any company. The ability to turn raw data into insights is highly valuable.

There are two major ways data can be used to make decisions. Data-driven decision-making or data-inspired decision-making. A lot of businesses sometimes mix up the two but they are very different.

Data-Driven Decisions (DDD)

This is when data is used to determine decisions regardless of external opinions or experiences. It simply means using facts to guide decision-making.

This type of decision-making triumphs when there is enough data to support the fact.

When the data is insufficient, the fact is inherently flawed and making decision based on it would most likely result in bad decision making.

Other drawbacks of DDD are reliance on historical data, a tendency to ignore qualitative data, vulnerable to biases during analysis.

Data-Inspired Decisions (DID)

This type of decision involves the entire party. Although similar to DDD, they give room for opinions and suggestions about the decision to be made based on the data. Slight deviations are allowed, experiences are considered, and so on.

DID can also avoid some of the drawbacks that DDD has.

When making decisions with data, there are generally two types of data at play: Qualitative and Quantitative data.

Qualitative data: is data that is often subjective and not measurable. These kinds of data are often non-numerical. They help provide us with context to our quantitative data and are useful to get a more ground understanding of what our data is telling us. We can get qualitative data from focus groups, in-person interviews, reviews and so on.

Quantitative data: is data that is objective and measurable. They can often be in numerical or a structured format. This kind of data helps us spot trends and anomalies in our data. Examples of ways we can get quantitative data includes surveys, polls, ratings, and so on.

You can think of it this way. If there were a group of people in a room, qualitative data could describe how they feel, what they look like, what clothes they are wearing, or their motivations of why they’re here. Whereas quantitative data about the same group may include the number of people in the group, their age, or the temperature in the room.

How to Solve Problems

Before solving any problem, we need to define it properly. There is a famous quote by Albert Einstein that says, “If I were to spend 1 hour solving the world’s problem, I would spend 55 minutes defining the problem and 5 minutes solving it.” This is the degree you need to take to define a problem. You don’t want to spend 5–10 hours solving a problem only to find out it was the wrong problem.

Defining the problem is only one piece of problem-solving. A lot of problem-solving involves thinking structurally. Thinking structurally means understanding the current problem, identifying gaps and opportunities, collecting and organizing information, and defining options available.

A clean way to practice structured thinking is through the scope of work. A scope of work is a document that defines a project’s problem, deliverables, timelines, milestones, and reports. It is an agreed-upon doc that allows any party involved to follow along with the project’s progress.

Do not confuse Scope of work with Statement of work. Statement of work defines what you are contributing to the company/business and what is expected of you. It includes objectives, guidelines, deliverables, schedules, and costs.

In problem-solving, you also need to understand the context related to the data. Numbers on a spreadsheet mean nothing without context. And depending on the context, the data could mean different things. It’s extremely important to be as intimate as possible with the data’s context because it would define the interpretation and result of the analysis.

However, nothing above would be important if we do not understand the limitations of our data.

Limitations of Data

A great DA knows the limitations of his/her data and prepares for them. Some limitations of data can include:

Incomplete/non-existent data: Incomplete data is data restricted by samples, time range, features, etc.

Whenever you encounter incomplete or non-existent data, you want to always be upfront about it with stakeholders. Without the required data, it would be difficult to reach an unbiased conclusion.

Misaligned data: Sometimes, the data you collect could have a different interpretation from what you believe it to be.

For instance, suppose you collect data from another team, and one of the features is number of course participants. You might assume that to be the entire population who enrolled in the course, when in fact, it was only those who completed the course.

So you want to be clear beforehand in order to avoid stories that touch the heart.

Dirty Data: This one should be obvious. You shouldn’t work with dirty data until it’s cleaned.

As a quick reminder, data cleaning is the process of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete data within a dataset.

Tell a clear story: When visualizing data, there are a few things to take note of:

• Compare data of the same data type to avoid inconsistencies.
• Set your Y-Axis to 0. It’s easy to see a 0.01 change as a major change when it is zoomed in.
• If your table is clearer without graphs, please don’t add graphs.
• Test for statistical significance. Sometimes two datasets will look different, but you will need a way to test whether the difference is real and important.
• Pay attention to the sample size of your data. Too little can skew results. Always try to collect more data.

Be the Judge: When you know the limitations of your data, it becomes easy to make sound judgments and allow for good decisions to be made in the company. Take the necessary steps to ensure your data is viable to use and make sound judgments.

COURSE 3 — Preparation Data For Exploration

Before we do anything with data, we need to ensure we have the right data. This involves going back to the ask phase and asking questions that narrow down the type of data to collect.

For instance, What type of data would be appropriate? What data would I use? How can I get this data? Is his data relevant to the problem at hand? Where would I need to store this data? What would be my data sources? How much data is needed?

All these questions help narrow down your options and help make data preparation as efficient as possible.

How to decide which data source is reliable?

1. Good data sources are reliable
2. They are original. Validate the data with the original source.
3. They are comprehensive. They contain all the necessary population data.
4. They are current and up-to-date. The more stale a data source is, the more the quality of the data decreases.
5. They are properly cited.

Types of data sources

First Party Data: This is data that you created using your own resources. For instance through surveys or website registrations. This is the most reliable source of data.

Second Party Data: This is data gotten from an external source who created/owns the data. For instance, Using Facebook API to get data on specific profiles.

Third Party Data: This is data that was gotten from an external source who doesn’t own the data i.e. it was gotten from another provider. This is the least reliable source of data among the three and should be avoided if possible. For example, buying a list of emails from a broker.

Depending on your use case, you can use any of the above data sources. However, third-party data should be considered when the consequences of incorrect data are low.

What type of data to collect?

There are majorly two types of data. Quantitative and Qualitative data.

Qualitative data
Qualitative data is data that cannot be measured or quantified. Examples are names, categories, genres, reviews, etc. They are usually subjective. Types of qualitative data are Nominal and Ordinal Data.

• Nominal Data is data that is categorized without a set order. Examples include Yes/No and Not-Sure questions, sex/gender, etc. These type of questions can be chosen in any order and there’s no dependence between the possible answers.
• Ordinal Data is data that is categorized within a set of orders or scales. Examples include the scale of 1–10 questions, ratings, Size of a shirt, etc. The possible values have a dependency on each other in the sense that one is bigger or smaller than the other.

Quantitative data
Quantitative data is data that can be measured and quantified. Examples are ratings, age, height, time, length, etc. There are different categories of quantitative data which include: Discrete, Continuous, Nominal, and Ordinal data types.

• Discrete data is quantitative data that has a limited number of values i.e. no continuous decimal places. Eg dollar bill, ratings, Maximum capacity allowed in an arena, etc.
• Continuous Data is a type of quantitative data that can have almost any numeric value and can have as many decimal values as possible. Examples include height, temperature, time, etc.

Internal data
This is data gathered within the company. It is the most secure and reliable source of data.

External data
External data is data obtained from outside the data. It becomes important when your analysis depends on as many sources as possible.

Structured data
Structured data is data organized in a certain format, like rows and columns. Eg tax returns, expense reports, etc.

Unstructured data
Unstructured data is data that cannot be stored as columns and rows in a relational database. Eg social media posts, emails, texts, etc.

During the collection of data, we need to ensure we include the metadata on the data.

Metadata

Metadata is data about data. It helps signify how and where data was created/collected. Some examples of metadata include the properties of a PNG file (created, size, ratio, etc).

Emails also contain hidden metadata that includes server names, IP addresses, HTML format, and software details. This image includes hidden email metadata such as the message ID and when the email was created.

This metadata helps data analysts to put data in context. It also helps them determine when data is accurate, precise, reliable, relevant, and timely. It is even more important than the data itself.

Metadata helps analysts ensure that they’re working with the right data and that the data is described correctly.

For example, a data analyst completing a project with data from 2022 can use metadata to easily determine if they should use data from a particular file.

Data Bias

Data Bias is data that skews results in a certain way. There are different forms of bias.

Some examples include -

• Sampling Bias: is when a sample of a dataset isn’t representative of the population. This can be avoided by choosing samples at random.
• Observer Bias: is when different people observe the same thing differently.
• Interpreter Bias: is the tendency to interpret situations or data positively or negatively.
• Confirmation Bias: is the tendency to identify or look for information that confirms our pre-existing knowledge/beliefs.

Before, during, and after the collection of data, we need to follow certain standards that protect the owners of the data. These standards are called Data Ethics.

Data Ethics

Data ethics is a set of rules and principles that should be adhered to when collecting, sharing, and using data. Some aspects of data ethics are consent, privacy, openness, transparency, and currency (being aware of how your data is used financially).

Balance security and analytics

Data security is the security measures we take to protect data from unauthorized access. The best way to secure data is to not collect it in the first place. But that doesn’t help analysts solve problems in any way.

So we want to find some balance between security and our analysis. Our data must be secured but in a way that also allows us to easily retrieve and use it.

Some common methods we can use are encryption, tokenization, authentication, authorization, etc. There are much other methods we can use to secure data but these are popular and well tested methods.

COURSE 4 — Process Data From Dirty To Clean

Data integrity is critical to successful analysis. This usually involves ensuring your data is clean, accurate, trustworthy, and reliable.

There are many ways your data could get compromised during analysis. Data can be compromised when it’s replicated, transformed, manipulated, transferred, etc. It can also be corrupted due to hacking or software issues/malfunction.

This is why you want to always ensure you have a backup of the original data. Never work on the original data. Instead, create a copy and work on that copy.

This gives you the option to revert changes if the need ever arises. It also helps you and everyone understand where and how the analysis is derived from.

Types Of Dirty Data

In data analysis, small mistakes in data can lead to big consequences.
So always take the time to clean up the data and soon enough it’ll become a habit, a second nature.

What to do when you find an issue with your data

I’m sure you’re all familiar with the term proxy. According to Google, A proxy is something/someone that has the authority to represent someone else.

When dealing with faulty/missing data, we can always resort to proxy data. This is data that closely relates to the original data.

No Data Issue
Suppose, we are performing an analysis on traffic in cities based in the US, but we do not have any data concerning a certain city.

Rather than excluding that data or spending exorbitant resources to collect that data ourselves, we could just use a city that has the same population as the city we do not have data on.

This is usually the common way of approaching such problems and saves us time.

Missing Data Issue
Suppose you are performing an analysis on loneliness crisis and there isn’t any data for those aged 18–24. You can perform the analysis but when reporting, state that the analysis only includes those aged 25 and above.

If there isn’t a clear group or category that is missing, or just random features are missing for random people, use proxy data to fill them up.

Wrong Data Issue
If you are performing an analysis on a certain data and find that some data results in errors when performing calculations on them, if the data set is large enough and the errors are infrequent relative to the size of the dataset, you can ignore the errors and performing your analysis.

If you have the wrong data because of misunderstood requirements, then get clearer on the requirements.

Important note: Sometimes data with errors can be a warning sign that the data isn’t reliable. Use your best judgment.

Common data-cleaning pitfalls

I believe an image says a thousand words:

Document the cleaning process

As a data analyst, it’s important you document your cleaning process in order for everyone who encounters the data to see the changes that lead up to it and also fix errors.

This documentation usually includes:

• The date of change.
• The person who made the change.
• Why the change was made.
• The person who approved the change.
• Data, files, formats, etc. that were changed.
• Version Number.
• Description of the change.

Let’s say you made a change to a formula in a spreadsheet because you observed it in another report and you wanted your data to match and be consistent.

If you found out later that the report was actually using the wrong formula, an automated version history would help you undo the change. But if you also recorded the reason for the change in a changelog, you could go back to the creators of the report and let them know about the incorrect formula.

If the change happened a while ago, you might not remember who to follow up with. Fortunately, your changelog will have that information ready for you!

Here are a few things to note:

• Changelogs are for humans, not machines, so write legibly.
• Each change should be on a different line
• Group the same types of changes. For example, Fixed should be grouped separately from Added.
• The release date of each version should be noted.

The world itself is a self-scalable phenomenon worth studying. Every living moment, the world is expanding, evolving, and adaptive. Within it, there exists independence in dependency and vice versa.

Everything and everyone working in their best interest and at the same time, working in the best interest of humanity as a whole.

No mistake or error made by a single person can completely take humanity off course.

Everything and everyone can be made redundant. Everything and everyone is reused. Everything and everyone exists in their most experienced and updated state.

These are some of the heuristics to follow when designing a scalable system.

Before we start any project, we should have a solid idea of how many users or tasks the project would need to handle. You should have this conversation with your boss if it’s at your job.

I prefer to overestimate the number of expected users by a margin of 10 times. If the expected users are 10, I craft the project with 100 in mind. If it’s 100, I expect 1000, and so on. This helps me prepare for unaccounted circumstances, which most likely there would be.

1. Caching

Ask yourself… If I was designing the world, and every time I needed to fetch something, I had to fetch it from the very source and compile it before properly accessing its most valid state, would that be a very efficient system? It essentially means, that for every minute you exist, you need to be recreated from scratch before you can perform a certain action.

Obviously not efficient.

This is why caching is useful. It allows us to access the most valid and useful form of an object. Obviously, the world is far better at updating its cache than we are but you get the point.

We use caching to offload the database as much as possible and access frequently accessed items as fast as possible without causing additional stress on the system.

Always ensure to store your cache in memory which is accessed via the RAM than in disk which is slower than accessing via RAM.

There are softwares that can help us handle caching effectively. Some of them include: Memcached, Redis, Varnish Cache, CDNs like Cloudflare, etc

2. Modularity

We have to break things down to their MVP — Minimum Viable Program. Breaking down the software to separate functionalities helps keep the code clean and maintainable. It allows for easy testing and deployment. It also allows us to easily add more modules and features to our software. All these enhance scalability.

3. Asynchronous and Non-blocking I/O

Running I/O operations asynchronously is a must. We do not want to have to wait for a task or request to finish before performing another task. Before we build our software asynchronously, we must carefully ensure the software would be best built using asynchronous methods.

We use asynchronous operations when all the tasks to be performed are independent of one another. That is, the input and result of one task aren’t dependent on the result or input of another.

We could use asyncio, futures, or threading modules in Python to achieve this.

4. Load Balancing

Load balancing is the distribution of traffic across different servers. A load balancer can be either software or hardware. It determines which server a particular task or traffic should operate in and its chosen server is usually based on criteria such as size, estimated time to complete, least busy server etc.

Using load balancers to distribute incoming traffic across multiple instances of the software ensures that no single instance is overwhelmed.

5. Fault Tolerant

When designing our application, we must ensure its able to withstand and quickly recover from any error it encounters. This is usually done using error-handling methods such as try-and-except/catch blocks.

No single function should have enough power or control to block our entire application from executing or performing other tasks.

We must ensure we design the software resilient to failures/errors by graceful degradation, failover mechanisms, exponential backoff, etc.

6. Database sharding

This involves splitting a database into separate smaller components (usually called shards) to reduce server load and increase response times. A database server can only handle so much load before it begins to slow down your application. When building scalable applications, we must take this into consideration and address this properly.

We can separate the database using different methods. By id (eg 1–10,000, 10,001 – 100,000, etc), using some hashing algorithm to determine which database to allocate a particular record to, separating records based on a column alphabetically eg (Database 1 — [1, Alexander], Database 2 — [2, Bright]).

When choosing our desired method, we must ensure it’s fit for both retrieval and deposition into the database. We do not want to use a certain method only to find out that to retrieve that data we have to search through every single database.

We would also want a method that enables our data to be evenly distributed among each database. This would often require knowledge of the problem context and use case.

Using database sharding allows us to scale horizontally rather than vertically which promotes scalability.

A good database sharding addresses all the above.

7. Use Microservices architectures

Using microservice architectures enables us to loosely couple our application which means we can decrease or increase each service capacity based on demand.

We can do this by either separating our services/components on different servers from the same provider or on different servers from different providers.

Rather than the traditional way, which is monolithic architecture, we can easily scale on demand.

Loosely coupling components also keeps the application running even if one component fails. So our application is almost guaranteed a 100% uptime.

The application is also easy to maintain since developers can easily remove a component and replace it with a new updated version without disrupting the entire software/application.

8. Monitoring and Auto-scaling

It’s critical in any application to have monitoring services integrated into our application. This would help us track important metrics such as failures, application-related errors, outages, performance, speed, etc.

When I developed my first mobile app, I used Sentry as my monitoring service. But when deploying to the cloud, your cloud provider usually has monitoring services out of the box for you. For instance, AWS has CloudWatch.

9. Choosing the right technology

This has to be the most critical aspect. The ability for your entire application to be scalable relies on the technology it’s involved with.

You want to ensure you choose tech stacks like Java or Python which allows us to follow basic OOP principles. These principles also make our projects highly scalable. Node Js is another tech stack that can enhance the scalability of our application.

They allow us to easily reuse and maintain our code base over time. They equally allow us to separate the concerns of our application which promotes readability and reusability.

10. Choose a horizontal scale over a vertical one

You might find that the system is limited to scale horizontally sometimes. Yet, these limits regularly extend databases in the same direction. Besides, with a horizontal scale, you can just add another server rather than upgrade the existing one. It will save you some money.

Bonus: Take work away from the core.

The number of clients will exceed the number of available services you have. So, to avoid any kind of bottlenecks while scaling the application up, distribute as much work as you can away from the core.

There are hundreds or probably thousands of Python best practices out there and depending on who you ask, you would get some slight variation on a given practice.

The internet has given everyone the right to voice an opinion. Including even me. But in this article, we will be dealing with 50 Python best practices that are set in stone.

These are practices that even God himself can’t tweak. These practices differentiate the pro from the amateur and a lot of them can also be adapted for various programming languages.

Most Python developers need somewhere to quickly test their code or debug errors. I developed a website called python-fiddle.com which you can use to quickly test out a code and it uses AI/LLMs to help find solutions to possible errors.

If You’re INTERESTED IN MKING MONEY ON UPWORK, JOIN OUR DISCORD: https://discord.gg/xPMWCAkqtq

LAW 1: Avoid Comments At All Cost

Comments have a way of being falsely true. They deviate the mind of the reader from what the code is actually doing to what someone else says it’s doing.

This can become very problematic as time passes and the code receives updates or changes. At some point, the comment becomes a lie and everyone now has to observe the truth through the lens of the lie.

Comments must be avoided at all costs. A comment forces the reader to inherit your thinking which at best is in the past. When a function or class changes, most likely, its comments do not change along with it. Most likely, they block the reader from thinking forward.

A comment signifies that the writer was mentally incapable of providing a well-descriptive class, function, or variable name. It exposes the lackluster attitude of the programmer and forces the team to inherit such an attitude.

Avoid comments at all costs.

In Law 14 and 15, you would learn when and when not to use a comment.

LAW 2: Do Not Use Type Attributes As A Name For Variables

Sometimes we want to specify that a particular variable is a string or an int. Some developers would therefore name the variable like this: name_of_variable_str or name_of_variable_int.

This can be quite redundant especially if the variable is intuitively a string. A name variable, for instance, can never be an int type.

But when the variable type isn’t intuitive, rather than specifying the type when naming the variable, the best way to do this is to use type annotations.

name_of_variable:str = value instead of name_of_variable_str = value.

This way, everyone can tell the variable is a string while keeping the code neat & concise.

LAW 3: Class Names Should Be Nouns.

It’s always best practice to keep your class names as nouns.

This is mostly because a class object is used to identify or represent a bunch of characteristics and actions. Similar to how a goat represents some characteristics like horns and actions like nod_nearby_humans.

It also makes the code very readable and less redundant. For example, Goat.get_horn_length() instead ofGetGoat.get_horn_length() .

LAW 4: Function Names Should Be Verbs

Functions are best named as verbs because they help the neighboring developer clearly understand what actions they perform.

It also eradicates the need for comments and allows any developer to mentally conceptualize without having to look at the raw code.

LAW 5: Functions Should Specify The Parameter And Return Type

When defining a function, you want to always specify what the arguments’ types are and also what data type the result of the function returns.

This would help both you and the devs in your team know what to expect without always having to use print statements to get a visual understanding.

Bad Practice

Good Practice

LAW 6: A function Must Perform One Functionality Only

Junior devs are fond of breaking this rule. Having a function perform one function is crucial because it helps expose where the bugs are located, enables reusability, and does exactly what the function name says.

You don’t want to do something like this…

(Note that the methods and attributes are just for example purposes.)

This checks if the address is valid and after checking returns the latitude and longitude. The function does two things. Checks if the address is valid and returns the geolocation of the address.

Here is a better way of doing this.

The above functions do only one thing and nothing more. Although it may seem more verbose, it’s a lot more concise and readable.

Knowing exactly what is “one” functionality can be a bit difficult, especially for new devs. You need to be very specific on what you want a function to do.

Usually, if you can extract or group some of the actions in a function as a whole other function, then it’s probably doing more than one thing.

Another way you can tell is if it has more than one level of abstraction…

LAW 7: Functions Should Be At The Same Level Of Abstraction

When we talk about functions being at the same level of abstraction, we’re referring to the idea that a function should perform a single, well-defined task. That task should be at a consistent level of abstraction throughout the function.

In other words, the function should focus on a specific level of detail or complexity, and all the functions’ operations should operate at that same level.

Good Practice

This function has statements that are of a low level of abstraction. Things like sum, len, etc.

Bad Practice

This function, for instance, has a bunch of levels of abstraction. get_numbers() is a high level of abstraction, the list comprehension is an intermediate level of abstraction, and sum is a low level of abstraction.

Basic REST API Principles That make you a 1% programmer.

LAW 8: A Function and It’s Argument(s) Should Be Like Brothers And Sisters

A function name should be closely tied to its arguments. You don’t want to have a function that seems to do one thing and the arguments passed do not correlate with the function name.

Bad Practice

write(True)

Good Practice

write(name)

The second one is more descriptive of what exactly the function is doing. It is clear to whoever reads this that we are writing a name.

The first one isn’t as explicit as the second. You have to make guesses and possibly even have to view the entire function.

LAW 9: Functions Should Be Small

A function is meant to be reusable. And the bigger the function gets, the less likely it is to be reusable. This also correlates to why a function should do only one thing. If it does only one thing, there’s a high chance it’s going to be small.

I plan to start posting content on linkedln too. so if you’re interested: My Linkedln: https://www.linkedin.com/in/alexanderobidiegwu

LAW 10: Avoid Noise/Redundant Words

There are times when devs use words that don’t provide any further clarification on a variable or function’s meaning. Things like this:

Without having prior knowledge of the function’s implementations, a developer seeing this doesn’t know which function to use.

LAW 11: Do Not Be A Dirty Programmer

Any senior dev would tell you he only feels clean when his codes are clean. This is because a dirty code makes up a dirty programmer. A clean code encourages everyone on the team to continue writing clean code. You must aim to always write clean code.

But what is a clean code? A clean code is well structured and arranged.

A clean code doesn’t hide bugs. It exposes anywhere a bug could hide to the programmer and makes room for an easy fix without complete refactoring.

LAW 12: Open Closed Principles

The open-closed principle (OCP) states that a class, method, or function must be open for extension but not modification. This means that any class, method, or function defined can be easily reused or extended for multiple instances without changing its code.

Let’s take for instance, we have a class called address.

This fails to adhere to OCP because whenever there’s a new country, we would need to write a new if statement to complement that. This might seem simple now but imagine we have 100 or more countries to take into account. How would that look?

This is where OCP comes into play.

This is a more robust solution because now we don’t need to modify either the class or its functions. If we ever want to consider a country and its capital, we can simply adjust our capital dictionary.

Another common example you would see is using class inheritance.

For example:

This is a wrong method because anytime we want to add a new payment method, we would always need to modify the PaymentProcessor class.

A better way to do this is:

This way, whenever we need to add a new payment option, say crypto or paypal, we wouldn’t need to edit or modify any class to achieve this. we could simply do this:

LAW 13: Liskov Substitution Principle

If we look at the previous principle, when making payments using crypto, we don’t exactly specify the cryptocurrency we are transferring. We only specify an amount. So suppose we want to specify a cryptocurrency, we would usually do something like this:

from abc import abstractmethod

class PaymentProcessor:

   @abstractmethod
   def pay_tax(amount, crypto):
      pass

Then when calling each payment processor, we declare the crypto argument as a None type or give it a default value to avoid passing in any argument if not needed. Both cases fail to adhere to the Liskov Substitution Principle.

This is because the parent class or the abstract class, contains an argument that isn’t relevant to most of our subclasses

The Liskov Substitution principle (LSP) states that “Objects of a superclass should be replaceable with objects of its subclasses without affecting the correctness of the program”.

To adhere to LSP, we would need to define the cryptocurrency within the CryptoPaymentProcessor class to avoid any irrelevant clash with other subclasses.

class CryptoPaymentProcessor(PaymentProcessor):
   def __init__(self, crypto):
      self.crypto = crypto
  
   def pay_tax(amount):
      print(f'Your {self.crypto} wallet is being processed for tax payment')
      print(f'You are to be charged {amount}')

LAW 14: Know When To Use A Comment

Every time you need to use a comment, be ashamed of your inability to express yourself in code. But there are some situations whereby the use of comments can actually help to express the underlying workings of a code than the code itself. Here are 5 best examples of “good” comments.

Informative comments
Making comments informative can always help express the code to the reader. A comment that highlights the return value of a function, for instance, would provide more clarity. But comments like this can be made redundant through the use of well-descriptive functions or variable names.

TODO comments
Comments like this help other programmers know that this is an unfinished function/task or it requires modification. Maybe there’s a better way to implement a certain function that wasn’t taken advantage of. Maybe the code fails periodically.

Regardless of your reason, comments like this provide more value than they take. TODO comments are also less likely to be left untouched as the code changes or improves because they are usually remembered to be removed once the task has been completed or modified properly.

Warning of consequences
Sometimes, we want to tell other developers about potential landmines. Stepping on these landmines could have some unforeseen consequences. And we want to all survive through the day. Comments can save the day in this situation.

Suppose a certain code takes time or has the possibility of overloading certain systems, a warning like #CONSUMES A LOT OF COMPUTING RESOURCES would be beneficial to a reader or other programmers.

LAW 15: When Is A Comment Bad?

Noise comments
These are comments that reinstate the obvious. They provide no further information and only increase the verbosity of the code. A lot of times, we skip through comments like this when we see one. Examples of noise comments are:

# adds to animal list
animal.append(dog)

Non Local information
One mistake a lot of programmers make when using comments is giving information about global information locally. When writing comments, ensure that it remains relevant only to the function or statement it is referencing. Anything outside of that should be pruned out.

Unobvious comments
We often write comments that seem obvious to us but not to someone else. The connection between your comments and the function referenced must be clear. They must both follow the same step or procedure at sync. You don’t want your comment to need to have its comment as well.

Short Functions
Most likely we do not need a comment for a short function. The shorter/smaller the function, the more likely it can be described with a good name. Hence, they are usually self-descriptive.

3 Ways To Run A Python Function Asynchronously

LAW 16: Keep Source Files Short

A source file should be between 100–200 lines with 500 lines at most. Unless you have very good reasons to choose otherwise. Keeping a source file short has various obvious benefits such as reusability and readability. They are also easier to maintain and update since we spend less time scrolling and trying to connect the dots.

LAW 17: Know When To Use Blank Lines

Blank lines are a way to tell the reader we are progressing into a new and separate concept. Each group of lines represents a complete thought. It helps readers understand when a thought ends.

LAW 18: Keep Related Lines Of Code/Functions/Classes Close

Go where you are valued

Can you remember when you had to scroll all the way to the top of the script, just to find out what a function does and how it relates to where it’s being called?

The worst part of this is you can’t understand the relation in one glance. It’s a back-and-forth movement. Once you’ve experienced this, you’ll understand how valuable it is to keep related code to each other.

A function/Variable should be as close to where it’s needed the most or has the most importance. It should be furthest away from where it has the least importance.

LAW 19: Know When To Use White Spaces

We use white spaces to show that two things are not strongly related and no white space to associate things that are strongly related. For instance when defining functions…

def create(name):
    print(name)

There exists no whitespace between the parenthesis between the function and the name variable. If there was, it would look very disjointed and uncoordinated…

def create (name):
    print (name)

Arguments passed into a function, should be separated to show that the arguments are separate.

LAW 20: Obey Team Rules

Almost every developer has his/her style. From the way we name our files, to the way we write print statements.

But when it comes to working with other devs, you want to dumb down those personal preferences and adopt the team’s preferences. Not everyone might be able to see beauty in your code the way you do.

LAW 21: Avoid Magic Numbers

A Magic Number is a hard-coded value that may change at a later stage, but that can be therefore hard to update.

For example, let’s say you have a Page that displays the last 50 Orders in a “Your Orders” Overview Page. 50 is the Magic Number here because it’s not set through standard or convention, it’s a number that you made up for reasons outlined in the spec.

Now, what you do is you have the 50 in different places — your SQL script (SELECT TOP 50 * FROM orders), your Website (Your Last 50 Orders), your order login (for (i = 0; i < 50; i++)) and possibly many other places.

# Bad
SELECT TOP 50 * FROM orders

# Good
NUM_OF_ORDERS = 50
SELECT TOP NUM_OF_ORDERS * FROM orders

LAW 22: Avoid Deep Nesting

Limit the levels of nesting within loops, conditionals, or functions to improve readability.

# Bad
if x:
    if y:
        do_something()

# Good
if x and y:
    do_something()

LAW 23: Avoid Temporary Variables

# Bad
temp_result = calculate(x, y)
final_result = temp_result * 2

# Good
final_result = calculate(x, y) * 2

LAW 24: Avoid Cryptic Abbreviations

Use descriptive names instead of cryptic abbreviations to enhance readability.

# Bad
def calc(x, y):
    pass

# Good
def calculate_total_price(quantity, unit_price):
    pass

LAW 25: Avoid Hardcoding Paths

Refrain from hardcoding file paths or URLs; use configuration files or environment variables instead.

# Bad
file_path = "/path/to/file.txt"

# Good
import os
file_path = os.getenv("FILE_PATH")

10+ Scalability Laws To Follow In Your Next Project.

LAW 26: Always use Try-Catch-Finally Statements

When writing code, it’s always best to include error handling also. This can help speed up the debugging process and increase the sophistication of the code while keeping it clean and manageable.

You want to use try-catch statements when a certain code is more likely to return an error.

Things like API requests, file handling, etc tend to fail or raise errors due to one reason or the other. Using try-catch statements for multiplication or division is just uncalled for and creates more problems than it solves.

LAW 27: Provide Context with Exceptions

Whenever you catch an exception, it should provide enough context for anyone who encounters it to know where the problem is from and debug accordingly.

Create informative error messages along with your exceptions which can be done when printing the error. Mention the context in which the operation failed and the type of failure.

Law 28: Avoid Using Multiple Exception Class

Have you ever seen a code like the one below…

try:
 pass
except ValueError:
 pass
except TypeError:
 pass
except IndexError:
 pass
except KeyError:
 pass
except FileNotFoundError:
 pass

This is extremely bogus and takes away (in verbosity, complexity, and maintenance) the additional help it provides in error handling.

It’s often better to use a more general exception to catch any sort of error we might come across. This type of exception, by default, includes the type of error we got.

try:
 pass
except Exception:
 pass

Only be specific about the type of error you want to catch, when you want all other errors to pass through.

LAW 29: A function must either mutate or return something, but not both.

Whenever we are creating a function, we should keep in mind what exactly that function is supposed to do. Does it mutate the arguments passed? Or does it need to return something?

If the function mutates the argument(s) passed, we only need to do that in that function and nothing else.

But what do I mean by mutation? If the function changes the contents of the argument(s) or changes the data type of the arguments, it mutates.

def changed(array):
    array.append('hello')

If the argument(s) is used to create another variable, then it doesn’t mutate. For instance, if an argument called time was used to calculate a distance, then it doesn’t mutate, which means distance can be returned in that function.

def calculate_distance(time, speed):
    distance = speed * time
    return distance

But there’s a way to get the best of both worlds. You can copy the argument(s) of the function and perform mutation on those. This way we avoid any side effects.

def changed(array):
    array_copy = array[:]
    array_copy.append(4)
    return array_copy

LAW 30: Not all function names should be verbs

Yes, I know. I previously said all function names should be verbs, which is true generally, but there are times when a function name should take the form of a noun. And knowing when it should take either is based on the previous law/ law 29.

If a function returns something without mutating its arguments, the function’s name should be a noun. If a function mutates the arguments and returns none, it should be a verb.

This is a common convention built into Python itself. Methods such as sort, and append are verbs because they mutate the data type and return None while methods like sorted, sum, product are all nouns because they don’t mutate any arguments passed and return a new copy of the data.

There are obviously exceptions to this and whenever you feel you’ve encountered one, feel free to fall back on using verbs.

LAW 31: Classes should be small

Yep! Classes should be as small as possible. Just like functions.

The only difference is that in functions, size is determined by the number of lines in that function while in classes, it is determined by the number of responsibilities in that class.

Usually, a class name represents the kind of responsibilities it might possess but when the name is ambiguous or too general, most likely we are giving it too much responsibility.

This takes us back to SRP (single responsibility principle) which states that a class should only have one reason — one responsibility — to change.

LAW 32: Classes should have a small number of instance variables

Instance variables, in case you don’t know, are variables that are defined when the class is defined or instantiated.

class Animal:
    def __init__(self, name):
        self.name = name #instance variable

If all our functions are related to the class responsibility, there’s no reason why we should have a lot of instance variables.

When we begin to have tons of instance variables, is when certain functions in the classes deviate away from the core responsibility of the class.

These functions tend to come with their own variables that other functions in the class do not need.

LAW 33: Your classes should be cohesive.

All the functions in your class should contain one or more of the instance variables. The more your functions relate to or contain the instance variables in your class, the more cohesive your class is.

LAW 34: Use the with Statement for Resource Management

Use the with statement to automatically manage resources like files or database connections, ensuring they are properly closed or released.

# Bad
file = open("example.txt", "r")
data = file.read()
file.close()

# Good
with open("example.txt", "r") as file:
    data = file.read()

LAW 35: Avoid Complex Ternary Expressions

Refrain from using overly complex ternary expressions; favor readability over brevity to make code more understandable.

# Bad
result = "even" if number % 2 == 0 else "odd" if number % 3 == 0 else "neither"

# Good
if number % 2 == 0:
    result = "even"
elif number % 3 == 0:
    result = "odd"
else:
    result = "neither"

10 Hard Python Projects For Intermediates To Boost Your Python Skills & Portfolio.

LAW 36: Use ‘is’ and ‘is not’ for Identity Comparison

Most of the time, we use == to check the comparison between two variables. This is usually okay for immutable data types like strings or integers because immutable objects with the same value are usually stored in the same memory location, so a memory location check is not needed.

But when dealing with mutable data types, such as list, dicts and custom objects, it’s often better to use the is comparator because it checks the subtype of the variable and the memory location.

The memory location of mutable objects are usually not the same due to the way Python works. Python stores mutable objects in different memory locations because they can be changed at any time and each of them must be independent of one another even though.

Strings, tuples, and integers cannot be changed once created.

Run this using the link below:
https://python-fiddle.com/saved/nV6iEIyBuHm2mevD9Bhg

# Example 2: Checking if two lists refer to the same object
list1 = [1, 2, 3]
list2 = [1, 2, 3]

# Less preferred: Using ==
if list1 == list2:
    print("Lists are equal in value")

# Preferred: Using is
if list1 is list2:
    print("Lists are the same object")

# Note: In this case, list1 and list2 are different objects with the same value,
# so using `is` would give a different result than `==`.

LAW 37: Dependency Inversion Principle

The Dependency Inversion Principle (DIP) is a fundamental principle of object-oriented design that promotes loose coupling between components and facilitates easier maintenance and extension of software systems.

It states that high-level modules should not depend on low-level modules, but both should depend on abstractions.

In other words, classes should depend on interfaces or abstract classes rather than on concrete implementations.

# Bad
class Logger:  
    def log(self, message):  
        with open('log.txt', 'a') as f:  
            f.write(message + '\n')  
  
class Calculator:  
    def __init__(self):  
        self.logger = Logger()
   
    def add(self, x, y):  
        result = x + y  
        self.logger.log(f"Added {x} and {y}, result = {result}")  
        return result

In the example above, we define the class Logger and directly create a new instance of it in our Calcuator class. This means that Calculator now depends on Logger class and if for any reason we change Logger class, we now also have to modify Calculator class.

And as you can see this also fails to adhere to the open closed principle (open for extension, closed for modification).

This tight coupling makes the code harder to test because we can no longer simply use a fake logger class when testing.

# good
from abc import ABC, abstractmethod  
  
class LoggerInterface(ABC):  
    @abstractmethod  
    def log(self, message):  
        pass  
  
class Logger(LoggerInterface):  
    def log(self, message):  
      with open('log.txt', 'a') as f:  
          f.write(message + '\n')  
  
class Calculator:  
    def __init__(self, logger: LoggerInterface):  
        self.logger = logger  
   
    def add(self, x, y):  
        result = x + y  
        self.logger.log(f"Added {x} and {y}, result = {result}")  
        return result

This promotes modularity because changes to one component do not necessarily require changes to other components as long as the interface remains consistent.

This modularity enables easier understanding, modification, and extension of the codebase.

LAW 38: Avoid Using ‘assert’ for Data Validation

Use assert statements for debugging and development purposes only; avoid using them for data validation in production code.

# Bad
assert x > 0, "x should be positive"

# Good
if x <= 0:
    raise ValueError("x should be positive")

LAW 39: Avoid Hard-Coded Numbers

Use named constants instead of hard-coded values. Write constants with meaningful names that convey their purpose. This improves clarity and makes it easier to modify the code.

def calculate_discount(price):  
    discount = price * 0.1 # 10% discount  
    return price - discount

The example below uses the hard-coded number 0.1 to represent a 10% discount.

This makes it difficult to understand the meaning of the number (without a comment) and adjust the discount rate if needed in other parts of the function.

def calculate_discount(price):  
    TEN_PERCENT_DISCOUNT = 0.1  
    discount = price * TEN_PERCENT_DISCOUNT  
    return price - discount

The improved code replaces the hard-coded number with a named constant TEN_PERCENT_DISCOUNT. The name instantly conveys the meaning of the value, making the code more self-documenting.

LAW 40: Follow the DRY (Don’t Repeat Yourself) Principle

Avoid writing the same code more than once. Instead, reuse your code using functions, classes, modules, libraries, or other abstractions. This makes your code more efficient, consistent, and maintainable.

It also reduces the risk of errors and bugs as you only need to modify your code in one place if you need to change or update it.

# Bad

def calculate_book_price(quantity, price):  
    return quantity * price  
def calculate_laptop_price(quantity, price):  
    return quantity * price

# Good

def calculate_product_price(product_quantity, product_price):  
    return product_quantity * product_price

LAW 41: Follow Established Code-Writing Standards

It’s important to follow commonly accepted conventions in terms of spacing, comments, and naming. Most programming languages have community-accepted coding standards and style guides, for example, PEP 8 for Python

Some common conventions are:

• Use snake_case for variable, function, and class names.
• Use spaces over tabs for indentation.
• Use 4 spaces per indentation level
• Limit all lines to a maximum of 79 characters.
• Break a line before a binary operator

LAW 42: Law of Demeter

The law of Demeter, in simple terms, means that a module/function/class can have knowledge or make reference to another module/function/class immediate neighbors but should not have any knowledge further than that.

By immediate neighbors, this means methods or functions or variables that can be directly accessed.

An example should clear this up…

class Order:
    def __init__(self, customer):
        self.customer = customer

    def get_customer_name(self):
        # Violation: Order knows too much about the customer's structure
        return self.customer.get_profile().get_name()

In this example, the Order class directly accesses the customer’s profile to retrieve the customer’s name. This violates the Law of Demeter because Order is reaching into the internal structure of the Customer object to access its profile and name.

It has gone past an immediate neighbor and now knows too much about the customer’s object.

class Order:
    def __init__(self, customer):
        self.customer = customer

    def get_customer_name(self):
        # Adherence: Order only interacts with its immediate collaborator
        return self.customer.get_name()

In this adherent example, the Order class only interacts with its immediate collaborator, the Customer object, and calls a method directly on it to retrieve the customer’s name.

It does not reach into the internal structure of the Customer object, thus following the Law of Demeter.

LAW 43: Readability over conciseness

Code needs to work and be understood by the machine executing it. However, other developers also need to be able to understand the code, especially if you are working on a project with multiple people.

That is why the readability of code is always more important than its conciseness when it comes to software development.

There is no point in writing concise code if other developers cannot understand it.

LAW 44: Keep Imports Clean

Import only the necessary modules and symbols to keep the import section clean and improve readability. When we import everything (*) from a module, all variables, functions, and classes are also imported which can make it very difficult to know where a certain function/class originated from and can also be a pain when using modern IDEs.

Imagine you want to write a function with the name get_file. You click g and your IDE recommends you a list of functions/classes/variables that start with the letter g. It can become quite chaotic.

This becomes more of a pain when you want to call that function. Your function name could be lost in between the recommendations and now your IDE becomes more of a problem than an efficient solution.

# Bad
from module import *

# Good
from module import symbol1, symbol2

LAW 45: Do Not Return Null/None

Usually, when we define functions, by default they return None if no return value is specified. But when we explicitly return None, we are indirectly telling the reader that the function can return something else except None.

And if this is not true, it can cause a lot of miscommunications.

LAW 46: Separate Construction From Its Use

Separation of concerns has been a very fundamental principle in software engineering. We need to know how to separate the construction of our software from how it’s being used.

This often means separating the startup process, — which is when our dependencies and objects are wired together — , from the run time logic, — which is when the logic of the application executes based on inputs from the user or other triggers — .

One common way to separate construction from its use is to construct the application logic in a file/function/module called main.

The main function builds the objects necessary for the application to run smoothly. This frees other modules from being tightly coupled to the application and promotes reusability and modularity.

LAW 47: Simple Design Contains All These Rules

Runs all the tests: A system might have a perfect design on paper but if there’s no way to verify that the system actually works as intended, then the design on paper becomes questionable.

Contains no duplication: Duplication is the primary enemy of a well designed system

Expresses the intent of the programmer

Minimizes the number of classes and methods

LAW 48: Avoid Nested Try-Except Blocks

Refrain from nesting try-except blocks excessively to prevent overly complex error handling logic.

# Bad
try:
    try:
        # Code that might raise errors
        pass
    except ValueError:
        # Handle ValueError
        pass
except Exception as e:
    # Handle any other unexpected errors
    pass

# Good
try:
    # Code that might raise errors
    pass
except ValueError:
    # Handle ValueError
    pass
except Exception as e:
    # Handle any other unexpected errors
    pass

LAW 49: Use Concurrency Only When Needed

It’s very easy to write bad code when implementing concurrency functionality.

It’s also very easy to write clean code when implementing very faulty concurrency functionality. Usually, you might not even be aware that it is faulty until a lot of stress is put on the system.

You want to pick your battles very smartly.

There are multiple reasons why your concurrency code might fail. Here are some:

Starvation: Starvation occurs when a thread or process is unable to gain access to a shared resource indefinitely, despite attempting to do so. This can happen when other threads or processes continually acquire and hold the resource, preventing the starving thread from making progress.

Deadlocks: Deadlocks occur when two or more threads or processes are blocked indefinitely, waiting for each other to release resources that they need. This can happen when each process holds one resource and waits for another resource held by another process, creating a cyclic dependency.

5 Concepts Every Python Engineer Should Know in 2024

LAW 50: Follow The 49 Rules

These laws are only there to guide you on your journey as a software engineer. You should abide by them whenever needed.

But as you grow in experience and skills, you want to be able to decide when it’s best to follow a certain rule and when it’s not.

This gut feeling/intuition only comes to those who have mastered their craft and if you’re a newbie or you just started your career 2 years ago, it’s usually best you follow these laws like it’s your only ticket to heaven.

Most Python developers need somewhere to quickly test their code or debug errors. I developed a website called python-fiddle.com which you can use to quickly test out a code and it uses AI/LLMs to help find solutions to possible errors.

Linear regression is by far one of the most impactful statistical methods used in the 21st century. From computer science to medicine, to geology, to aeronautics, to behavioral and social science, and many more.

Everyone involved in Machine Learning or Data Science must have a strong understanding of linear regression. It’s almost a default. This article aims to help you do justice to that.

But in order to do justice to that we’re going to need to go all the way back to the 1800s and 1900s, to understand the thinking behind the thinking. Get a grasp of how linear regression became what it is today.

We won’t be doing a lot of maths or graphs here. Not yet. This article’s existence is mainly to please your intuition. To see Maths as a means by which humans communicate non-verbally at a global scale.

We’ll be using this Python fiddle to implement any graphs or plots.

Get a coffee and let’s dive in!

THE FEAR OF ERRORS

Before diving in, we need to briefly touch on the concept of errors in statistical theory.

Normally we see errors as a mistake, but in statistics, an error is simply just the difference between an estimated value and the actual true value.

For a more in-depth analysis on errors, read this article.

Wikipedia:
In statistics, “error” refers to the difference between the value that has been computed and the correct value.

Oxford:
The quantity by which a result obtained by observation or by approximate calculation differs from an accurate determination

Handling errors has always been a problem. Astronomers since the 18th century had a deep-rooted fear of combining observations.

The fear was not based on observations under similar circumstances, as those were easily solved using mean, but by observations made under varying circumstances (non-standardized observations).

Combining such observations to get an estimated “truth value” was faced with serious skepticism.

How this truth value is determined is left to the observer or analyst.

Theodore M. Porter best describes the problem:

“Astronomers had been debating for several decades the best way to reduce great numbers of observations to a single value or curve, and to estimate the accuracy of this final result based on some hypothesis as to the occurrence of single errors.”

This simply means astronomers had difficulties finding a way to standardize varying ranges of observations (eg 1 and 1000), and even after standardizing it to a single range or value, finding a way to properly determine the accuracy gotten from this range or value.

If you don’t know what standardizing means, it’s simply bringing down values within similar ranges. (eg 1 and 1000 * 1/2000 are close in range)

Taking the mean was not an option because the mean value was too far away from most of the individual observations.

Most also feared that the errors within each observation would multiply, not compensate, and move greatly away from the supposed “truth value”.

How these observations got contaminated with errors could vary which made the problem more relevant. It could be faulty instruments, human error, wrong calculations, atmospheric conditions, etc.

A rule was needed. One that would account for all observations and their variations.

THE INVENTION OF LEAST SQUARES

A man named Legendre came out with a formula that accounts for all observations and their variations. The method of least squares.

A Brief on Legendre
He was born in Toulouse and the family moved to Paris when he was very young. He came from a wealthy family and he was given a top-quality education in mathematics and physics at the Collège Mazarin in Paris.

The method was highly accepted across all domains. At that time, no statistical method had as much of an impact as this did. It began the acceptance of errors.

Legendre suggested that the sum of the squared deviations around the mean would provide some grand estimate of the error variance on the entire observations.

Remember…
Error is simply just the difference between an estimated value and the actual true value.

We square the deviations because we do not care about direction. We just want a total or an aggregate and hence squares remove any negative operator to make it possible for us to get a grand estimate.

We also square to identify and penalize outliers easily.

For example, if we have a dataset of [10,46,100,2345,832] and we calculate the mean, we get 666.6.

To calculate the sum of the squared deviations, we are going to….

(10–666.6)² +(46–666.6) + (100–666.6)² + (2345–666.6)² + (832–666.6)²

431123.56 + 385144.36 + 321035.56 + 2817026.56 + 27357.16

Total deviation = 3981687

This is an aggregate of how much the dataset deviates from the mean. By quite some margin as we can see (mostly due to the square).

This helps us to know how well our datasets vary from each other. Also, by squaring the difference, we put more weight on outliers which can help us quickly figure out any variables that influence our prediction or model heavily. In our case, 2817026.56 or 2345 is the outlier.

Using graph

Suppose we have all these data points. And we try to plot a best-fit line in order to predict new values…

Our error is the difference between the actual data points (red points) and the line…

Once we get the difference of each point, we square each difference up and add them to get the sum of squared deviations.

With this aggregate value, we can now try to find the best-fit line (assuming this is not already it), by moving closer to values that drastically influence our aggregate value in order to reduce their deviation.

Our goal with the least square is to minimize the total deviation to as minimal as possible.

This method’s breakthrough was that “errors” could now be combined, but more importantly, minimized through a method that would enhance prediction, not through multiplying error, but reducing the overall quantity of it.

The method gained wide appeal and by the early 19th century, was used regularly in the fields of astronomy, geodesy, and eventually spread to the social sciences, including psychology.

One issue with the sum of squared deviation is that it assumes that your data point follows a normal distribution. eg {20, 18, 23, 22, 15, 24, 20, 29, 16, 19} rather than a skewed distribution eg {25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100}.

By normal distribution curve, I mean most data points in a set revolve around the mean or some central tendency (eg median) with fewer and fewer data points as they go further and further away from the center. For instance, population, age, weight, height, house prices, etc.

Don’t bother about the mathematical complexities of the image above, just understand how it relates to a normal distribution curve/dataset.

This is particularly one of the reasons why least square is used with regression analysis.

THE FALSE INVENTION OF REGRESSION

A man named Bravais Auguste came first to find the regression line but had no idea he did so. He was a physicist and a professor of astronomy. Most of his works were aimed at crystallography.

He is best known for a paper he wrote in 1846 titled “Analyse mathématique sur les probabilités des erreurs de situation d’un point” translated to “Mathematical analysis on the probability of errors of a point”.

He found what would eventually be coined the “regression line.” He did so by investigating how the various elliptical (oval-shaped circles) areas of the frequency surface vary given various directly observed quantities.

Frequency surface means the frequency or quantity of values in different parts of distribution (both extremes and the middle).

Through this, he found the line of regression, but did not realize it, and thus could not “make the leap”. This was mainly because he didn’t have the goal in mind.

At the time, getting rid of errors from observations was the highest priority and thus, bravais goal was to do just that. His motivation was to show how the errors obtained from observations were independent, not associated.

This blindsided Bravais to the idea of correlation.

He was able to get the line (i.e., “OM”) which is similar to the common regression line. But this is not a result of observing x and y and determining their association, but of the fact that x and y are functions of certain independent and directly observed quantities”

But since he managed to somehow derive the regression line, shouldn’t he be crowned for it? I mean, he was one of the first to study the existence of two or more errors, something that only a few at the time had done before.

Second, he produced the product term for the correlation coefficient and discovered the mathematical equation for what 30 years later was coined the regression line.

Product term for the correlation coefficient
Multiplication of deviations X and deviation of Y divided by the standard deviation

Third, he discussed both of these in terms of the normal surface of correlation and identified the varying shapes of the ellipses that would exist given varying quantities and accompanying errors.

Well, let’s talk about the real father — Galton.

THE INVENTION OF REGRESSION

Galton observed that the mean value of inherited characteristics deviates away from the mean of the mid-parents(average height of both parents) and towards the mean of the population.

Suppose you have tall parents, your height wouldn’t be directly a result of the mean of your parent’s height but more towards the mean height of the entire population.

So if you have extremely tall parents, you are mostly likely to be shorter in stature and if you have extremely short parents, you are most likely to be taller.

The main thesis is that deviations in inherited characteristics could be explained by their regression towards the mean value. Extreme deviations did not produce equally extreme characteristics/deviations but in general, deviations of a lesser degree regressing towards the central tendency, the mean.

A brief on Galton’s motivation…

Galton possessed a penchant for observations, some would even say an “obsession.” In short, Galton measured everything he could, from wind direction to fingerprints, to what would form the empirical roots of his later discoveries — physical attributes such as heights and stature. As Stigler (1986) remarks, Galton was greatly influenced by Quetelet in that he almost “rejoiced” in Quetelet’s concept of a theoretical law of deviation from an average (i.e., “error theory”). Galton even appears to have foreshadowed, using the concept of the error curve, the basis of modern hypothesis-testing, that of determining whether observed values arise from a single population, or from various populations.

He tried to graphically display the distribution of human physical attributes, with the hope of somehow showing that these attributes are inherited. This somehow plagued Galton’s mind which led him to perform an experiment that would calculate a reversion coefficient.

A reversion coefficient is a coefficient of correlation between two variables. This coefficient determines the rate of reversion to the mean and can help with forecasting.

He took a data set of 928 offspring and their parents to produce a Table of Correlation. Such a table plotted the heights of mid-parents against the heights of offspring.

For instance, if a parent was 70 inches in height, and the adult child had a height of 67, a checkmark was placed in the corresponding cell where the two entries formed a 90° angle. (female’s heights were multiplied by 1.08 to normalize the dataset.) These checks were then tallied to produce the numbers in the graph below…

What Galton noticed next was significant to his discovery.

He found that when line segments were drawn that connected each point where the values were the same, the lines formed an ellipse and the center point of these ellipses was the mean of values on the ordinate and abscissa.

He further found that at each point where the ellipse was touched by a horizontal tangent, all points lay in a straight line and each line was inclined to the vertical in the ratio of 2/3.

Those points where the ellipses were touched by a vertical tangent also lay in a straight line but this time inclined to the horizontal in the ratio of 1/3.

By measuring the heights of hundreds of people, he was able to quantify regression to the mean and estimate the size of the effect. Galton wrote that “the average regression of the offspring is a constant fraction of their respective mid-parental deviations”.

This means that the difference between a child and its parents for some characteristic is proportional to its parents’ deviation from typical people in the population.

If its parents are each two inches taller than the averages for men and women, then, on average, the offspring will be shorter than its parents by some factor (which, today, we would call one minus the regression coefficient) times two inches.

For height, Galton estimated this coefficient to be about 2/3: the height of an individual will measure around a midpoint that is two-thirds of the parents’ deviation from the population average.

By doing this, Galton confirmed his invention of statistical regression. Not mathematically, but through empirical data and observation.

One mind-blowing fact was that Galton was not a mathematician. He had no deep roots in maths and even before he concluded his experiment, he sent the data to a mathematician called Dickson to reproduce mathematically what he produced empirically. To his amusement, it was the same result.

Although Auguste Bravais touched statistical regression and correlation, Galton made it law.

Galton also notes that due to the law of regression, he’s very much against the current practice of proportional rescaling because it ignores the effect of regression.

If a thigh bone is 5% greater than the average thigh bone, we should not infer that a man is 5% taller than the average person. Such an inference would ignore the effect of regression and would tend to overestimate by an amount that is greater than even those on the extreme of the distribution curve.

CONCLUSION

Perhaps there is no better example in the history of statistics than the one before us to show that statistical techniques do not simply arise out of mathematical manipulation, but are often spun out by a social-political cause or other equivalent goal.

Hence, in this context, the statistical technique is but a “weapon” in the pocket of artillery of the proponent. Since Bravais had no particular reason for finding the technique of correlation, he did not.

Similary, since Galton’s theory of inheritance relied heavily on him finding a “mechanism” of inheritance, he had all the reason and motivation for discovering the technique of regression and correlation, and so he did.

I was a fraud three days ago. I thought I fully understood the concepts behind REST API, REST / RESTful. Until a colleague of mine was trying to communicate verbally to me and all I could hear was gibberish.

Every 3 words spoken was followed by a mental freeze. Every tacit word was a foreign language. It was a spit on my reputation. One I hoped to dissolve away.

I decided to dedicate this article to do just that. To save me from who I was 3 days ago. And not just me, but everyone else who perchance comes across such a harrowing situation.

Together, we endure this brief period of redemption. In 5 minutes, we enjoy salvation.

Contact me on Upwork for software development jobs relating to Python/SQL.

What is an API?

Application Programming Interface

An API is an interface that enables communication between different applications. Interface doesn’t mean software interface i.e. the front end, but a shared boundary that allows different components of a computer system to exchange information.

Interfaces help components communicate with each other and function independently.

For example, Open the following link in your browser to request a random computer question from the Open Trivia Database:

https://opentdb.com/api.php?amount=1&category=18

If you open it, you should see a JSON like this

{
  "response_code": 0,
  "results": [
    {
      "category": "Science: Computers",
      "type": "multiple",
      "difficulty": "easy",
      "question": "What does GHz stand for?",
      "correct_answer": "Gigahertz",
      "incorrect_answers": [
        "Gigahotz",
        "Gigahetz",
        "Gigahatz"
      ]
    }
  ]
}

This API helps a client (you currently) to communicate with the application’s server in order the retrieve the result when amount=1 and category=18.

What is REST?

Representational State Transfer

REST is simply a software/architectural design style that is used to guide how a client should communicate with a server. It can also be used for server-to-server communication.

In general, the one requesting the information is the client and the one (capable of) providing that information is the server.

But this doesn’t necessarily sound very intuitive given the full name, “Representational State Transfer”. What the heck could that mean?

Representational — To stand in for a particular item, object or person.
State — A given form of an entity at any particular point in time.
Transfer — To move something from one place to another.

So what exactly are we representing? A resource. A resource is simply data located on your server. This could be an image, HTML page, audio, video, etc.

So we are trying to represent that resource on an application. Not the exact state of a resource but a representation of it.

It’s like going to a high-end restaurant and while eating “high-end food”, you decide to take a picture of the food, just so others can see and use it in whatever way that’s relevant to them.

But ours is much closer to home than that. Instead of just seeing the food, we can actually get a taste of what the food is like or even eat it, but we’re not necessarily eating the exact one you ate in the restaurant. Just a representation (an image representation in this case). You get me?

So the state of the resource in the server is constant. But just the representation of it keeps changing.

This ability to transfer a resource using a representation of the resource state is what is called REST.

But what does a representation look like exactly? Let’s first see some examples of what a representation can be.

Some Examples of Representations

• JSON
• XML
• HTML
• IMAGE
• Video

There are many more forms of representations, which usually depend on the data being sent or received, but the most common type of representation is JSON.

But if you remember, at the beginning of this blog, I defined REST as “REST is simply a software design style that is used to guide how a client should communicate with a server”.

To put that into context, when we want to access a certain resource from a server, the server sends us an appropriate representation used to access that resource eg JSON, XML, etc. This is done using the content-type parameter in the header object in the REST API.

For example, to make sure that when our REST API app responds with JSON, clients interpret it as such, we should set content-type in the response header to application/json after the request is made.

You can check this using developer tools in the browser. Let’s load up YouTube and see what the content type is:

As we can see the content-type is text/html for the youtube webpage. I think that makes intuitive sense.

The process of using a REST design principle is simply being RESTful. They both mean the same thing. RESTful is just the degree to which an application follows REST design principles.

Principles of REST

Client-Server

The first principle is to separate the concerns of the user/client and the concerns of the server. By separating the two, we improve the scalability of the server and the portability of the application.

Scalability is improved on the server because the server can afford to serve multiple users at once and simplify its server components while portability is improved on the applications because it can easily be deployed on multiple platforms.

This can also help both parties add new technologies and evolve without affecting how they communicate with each other.

If it were otherwise, adding new technologies would be difficult because it would obstruct the flow of the users i.e. every update/change would require users to understand the technicalities and how to adapt to the changes.

Stateless

The second principle is stateless. If you allow yourself a brief moment into the abyss of information provided on the internet relating to this principle, you would surely come out exhausted.

Stateless is simply a guiding principle whereby each request from a client must contain all necessary information for the server to process that request without taking advantage of any state or context (eg sessions) stored in the server.

Therefore an application must avoid storing a client’s state/context on the server but on the client side (eg using cookies on browsers).

The advantage of this is that our servers can free up resources and handle more load. It enhances scalability and reliability is improved because it eases the task of recovering from partial failures.

Cache

The third principle is cache. Clients need to be able to cache representations in order to improve network efficiency and reduce latency.

Cache requires that representations be explicitly or implicitly set as cacheable. If a representation is cacheable, then a client cache is given the right to store the representation for a period of time.

A benefit of cache is that it improves speed and user-perceived performance. It also reduces server load by instead serving cached data, rather than requesting for a new one every time.

One drawback with caching representations is stale data. When there’s an update to a representation or application, the client won’t be able to view the update until the cache expires or is explicitly removed.

But this is only a problem if the data cached is frequently updated or requires a critical update.

Uniform Interface

This is a major feature that distinguishes REST from other network-based architectures. It is the foundation of REST.

It suggests that there must be a generalized way for servers to transfer information or communicate with clients independent of device or application.

That is, every client should be able to communicate and receive representations from the server in the same way, irrespective of the client’s device or specialized needs.

It’s like having everyone speak English instead of each person developing their own set of language that suits them.

In order for such an interface to be adopted, a few standards must be followed:

• Identification of resources: You use the URI (Unique Resource Identifier) to identify the resource required.
• Manipulation of resources through representations: Using HTTP methods to describe the type of communication with the resource. So for example GET means that you want to retrieve data about the URI-identified resource. You can describe an operation with an HTTP method and a URI.
• Self-descriptive messages: Self-descriptive messages mean that when transferring information or communicating, the media type of the representation must be clearly stated. This allows the end user to be able to understand how to parse and effectively use the data. MIME types are used when making messages self-descriptive. It consists of a type (eg application, text, etc) and a subtype (json, css, html) usually separated by a slash (application/json, text/html). This is usually set in the content_type (server) or accept (client) header parameter. Most libraries do this for you automatically.
• Hypermedia as the engine of application state (HATEOAS): It’s a principle that suggests including hypermedia links in API responses. It gives clients related URI and resources that can be requested and accessed. It is used to provide clients with a means to access URIs without the need to follow the application’s default structure of URI paths. For instance, if you request a blog resource (/blog), in the body of the API response, it should contain links to access comments of the blog (/blog/comments) or likes of the blog (/blog/likes).

Advantages of REST API

Widely adopted

REST API has massive users and due to this, there’s a community constantly updating documentation and building libraries that improve the speed of deployments when developing REST APIs.

Scalability

REST is highly scalable due to its approach when communicating and responding to clients.

Disadvantages of REST API

Lack of in-built security Features

REST APIs send data using HTTP(s). This is often enough for most services but when dealing with highly sensitive information (eg governmental information), it is usually an inadequate measure. This is because HTTP only encrypts data in transit but not at rest or destination. So once the data reaches its end user, it becomes highly vulnerable to attacks.

Lack of Standardization

Since REST is just a design principle but not a protocol, implementations can sometimes be inconsistent. Some might decide to adamantly follow its principles while some lethargically.

Best Practices for REST APIs

Use Nouns instead of Verbs in endpoint paths

We shouldn’t use verbs because HTTP methods (eg GET, POST) are already verbs. It would be redundant to use verbs. Instead, use nouns that describe the resource to be accessed or manipulated.

For instance, we want to get all articles www.example.com/articles instead of www.example.com/getAllArticles.

Use logical nesting on endpoints

We want to nest paths that are closely related to one another. We want to go through the house door before entering our rooms.

For instance, if we want to get the comment of an article, we can simply use the endpoint:

www.example.com/articles/12/comments/32

This points to comment 32 of article 12. This link can also be passed as a hypermedia link in an API response to www.example.com/articles/12.

But after some point of nesting, usually 2 to 3 levels, it can start to get a bit bogus and unreadable. We should instead consider creating a separate endpoint for accessing that particular resource.

So suppose we want to get the author of the comment, instead of using www.example.com/articles/:12/comments/32/author/JohnDoe, we can create an endpoint www.example.com/user/:users to get the author’s information.

Use Plurals

When creating endpoints, we want to always use plurals to refer to resources and then an `id` or unique identifier to specify the exact resource or information you want.

Wrong Format — www.example.com/article/12
Correct Format — www.example.com/articles/12

Handle Errors

We should always return custom error messages when there’s a HTTP request failure. This message would help would help the user understand what went wrong especially when business or application context is needed.

For example, instead of returning 401 Unauthorized, we could return Invalid token.

Use filtering, sorting, and pagination

Due to the extremely large amount of data that a server might have, it’s best to also apply some filtering or sorting to query to reduce server load and wait time. This would increase efficiency and avoid unnecessary calls.

We also need ways to paginate data so that we only return a few results at a time. We don’t want to tie up resources for too long by trying to get all the requested data at once.

Cache Data

Repeated requests over the same resource within a certain period can be redundant. It can also increase server load and therefore cost.

It’s a much wiser approach to cache frequently accessed resources to increase application speed and enable the server to delegate resources for other tasks.

There are many kinds of caching solutions like Redis, in-memory caching, and more. We can change the way data is cached as our needs change.

If you are caching data, you want to add the cache-control parameter in your header to signify that the resource is to be cached. This will help users effectively use your caching system.

Versioning our APIs

To make large/critical updates to our APIs over time, it’s recommended to version our APIs. This means any major change comes with a semantic version eg 1.0, 1.0.1, 2.0, 1.4.6, etc

This would allow us to provide services to our old users who still use our previous versions while servicing new clients with up-to-date technologies.

It also allows us to phase out an older version gradually and move clients to the newer version.

The v1 endpoint can stay active for people who don’t want to change, while the v2, with its shiny new features, can serve those who are ready to upgrade.

This is especially important if our API is public. We don’t want to break third-party apps when making changes to our APIs.

Versioning is usually done with /v1/, /v2/, etc. added at the start of the API path.

Some companies also prefer to have a sub-domain when accessing their API. For instance, instead of www.example.com/api/v1, we use www.api.example.com/v1. But anyone is fine.

Security Tips for REST API

1. Always serve across HTTPS
2. Avoid Reinventing the Wheel.
   Use frameworks that have been used and tested. Developers not familiar with designing secure systems often produce flawed security implementations when they try to do it themselves, and they leave their APIs vulnerable to attack.
3. Don’t use a single key-based authentication, unless your API is free and public.
4. Use Hash Message Authentication Code (HMAC) to encrypt because it’s the most secure. (Use SHA-2 and up, avoid SHA & MD5 because of vulnerabilities.)
5. Your API should not leak exception information.
   If there is a data retrieval error in the database, your service should map it to a client-friendly exception of some kind. This is because, it gives the user some information about how your database is structured which could lead to vulnerabilities.
6. Avoid using only Keys to handle authentication.
   Some services only use keys without a secret which is practically verifying a username without a password.

It only takes 5 minutes to be worth $140k

Everyone wants the big paying job and the prestigious title, of “Expert”. But it’s not so easy to get. Today, I’m going to show you in just 5 minutes how to differentiate your work as an SQL developer.

There are various CLAUSES in SQL that are only known to those who are really knowledgeable in SQL. When I was starting, I had no idea these clauses existed. Mostly because a lot of beginner courses out there don’t include these.

There’s one important concept that you would need to know before becoming worthy of $140k and to fully grasp the entirety of this article. Window Functions.

Window Functions

These are functions that perform computations across a table row. They can be used to create new values that consider other values in other rows in the table when performing calculations. They are similar to other aggregate functions (eg GROUP BY) but they cause rows to become grouped into a single output row.

You can’t use window functions and standard aggregations in the same query. More specifically, you can’t include window functions in a GROUP BY clause.

Now that we’re done with that, Here are 5 SQL CLAUSES and how they differentiate.

OVER

OVER ([PARTITION BY columns] [ORDER BY columns])

We define the window frame using the OVER() clause. A window frame is a set of rows that depend on the current row. The set of rows could change depending on the row being queried.

The PARTITION BY defines the condition for each row to be part of the window frame.

ORDER BY defines the order of the records in the window frame.

Junior SQL developer

-- Junior (Without OVER() Clause)
SELECT 
  EmployeeID, 
  Salary, 
  (SELECT AVG(Salary) FROM Employees) AS AvgSalary
FROM 
  Employees;

Mid-Level/Expert SQL developer

SELECT 
  EmployeeID, 
  Salary, 
  AVG(Salary) OVER(PARTITION BY DepartmentID ORDER BY Salary DESC) AS AvgSalary
FROM 
  Employees;

NTILE

You can use NTILE to determine what percentile (or quartile, or any other subdivision), a row ranks in respect to other rows in your table.

Junior SQL developer

SELECT 
  EmployeeID, 
  Salary, 
  CASE 
    WHEN Salary <= (SELECT MAX(Salary) FROM Employees) / 100 THEN 1
    WHEN Salary <= (SELECT MAX(Salary) FROM Employees) * 2 / 100 THEN 2
    ELSE 100
  END AS SalaryPercentile
FROM 
  Employees;

Mid-Level SQL developer

SELECT 
  EmployeeID, 
  Salary,
  NTILE(4) OVER(PARTITION BY DepartmentID ORDER BY Salary) AS SalaryQuartile, 
  NTILE(5) OVER(PARTITION BY DepartmentID ORDER BY Salary) AS SalaryQuintile,
  NTILE(100) OVER(PARTITION BY DepartmentID ORDER BY Salary) AS SalaryPercentile
FROM 
  Employees;

Expert SQL developer

This should only be used when working with the same window frame. e.g PARTITION BY DepartmentID ORDER BY Salary

SELECT 
  EmployeeID, 
  Salary,
  NTILE(4) OVER ntile_window AS SalaryQuartile,
  NTILE(5) OVER ntile_window AS SalaryQuintile,
  NTILE(100) OVER ntile_window AS SalaryPercentile
FROM
  Employees;
WINDOW ntile_window AS (PARTITION BY DepartmentID ORDER BY Salary)

--WINDOW clause should always come after WHERE clause

LEAD OR LAG

LEAD(expr-column to lead, offset — rows to skip, default)
LAG(expr-column to lag, offset — rows to skip, default)

Assuming you wish to get all values from previous rows or following rows while performing some query, Lead or Lag is the way to go! As the name suggests, Lead considers n number of following rows while lag considers n number of previous rows.

It can be used to subtract previous row value from the current row value

Junior SQL developer

SELECT 
  e.EmployeeID, 
  e.Salary, 
  (SELECT TOP 1 Salary FROM Employees WHERE DepartmentID = e.DepartmentID AND Salary < e.Salary ORDER BY Salary DESC) AS LeadSalary
FROM 
  Employees e;

Mid-Level/Expert SQL developer

SELECT 
  EmployeeID, 
  Salary, 
  LEAD(Salary) OVER(PARTITION BY DepartmentID ORDER BY Salary DESC) AS LeadSalary
FROM 
  Employees

ROW_NUMBER

row_number displays the number of the row (when specified, according to the order by clause in the window statement). It does not require any argument.

Junior SQL developer

SELECT 
  EmployeeID, 
  Salary,
  (SELECT COUNT(*) + 1 FROM Employees e2 WHERE e2.Salary > e1.Salary) AS RowNum
FROM 
  Employees e1;

Mid-Level/Expert SQL developer

SELECT 
  EmployeeID, 
  Salary,
  ROW_NUMBER() OVER (ORDER BY Salary DESC) AS RowNum
FROM 
  Employees;

RANK() and DENSE_RANK()

Rank and Row_Number are a bit similar. They differ in the case that two or more rows contain the same value. It ranks rows by column values within a partition

SELECT 
  EmployeeID, 
  Salary, 
  RANK() OVER(PARTITION BY DepartmentID ORDER BY Salary DESC) AS LeadSalary
FROM 
  Employees

For example, if you order by Salary and have two rows with the same salary, it gives them the same rank (eg 2) but it also skips the next rank number (in this case 3) and continues with the next number in the sequence (4).

DENSE_RANK() does all that RANK() does except it doesn’t skip a rank. Identical rows would still have the same rank.

Junior SQL developer [RANK]

SELECT 
  EmployeeID, 
  Salary,
  (SELECT COUNT(DISTINCT Salary) FROM Employees e2 WHERE e2.Salary >= e1.Salary) AS Rank
FROM 
  Employees e1;

Expert SQL developer [RANK]

SELECT 
  EmployeeID, 
  Salary,
  RANK() OVER (ORDER BY Salary DESC) AS Rank
FROM 
  Employees;

Junior SQL developer [DENSE_RANK]

SELECT 
  EmployeeID, 
  Salary,
  (SELECT COUNT(DISTINCT Salary) FROM Employees e2 WHERE e2.Salary > e1.Salary) + 1 AS DenseRank
FROM 
  Employees e1;

Mid-Level/Expert SQL developer [DENSE_RANK]

SELECT 
  EmployeeID, 
  Salary,
  DENSE_RANK() OVER (ORDER BY Salary DESC) AS DenseRank
FROM 
  Employees;

Timeline to finish each project — 1 month

Do you want to fast-track your learning? Tired of not being confident enough to tag yourself as an expert? Or maybe you do, but deep down we both know that’s not really true.

This projects aim to clear those doubts. Obliterates them overnight. But this would need discipline and willing brain wrecking.

Here are 10 projects that would make you stand out among the thousands of Python developers.

Contact me on Upwork for software development jobs relating to Python.

1 . Sitemap generator

A sitemap is a file where you provide information about the pages, videos, and other files on your site, and the relationships between them. Search engines like Google read this file to crawl your site more efficiently. — Google Developers

If you’ve ever had your website or personally hosted blog, you would have come across a situation where some of your blog posts aren’t indexed at all or just not indexed fast enough.

Google offers a way to fix that using site maps. This helps google crawl the sites faster and prioritizes your website pages over other websites. We can do this using scrapy and selenium.

These include pages from clickable elements eg dropdowns. Get a random site or your website and generate a sitemap for it. To understand the format, you can generate a sitemap for a website or mini-blog.

I’d leave a solution to this on my Git Hub. You can check that out when you want. Good luck.

2. Local Chatting system

If you’re familiar with Tech with Tim on YouTube, he was able to land a job at Shopify by building a well-structured, neat, and error-free chatting system.

This chatting system allows anyone to easily join locally, and chat with each other. To mark this project completed, it must be error-free, functions without clear naming should have comments, and it must be reusable.

This project would require some knowledge of socket.io, python, and Django which you can always learn. Once you are done, check out the solution here.

50 Coding Laws That Would Make You A Decent Programmer.

3. Build a CAPTCHA Maker

To know your enemy, you must become your enemy. — Sun Tzu.

If you’ve ever scraped websites, you would constantly come across captchas, obstructing you from trying to save the world.

This project is not necessarily to enable you to find loopholes (unlikely you would find either way) but to see things through the eye of web scraper’s greatest nemesis. The captcha.

Build a captcha maker that can be used to verify human presence before a form is submitted.

4. Build a Simple Blockchain in Python

When you hear blockchain, what first comes to mind? Crypto? Decentralization? Sam bankman fried? or just a block of chains?

Either way, I doubt blockchain technology is ever going anywhere. If you don’t understand what a blockchain is, it’s just a digital ledger where records of transactions/input(s) are kept, stored, and hashed, to prevent future changes and to ensure the integrity of the data timelessly.

Block - A space where we store transactions
Chain - A set of linked records

Each block is built or connected to the other making every block dependable on one another. This creates an irreversible chain of blocks, hence the name blockchain.

We are going to create a blockchain that can store user interactions (eg users’ changes of site’s color or pages visited) on a site.

3 Ways To Run A Python Function Asynchronously

5. Work Hours Monitor

This can be one of the best productivity tools to wield. Everyone has a certain period at which they work. We can try to track ourselves manually by setting an alarm clock or downloading some software but the timer/software has no knowledge about the times you become mentally disengaged with your work or take a quick piss in the toilet and this can create a bias where you believe you work longer than you actually do.

Creating our custom work monitor can help curb these issues. One way is to keep track of applications or websites that we often use when working.

Once any of them are active and in the foreground, we can begin or continue counting. Otherwise counting should either pause or stop.

6. GIF Generator

From pieces of images or a short clip of video, try to make a smooth-looking gif.

5 Concepts Every Python Engineer Should Know in 2024

7. Extract questions from PDFs

As the name suggests, we would find a way to extract all questions from a bunch of PDFs. What would a question look like? Does it end with a question mark? What are the most appropriate words to start a question? (eg. why).

You can use the PyPDF2 or PyMuPDF Python Package.

Here’s the solution.

8. Turn on & off LED light using python

This project would help you understand how Python can easily communicate with hardware devices like sensors.

You can get the hardware components for as little as $100 from Arduino. This would open your mental horizon to possible ways facial recognition devices could work.

The history of Averages and why it’s so Perfect.

9. Bulk Thumbnail Creator

Picture processing can take a bit of time for some transformations. Especially if the image is large. Create an image program that can take hundreds of images and convert them to a specified size in the background thread while you do other things.

For added complexity, have one thread handling re-sizing, have another bulk renaming of thumbnails, etc.

10. Develop a Chrome extension

Build a Chrome extension that would communicate with your work hours monitor to monitor your browser activity during working hours.

This ensures that the work hours counted are only for websites you have specifically associated with your working environment. Eg stackoverflow.

GOODLUCK!

Follow on Linkedln:@alexander
Follow on Twitter: alexander