Imagine you’re a banker analyzing customer savings trends or a manager looking at sales performance. You’ve got mountains of numbers, but how do you make sense of it all? This is where descriptive statistics come in. They help you turn raw data into actionable insights, summarizing the essential details so you can make smarter decisions.

Think of descriptive statistics as a toolkit: measures like averages tell you what’s typical, while measures of spread show how much variation exists. By using these tools, you can quickly spot trends, compare different datasets, or pinpoint areas that need your attention. For example, you could identify the average account balance of your customers or understand the variability in loan repayments.

In this blog, we’ll break down key concepts like mean, median, standard deviation, and more — all with relatable examples you can apply directly to your work or everyday life. Ready to get practical and put your data to work? Let’s dive in!

Before we start, we should know the types of data that generally categorized in statistics.

Types of Data in Statistics

Quantitative Data (Numerical)

• Definition: Data that can be measured numerically.
• Examples: Account balance, loan amount, interest rate.

Qualitative Data (Categorical)

• Definition: Data that describes qualities or characteristics.
• Examples: Account type (Savings, Checking), customer satisfaction level (Low, Medium, High).

Types of Quantitative Data

• Discrete Data: Countable values, e.g., number of loans taken.
• Continuous Data: Measurable values, e.g., account balances.

That’s it. Hope you have understood the short introduction about the data. Now we can proceed with the types of statistics.

Types of Statistics

There are generally two types of statistics -

• Descriptive Statistics: Descriptive statistics summarizes and organizes data in an informative way. Consider a dataset containing monthly account balances of 1,000 customers. Descriptive statistics would help summarize the average balance, the most frequently occurring balance range, and the overall distribution of account balances.
• Inferential Statistics: Inferential statistics uses a random sample of data taken from a population to make inferences or predictions about the population.

Now let’s deep dive into descriptive statistics that we need to understand data in more meaningful way.

What is Descriptive Statistics?

Descriptive statistics are methods used to summarize, organize, and present data in a clear and understandable way.

Instead of drawing conclusions or making predictions, descriptive statistics focus on describing the main features of a dataset.

Measure of Central Tendency

Central tendency is a statistical measure used to determine a single score that defines the center of a distribution. The purpose of central tendency is to provide a single value that represents the entire dataset, giving a sense of the “average” or “typical” value within the data.

The three main measures of central tendency are:

1. Mean (Arithmetic Average)
2. Median
3. Mode

Each measure has its own strengths and is used in different contexts, depending on the nature and distribution of the data.

Mean (Arithmetic Average)

The mean is the sum of all data points divided by the number of data points. It is one of the most common and widely used measures of central tendency.

Suppose you want to calculate the average balance in savings accounts. If the balances are 1,000, 2,500, and 3,000, the mean balance is:

Properties of the Mean:

• Sensitive to Outliers: The mean can be heavily influenced by extremely high or low values. For instance, if one client has a very large bank account balance compared to others, it can distort the mean, making it less representative of the typical account balance.
• Use Case: The mean is suitable for datasets without extreme values and when every value in the data contributes equally to the final result, such as average transaction amounts.

Median

The median is the middle value of a dataset when it is ordered from smallest to largest. If there is an even number of data points, the median is the average of the two middle values.

How to Calculate:

• Arrange the data in ascending order.
• Identify the middle value. If there are two middle values, calculate their mean.

Suppose we have loan amounts of 5,000, 15,000, 20,000, 25,000, and 100,000. The median loan amount is 20,000, as it is the middle value in the ordered list.

Properties of the Median:

• Not Affected by Outliers: The median is robust to outliers and skewed data. It provides a better measure of central tendency when data is not symmetrically distributed.
• Use Case: The median is particularly useful when analyzing income data, property prices, or loan amounts, where a few very high or very low values can skew the mean.

Mode

The mode is the value that appears most frequently in a dataset. A dataset may have one mode, more than one mode, or no mode at all.

Types of Mode:

• Unimodal: One mode
• Bimodal: Two modes
• Multimodal: More than two modes

If a bank analyzes the number of times customers visit branches in a month, and most customers visit twice, then the mode is 2 visits.

Properties of the Mode:

• Useful for Categorical Data: The mode is the only measure of central tendency that can be used with nominal data, like the most common type of loan (e.g., personal loan, mortgage, auto loan).
• Use Case: Mode is useful for identifying the most common transaction type or the most frequent customer complaint.

Choosing the Right Measure of Central Tendency?

Since there are 3 ways we can measure the central tendency. Which one is best?

Mean is appropriate for:

• Normally distributed data.
• Financial metrics like average loan size, average account balance, etc., when outliers are not a concern.

Median is preferred for:

• Skewed distributions or when outliers are present.
• Income or wealth data analysis, where large variations exist.

Mode is useful for:

• Analyzing categorical data.
• Identifying the most common transaction types or services used.

Measures of Variability (Spread)

Variability, or the spread of data, refers to how spread out or dispersed the values in a dataset are. Understanding the variability is crucial because it provides insights into the consistency, reliability, and predictability of data. In banking, variability can help identify risk, understand customer behavior, or assess the stability of financial metrics.

Key Measures of Variability:

1. Range
2. Interquartile Range (IQR)
3. Variance
4. Standard Deviation

Range

The range is the difference between the highest and lowest scores in a data set and is the simplest measure of spread.

If a bank’s loan amounts range from 5,000 to 100,000, the range is 100,000–5000 = 95000 Rs.

Advantages:

• Simple to calculate and understand.
• Provides a quick sense of the spread.

Disadvantages:

• Very sensitive to outliers.
• Doesn’t give any information about the distribution of values within the range.

Use Case: Useful for getting a quick idea of the spread in loan amounts or transaction values but not reliable for datasets with outliers.

Interquartile Range (IQR)

The interquartile range (IQR) is a measure of variability, based on dividing a data set into quartiles.

A quartile is a type of quantile. The first quartile (Q1) is defined as the middle number between the smallest number and the median of the data set. The second quartile (Q2) is the median of the data. The third quartile (Q3) is the middle value between the median and the highest value of the data set.

Let’s take an example:

List = [25, 33, 14, 31, 54, 76, 57, 87, 81]

Let’s find the median first:

Median = 54 , it separates the given data sets into to equal halves

So Q2=54(median of whole table)

Q1=14(median of upper half, from row 1 to 5)

Q3=57(median of lower half, from row 5 to 9)

For the above example :

IQR(Inter Quartile Range ) = Q3 — Q1 = 57–14= 43

One of the application of IQR is → The interquartile range is often used to find outliers in data.

Variance

Variance measures the average squared deviation of each data point from the mean. It quantifies the overall spread of the data.

If analyzing monthly customer deposits, a high variance indicates that deposit amounts are highly inconsistent.

Advantages:

• Uses all data points, providing a comprehensive measure of spread.

Disadvantages:

• Squared units make interpretation less intuitive.
• Sensitive to outliers.

Standard Deviation

The standard deviation is the square root of the variance, providing a measure of spread in the same units as the original data.

If the standard deviation of monthly transaction amounts is high, it means that customers have varying transaction behaviors.

Advantages:

• Easier to interpret than variance because it’s in the same units as the data.
• Widely used in statistical analysis.

Disadvantages:

• Still sensitive to outliers.

Use Case: Standard deviation is crucial for assessing the risk associated with investment portfolios or analyzing customer spending variability.

Sample Vs Population

Population

In statistics, a population refers to the entire set of individuals or items of interest in a study. It is the complete group that you want to draw conclusions about.

Examples:

• All bank account holders in your financial institution.
• Every loan application submitted to the bank in a year.
• The entire set of transactions made by customers over a specific period.

Characteristics:

• Populations are often very large and sometimes even infinite (e.g., all possible future transactions).
• Describing or analyzing the entire population is often impractical or impossible due to constraints like time, cost, or accessibility.

Sample

A sample is a subset of the population selected for analysis. It represents the population and is used to make inferences about the overall group.

Examples:

• A selection of 1,000 bank account holders from a total of 100,000.
• Analyzing 500 randomly selected loan applications to understand approval rates.
• Studying 10,000 transactions out of a total of 1 million to detect patterns.

Characteristics:

• Samples are smaller and easier to analyze compared to the full population.
• Proper sampling methods ensure that the sample represents the population well, reducing bias and increasing the reliability of inferences.

Key Differences Between Population and Sample

1. Size: The population is the complete set, while the sample is a smaller, manageable subset.
2. Notation: In statistics:

• Parameters (values that describe the population) use Greek letters (e.g., μ for mean, σ for standard deviation).
• Statistics (values that describe the sample) use Latin letters (e.g., x̄ for sample mean, s for sample standard deviation).

Example:

• If we measure the average account balance for all account holders, it’s a population mean (μ).
• If we calculate the average from a sample of 1,000 account holders, it’s a sample mean (x̄).

Why Do We Use Samples Instead of Populations?

1. Practicality: Analyzing an entire population is usually expensive, time-consuming, and sometimes impossible.
2. Efficiency: With the right sampling techniques, we can draw reliable conclusions about the population using a sample.
3. Resource Constraints: Limited resources, like time and money, make it more feasible to work with samples.

Sampling Techniques

There are few sampling techniques that needs to be studied →

1. Random Sampling
2. Stratified Sampling
3. Systematic Sampling
4. Cluster Sampling
5. Convenience Sampling

Let’s understand each with examples:

Random Sampling:

• In random sampling, every member of the population has an equal chance of being selected. This method ensures that the sample is unbiased and represents the population accurately
• Imagine a bank wants to analyze the average account balance of its customers. The bank randomly selects 1,000 customers from its entire customer base, giving each customer an equal chance of being chosen.

Stratified Sampling:

• In stratified sampling, the population is divided into subgroups (strata) based on a specific characteristic. Samples are then taken from each stratum proportionally.
• A bank wants to understand loan approval rates among different income groups. The population is divided into income brackets (e.g., low, middle, high income), and samples are taken from each group proportionally to ensure all income levels are represented.
• It ensures all subgroups are represented.

Systematic Sampling:

• In systematic sampling, every k-th item from the population list is selected after a random starting point.
• A bank wants to review customer service satisfaction. If there are 10,000 surveys, the bank may select every 100th survey response to form a sample of 100 responses.

Cluster Sampling:

• In cluster sampling, the population is divided into groups or “clusters,” which are often based on geographical or natural divisions. Instead of sampling individuals from all clusters, the researcher randomly selects entire clusters and then collects data from every individual within those selected clusters.
• This method is particularly useful when dealing with large and geographically dispersed populations.

Imagine a bank wants to analyze customer satisfaction across multiple branches in a large country. Instead of sampling customers from every branch, the bank can use cluster sampling by:

1. Dividing all branches into clusters based on their locations (e.g., city or region).
2. Randomly selecting a few clusters (e.g., a few cities or region).
3. Surveying all customers in the selected clusters.

This approach saves time and reduces costs compared to trying to sample individuals from each branch.

Convenience Sampling:

• Convenience sampling involves selecting individuals who are easiest to reach or sample.
• A bank might use convenience sampling to gather feedback from customers visiting a specific branch because it’s easy and cost-effective.

Footnote →

Descriptive statistics serve as a solid foundation before diving into more advanced statistical concepts. With this understanding, you can now extract meaningful insights from data. I’ve covered various topics in statistics across my other blogs, which you can explore for further learning. If you have any questions, don’t hesitate to reach out — I’m happy to help!

In Part-I of this series, we explored a variety of performance measurement models tailored for classification problems within the realm of Machine Learning. These metrics, including Accuracy, Confusion Matrix, Precision & Recall, and others, play a crucial role in evaluating the effectiveness of classification algorithms. But as we transition from the discrete outcomes of classification to the continuous nature inherent in regression problems, our approach to performance measurement must evolve accordingly.

In Part II, we shift our focus to regression-related performance metrics. Regression tasks, which predict continuous outcomes based on input features, require distinct metrics for assessing model performance. Here, we will delve into critical metrics such as →

• SSE
• MSE
• RMSE
• MAE
• MAPE
• MPE
• R Squared & Adjusted R Squared
• MAD
• Distribution of error.

These performance metrics provides insight into the accuracy and efficiency of regression models. Understanding these metrics is essential for developing, evaluating, and refining predictive models that are crucial in countless applications from finance to healthcare.

Let’s get started, Shall we?

SSE (Sum of Squared Error)

It is defined as →

SSE is the summation of the squared differences between the predicted values and the actual values.

A lower SSE value indicates a better fit between the predicted and actual values. SSE is sensitive to outliers because squaring large deviations increases their impact on the total error.

Example

Let’s create a simple example to calculate the Sum of Squared Error (SSE) for a series of basketball game score predictions. We’ll assume we have predicted scores for five games and then compare these predictions to the actual scores of the games.

• Error column shows the difference between the Predicted Score and the Actual Score. This represents how much our prediction deviated from the actual outcome.
• Squared Error is the square of the Error. Squaring ensures that all errors are positive (since negative errors would indicate underestimation, but when squared, reflect the magnitude of the error just the same as overestimations).

What Does SSE Tell Us?

The SSE of 95 is a way of aggregating all the errors our predictions made into a single metric. It gives us an idea of the total error magnitude across all our predictions. Here’s why SSE is important:

• Total Error Measurement: SSE provides a cumulative measure of error across all predictions. A lower SSE suggests better model accuracy.
• Sensitivity to Large Errors: Because errors are squared, large deviations from actual values are given more weight, making SSE particularly sensitive to outliers. This can be both a strength and a weakness, depending on the context.
• Model Comparison: Within the same dataset, SSE allows for the comparison of different models or prediction methods. The model with the lowest SSE is typically considered more accurate or better fitting to the data.

Limitations of SSE:

While SSE is useful for understanding and comparing model errors, it’s not without limitations. For instance, it doesn’t account for the number of data points or the variability of the data. Therefore, it’s often used in conjunction with other metrics, like Mean Squared Error (MSE), which normalizes SSE by the number of observations, making comparisons across datasets more meaningful.

Mean Squared Error (MSE)

One downside that we observed with SSE is that it is depend upon the number of data points.

Therefore, it is not easy to compare SSE between N = 100 and N = 1000. In case of N = 1000, the error will be bigger simply due to making more predictions.

MSE is the average of SSE; it scales the SSE to the sample size, providing a measure of the average deviation of the predictions.

It’s used to compare the accuracy of different models on the same dataset. MSE penalizes larger errors more than smaller ones due to the squaring process, making it useful when large errors are particularly undesirable.

Example

Let’s borrow the example from SSE → A Basketball score prediction!

To find the MSE, we’ll take the sum of all the squared errors (which we found to be 95 when discussing SSE) and then divide by the number of observations (games) to calculate the average.

This means that, on average, the squared error per game is 19.

Understanding MSE:

• This metric provides a standardized way of looking at the error magnitude, allowing for easier comparison across models or datasets with the same scale.
• MSE’s sensitivity to large errors makes it a useful tool for identifying models that might be making significant mistakes in some predictions, even if they perform well on average.
• Remember, MSE is particularly useful because it puts the error metric back on a scale that is somewhat more interpretable, being in the squared units of the outcome variable. However, because it’s in squared units, direct comparisons to the original predictions or outcomes require taking the square root, which leads us to the RMSE for a more directly interpretable metric.

Limitations of MSE:

• Scale Dependency: Because it’s an average of squared errors, MSE’s value is directly influenced by the scale of the dataset. Larger values in the data will typically lead to larger MSE values, making it challenging to compare performance across datasets with different scales.
• Interpretation Difficulty: The squared term in MSE makes it a bit more difficult to interpret in the context of the original data units. This is why the root mean squared error (RMSE) is often preferred for its interpretability.

Root Mean Squared Error (RMSE)

Up until now we have seen two performance metrics that is → SSE and MSE. The major disadvantage of these two metrics is that they don’t have intuitive units.

Imagine you are forecasting temperature in Degree Celsius, but using the squared error, we will get Celsius squared. You must understand that there is no such thing as Celsius squared or it has no meaning to it. Therefore one way to express this error metric in the unit that make sense is to take Squared Root of the Mean Squared Error. In short, calculate the RMSE.

Example

Let’s borrow the example from MSE → A Basketball score prediction!

This calculation tells us the average error (in the same units as the scores) that our model’s predictions have from the actual scores.

Given the square root of 19, the RMSE would provide a direct sense of the model’s prediction accuracy on an average per-game basis, emphasizing the magnitude of errors and penalizing larger errors more significantly than smaller ones. This makes RMSE a highly insightful and widely used metric for evaluating the performance of prediction models. ​

Understanding RMSE:

RMSE provides a measure of how well a model can predict the value of an outcome. Here’s why it’s valuable:

• Scale Compatibility: RMSE is expressed in the same units as the predicted and actual values, making it easier to interpret than MSE.
• Sensitivity to Large Errors: Like MSE, RMSE emphasizes larger errors more significantly than smaller ones, due to the squaring of errors before averaging. The square root helps to moderate this effect but still retains the property of penalizing large errors more heavily.
• Model Comparison: RMSE is widely used for comparing forecasting accuracy of various models. A lower RMSE indicates a better fit to the data.

Limitations of RMSE:

• Still Sensitive to Outliers: While the square root helps moderate the impact of large errors, RMSE remains sensitive to outliers. This can sometimes lead to misleading interpretations if the dataset contains many outliers.
• Not Normalized: RMSE does not normalize for the number of observations, so while it’s excellent for comparing models on the same dataset, it’s not as useful for comparing across datasets of different sizes or scales.

Mean Absolute Error (MAE)

MAE measures the average magnitude of errors in a set of predictions, without considering their direction.

It is less sensitive to outliers than MSE and RMSE since it does not square the errors.

MAE provides a direct measure of average error magnitude and is interpreted as the average distance from the predicted values to the actual values.

Example

Let’s borrow the example from RMSE → A Basketball score prediction!

• The MAE of 4.2 points tells us that, on average, our predictions were off by 4.2 points from the actual scores.
• This metric is useful because it gives us a direct measure of prediction accuracy on the same scale as the original data, and it’s robust against outliers. The MAE doesn’t penalize large errors more than smaller ones, as each error contributes linearly to the total error measure.

Mean Absolute Percentage Error (MAPE)

Imagine you’re guessing the price of several items. After all your guesses, you check the actual prices to see how off you were with each guess. MAPE helps you understand, on average, by what percentage you were wrong with your guesses. This makes MAPE very intuitive and easy to interpret, especially for non-technical stakeholders.

Let’s create a simple example to calculate the Mean Percentage Error (MPE) for a series of basketball game score predictions. We’ll assume we have predicted scores for five games and then compare these predictions to the actual scores of the games.

This means that, on average, the predictions were off by about 3.99% from the actual scores, without considering whether they were overestimations or underestimations. This percentage gives us a sense of the average size of the prediction errors in terms of their proportion to the actual scores, offering a straightforward way to understand prediction accuracy across the set of games. ​

Here’s how it works:

1. Subtract your predicted score from the actual score to find the difference for each item. This difference tells you by how much you overestimated or underestimated.
2. Turn these differences (errors) into absolute value (absolute errors).
3. Turn these absolute errors into percentages by comparing them to the actual score. This tells you, for each item, by what percent your predicted score was too high or too low.
4. Finally, calculate the average of these percentages. This average is your MAPE. It gives you a single percentage that represents how accurate your guesses were across all items, without getting skewed by whether you consistently overestimated or underestimated.

So, MAPE essentially tells you, “On average, your guesses were off by this percentage.” A lower MAPE means your guesses were closer to the actual prices, while a higher MAPE means they were further off.

Advantages:

• Scale-independent, facilitating comparisons across different datasets.
• Intuitive interpretation in terms of percentage errors.

Disadvantages:

• Can be heavily skewed or undefined when actual values are zero or close to zero.

Mean Percentage Error (MPE)

Similar to MAPE, MPE calculates the average of percentage errors; however, unlike MAPE, MPE includes the sign, providing information on whether the model tends to under- or over-estimate.

MPE can be particularly informative when you’re interested in understanding the bias of the forecast errors.

Imagine you’re trying to predict the scores for a series of basketball games. After all the games, you compare your predicted scores to the actual scores to see how you did. MPE helps you figure out, on average, whether your predictions were too high or too low and by what percentage.

What this tells us is that, on average, the predictions were very slightly overestimated by about 0.32%. The individual game calculations show us by what percentage each game’s score was over or under-predicted, while the MPE gives us an overall average of prediction accuracy across all the games.

Understanding MPE:

MPE is particularly useful for identifying bias in predictions. Here’s what it tells us:

• Positive MPE: Indicates that, on average, the model overestimates the actual values. The model’s predictions are higher than they should be.
• Negative MPE: Indicates that, on average, the model underestimates the actual values. The model’s predictions are lower than they should be.
• MPE Close to 0%: Suggests that the model, on average, accurately predicts the actual values, although individual predictions may still be off.

In practice, MPE can offer valuable insights into a model’s predictive performance, especially regarding systematic biases. However, it’s often used alongside other metrics to provide a more comprehensive view of model accuracy and reliability.

R² (Coefficient of Determination)

In regression related problems, the metric R² used to determine “How good is our regression model?”

R-squared, also known as the coefficient of determination, is a statistical measure that represents the proportion of the variance for a dependent variable that’s explained by an independent variable or variables in a regression model.

It provides an indication of the goodness of fit of a model and is often used to analyze how well the model predicts or fits the actual data.

Before explaining R², there are some prerequisite terms that we need to understand. Those are -

1. Errors in the Regression
2. SSTotal (Mean Regression Model)
3. SSResiduals

Let’s understand these concepts.

Errors in the Regression:

An error in regression model is defined as →

An error in regression is nothing but the difference between Actual Value & Predicted Value.

Error = Actual — Predicted

Error is most important concept in regression problems. Hence we have revisited it once again.

SSTotal (Mean Regression Model):

• It is the differences between observed values and the mean of observed values

• This is nothing but the most basic regression model called → Mean Regression Model.
• Given any query point, the model predicts average value of Yi as Yq as shown in below:

• Therefore it is nothing but the sum of squared error using simple mean model

SSResiduals:

• It is the sum of squares of residuals (the differences between observed and predicted values).

• It represents the variations in the dependent variable that cannot be explained by the model.

Now let’s proceed towards the calculation of R²

Calculation & Interpretation of R²

Let’s understand case by case.

Case 1: When SSResiduals = 0

• When SSResiduals is zero or close to zero which means Actual is equal predicted.
• This is the best regression case when R² is 1 or close to 1

Case 2: When SSResiduals < SSTotal

• If SSResidual is less than SSTotal the R² lies between 0 & 1.
• This is the typical case

Case 3: When SSResiduals = SSTotal

• It means sum of square of residuals is same as simple mean model.
• In this case R² = 0

Case 4: When SSResiduals > SSTotal

• This is the worst case in regression.
• In this case, R² is Negative.

Case 5: When SSTotal = 0

• It indicates a special and very specific scenario in your data. This occurs when all of the observed values, yi​, are identical.
• In other words, there is no variance in your dependent variable.
• Each yi​ is equal to the mean of the dependent variable, yˉ (y bar)​, resulting in each term of the SSTotal​ sum being zero.
• Since each datapoint yi is yˉ (y bar)​, the, just ere is no need of regression, just return yˉ (y bar)​ for any new query point.

Examples

Let’s consider a dataset where we’re trying to predict a person’s weight based on their height. In this scenario:

• If R² = 0.8, it suggests that 80% of the variance in weight can be explained by height, indicating a strong relationship.
• If R² = 0.2, it suggests that only 20% of the variance in weight can be explained by height, indicating a weak relationship.

Limitations and Considerations

• Overfitting: A high R² value does not necessarily mean the model is good. A complex model might overfit the data, capturing noise rather than the underlying relationship.
• Not a measure of accuracy: R² does not indicate whether the predictions are close to the observations; it only measures the proportion of variance explained.
• Comparability: R² alone should not be used to compare models with different dependent variables or datasets.

Adjusted R²

Adjusted R² is a refinement of the R² statistic, designed to provide a more accurate measure of the goodness of fit for regression models, especially when comparing models with a different number of predictors.

Let’s dive into the details.

Definition and Formula

Adjusted R² adjusts the R² value to account for the number of predictors in the model. Unlike R², which always increases as you add more predictors, adjusted R² can decrease if the additional predictors don’t improve the model significantly.

This makes it a more reliable statistic for comparing models with different numbers of independent variables. The formula for adjusted R² is:

Where:

• R² is the original coefficient of determination.
• n is the sample size, i.e., the number of observations.
• p is the number of predictors (independent variables) in the model.

Interpretation

The adjusted R² value provides a measure of how well the independent variables explain the variability of the dependent variable, adjusted for the number of predictors.

It penalizes the addition of irrelevant predictors, which is a significant advantage over R².

• A higher adjusted R² indicates a better model fit, taking into account the number of predictors.
• Unlike R², the adjusted R² can decrease if the contribution of the new predictors does not outweigh the penalty for increasing the number of predictors

Why Use Adjusted R²?

1. Model Comparison: It is particularly useful when comparing models with a different number of independent variables. It helps in selecting the model that has the right balance between goodness of fit and model complexity.

2. Penalty for Extra Predictors: Adjusted R² incorporates a penalty for adding predictors that do not contribute to an increase in R² commensurate with the loss of degrees of freedom. This discourages overfitting by penalizing complex models that don’t necessarily provide better explanatory power.

3. More Accurate Measure of Fit: For models with many predictors, adjusted R² gives a more accurate measure of how well the model generalizes, potentially avoiding the misleading interpretation that adding more variables makes the model better.

Example

• Consider two models predicting house prices: Model A uses 2 predictors (square footage and number of bedrooms)
• Model B uses 5 predictors (adding in age of the house, distance to the city center, and local crime rate).
• If both models have similar R² values, Model A might be preferred due to its simplicity unless Model B has a significantly higher adjusted R², indicating that the additional complexity of the model is justified by a proportional increase in explanatory power.

Median Absolute Deviation (MAD)

The main concern that a single large error (residual) can significantly affect the sum of squared residuals SSResiduals, which in turn affects the R² value. R² is not robust in the presence of outliers, as outliers will disproportionately influence the sum of squares due to the squaring of the residuals. This is where MAD can be particularly useful.

By using the median of the absolute deviations rather than the mean (which would be more influenced by outliers), the MAD provides a measure of dispersion that is more resistant to the influence of outliers.

Let’s take a look at the error (ei)given below →

• If one of the ei is large (Let’s say e4 is large) then it has high chances that It will affect the R². As we have studied, R² isn’t robust and it is affected by outliers.
• MAD can be used to tackle the problem as it is not easily gets affected by outliers.
• If we think eq is random variable, we can compute it’s median and MAD as well.

• If we find out median (eq) and MAD (eq) are small then we understand our errors are also small.

Distribution of Errors

In last chapter we’ve seen to get more information about errors with the help of mean, median, standard deviation and MAD. But we can also use probability distribution to get much more information about the errors. (We will see how to utilize PDF & CDF)

Final Note

That’s a wrap! I’ve thoroughly covered performance measurement techniques, with Part I focusing on classification performance measurements and Part II on regression performance techniques. I’ll keep updating this blog with new content as needed. If you enjoyed reading, please clap and consider following me.

Doesn’t matter whether you are Data Scientist or not, you indulge in measuring the performance or (in short) measuring the quality of things you deal with. And that’s the ultimate truth. Remember, we used to get performance cards during school days to get them signed by parents? Even today, if we visit a supermarket for grocery shopping, we do quality checks many times before we put them in our carts.

We are 360⁰ involved in quality checks or performance measures. Similar fashion, as a data scientist, check the quality of the model that you built. And that’s where all performance measurement models come into the picture. In this section, we will discuss performance measures for a classification problem.

Introduction

In the Data Science life cycle, measuring the performance of the model you built comes at the end of the life cycle. Your task is to first understand the business problem, collect data as per the business problem. Once you wrap up the data collection phase, then you go with the data preparation and exploratory data analysis phase. Once we understand the data completely (regarding our problem), we build machine learning models.

Once we have done with building machine learning models next step is to verify those models or measure the performance of those models to find out good performing models for our goal. This stage is nothing but the model evaluation stage, as shown in the above figure.

In this blog post, we will focus on metrics related to classification problems. There are metrics related to regression problems as well, but that we discuss that in Part-II of this blog post. For now, we will cover below performance metrics →.

• Accuracy
• Confusion Matrix
• Precision & Recall
• F Beta
• Kohen’s Kappa
• ROC Curve
• Log Loss

Let’s start with Accuracy.

Accuracy

Accuracy is one of the popular metrics for evaluating classification models. In classification problems, we calculate accuracy in two ways →.

Problems with Accuracy Measure

There are issues with accuracy measures and hence it is not a widely used performance metric in data science. Let’s discuss the problems faced by accuracy measures.

Problem 1 → Imbalanced Data

Accuracy is a good performance measure when you have perfectly balanced data. But that’s not the reality always. You may or may not have perfectly or somewhat balanced data to accept the accuracy as the performance metric.

Let’s say, you have got a classification dataset about Xi and the class label of that dataset is (Yi) “Has cancer” or “No Cancer”. Basically, you are trying to classify whether a patient will be diagnosed with cancer or not?. Let’s look at the nitty-gritty details of the dataset.

If your dataset is above mentioned one then given query point (Xq), the model will classify it to the “Negative” category with an accuracy of 90%. With such a faulty model, we get pretty good accuracy. This is a concerning thing regards to an accuracy performance metric.

When there is a case of imbalanced data, never use accuracy as a model performance trait.

Problem 2 → Predicted Class Label Vs Probability Score

Many algorithms can provide probability scores of their prediction as well as predict class labels. Let’s use this functionality of models to illustrate this concept.

Let’s say you have a dataset with two features is X & Y. (Here we are predicting Y based on variable X). Let’s use two models (Let me call them M1 & M2) to do the predictions.

Observations →

• Model M1 & M2 return probability scores & also, depending upon the probability score, we have labelled them in the above problem.
• For X1, the original class label is 1. With the help of M1 (P (Y=1|X1) we got the probability of 0.93 and with M2, we got the probability of 0.62. Both model labelled Y as 1 due to (P (Y=1|X1) > 0.50.
• For X2, the original class label is 1. With the help of M1 (P (Y=1|X2) we got the probability of 0.82 and with M2, we got the probability of 0.74. Both model labelled Y as 1 due to (P (Y=1|X2) > 0.50.
• For X3, the original class label is 0. With the help of M1 (P (Y=1|X3) we got the probability of 0.12 and with M2, we got the probability of 0.45. Both the model labeled Y as 0 due to (P (Y=1|X3) < 0.50.
• For X4, the original class label is 0. With the help of M1 (P (Y=1|X4) we got the probability of 0.15 and with M2, we got the probability of 0.42. Both the model labeled Y as 0 due to (P (Y=1|X4) < 0.50.

Models M1 and M2 predicted class labels accurately as predicted and actual class labels are the same in all situations.

By observing the table, we can conclude that model M1 is better performing model than model M2 by looking at their probability score. Unfortunately, performance measures like accuracy cannot use the probability score, it can only use the class labels. Model M1 and M2 has same accuracy, but looking at probability score, we know M1 is better than M2.

Confusion Matrix

There is another classification performance metric known as the confusion matrix used widely in classification problems in statistics and machine learning. We define the confusion matrix as →.

In machine learning or in statistical classification, a table of confusion (sometimes also called as Error Matrix) is a table with two rows and two columns shows the performance of the classification algorithm by comparing ‘Actual Vs Predicted Class Labels’.

When we have a binary classification problem, then the grid of the confusion matrix is 2x2. (Confusion Matrix is mostly used in Binary Classification Problems)

Like Accuracy, the Confusion Matrix does not process the Probability Score!

If your model is sensible, all principle diagonal elements should be high and off-diagonal elements should be low.

Understanding TN TP FN FP

A confusion matrix is built with the help of the following concepts →

• TN (True Negative),
• TP (True Positive),
• FN (False Negative),
• FP (False Positive)

Now, let’s understand each of these concepts briefly.

True Positive (TP)

• The model has predicted positive and actual is positive too.
• Ex → Model has predicted a woman is pregnant, and she actually is.

True Negative (TN)

• The model has predicted negative, and actual is negative too.
• Ex → Model has predicted that a man is not pregnant, and he actually isn't.

False Positive (FP)

• The model has predicted positive but the actual is negative.
• This is nothing but a Type-I error.
• Ex → Model has predicted that a man is pregnant but actually he isn’t.

False Negative (FN)

• The model has predicted negative but actually is positive.
• This is the case of Type-II error.
• Ex → Model has predicted woman is not pregnant, but she actually is.

By using the above information, we can calculate the list of rates from the confusion matrix for a binary classifier. Let’s understand these rates now.

Various Performance Metrics Derived From Confusion Matrix

Let’s understand the various performance metrics that can be calculated from the confusion matrix itself with the help of a toy example.

Accuracy

• Overall, how often is the classifier is correct?
• Accuracy = (TN + TP) / (TN + TP + FN + FP) = 175 / 200 = 0.87

Misclassification Rate

• Overall, how often your classifier is wrong?
• Misclassification Rate = (FN + FP) / (TN + TP + FN + FP) = 25/200 = 0.125

True Positive Rate (TPR)

• When it’s actually “Yes”, how often does your model predict “Yes”.
• This is also known as “Sensitivity” or “Recall”
• TPR = TP / P = TP / (FN + TP) = 100 / 105 = 0.95
• It is equivalent to 1-FNR

True Negative Rate (TNR)

• When it’s actually “No”, how often does your model predict “No”?
• It is also known as “Specificity” or “Selectivity”
• TNR = TN / N = TN / (TN + FP) = 75 / 95 = 0.78
• It is equivalent to 1-FPR

False Positive Rate (FPR)

• When it’s actually “No”, how often does your model predict “Yes”?
• It is also known as “Type-I error” or “Fall-out”
• FPR = FP / N = FP / (TN + FP) = 20 / 95 = 0.21
• It is equivalent to 1-TNR

False Negative Rate (FNR)

• When it’s actually “Yes”, how often does your model predict “No”?
• It is also known as “Type-II error” or “Miss Rate”.
• FNR = FN / P = FN / (FN + TP) = 5 / 105 = 0.04
• It is equivalent to 1-TPR

Balanced Accuracy (BA)

• It is a simple arithmetic mean of “Sensitivity” and “Specificity”.
• BA = TPR + TNR / 2 = 0.95 + 0.78 / 2 = 0.86

Your Model will be good, if TPR & TNR is high and FNR & FPR is low. In simple words, all the principle diagonal elements must be high and off-diagonal element must be low.

Precision & Recall — We Care About Your Positive Class!

These are the metrics also calculated from the confusion matrix only. The reason I have kept them separate as topics is that they are highly important metrics in binary classification problems.

Let’s discuss them now.

Precision

• Precision is the concept is taken from Information Retrieval and Pattern Recognition.

Precision, which is also known as Positive Predictive Value (PPV) defined as →

It is a fraction or percentage of relevant instances among the retrieved instances.

• In our binary classification problem, we can say precision is, “All the points that the model declared or predicted to be positive, what % of them are actually positive?”
• Precision is actually the ratio of True Positive (TP) to True Positive & False Positive.

Example →

For fishing, you throw a fishing net into the river/pond/ocean. After some time you pull it back considering that you have trapped enough fish. But what you get is some fish and some plastic (Garbage). So in this case your precision would be →

“All the stuff that you caught, what % them are actually fish?”

Recall

• The recall is nothing but the True Positive Rate (TPR) is defined as →

It is fraction, or percentage, of relevant instances that were retrieved from total relevant instances.

• In binary classification lingo, we can say, “When it’s actually ‘Yes’, how often does your model predict ‘Yes’?
• It is also known as “Sensitivity”.

Example →

In the above fishing example, the recall would be →

“Out of all the fish in the ocean, what % of fish did you catch?”

Precision & Recall are very much important when you care about positive class in classification problems.

Example: Medical Diagnosis for a Disease

Imagine you have developed a machine learning model to diagnose a rare but serious disease, where:

• “Positive” means the model predicts the disease is present.
• “Negative” means the model predicts the disease is not present.

Now, consider the following outcomes from testing 1000 patients:

• True Positives (TP): 80 patients truly have the disease, and the model correctly diagnoses them as having the disease.
• False Positives (FP): 20 patients do not have the disease, but the model incorrectly diagnoses them as having the disease.
• True Negatives (TN): 880 patients truly do not have the disease, and the model correctly identifies them as not having the disease.
• False Negatives (FN): 20 patients have the disease, but the model incorrectly identifies them as not having the disease.

Precision (Quality of Positive Predictions):

In our example, precision represents how accurate the model is when it predicts that a patient has the disease:

So, the precision of your model is 0.80 or 80%. This means that when the model predicts the disease, it is correct 80% of the time.

Recall (Capturing Actual Positives):

Recall, on the other hand, represents how well the model can identify patients who actually have the disease:

So, the recall of your model is also 0.80 or 80%. This means that out of all the patients who truly have the disease, the model correctly identifies 80% of them.

Interpretation and Use-Case:

• Precision: If the disease is one where the treatment has severe side effects, you might prioritize precision. This is because you want to be very sure that patients diagnosed with the disease actually have it before subjecting them to the treatment. In our example, a precision of 80% is fairly high, which is good if the treatment has significant risks.
• Recall: If the disease is extremely dangerous or infectious, you might prioritize recall. This is because you want to ensure that nearly all individuals who have the disease are identified, even if that means some false positives. In our example, a recall of 80% means that we’re capturing a good proportion of actual cases, but we might want to improve this if the disease is severe and missing a case could be deadly.

F-βeta Score

In the last topic, we’ve discussed the most important metrics in classification that is Precision & Recall. If you could remember, we have stated also that →

• When we care more about the False Positive (FP) than False Negative (FN), we use Precision.
• When we care more about the False Negative (FN) than False Positive (FP), we use Recall.

But what if in the situation, you need both metrics at the same time? If yes, instead of stating each of these metrics separately, Is there any way to combine these two metrics? → Yes, and the answer lies in the F-βeta concept. Let’s discuss it.

With F-βeta, a usually most important task is to select the value of βeta. Let’s take various βeta values and discuss various cases.

When βeta = 1

• When we have βeta = 1 then the above F-βeta equation becomes F1-Score.
• F1 Score is nothing but the Harmonic Mean between Precision & Recall.
• If Precision & Recall both are equally important in the problem statement or aim, we use βeta = 1.

When βeta = 0.5

• In some cases, False Positive (FP) has more impact than False Negative (FN) then we reduce the βeta value to 0.5
• It is also known as the F0.5 Score.
• Use βeta = 0.5 whenever a Type I error is more important than Type II.

When βeta = 2

• In some cases, False Negative (FN) has more impact than False Positive (FP) then we increase our βeta value to 2.
• It is known as the F2 Score.
• Remember we use βeta value > 1 when type II error has more impact than type I error on the model.

Cohen’s Kappa

Cohen’s Kappa score is a famous evaluation metric for measuring the performance of classification problems in machine learning. It was proposed by a famous statistician, Jacob Cohen, and later it was accepted globally.

Cohen’s Kappa score is based on the principle called → “Agreement”.

Here Agreement is nothing but coming to the same opinion by two different human beings.

For example, consider a conversation with 2 persons-

James: “Titanic is an excellent movie. I like to watch it whenever it is getting telecasted on TV”.

Sam: “Yes. Titanic is a gem. Especially the story and screenplay are outstanding”.

Here, Both James and Sam have the same opinion regarding the movie which means their agreement is strong.

Similarly, the Chance of agreement is the probability of 2 people will have an opinion on a subject purely based on random chances but not because of any mathematical fact

Thus Kappa coefficient (K) takes the agreement by chance into account. Now let’s see how do we calculate the value of the Kappa coefficient.

Here →

• a → The total number of instances coded as “correct” in both actual and predicted.
• b → The total number of instances coded as “Incorrect’ in actual but “correct” in predicted.
• c → The total number of instances coded as “Correct’ in actual but “Incorrect” in predicted.
• d → The total number of instances coded as “Incorrect” in both actual and predicted.

To calculate Kappa Coefficient, we need to calculate →

1. Total Frequency → It is nothing but the addition of TN + FP + FN + TP. That is a + b + c + d.
2. Expected frequencies ( ef ) of a & d → If we would have purely guessed, how many correct should be in that column?

Now let’s calculate the Cohen’s Kappa Coefficient (K) with the help of the above-explained concepts.

Interpretation

Receiver Operating Characteristics (ROC) And Area Under The Curve (AUC)

A ROC curve (Receiver Operating Characteristics Curve) is a graph showing the performance of the classification models at all classification thresholds.

This curve plots two parameters →

• True Positive Rate (TPR)
• False Positive Rate (FPR)

ROC was designed by Electrical Engineers during WWII to find out how well their missiles are working. Well, Data Science is a combination of many areas, this might be taken from electrical engineers. Who knows? :D

ROC is designed to use in Binary Classification tasks. It is the probability curve that plots the TPR against the FPR at various threshold values (by doing this) and essentially separates signal from noise.

AUC represents the degree or measure of separability. It tells us how much your model is capable of separating two classes.

The higher the AUC, better is the performance of the model on separating two classes that is positive and negative.

How to measure the performance of the model using AUC?

It is important to interpret or understand the value of AUC. The Value of AUC varies from 0 to 1. Let’s take case by case to understand the interpretability of AUC.

Case 1 →

• When you’ve AUC values that are near 1 it means your model is performing well. The model has a good value of separability.
• As we know that ROC is the curve of probabilities. Therefore, plotting those distributions would be a wise thing to do.

• In above picture, there are two classes ( +ve class = Orange & -ve class = Blue) aren’t overlapping with each other. The model is excellent, as it shows the best separability between positive and negative classes.
• We can perfectly distinguish between the class.
• This is a rare scenario when we have AUC = 1.

Case 2 →

• In the above case, we have seen the “Best Case” which you will never encounter in real life. (At least it never happened with me)
• In real life, you will have AUC between 0.50 to 1.0. This would be a typical case where minor overlapping of classes can be seen.
• With the presence of overlapping as well, these models show good class separability.

• When there is possible overlap in distinguishing the two classes, we introduce something known as type-I and type-II errors.
• We can move our threshold to minimize or maximize the type-I or type-II errors.
• When AUC =0.75, it means there is a 75% chance model will distinguish between the positive class and the negative class.

Case 3 →

• In machine learning, the worst situation is when the model cannot distinguish between the two classes.
• This situation arises when we get AUC = 0.50, which literally means the model has no separation capabilities to distinguish between classes.

• A model with AUC = 0.5 is nothing but a random model.

Case 4 →

• When AUC is approximately 0, the machine learning model interchanges the classes.
• It means the model is predicting positive class as a negative class and negative class as a positive.
• AUC of a model less than 0.5 can be improved by swapping the class labels.

How do we construct ROC Curve?

The ROC (Receiver Operating Characteristic) curve is a graphical representation used to evaluate the performance of a binary classifier system as its discrimination threshold is varied.

It is a plot of the true positive rate (sensitivity) against the false positive rate (1 — specificity) for the different possible cut points of a diagnostic test. (TPR Vs FPR)

Here’s a step-by-step guide on how to calculate and interpret the ROC curve, using an example:

1. Prepare Your Data:

First, you need a set of observations (actual outcomes) and the corresponding predictions (usually probabilistic or score-based) from your binary classifier. For this example, assume we have the following data:

2. Rank Predictions:

Rank the individuals by their predicted probabilities (or scores), from highest to lowest.

3. Determine Thresholds:

For each unique predicted probability (which will act as a threshold), determine the True Positive Rate (TPR) and False Positive Rate (FPR):

• Threshold: The predicted probability at which you decide to differentiate between the classes.
• True Positive Rate (TPR): TP / (TP + FN), where TP is the number of true positives and FN is the number of false negatives.
• False Positive Rate (FPR): FP / (FP + TN), where FP is the number of false positives and TN is the number of true negatives.

4. Calculate TPR and FPR at Each Threshold:

The thresholds typically start from 1 (or the maximum predicted probability) and go down to 0, but they will be based on the unique values from the predictions in our example:

Please Note →

• TPR (Sensitivity) = TP / (TP + FN)
• FPR (1 — Specificity) = FP / (FP + TN)

At threshold 1, no predictions are positive, so TPR and FPR are 0. At threshold 0, all predictions are positive, so TPR and FPR are 1.

5. Plot the ROC Curve:

Plot the FPR on the x-axis and the TPR on the y-axis for each threshold.

Here’s the ROC curve drawn based on the data from the table. This curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at different threshold settings. The dashed line represents a classifier with no discriminative ability (random chance), and the curve demonstrates how well the classifier can distinguish between the two classes.

Log Loss (Cross Entropy Loss)

Log loss, also known as logistic loss or cross-entropy loss, is a performance metric used in statistical models, particularly for evaluating the outcomes of binary classification models like logistic regression.

Unlike other metrics such as accuracy or the ROC curve, log loss provides a measure of uncertainty for the predictions made by a model, penalizing false classifications.

A log loss is important classification metric based on probabilities that we get when we apply model. As already stated, a log loss function generally used in binary classification problems but we can further extend it to multiclass classification problems as well.

Log loss measures the unpredictability of the probability estimates — meaning, it quantifies how much the predicted probabilities diverge from the actual class labels.

It is a “cost” or “loss” because lower values are better and represent a model with better predictive accuracy.

The formula for log loss for a binary classification model is:

Key Characteristics of Log Loss:

• Penalizes Confidence: Log loss heavily penalizes being confident and wrong. For example, a prediction of 0.99 for the wrong class would incur a much higher loss compared to a less confident prediction, like 0.51.
• Sensitive to Probability Estimates: Unlike accuracy, which cares only about the final prediction, log loss evaluates how close the predicted probabilities are to the actual class labels.
• Range: The log loss is always non-negative, and a perfect model would have a log loss of 0. However, there is no upper bound, which means the worse the model’s predictions are, especially with high confidence in the wrong class, the higher the log loss.

Example:

Note → Log Loss function measures the uncertainty of probabilities of the model by comparing them to true labels.

Let’s plug the values of above table in the log loss formula given above. Below table is resultant table —

Let me explain the above log loss values →

• The log loss values for X1 & X3 are small because it has high probability and therefore low loss (As a probability is closer to 1)
• The log loss values for X2 & X4 are high because 0.6 and 0.4 is not closer to actual label 1 and 0 respectively. Therefore a log loss value for X2 and X4 will be higher compared to X1 and X3.

Key Takeaways

• A lower log loss (closer to 0) indicates a better model prediction. It means the predicted probability is very close to the actual label.
• A higher log loss indicates a poorer prediction, as the predicted probability diverges significantly from the actual label.

Final Note

That’s it. These are mainly used performance measurements model for classification problems. As I have already stated, this blog post would serve as Part — I in the series of two. In the sequel, we will talk about performance classification models for regression problems. Stay tuned for further updates.

Best,

Akash Dugam

TaDaaa!

Let’s shed some light on what we will learn about this algorithm. In short, let’s check the agenda.

• Introduction to K Nearest Neighbour
• Geometric Intuition of K-Nearest Neighbour (KNN)
• How does K-Nearest Neighbour identify the neighbours?
• Decision surfaces in K-Nearest Neighbour
• Overfitting and Underfitting from the sense of K-Nearest Neighbour.
• Pseudocode for K Nearest Neighbour
• Weighted K Nearest Neighbour
• Regression Using KNN
• Probabilistic class labels using KNN

Introduction to K Nearest Neighbors (KNN)

K Nearest Neighbors, KNN for short is one of the simplest and easy to learn algorithm in machine learning. It is a supervised machine learning algorithm which is mostly used to solve classification related problems. Most of the classification related algorithm can perform well in regression problems and K Nearest Neighbor is one of them.

Let’s say you have 10 dogs and your job is to classify them into ‘Puppies’ and ‘Adults’. If I ask you how’d you do it? Just looking at them you can tell us whether its puppy or adult. This has happened because you have stored characteristics of ‘Puppies’ and ‘Adult’ dogs in your mind. But how do you think the machine will classify them? For this purpose like your mind have been trained we need to train a machine to identify that. That is where machine learning algorithms come into the picture. We train the machine with the help of the K Nearest Neighbour algorithm which will help the machine to remember characteristics of ‘Puppies’ and ‘Adult Dog’. The similar example can be found here. Please do watch below video for a better understanding.

Now you might be wondering, how does one algorithm can make a machine learn? Don’t worry, we will learning that as we progress in this KNN tutorial. Now let’s understand how does KNN works from a geometric perspective.

Geometric Intuition of K-Nearest Neighbour (KNN)

There are two classes shown in the above diagram. These classes could be anything.

• Data Points associated with class 1 are represented with Orange color.
• Data points associated with class 2 are represented with Blue color.
• There is a decision surface shown in the above figure used to divide two classes into the region.

Let's assume we need to classify X (a white point shown in the above diagram) into its right category. In simple words, which class point X belongs to? So basically we are determining its class.

Now let's understand step by step procedure to determine the class of the X that shown in the above diagram.

Step 1: → Find k nearest points to X in the dataset.

• ‘K’ in K Nearest Neighbor algorithm is the number of neighbors of X that we need to evaluate before deciding its class.
• Its just number. If we assume K = 5, the algorithm is going to check its nearest 5 neighbours or points.
• ‘K’ in KNN is hyperparameter, we can’t just assign it to any number we need to decide its value by doing cross-validation. Don’t worry for now we will see those concepts later.

Step 2: → Find Corresponding Classes of K Nearest Neighbors

• Let say we have selected K = 5 in the above step that makes us investigate the properties or characteristics of 5 nearest neighbors.
• If could observe in the above diagram we can clearly see 5 nearest neighbors are marked inside the circle.
• In classification problems, every Xi comes with Yi. It means for every data points there is a class associate with the data point.

• As shown in the above diagram, for 5 neighbours there are 5 class labels associated with it.

Step 3: → Majority Vote

• Now we will classify the given point either of the categories based on the majority.

• Look at the table above, points X1 and X2 belongs to class 2 but rest of the points X3, X4, X5 belongs to class 1.
• Now if we apply the majority vote concept what we get is the given point X is belongs to class 2 as there 3 class 2 points in the neighbors of X.

Note: K value should be always odd or majority vote concept will not work.

• To demonstrate the above claim, let's take the K = 4 (Even number) for a moment. If we assign even value for K then the majority vote concept will fail as there could be the situation of a tie of votes between two classes. Look at the table below

• In the above case, we cannot classify the query point X into its class as there is no winner as its tie of votes between classes.

How does K Nearest Neighbour identify the neighbors of given query point?

We have seen a bit part of this in the above concept. Finding K will help us to identify neighbors. But wait a minute! Have we discussed, how to find “Nearest” neighbor? I am emphasizing nearest here as we are only concern about the nearest neighbor of the query point. Let’s say ‘Arjun’ and ‘Abhimanyu’ two of my friends stay close to my house. If I want to determine which one is closest among these two? I would I determine that?

Its very simple. Take the distance between your house to your friend's house and compare it. You will have an answer. In a similar fashion, K nearest neighbour algorithms calculate the distance between query point to its neighbour and choose the lowest K points as the neighbour.

The following distance measure techniques that we are going to check

• Euclidean Distance
• Manhattan Distance
• Minkowski Distance
• Hamming Distance
• Cosine Distance

Let's understand these concepts in brief as it is necessary to understand if you wish to know about the internal implementation details of K Nearest Neighbors.

Euclidean Distance (L2):

Let X1 and X2 are two points in 2D space and “d” is the distance between them as shown in the above diagram.

We can find this distance “d” with the help of Pythagoras theorem.

This distance in statistics is known as Euclidean Distance between two points. As for understanding purpose, we took an example of 2D by considering only two features F1 and F2. But in real life, we might have many more features and it is not restricted to two. In such cases, the formula changes. Now we will see how does Euclidean distance behave in d-dimensions.

Note: In both cases in above we have calculated the distance between two and d points. But how do we can represent the same in vector format? After all, every point in dimension can be represented as a vector.

In above equation 2 represents its norm (L2-Norm). As distance is always between two points and the norm is always between two vectors.

Manhattan Distance (L1):

The Manhattan distance between two vectors (or points) X1 and X2 is defined as : →

This is known as Manhattan distance because all paths from the bottom left to the top right of this idealized city have the same distance as shown below

This distance is also noted as L1 -Norm of a vector and it is represented as shown below : →

Minkowski Distance (Lp):

Minkowski Distance between d vectors or points is defined as

The Minkowski distance is a metric in a normed vector space which can be considered as a generalization of both the Euclidean distance and the Manhattan distance.

This distance metric is also known as Lp-Norm of vectors and it is given by the following formula.

We made a big statement above as Lp norm is a generalization of L1 and L2 norms. How can we prove that?

P = 1

• When P = 1 it takes L1 norm of a vector which is also known as Manhatten Distance.

P = 2

• When P = 2, it takes L2 norm of a vector which is also known as Euclidean Distance

Hamming Distance:

The concept of Hamming Distance is taken from the Information Theory. It generally used in Text Processing or when the problem statement deals with binary vector or Boolean vector.

We define Hamming Distance in the following way →

Hamming distance between two strings of equal length is the number of positions at which the corresponding symbols are different.

Let’s understand this concept with the help of an example shown below.

Thus

The Hamming Distance (X1, X2) is nothing but the number of location or dimensions where binary vector differs.

Therefore, Hamming Distance (X1, X2) = 3 as it differs in 3 location as shown in the above diagram.

Cosine Distance & Cosine Similarities

Cosine Distance is based on Cosine Similarities. There is a relationship between cosine similarities and cosine distance that we need to understand first.

“As distance increases, similarity decreases and vice versa.”

So there is an inverse or opposite relationship between similarity and distance. If you think intuitively, as the distance between two points increases which means they are not close to each other which ultimately states these two points are dissimilar.

Let cosine similarity lies between [-1, 1] then our cosine distance actually makes sense.

• -1: Dissimilar
• 1: Similar

→ If cosine similarity = 1 then, cosine distance = 1–1 = 0. This actually makes sense as similarity increases the distance reduces.

→ If cosine similarity = -1 then, cosine distance = 1-(-1) = 2. This too make sense as similarity decreases; distance increases.

This is what the relationship between Similarity and Distance that we should aware of. Remember these concepts not just limited to KNN algorithms but these concepts are heavily used in other areas of machine learning. Now let's understand cosine similarity in details.

We can find the distance between two given vector i.e. X1 and X2 using Euclidean Distance. Cosine similarity says instead of taking geometrical distance, take an angular distance. In cosine similarity, we measure the degree of angle between two vectors.

Therefore,

Cosine Similarity = Cos θ ….Here θ is the angle between X1 and X2 as shown in the above diagram.

If X1 and X2 are unit vectors then ||X1|| = ||X2|| = 1

Therefore,

If the angular distance between two vectors is more, those vectors are considered as less similar and vice versa. Now let's understand how does the cosine metric is different than Euclidean distance.

From the above diagram, we can say that →

• Cosine Similarity (X1, X2) = Cos θ
• Cosine Distance (X1, X2) = 1- Cos θ

Similar way we can find the distance between X2 and X3 i.e. D13 as shown in the above diagram

• Cosine Similarity (X2, X3) = Cos θ = Cos 0 = 1
• Cosine Distance (X2, X3) = 1 -1 = 0

As X2 and X3 are two vectors are in the same direction therefore the angle between them is 0. That is Cos θ = Cos 0 = 1. In other words, It means they are very similar. But if you go with Euclidean distance then the result would be different.

D13 > D12

But, Cosine Distance (X2, X3) < Cosine Distance(X1, X2)

Now let's understand the properties of Cosine Similarities with the help of below diagram.

• If X1 and X2 are in the same direction then cosine similarity = 1 and cosine distance = 0
• If X2 lies perpendicular to X1 then cosine similarity (X1, X2) = 0 and cosine distance = 1-cosine similarity = 1
• If X2 and X1 is completely opposite to each other hence cosine similarity = -1 and cosine distance = 1-(-1) = 2

Decision Surfaces in K Nearest Neighbors:

Until now we’ve discussed how does KNN works and also looked at various distance measure techniques that KNN uses internally. In this tutorial, we will be understanding everything about the decision surfaces in KNN.

Remember we haven’t learned about how to find the right value of K yet. Don’t worry about that. We will be learning that concepts in the subsequent lesson. In this section, we will be understanding how does changing the value of K affects the decision surfaces in KNN.

Basically, if you think intuitively K in KNN is often referred to Hyperparameter. Its basically controls the learning process of machine learning algorithms.

In the above diagram,

• Fig 1 shows the dataset is divided by the smooth curve into two classes. That is ‘Class 1 and Class 2’. The decision surface in fig 1 caused two misclassifications of points.
• Fig 2 shows the dataset is divided by a non-smooth curve into two classes. That is ‘Class 1 and Class 2’. The decision surface in fig 2 makes no mistakes and there is no misclassification error that can be seen.

There are two curves here which separates dataset into two different classes as we have discussed. These curves are nothing but ‘Decision Surfaces’. We call it as decision surfaces as it decides something it belongs to class 1 and something belongs to class 2. Decision surface can be thought of as a model that we are building. It's the main crux in many machine learning algorithms as it decides or separates two or more classes.

Thus changing the value of K will defiantly change the structure of these decision surfaces. Let's understand these impact by observing below cases.

Case 1:-

• Let r1, r2 and r3 are the query points in our dataset. (Points with no color)
• In case 1, we have K =1. Thus to classify the given query point its 1 nearest neighbour will be investigated.
• Therefore, r1 will be classified into class 1, r2 will be classified into class 2 and r3 will be in class 1.

Case 2:-

• Let r1 and r2 are the query points in case 2 scenario.
• In case 2, we have K = 5. Thus to classify the given query point its 5 nearest neighbours will be investigated and majority voting concepts will be used to determine its class.
• r1 → Majority of nearest neighbour is labelled as a class 1and hence the query point r1 will be labelled as class 1.
• r2 → Majority of nearest neighbour is labelled as a class 2 and hence the query point r2 will be labelled as class 2

As ‘K’ increases, the smoothness of curve also increases. It means higher the K value smoother the curve.

Case 3:-

• Since we have not decided the right value of K hence it can be any number. We can take K = n which is nothing but all the values of the dataset.
• Let r1, r2 and r3 are the query points shown in the above diagram.
• If we want to classify these query points then its “n” nearest neighbours will be investigated.
• In case 3, we have the situation of an unbalanced dataset. There are more class 1 points then class 2.
• Since the determining factor, K =n and we have the situation of unequal class weights, in such scenario majority class will be classifying query points into class 1.
• Thus when K = n, it doesn’t matter where is the query point (Near to cluster of points belongs to class 1 or near to a cluster of points belongs to class 2) it will be classified according to where is the majority of points lies.

This is how the value of K affects the decision surfaces in machine learning algorithms.

Overfitting and Underfitting from the sense of K-Nearest Neighbour:

In this section, we will understand how does the value of K cause the ‘Overfitting and ‘Underfitting’ problems in the machine learning algorithm.

Case 1:- Overfitting

• If you observe the above diagram there are two points that fall in the wrong category. The one class 2 point is in class 1 region and one class 1 point in class 2 region.
• These points are away from the bunch of same category points. These points could be erroneous or noisy or an outlier. Since KNN with K=1 will try to classify them into the correct category. In other words, it tries to overfit every small error that exists in the dataset.
• Thus we can say, this model works very hard to fit every point in the dataset. This way it makes no mistakes in Training Data.
• Since it does not make any mistake on training data we can say overfitting works very accurately on training data but it works poorly on unseen data.
• Overfitting always have extremely non-smoothing surfaces (As shown in Fig 1)

Case 2:- Underfitting

• Underfitting scenario arises mostly when the K values in KNN is equal to ’n’. (Shown in 3rd diagram )
• If K=n, then we can say every point belongs to the majority class. In this case, the algorithm is underwriting, it is not trying well to fit the dataset.
• Generally, in the case of underfitting, the algorithm works poorly on training as well as test data.

Case 3:- Well-Fitting

• In a well-fitting scenario, the algorithm tries to balance things between ‘Overfitting & Underfitting’
• It neither tries to accommodate every point in the dataset into its class nor it will fit every point into one class.
• In a well-fit scenario, the algorithm performs well on training as well as unseen data.
• In the above diagram, fig 2 shows, it is okay to make a few mistakes as long as the majority of points classifying into the correct category.
• This is a robust case and less prone to an outlier.

How do we find the correct value of K?

As we have seen in a couple of the above section that there are a lot of things that depend upon the value of K. Be it decision surfaces and scenarios of overfitting and underfitting etc. Let's find the right value of K that will free us from the issues that we discussed earlier.

Case 1:- Comparing Accuracy on Different K-Level

In this case, we assign multiple random values for K and fit a model. After fitting a model accuracy parameter will be considered for each model to determine the right value of K. Take a look at the table below →

In the above table, K=9 gives good result compared to other values of K. This is a trial and error method where we randomly select K’s value and compare them on the basis of each K’s accuracy.

By using K=9, we get the accuracy of 0.85 which good compared to other accuracies mentioned in the table. There is a problem with this methodology though. As the objective of any ML algorithm is to perform well on future unseen data points. When an algorithm performs well on future unseen data points then it is called ‘Generalization’. Thus the objective of any machine learning algorithm is to get good ‘Generalization Accuracy’. Above explained method will not hold true for generalization. As to determine ‘K’ we are using ‘Test Data’ and to determine ‘Nearest Neighbour’ we are using ‘Train Data’.

Since we are using test data to find the right value of K and train data to find the Nearest Neighbour we cannot generalize the accuracy that we got will hold true for unseen data points. This problem is pretty much common and we use something called ‘Cross-Validation’ to answer this issue.

Case 2:- Using Cross-Validation to find the value of right K

As shown in the above diagram, we use to train and test both the dataset to find the function of KNN. In such a case, we do not test the model on the completely unseen dataset as both train and test dataset are used to find the nearest neighbour and K value respectively. Now let’s check how to address this issue.

In the above case, we used train data to find the nearest neighbour, we used CV data get information about right K value. In this case, we did not use test data and it is completely unseen for us. Once we find a function of KNN with the help of train and cv data then we apply it on test data to see how does it perform with the help of the accuracy parameter. Since in this case ‘Test Data’ is completely unseen for us so we can conclude → Given a future unseen point, the accuracy will remain the same. The accuracy that we get using test data is considered as ‘Generalization Accuracy’.

This is one of the best methods is available to find the correct value of ‘K’ in the KNN algorithm. But there is one issue with this method. Let’s discuss that in details.

In the cross-validation method, we break the dataset into 3 part instead of 2. You can imagine breaking dataset this way →

There are scenarios where the ML algorithm needs more data in the training phase. You can call it thumb rule but more training data, better is the trained model. Is there anything that we can modify and take a benefit of 80% of the training data?

Yeah! Certainly. This is where K-Fold Cross-Validation comes into the picture. Let’s understand it in the next section.

Case 3:- k-Fold Cross-Validation

This is the state of the art technology to carry out cross-validation. As we’ve discussed in case 2, more training data always beneficial for a machine-learning model to learn patterns from data. k-fold cross-validation is a special case of cross-validation where we iterate over a dataset set k times. In each round, we split the dataset into k parts: one part is used for validation, and the remaining k−1 parts are merged into a training subset for model evaluation.

Step 1 →

Instead of 3, break the dataset into Training Set and Testing Set only. Here we will be using train data to compute K’s value and nearest neighbours as well.

Step 2 →

‘k’ in ‘k-Fold’ denotes the given training data set is split into a ‘k’ number of sections or folds where each fold is used as a testing set at some point. Let’s say you have training data and you decided to break it into 4 then it will be called ‘4-Fold’ cross-validation technique. The value of fold is up to the user but generally, ‘10-Fold’ is followed everywhere. For simplicity, we will use the 4-Fold CV to understand it easily.

Let’s break training data into 4 equal parts and name it D1, D2, D3, and D4 as shown in the above diagram. Each of these chunks constitutes 20% of the total training data.

Step 3 →

In this process, we are using a total of 80% of the data to train our model and 20% of data to test. We’ve used D1, D2, D3, and D4 chunks 3 times to train the model and 1 time test the model. Since we are repeating each K-Level 4 times, we can call this method as a 4-Fold CV.

On each K-Level we have taken the average of multiple accuracies to represent a single number. This is very important to get one value for all. Next step would be comparing the average of accuracies on each K-Level.

Let's consider X3 is highest among the rest of the accuracies. Thus K = 3 will be selected as ‘Best K’ for our model.

Major Issue with k-fold CV:-

In k-fold cross-validation, we repeat the process k times to get optimum accuracy. This process is expensive and it takes some load on your CPU.

Pseudocode for K Nearest Neighbor

If you are coming from the computer or IT background then you might have a habit of understanding any algorithm in the form of Pseudocode. It simplifies the understanding process. Let’s take benefit of this technique to understand the KNN algorithm.

Weighted K-Nearest Neighbor

The weighted K-Nearest Neighbor is considered a modified KNN algorithm which assigns a weight to nearest neighbors of a query point. These weights are taken into consideration while labelling the query points.

Let’s understand this with a simple example that illustrated below —

Given the query point Xq we want to determine its class label Yq. In other words, let’s categories this point in either of the class. We will take help of two method to classify this point

• KNN
• Weighted KNN

KNN →

• Let’s consider K = 5
• Let X1, X2, …., X5 nearest neighbour of Xq.

Lets investigate the class labels of these nearest neighbors of Xq.

As per KNN algorithm, the majority vote concept is taken into account and the point Xq will be classified into positive category.

Let’s try to solve same problem with the help of weighted KNN.

Weighted KNN →

• As we discussed, In weighted KNN we assign weights to nearest neighbours and depending upon the weights we decide its class labels.
• Let K = 5 in this case as well and X1, X2, X3, …, X5 are the nearest neighbours of Xq.
• In weighted KNN, depending upon the distance weight is assigned to the nearest neighbours of the query point.

• In above table for sake of understanding the concepts we have assumed distance. Note that its not actual distance that we calculated.
• If you look closely, the points X1 and X2 are closest to query point Xq than other points. If I ask you question → Is it valid to give more importance (weight) to the class which are closest?
• In weighted KNN we tends to give more importance to the neighbours which are closest. This is the main crux of weighted KNN.
• There are more weighing scales that we could use but I simply prefer simple inverse relationship that’s why the equation became weight = inverse of di.
• If we take sum of the distance separately for each class then its 15 for ‘Negative Class’ and 1.75 for ‘Positive Class’. (As shown in above table)
• Since 15 > 1.75, we can say Xq belongs to ‘Negative Class’. This is all about weighted KNN.

Regression Using KNN

Most of the times classification problems can be easily converted into regression problems. Though, this completely depends upon the algorithm itself. In the case of KNN we can easily modify the algorithm and use it for regression analysis.

In regression analysis, given Xq we are not predicting it’s class. But we predicts real value. There are few useful steps are followed to achieve this. Let’s see one by one.

Step 1 →

• Given Xq we need to find its nearest neighbours.
• (X1, Y1), (X2, Y2), …, (Xk, Yk)

Step 2 →

• Once we have all Yq’s we need to one representative number.
• Representative number can be found with the help of ‘Mean’ or ‘Median’ strategy.
• We need to take mean or median of all Yq’s as its real number and not a class label.
• Mostly prefer ‘Median’ as its less prone to outliers.

Probabilistic class labels using KNN

• This is example of two class classification. Here yi belongs to {0, 1}
• Let’s assume the K value i.e. K = 7. With the help of K value, we need to classify the Xq.
• If we simply apply the majority vote concept for this case, the query point Xq will fall into negative category. (As -ve: 4, +ve: 3)
• Out of 7 points, we have 4 negative and 3 positive. Difference between those points is very small. Thus is fair to say that → Xq is negative point by simple majority vote concept? Does it make sense?
• In order to mitigate above issue, we can use something called ‘Probabilistic Class Labels’.
• Let’s take the example (stated in image 2 above), Here in second scenario, there are 7 negative points and 0 positive points. Now if we go by majority vote concept, Xq’ will be classified to negative class.
• If you observe both the scenario, Xq’ is more certain of being in negative class than Xq. Is there any way to quantify these uncertainty? Yes, by using probabilities. (Look at the equations below)

Footnote →

This blog post delves into the K-Nearest Neighbors (KNN) algorithm, a foundational technique in machine learning used for classification and regression tasks. KNN is highlighted for its simplicity and effectiveness, operating on the principle that similar things exist in close proximity.

The post begins by introducing the basics of the KNN algorithm, explaining how it classifies new data points based on the majority vote or average of its K nearest neighbors from the training dataset. It emphasizes the importance of choosing an appropriate value of K, as it significantly impacts the model’s performance, affecting its bias and variance.

We have also discussed different ways to measure distance between data points, a critical aspect of the KNN algorithm, including Euclidean, Manhattan, and Minkowski metrics. The post also covers the pros and cons of KNN, noting its ease of understanding and implementation as advantages, while pointing out its sensitivity to irrelevant or redundant features and the curse of dimensionality as drawbacks.

This is not the end of KNN but it is enough to have understanding of the nitty gritty details of the ML algorithm.

Signing off…!!!

Unlike traditional databases, vector databases are designed to efficiently handle, store, and retrieve high-dimensional vector data, which is essential for powering advanced machine learning models. This article aims to shed light on the critical role of vector databases in Generative AI, exploring their foundational concepts, technological nuances, and the challenges they help overcome. As we delve into this topic, we will unravel how these databases are not just a backend utility but a driving force in the ongoing AI revolution.

The ABCs of Vector Databases!

In the ever-evolving world of technology, particularly in the field of artificial intelligence (AI), there’s a growing need to understand and manage complex data. This is where vector databases come into play. But what exactly are vector databases?

Imagine you have a large collection of photos, and you want a computer to sort and understand them just like a human would. Traditional databases are like meticulously organized photo albums, with each picture neatly labelled and categorized. However, when it comes to AI, particularly Generative AI, things get a bit more complex.

Generative AI doesn’t just sort data; it tries to understand patterns, create new content, and make decisions. For this, it needs to understand data in a way that’s more like how our brains perceive things, not just as simple labels or categories. This is where vector databases shine.

A vector database stores data in a format that AI can easily understand and use. Let’s go back to the photo example.

In a vector database, each photo is transformed into a “vector” — a list of numbers that represent all the intricate details and features of that photo, much like a unique fingerprint.

These vectors help AI to not only recognize photos quickly but also to understand subtle patterns and similarities between them.

In essence, vector databases translate complex, real-world data into a language that AI can efficiently process and analyze. They are essential for advanced AI tasks like image recognition, language processing, and creating new content, which is the heart of Generative AI. By using vector databases, AI systems can operate more intuitively and effectively, mimicking the human ability to understand and interpret complex information.

Here is the quick summary →

• A vector database is a type of database designed to efficiently store and manage vector data.
• Vector data, in the context of computer science and artificial intelligence, refers to data represented as arrays of numbers, often known as vectors.
• These vectors are mathematical constructs that can encode various forms of complex information, such as the features of an image, the semantics of a piece of text, or the characteristics of audio.
• Therefore we can say, vector databases are specialized databases optimized for storing, managing, and retrieving vector data, which is essential in various AI and machine learning applications. They provide a way to efficiently handle the complex and high-dimensional data that these applications rely on.

Vector DB Vs Relational DB Vs Non Relational DB

It is vital to understand the difference between various DB’s that are available in the field of computer science. Below table distinguishes vector databases with other available.

Vector Embeddings

Let’s start with the basics. First, I’ll explain what a vector is, and then we’ll talk about Vector Embeddings.

A vector is a math concept that has both size and direction. You can think of it like a point in space, with the direction being like an arrow going from the starting point (0,0,0) to where the point is in space.

When we look at a bunch of vectors in one space, we can say that some are closer to one another, while others are far apart. Some vectors can seem to cluster together, while others could be sparsely distributed in the space.

Hope you have understood the concept of vector. Now let’s proceed towards understanding the concept of → “Vector Embeddings”

In the field of machine learning (ML), a fundamental concept often encountered is Vector Embeddings, especially when dealing with non-numeric data. These embeddings are crucial for ML engineers as they form the foundational elements of numerous ML and deep learning (DL) algorithms. In this discussion, we will explore what Vector Embeddings are and how they are utilized in artificial intelligence (AI).

Vector embeddings are a way to convert words and sentences and other data like images or audio into numbers that capture their meaning and relationships.

Let’s take a look at below example -

In simpler terms, when we deal with text data, different words like “cat” and “kitty” might have similar meanings, but they are made up of different letters, so they look very different to a computer. To help a computer understand that these words are similar in meaning, we use something called vectors. Vectors are a way to represent things in the computer. They help to show the computer that “cat” and “kitty” are similar, even though they are spelled differently.

How Vectors are Embedded?

• For Text: Words or phrases are converted into vectors using algorithms like Word2Vec, GloVe, or BERT. These algorithms analyze a large corpus of text and learn to represent each word by a vector such that words with similar meanings have similar vectors.
• For Images: Techniques like Convolutional Neural Networks (CNNs) can be used to transform pixel data into vectors, capturing features like edges, textures, or more complex patterns.

Example —

Let’s walk through the process of converting a sentence into a vector embedding using a machine learning approach. We’ll use a simplified example to make the concept clear.

Sentence: “The cat sat on the mat.”

The above image illustrate the process of the creating vectors through the sentences. In similar fashion (with few changes here and there), the vectors will be created for image, audio etc.

Type of Vector Embeddings

Use-cases of Vector Embeddings

Vector embeddings are used in many domains. Let’s look at few example where vector embeddings are useful.

• Semantic Search: Traditional search engines match keywords, but with vector embeddings, semantic search understands and finds results based on the real meaning behind a query.
• Question-Answering Apps: By training a model with question-answer pairs, we can make an app that answers new questions it’s never seen before.
• Image Search: Vector embeddings are great for image search tasks. There are many ready-to-use models like CLIP and ResNet, each good for different tasks like finding similar images or detecting objects.
• Audio Search: We turn audio into a visual representation (audio spectrogram) and use vector embeddings to find similar sounds.
• Recommender Systems: We can make embeddings from structured data about things like products or articles. These are often custom-made for your specific needs. Sometimes, they can also include unstructured data like images or text descriptions.
• Anomaly Detection: By using large datasets of labelled sensor data, we can make embeddings to spot unusual or abnormal events.

Similarity Measures (Distance Metrics)

Similarity measures, also known as distance metrics, in vector embeddings are crucial for many applications in machine learning and data science. They provide a way to quantify how similar or dissimilar two data points are in the vector space. Understanding these measures is essential for tasks such as clustering, classification, recommendation systems, and more.

Here each of these datapoints are mapped into the 2D vector space as illustrated in above image. If you observe carefully, fruits like Apple, Banana and Orange falls within the same region as they belongs to fruit category. Look at their embeddings, they are very close to each other as well. Same goes with category like Veggies and Flowers.

Since human understand these are fruits and needs to be clustered into fruit category. In order comprehend same things by computer / AI we need to measure the distance or similarity of vectors. Now you may ask, how do we measure the similarity or distance between datapoints / vectors? Take a look at below —

• Euclidian Distance
• Manhattan Distance
• Hamming Distance
• Cosine Similarity

These are the commonly used similarity or distance metrics in vector embeddings.

Euclidean Distance (L2):

Let X1 and X2 are two points in 2D space and “d” is the distance between them as shown in the above diagram.

We can find this distance “d” with the help of Pythagoras theorem.

This distance in statistics is known as Euclidean Distance between two points. As for understanding purpose, we took an example of 2D by considering only two features F1 and F2. But in real life, we might have many more features and it is not restricted to two. In such cases, the formula changes. Now we will see how does Euclidean distance behave in d-dimensions.

Note: In both cases in above we have calculated the distance between two and d points. But how do we can represent the same in vector format? After all, every point in dimension can be represented as a vector.

In above equation 2 represents its norm (L2-Norm). As distance is always between two points and the norm is always between two vectors.

Manhattan Distance (L1):

The Manhattan distance between two vectors (or points) X1 and X2 is defined as : →

This is known as Manhattan distance because all paths from the bottom left to the top right of this idealized city have the same distance as shown below

Imagine you are in a city where the streets are laid out in a perfect grid, and you want to travel from the bottom left corner of the city to the top right corner. Unlike in a diagonal straight line (which would be the Euclidean distance), in a grid system like Manhattan, you must travel along the streets, which means moving either horizontally or vertically.

In such a grid, all paths from the bottom left to the top right corner (assuming you only move right or up, without backtracking) have the same length in terms of distance traveled. This is because you’re simply switching the order in which you travel across the grid’s horizontal and vertical lines, but the total number of lines crossed remains constant.

This distance is also noted as L1 -Norm of a vector and it is represented as shown below : →

Hamming Distance:

The concept of Hamming Distance is taken from the Information Theory. It generally used in Text Processing or when the problem statement deals with binary vector or Boolean vector.

We define Hamming Distance in the following way →

Hamming distance between two strings of equal length is the number of positions at which the corresponding symbols are different.

Let’s understand this concept with the help of an example shown below.

Thus the hamming distance (X1, X2) is nothing but the number of location or dimensions where binary vector differs.

Therefore, Hamming Distance (X1, X2) = 3 as it differs in 3 location as shown in the above diagram.

Cosine Distance & Cosine Similarities:

Cosine Distance is based on Cosine Similarities. There is a relationship between cosine similarities and cosine distance that we need to understand first.

“As distance increases, similarity decreases and vice versa.”

So there is an inverse or opposite relationship between similarity and distance. If you think intuitively, as the distance between two points increases which means they are not close to each other which ultimately states these two points are dissimilar.

Let cosine similarity lies between [-1, 1] then our cosine distance actually makes sense.

• -1: Dissimilar
• 1: Similar

→ If cosine similarity = 1 then, cosine distance = 1–1 = 0. This actually makes sense as similarity increases the distance reduces.

→ If cosine similarity = -1 then, cosine distance = 1-(-1) = 2. This too make sense as similarity decreases; distance increases.

This is what the relationship between Similarity and Distance that we should aware of. Remember these concepts not just limited to Vector databases but these concepts are heavily used in other areas of machine learning. Now let’s understand cosine similarity in details.

We can find the distance between two given vector i.e. X1 and X2 using Euclidean Distance. Cosine similarity says instead of taking geometrical distance, take an angular distance. In cosine similarity, we measure the degree of angle between two vectors.

Therefore,

Cosine Similarity = Cos θ ….Here θ is the angle between X1 and X2 as shown in the above diagram.

If X1 and X2 are unit vectors then ||X1|| = ||X2|| = 1

Therefore,

If the angular distance between two vectors is more, those vectors are considered as less similar and vice versa.

Vector Indexing

Vector is the method of organizing and managing vector embeddings to enable efficient retrieval, search, or comparison operations.

In short, we can say → Vector indexing is nothing but organizing the similar vectors together.

How Vector Indexing Works

1. Storing Vectors: Initially, vector embeddings are created from data (like text, images, etc.). These vectors are multi-dimensional and represent the features of the data.
2. Index Creation: An index is built to organize these vectors in a way that optimizes search and retrieval operations. This index can be thought of as a specialized database for vectors.
3. Searching the Index: When a query is made (which is also converted into a vector), the indexing system efficiently searches through the vectors to find the most relevant or similar ones.

Now let’s take an example to explain this concept thoroughly.

Example

Let’s consider an example with a text search application:

Dataset:

Suppose we have a large collection of documents. Each document is converted into a vector embedding using techniques like TF-IDF, Word2Vec, or BERT.

Building the Index:

• The vectors from all documents are stored in an index.
• The index might use algorithms like k-Nearest Neighbors (k-NN) or more complex structures like trees or hash tables to organize these vectors.

Performing a Search:

• A user inputs a search query, like “machine learning.”
• The query is also converted into a vector using the same method used for the documents.
• The index quickly searches for and retrieves document vectors that are closest to the query vector in terms of cosine similarity or another similarity metric.

Retrieving Results:

• The documents corresponding to the top matching vectors are retrieved and presented to the user.
• This process is much faster than comparing the query vector to every document vector individually.

Importance of Vector Indexing

• Efficiency: Vector indexing allows for efficient retrieval in large datasets, reducing the search space and computational load.
• Scalability: It makes it feasible to work with very large datasets that would otherwise be impractical to handle.
• Accuracy: By using sophisticated indexing algorithms, the accuracy of retrieval can be enhanced, ensuring more relevant results for queries.

Architecture of Vector Database

The architecture of a vector database is designed to efficiently store, manage, and retrieve high-dimensional vector data, often used in machine learning and data science applications.

Let’s understand this with the help of below pointers →

• Regardless of the content type such as images, emails or text documents, we utilize the embedding model to convert them to vector embeddings.
• These embeddings are then loaded into vector databases where they are efficiently clustered & indexed based on their similarity.
• When the application issues a query, we use the same embedding model to create embeddings for the query and use those embeddings to query the database for similar vector embeddings.
• The vector database performs a search based on the query vector and retrieves the relevant results for the application.

In order to illustrate further consider the below example in IT threat detection application.

We have the activities represented as → Regular, Suspicious and Malicious in vector dimension space. By querying the vector database, we can determine if activity close to regular activity or suspicious activity or malicious activity. This allows us to classify the web traffic based on the proximity to vector space.

If we take the distance between query vector (?) with regular, suspicious and malicious it is evident that the query vector is near to malicious vector. Here query vector is the web traffic. Therefore we can consider the query vector (a web traffic) is malicious one. Therefore, it is like classification or anomaly detection.

So by using the vector database, we can establish common ground a common ground for managing different types of unstructured data and this is why vector database is needed and indeed it is solving the fundamental problem of unstructured data.

How does a vector database work?

In traditional databases, the queries typically aim to find rows where the values exactly match the specified query criteria. On the other hand, in vector databases, the approach is different: we use a similarity metric to identify a vector that most closely resembles our query.

A vector database works by efficiently storing, indexing, and retrieving high-dimensional vector data, which is often used in applications involving machine learning, AI, and big data analytics. Here’s an overview of how it operates.

Step 1 → Data Conversion to Vectors

• Input Data: The data can be of various types like text, images, audio, etc.
• Conversion Process: This data is transformed into vectors using machine learning models. For instance, text can be converted using models like BERT or Word2Vec, images with CNNs, etc. Each vector is a numeric representation of the data, capturing its essential features.

Step 2 → Indexing the Vectors

The main purpose of the indexing is to enable efficient searching within the vector databases. To do this, various algorithm is used to index these vectors which includes —

• k-Nearest Neighbours (k-NN): For finding vectors closest to a query vector.
• Locality-Sensitive Hashing (LSH): For grouping similar vectors.
• Hierarchical Navigable Small World (HNSW): A graph-based approach for indexing.

The indexing creates a structure that makes it easier and faster to search through a large number of vectors.

Step 3 → Query Processing

• Query as a Vector: When a user has a query (like a text query or an image), it is also converted into a vector using the same method as the database vectors.
• Searching: The vector database uses the index to quickly find vectors that are most similar to the query vector.
• Similarity Metrics: The search relies on similarity metrics like cosine similarity, Euclidean distance, etc., to determine the ‘closeness’ of vectors.

Step 4 → Retrieving Results

• Matching Vectors: The database retrieves the data items corresponding to the vectors that closely match the query vector.
• Use Cases: This could be, for example, finding similar images, recommending products, or searching for similar documents.

Step 5 → Post-Processing (Optional)

• Refinement: The initially retrieved results might undergo further processing like re-ranking based on additional criteria or filters.
• Accuracy vs. Speed: There’s often a trade-off between the accuracy of the results and the speed of retrieval. More accurate results might take longer to retrieve.

Example

Imagine a vector database used for a photo search application:

• User Uploads an Image: The image is converted into a vector.
• Database Search: The vector database quickly scans through its indexed vectors to find images whose vectors are most similar to the uploaded image’s vector.
• Result: The user is presented with images that are visually similar to the uploaded image.

In essence, a vector database is specialized in handling complex, high-dimensional data that traditional databases aren’t equipped to manage efficiently. Its ability to rapidly process and retrieve similar items based on vector proximity makes it invaluable in fields like AI and data science.

Enough of the theory now. Let’s get our hands dirty with some practical’s / coding.

Journey of Vector Databases with Pinecone!

Pinecone is a specialized database service designed to efficiently handle vector data, which is commonly used in machine learning applications. This service is particularly useful for tasks that involve searching and comparing vectors, such as in recommendation systems, image recognition, and natural language processing tasks.

Vector databases like Pinecone are crucial in the field of machine learning because they enable quick and accurate retrieval of similar items from large datasets. For instance, in a recommendation system, Pinecone can quickly find products or content similar to a user’s interests by comparing their preference vectors with those in the database.

Pinecone stands out for its scalability and performance, offering high-speed vector search capabilities. This is essential for real-time applications where rapid data retrieval is critical. Additionally, its user-friendly interface and integration capabilities make it accessible for both experienced developers and those new to vector databases.

Environment Setup & Installation

To begin incorporating Pinecone into your GenAI project, the initial step is to set up an account with Pinecone. This can be done by visiting their official website.

After establishing your account with Pinecone, you can proceed to configure your projects and generate the necessary API keys. These keys will be essential for performing various operations and implementing use cases with Pinecone.

As we are utilizing Python to execute operations in Pinecone, it’s necessary to install Pinecone using the ‘pip’ command.

> pip install pinecone-client

Follow official guide on installation → https://docs.pinecone.io/docs/python-client

Hello World with Pinecone

Let’s play with Pinecone database through Python.

# Import Pinecone Library
import pinecone

# Initialize API Key
pinecone.init(api_key="a4345490-b8s9-48e7-8f95-1bce784d353d",
              environment="gcp-starter")

# Create an index
pinecone.create_index(name="hello-world",
                      dimension=3,
                      metric="cosine") # cosine is default

# check on all the indexes
pinecone.list_indexes()

# This will return ['hello-world'] as output.

CRUD with Pinecone!

CRUD operations, which stand for Create, Read, Update, and Delete, are fundamental in managing data in databases or data storage services. When working with Pinecone, a vector database that’s often used for machine learning applications, these operations are adapted to handle vector data. Let’s get started with Create.

Create:

Read:

Update:

Delete:

Concept of Index & Collections

Let’s delve into the crucial concept on an index and collection.

An index in Pinecone is a named collection of vectors. These vectors are essentially high-dimensional data points, typically generated as outputs from machine learning models.

Functionality

• Storing Vectors: An index holds a set of vectors. For example, in an index named ‘test’, which has three dimensions, there are multiple vectors stored within these dimensions.
• Querying: The primary function of an index is to allow querying. You can search through the vectors in an index to find matches or similar vectors based on specific criteria or algorithms.

A collection is a static copy of an index in Pinecone. It’s a snapshot or a backup of the state of an index at a given point in time.

Characteristics and Use:

• Static Nature: Unlike an index, a collection is static. You cannot query a collection like you can with an index. It’s essentially a read-only copy of the index data.
• Purpose: The main purpose of a collection is for backing up an index. It serves as a means to preserve the state of an index before making changes.

Managing Index and Collection

We have already seen the concept of Index and Collection above. Let’s get some idea on → How to manage them?

Creating an Index:

You start by creating an index (e.g., ‘test’) where vectors are stored and can be queried.

Modifications and Backup:

• When you need to modify the index (like changing the index type or distance metrics), you first create a static copy of the current index as a collection.
• This backup ensures that the original data is not lost during the transition.

Creating a New Index from Collection:

• You can create a new index with the desired changes (e.g., different index type or distance metrics).
• This new index can be populated with vectors from the collection.
• Multiple indexes can be created from the same collection if needed.

Querying New Index:

Once the new index is set up and populated, it’s ready for querying. The collection remains as a static backup.

Let’s see how index can be managed with Python!

### Import Pinecone Library
import pinecone

### Pinecone Initialization
pinecone.init(api_key="a4345490-b8s8-48e7-8f95-1bce784d353d",
              environment="gcp-starter")

### 1. Create an Index
pinecone.create_index(name="crud",
                      dimension=3,
                      metric="cosine",  # cosine is default
                      pod_type="s1")

### 2. Retrieve List of Pinecone Database Indexes
pinecone.list_indexes() # --> This will return ['crud']

### 3. Connect to Index
con = pinecone.Index('crud') #We need to describe pointer to connect with Indexes

### 4. Describe the Index
pinecone.describe_index(name='crud')

### This will return below output. Its information about index that you created.
""" IndexDescription(name='crud', metric='cosine', replicas=1, dimension=3.0, 
shards=1, pods=1, pod_type='starter', status={'ready': True, 'state': 'Ready'}, metadata_config=None, source_collection='')"""

### 5. Delete an Index
pinecone.delete_index(name='crud')

Backing Up Indexes with Collections

As described above, we need Collections to have backup of Indexes. Let’s create Collection first then will move towards creating the backup.

### Step 1 - Let’s create an index
pinecone.create_index(‘test’, dimension=3,
                       metric=’dotproduct’, pod_type = ‘s1’)

### Step 2 - Lets Create Collection from Index
pinecone.create_collection(name = "my-collection", source = "test")

# You can list all collections from database
pinecone.list_collections() # --> This will return [my-collection]

# Describe the Collection
res = pinecone.describe_collection("my-collection")
res.dimensions # --> It will return 3 as dimension

# Delete the Collection
pinecone.delete_collection("my-collection") 

### Step 3 - Continue from Step 2
# Now once you create an collection, you can visit the WebApp to 
# create the index from collections.

Now once you create an collection, you can visit the dashboard in pinecone WebApp to create index from the created collections.

Vector Partitioning

Vector partitioning is a technique used in database management, especially in systems dealing with large-scale, high-dimensional data such as those found in machine learning applications. It’s particularly relevant for databases that store and manage vectors, which are arrays of numbers representing various features or attributes.

Why Partition Vectors?

Partitioning helps manage large datasets by dividing them into smaller, more manageable segments. This division can improve query performance, enhance scalability, and facilitate better load balancing across the database system.

Consider a database containing emails where each email is represented as vectors for its subject, body, and other metadata.

• Without Partitioning: Searching through this unpartitioned data might be slow and inefficient, especially as the database grows.
• With Partitioning: By partitioning these vectors into different categories (e.g., subjects, bodies), the database can quickly access and search through specific types of data, improving performance.

Methods of Partitioning

1. Namespaces:

• Different types of vectors are stored under the same index but categorized into different namespaces.
• With email data, each email is converted into vectors representing the subject, body, and other information. These vectors are then categorized into different namespaces (e.g., subject, body, other) within the same index

2. Separate Indexes:

• Each category of vectors has its own dedicated index.
• Instead of a single index for all email vectors, separate indexes are created for subjects, bodies, and other information.

3. Metadata Tagging:

• Vectors are tagged with metadata to indicate their type or category, still residing in the same index.
• In the e-commerce example, a vector for a shirt might have metadata tags like “category: clothing” and “color: blue.”
• A vector from an email’s subject might have metadata like “data type: subject”, while a body vector might have “data type: body”.

Benefits of Vector Partitioning

• Improved Query Performance: By organizing data into relevant partitions, the system can quickly access the necessary subset of data, speeding up search and retrieval processes.
• Scalability: As data grows, partitioning makes it easier to manage and scale the database across multiple nodes or servers.
• Load Balancing: Distributing data across different partitions or nodes allows for more efficient use of resources and prevents any single node from becoming a bottleneck.
• Fault Tolerance: Partitioning can enhance the robustness of the database by isolating issues within a single partition, preventing them from affecting the entire database.
• Optimized Resource Utilization: Partitioning can lead to more efficient use of storage and processing resources. By organizing data into partitions based on similarity, related data is stored and processed together, optimizing resource use.

Final Note

This extensive article explores vector databases, which are at the core of Generative AI’s growth. we have emphasized the critical role of vector databases in managing complex data structures essential for advanced machine learning models. The article delves into foundational concepts, technological nuances, and the challenges overcome by vector databases.

Key concepts discussed include:

1. Vector Databases: These are specialized databases optimized for handling, storing, and retrieving high-dimensional vector data. Unlike traditional databases, vector databases efficiently manage data represented as arrays of numbers (vectors), crucial for AI and machine learning applications.
2. Vector Embeddings: This section explains how non-numeric data, like text or images, is converted into vectors that capture meaning and relationships, using algorithms like Word2Vec, GloVe, or BERT for text, and Convolutional Neural Networks for images.
3. Similarity Measures (Distance Metrics): The article discusses various metrics like Euclidean Distance, Manhattan Distance, Hamming Distance, and Cosine Similarity, which are used to measure the similarity or dissimilarity between vectors.
4. Vector Indexing: It covers how vector embeddings are organized and managed for efficient retrieval, highlighting processes like storing vectors, creating indexes, and searching the index.
5. Architecture of Vector Database: The architecture is designed for efficient management and retrieval of high-dimensional vector data, with applications in machine learning and data science.
6. Practical Applications with Pinecone: Pinecone is introduced as a specialized database service for handling vector data. The article guides through environment setup, installation, and practical operations in Pinecone, including CRUD operations, managing indexes and collections, and vector partitioning.
7. Benefits of Vector Partitioning: The article explains the advantages of partitioning vectors, such as improved query performance, scalability, load balancing, and fault tolerance.

Overall, the article provides a comprehensive understanding of vector databases, their importance in AI and machine learning, and practical insights into using a vector database service like Pinecone.

Please follow me on Medium as more Generative AI content is coming on your way. :)

TaaDaaa!

This blog is designed to increase your understanding of the DevOps and MLOps domains and how large companies effectively use them in their projects. Also, the blog is written in easy-to-understand language, making it beginner-friendly. I have included examples wherever necessary, and the contents are structured to help you quickly find what you’re looking for. Let’s take a quick look at what this blog covers.

1. Introduction to DevOps
2. DevOps Pipeline
3. Continuity Matters

As this is an introductory blog post, I will keep it brief and straightforward. The more complex topics will be covered in subsequent parts or series of this blog. Follow me here to stay tuned for future updates. :)

Introduction to DevOps!

The main idea behind DevOps is to improve the collaboration and communication between the development and operations teams in software development. Traditionally, the development team would create the software, and then the operations team would deploy and maintain it. However, this approach can lead to delays, errors, and miscommunication between the two teams.

DevOps = Software Development + Operations

DevOps brings together the development and operations teams to work together throughout the entire software development process. This collaboration helps to streamline the process and ensure that everyone is working towards a common goal.

Let me give you an example to help you understand better -

Imagine a small software development team working on a new mobile app for a startup company. The developers are working quickly to build out the features and functionality of the app, while the operations team is focused on setting up the infrastructure for deployment. However, the two teams are working in isolation, without communicating or collaborating effectively.

When the app is launched, it becomes immediately clear that there are performance issues and user complaints about slow loading times and crashes. The development team identifies some bugs and makes fixes, but deploying the updated version is a slow and difficult process due to a lack of coordination and automated deployment processes.

As a result, the updated app is not delivered to users for several days, leading to further frustration and lost revenue for the startup.

DevOps Pipeline

A DevOps pipeline, also known as a continuous delivery pipeline or release pipeline, is a series of automated steps that a software application goes through, from development to deployment.

The DevOps pipeline is designed to ensure that each change to the application is thoroughly tested and validated before it is deployed to production.

As I have described already, DevOps is a practice that brings together software development and operations. During the development stage, software developers write, test, and debug code. On the other hand, in the operations stage, the developed software is deployed to production and monitored to ensure efficient performance.

Let’s now talk about the various pipeline stages of DevOps!

Plan Phase (Dev)

The Plan phase is the initial stage of the DevOps lifecycle, during which teams plan and prioritize the work that needs to be done. It’s the phase where a Product Manager or Project Manager proves their worth.

The requirements related to development will be extracted from the stakeholders and customers to build the product roadmap which will serve as a guide for future development activities.

In short, this phase includes the →

• Defining the project scope,
• Determining the requirements,
• Setting the goals and objectives
• Creating a roadmap to guide the development process.

Some of the activities that take place during the Plan phase include:

1. Defining the project scope: The project scope outlines the boundaries and constraints of the project, including the features and functionality that will be included.
2. Determining the requirements: The requirements specify the functional and non-functional requirements of the software, including user stories, use cases, and acceptance criteria.
3. Setting goals and objectives: The goals and objectives define what the project is intended to achieve, including business outcomes, customer satisfaction, and other key performance indicators (KPIs).
4. Creating a roadmap: The roadmap provides a high-level view of the development process, including the timeline, milestones, and key deliverables.

Example:

One example of the Plan phase in action might involve a team developing a new e-commerce platform.

During the Plan phase, the team might work on defining the project scope by specifying the features that the platform should include, such as a shopping cart, product search functionality, and secure checkout.

They might also determine the requirements for the platform, such as the need to support multiple payment options and integrations with third-party services.

Required Tools:

• Microsoft Office, Google Documents
• Microsoft Project (Project Management Tools)
• JIRA, Asana, Mantis (Task Management tool)

During the Plan stage, the product roadmap is created by gathering feedback and requirements from stakeholders and customers. The product roadmap is then broken down into smaller units such as Epics, Features, User Stories, and Backlogs of tasks based on the requirements.

A ticket management system such as Jira, Azure DevOps, or Asana can be used to record and track the product roadmap. These tools offer a variety of features that aid in monitoring project progress, milestones, and issues. The tasks listed in the Backlog are used to plan the sprint and allocate them to teams to kickstart the development.

Before the development team starts writing code, the tasks are prioritized and organized into sprints. A sprint is a short, time-boxed period during which a team works to complete the tasks allocated to them. Typically, a sprint lasts between one and four weeks, and each sprint is focused on delivering a specific set of features or improvements.

During the sprint planning, the development team will evaluate each task in the Backlog, estimate the time and resources required to complete it, and prioritize it based on its importance. The team will then allocate the tasks to individual team members, considering their skills, expertise, and workload.

Once the sprint has begun, daily meetings or stand-ups are conducted to track progress, identify and address any obstacles or issues, and ensure that the project is on track. This allows the team to stay in sync and adapt quickly to any changes or challenges that arise during the development process.

In summary, the Plan stage is critical in establishing the roadmap for the project, breaking it down into manageable tasks, and prioritizing them for allocation to the development team during the sprint planning.

Code Phase (Dev)

The Code phase is an essential stage in the DevOps lifecycle where development teams write and build the code for the software application. This stage involves designing, coding, testing, and integrating the application code.

During the Code phase, the development team uses various coding languages and tools to write the application code that meets the requirements defined in the Plan phase.

Once the code is written, the team will test it thoroughly to ensure it is functional and free of bugs before moving on to the next stage.

The Code phase may involve the following activities:

1. Designing the application architecture: The team needs to design the application architecture, including the framework and the software stack required to run the application.
2. Writing code: Once the application architecture is designed, the development team starts writing the code for the application.
3. Code testing: The team tests the code to ensure it is functional and free of bugs. This includes unit testing, functional testing, integration testing, and performance testing.
4. Code review: The team performs code reviews to ensure code quality and maintainability.
5. Integration: Once the code is tested and reviewed, the team integrates it with other components of the software application.

Example:

Let’s say a development team is building a mobile app for a new ride-sharing service. In the Code phase, the team would use programming languages such as Java or Swift, as well as tools and frameworks such as Android Studio or Xcode, to write the code for the app. They would also use development methodologies such as Agile or Scrum to manage their work and ensure that the code is delivered on time.

Once the team has written the code, they would then move on to testing it. They would use testing frameworks such as Appium or Espresso to perform unit tests, integration tests, and end-to-end tests to ensure that the app functions as intended and is free of bugs.

After the testing phase, the team would then perform code reviews to ensure the quality and maintainability of the code. They would use code review tools such as GitHub or Bitbucket to collaborate with other team members and ensure that the code is clean, efficient, and adheres to best practices.

Required Tools:

Some tools that can be used during the Code phase are:

• Code editors such as Visual Studio Code, Atom, or Sublime Text
• Version control systems such as Git, SVN, or Mercurial
• Code review tools such as GitHub, Bitbucket, or GitLab
• Testing frameworks such as JUnit, Selenium, or Jest
• Continuous integration and delivery (CI/CD) tools such as Jenkins, Travis CI, or CircleCI

Build Phase (Dev)

The Build phase in the DevOps lifecycle is where the development team actually starts building the software. The software code is written, tested, and prepared for deployment in this phase.

The first step in the Build phase is committing the code to a shared repository. The developer writes the code and commits it to a code repository like Git or SVN. Once the code is committed, the developer creates a pull request to merge the new code with the shared codebase.

Then, another developer reviews the code changes to ensure that there are no issues. The review process is quick and lightweight, but it’s effective in identifying potential issues early. If the reviewer approves the code changes, the code is merged with the shared codebase. If there are issues, the reviewer rejects the code changes, and the developer is notified to resolve the issue.

In parallel, an automated build process is triggered, which builds the codebase and runs a series of tests to identify any issues. These tests include end-to-end, integration, and unit tests. If the build fails or any of the tests fail, the pull request fails, and the developer is notified to fix the issues.

By continuously checking code changes into a shared repository and running builds and tests, integration issues can be minimized, and breaking bugs can be identified early in the development lifecycle. This approach ensures that the software is always in a deployable state, and the team can deploy it to production with confidence.

Example:

In the previous Code phase example, the development team built a ride-sharing app. During the Build phase, the team would use build tools such as Gradle or Xcode to compile the source code and create an executable version of the app that can be installed on a mobile device.

Additionally, the team may also use tools to automate the build process, such as Continuous Integration and Delivery (CI/CD) tools like Jenkins or CircleCI. These tools can automatically compile the code and create new builds of the app whenever changes are pushed to the source code repository.

Required Tools:

Some tools that can be used during the Build phase are:

• Build automation tools such as Apache Maven, Gradle, and Ant
• Continuous Integration and Delivery (CI/CD) tools such as Jenkins, Travis CI, or CircleCI
• Containerization tools such as Docker and Kubernetes
• Artifact repository tools such as Nexus and JFrog

Test Phase (Dev)

After a successful build, the code is automatically deployed to a test environment where it undergoes more thorough testing outside of the development environment.

The Test phase in the DevOps lifecycle is where the application is tested thoroughly to ensure that it meets the desired quality and functional requirements. The primary goal of this phase is to identify and fix any issues or bugs before the application is deployed to production environments.

After the application is deployed to the testing environment, a series of tests are conducted to ensure the software meets the necessary requirements and works as expected.

These tests can be both automated and manual. In manual testing, the application is used by individuals in a way that simulates how a customer would use it (UAT). This allows for any issues or improvements to be identified before the software is deployed to production.

These tests ensure that the application is performing as expected, that all features are working as intended, and that the application is stable and secure.

Required Tools:

• Selenium
• Load testing tools such as Apache JMeter or Gatling for performance testing.
• Bug tracking and management tools such as JIRA or Trello to manage and track bugs and issues during testing.
• Emulators or real devices for testing the application on different platforms and devices.

Release Phase (Ops)

Once the testing has been successfully completed, the build is released as RC (Release Candidate) which is ready for being deployed in production.

In this phase mostly automation kicks in. Sometimes the organization may choose to automate the deployment of any build that passes the build phase. Additionally, developers can leverage feature flags to disable new features that are not yet ready for release to customers. This approach represents the pinnacle of DevOps and enables organizations to deploy numerous releases of their products daily, resulting in rapid time-to-market and continuous improvement of their products.

Required Tools:

Some tools that can be used in the Release phase include:

• Release Management tools such as JIRA or Trello to manage the release process and track any issues or bugs.
• Monitoring tools such as Prometheus or Grafana to monitor the application’s performance and usage.

Deployment Phase (Ops)

The Deploy phase is the phase where the application is actually deployed to the production environment, making it available to the end-users. This phase is closely tied to the Release phase, which is where the application is packaged and prepared for deployment.

Example:

Here’s an example of the Deploy phase for a ride-sharing app:

1. Build the Deployment Package: Before deploying the application, the development team will build a deployment package that contains all the necessary files and configurations for the application to run. This package may be a Docker container or a package created with tools like Ansible, Chef, or Puppet.
2. Configure the Environment: The development team will ensure that the production environment is configured correctly to run the application. This may involve setting up load balancers, configuring network settings, and installing any necessary software.
3. Deploy the Application: Once the deployment package is built and the environment is configured, the development team can deploy the application. They may use tools like Kubernetes, Docker Swarm, or AWS Elastic Beanstalk to manage the deployment.
4. Rollout Strategy: To ensure that the deployment is successful, the development team will follow a rollout strategy. This may involve deploying the application to a small group of users initially and gradually increasing the user base as the application stability is confirmed.
5. Monitoring and Feedback: After the deployment, the development team will monitor the application and gather feedback from end users to identify any issues or bugs that may arise. They may use monitoring tools such as Prometheus or Grafana to monitor the application’s performance and usage. They will also gather feedback from end-users through feedback channels such as in-app feedback, social media, or customer support.

By using these tools and techniques, the development team can ensure that the ride-sharing app is deployed to end-users in a controlled and efficient manner. They can also ensure that any issues or bugs that may arise during the deployment are identified and addressed promptly, leading to a better user experience and improved adoption of the application.

Required Tools:

Some tools that can be used in the Deploy phase include:

• Configuration Management tools like Ansible, Chef, or Puppet to configure the environment.
• Containerization tools like Docker to build and deploy the application.
• Container orchestration tools like Kubernetes or Docker Swarm to manage the deployment.
• Monitoring tools like Prometheus or Grafana to monitor the application’s performance and usage.

Operate Phase (Ops)

In this stage, we know that the build is deployed and it is ready to be used. In this phase, the operation teams make sure that things run smoothly as expected. They also take care of server configuring and provisioning using tools like Chef.

Monitor Phase (Ops)

The Monitor phase of DevOps is the final stage in the DevOps cycle, but it is by no means the least important. In fact, monitoring is crucial for ensuring that the application runs smoothly in production and that any issues are identified and addressed as quickly as possible.

During this phase, the operations team is responsible for monitoring the application and the infrastructure that supports it. This includes monitoring the servers, databases, network, and any other components that the application relies on. By monitoring these components, the operations team can identify any potential issues before they impact the end users.

To achieve effective monitoring, it is essential to have the right tools in place.

These tools can include:

1. Monitoring Tools: There are a variety of monitoring tools available that can be used to monitor the application and infrastructure. These tools can range from essential tools like Nagios, Zabbix, and Prometheus to more advanced tools like Datadog, New Relic, and Dynatrace.
2. Logging and Analytics Tools: Logging and analytics tools can be used to gather data on the application and infrastructure. This data can then be used to identify trends and patterns that can help to identify issues before they become critical. Examples of logging and analytics tools include ELK Stack, Splunk, and Graylog.
3. Alerting and Notification Tools: Alerting and notification tools can be used to notify the operations team when issues arise. These tools can be configured to send notifications via email, SMS, or other messaging systems. Examples of alerting and notification tools include PagerDuty, OpsGenie, and VictorOps.
4. Dashboard and Visualization Tools: Dashboard and visualization tools can be used to provide a real-time view of the application and infrastructure. These tools can be used to monitor key metrics such as CPU usage, memory usage, network traffic, and more. Examples of dashboard and visualization tools include Grafana, Kibana, and Tableau.

During the monitoring phase, it is important to define and monitor key performance indicators (KPIs) relevant to the application and business goals. These KPIs can include metrics such as uptime, response time, error rate, and user engagement. By monitoring these KPIs, the operations team can identify any issues that may be impacting the end users and take action to resolve them.

Continuity Matters!

You probably heard about these words somewhere → “Continuous Integration / Continuous Delivery / Continuous Deployment / Continuous Feedback” etc. These concepts float around the DevOps pipeline. Therefore I thought you should be familiar with these concepts.

Continuous Integration

Continuous Integration (CI) is a software development practice where developers frequently merge their code changes into a central repository, and automated builds and tests are run to check for any integration errors.

This is how it works →

1. Developers write and save their code changes to a shared code repository, like GitHub or Bitbucket.
2. A CI tool (such as Jenkins, Travis CI, or CircleCI) monitors the code repository for any changes. When it detects new code changes, it pulls the latest code from the repository.
3. The CI tool builds the code by compiling it into an executable program. This ensures that the code is free of syntax errors and other common mistakes.
4. The CI tool runs a suite of automated tests to verify that the code works as intended and does not introduce any new bugs or errors.
5. If any errors or bugs are found, the CI tool reports them back to the developers, who can then fix them and repeat the process until the code is working properly.
6. Once the code passes all tests, it is considered “clean” and can be deployed to production.

In summary, CI helps to improve software quality and accelerate the development process by automating the process of building and testing code changes. By catching errors early on and ensuring that code is always in a stable, working state, CI makes it easier and faster to deliver high-quality software to end-users.

Continuous Delivery

Continuous Delivery (CD) is the practice of automating the entire software delivery process so that code changes can be delivered to production quickly and reliably. Consider it an extension of Continuous Integration (CI).

Here’s an example to help explain how it works:

Let’s say you’re part of a team working on an e-commerce website. You’ve made some code changes to the checkout process to make it more user-friendly. Now you want to deliver those changes to production so that customers can start using them.

Here’s how Continuous Delivery can help:

1. You commit your code changes to the shared code repository.
2. The Continuous Integration (CI) system runs tests on your code changes to ensure that they’re stable and error-free.
3. If the tests pass, the CD system kicks in and automatically packages your code changes into a release candidate.
4. The CD system then deploys the release candidate to a staging environment, which is a production-like environment used for testing.
5. Automated tests are run on the staging environment to ensure that everything works as expected.
6. If the tests pass on the staging environment, the CD system deploys the release candidate to production.
7. The CD system then monitors the production environment for any errors or issues and rolls back the changes if any issues are detected.

In this example, Continuous Delivery ensures that your code changes are delivered to production quickly and reliably, while also minimizing the risk of errors or issues. It also enables your team to release changes to production more frequently, allowing you to deliver value to your customers faster.

Continuous Deployment

Continuous Delivery (CD) and Continuous Deployment (CD) are both practices that are often used in conjunction with Continuous Integration (CI) to automate software delivery. While they are similar in some ways, there is a key difference between the two:

Continuous Delivery (CD) is the practice of automating the entire software delivery process so that code changes can be delivered to production quickly and reliably, but it still requires a human decision to be made to deploy the code changes to production.

In contrast, Continuous Deployment (CD) is the practice of automatically deploying code changes to production as soon as they pass all tests in the CD pipeline. In other words, the CD takes automation one step further by eliminating the need for a human decision to deploy the code changes.

To illustrate the difference, let’s take the example of an e-commerce website where a team has made some code changes to the checkout process. With Continuous Delivery, the CD system packages and deploys the code changes to a staging environment where automated tests are run. If the tests pass, a human makes a decision to deploy the changes to production. With Continuous Deployment, however, the CD system automatically deploys the changes to production without the need for a human decision.

In summary, the key difference between Continuous Delivery and Continuous Deployment is that Continuous Delivery still requires a human decision to be made to deploy the code changes to production, whereas Continuous Deployment automatically deploys the changes without human intervention once they pass all tests in the pipeline.

Continuous Feedback

Continuous Feedback is a key component of the DevOps process that involves continuously gathering and analyzing feedback from users and systems to inform ongoing software development efforts.

Here’s an example to help explain how it works:

Imagine that you’re part of a team working on a mobile app that helps people track their fitness goals. Here’s how Continuous Feedback can help improve the app:

1. The team releases a new version of the app that includes a feature to track water intake.
2. The app is released to a subset of users, who are asked to provide feedback on the new feature.
3. The team uses analytics tools to track how users are interacting with the app, including how often they use the new water-tracking feature.
4. Based on user feedback and analytics data, the team identifies areas for improvement and makes changes to the app, such as adding more intuitive controls for the water tracking feature.
5. The team releases a new version of the app that includes the changes, and the process starts again with gathering feedback on the updated version.

By continually gathering and analyzing feedback from users and systems, the team is able to make data-driven decisions about how to improve the app. This process helps ensure that the app meets the needs of its users and stays relevant over time.

Continuous Feedback is not just limited to user feedback. It can also involve gathering feedback from systems, such as monitoring tools that track the performance and stability of the app. This feedback can help the team identify and address issues before they impact users.

Overall, Continuous Feedback is a crucial part of the DevOps process that helps teams deliver high-quality software that meets the needs of its users. It enables teams to continually learn and improve based on feedback from users and systems, leading to better software and happier users.

Footnote

I am grateful for your interest in perusing this introductory blog post and trust that it has proven to be both captivating and enlightening. It gives me great pleasure to have been able to provide you with valuable educational content.

It is essential to note that this blog post serves as a precursor to an upcoming series of blogs that explore more intricate topics such as Dockers, Jenkins, and Travis CI. These topics hold a central position in the realm of DevOps and possess the potential to make a significant impact on software development and delivery processes.

I encourage you to follow me closely so that you may stay informed of any new updates or releases in the series. My unwavering commitment to presenting content that is informative and accessible ensures that this series will appeal to both beginners and experienced professionals alike.

Let’s get acquainted with each other on LinkedIn.

Tadaaa!

I have used the techniques below to optimize memory use and speed up the computations.

But, wait. How do we find out the memory uses in the first place? This is very simpler than you think. We can use info() or memory_usage(). The main difference between the two is info() doesn’t show a detailed report on memory uses whereas the memory_uses() method showcases a detailed view of the memory usage.

Now let’s talk about the methods that will help in optimizing memory usage.

1. Use of inplace assignment
2. Read what you require
3. Change the data type of the column
4. Parallelize your operation
5. Read in chunks!

Let’s talk about each of these techniques in brief.

Use of inplace assignment

Most of the time we tend to use standard assignment in data manipulation. We manipulate the data and save that manipulation in a separate copy of the dataframe which is exactly what standard assignment means. Take a look at the example below.

Due to the use of the standard assignment technique, two distinct dataframe lies in the same environment causing an increase in the memory. Intuitively, the DF (original dataframe) is no use when we have decided to go with DF2 (manipulated dataframe). This very problem gets solved when you decide to use the ‘inplace assignment’ technique than a standard one.

In the inplace assignment technique, multiple copies of the dataframe won’t be created avoiding the increase in memory with a useless dataframe.

Read what you require

The name itself means, accepting the columns that you required and abandoning the least interesting columns. Sometimes, you might be in a situation where you have high-dimensional data but your interest lies in certain (let's say 2 or 3) columns. In such cases, reading all irrelevant columns cause heavy memory use than reading the interesting columns.

Let me elaborate with one example, let’s say you have medical data of many patients. Your job is to calculate the BMI. Your dataset is high dimensional (Think of 3M rows and 300 columns). But you required only the information of the patient's name, height, and weight. That’s it. Therefore instead of reading the entire data, it’s advisable to read what you required.

Change the data type of the column

We all know how pandas work in assigning data types. By default, pandas assign the highest memory datatype to columns. But the question is do we required to have the highest memory type every time? Let’s find out.

Inspecting numerical columns:

Let me be honest with you, you don’t require to have int64 or float64 datatype every time. Refer to the below table to understand why I am saying this to you.

As we discussed already when you have an int or float column by default the pandas will assign an int64 or float64 data type. But we need to inspect the column and find the minimum and maximum value of the integer column to decide the range. Once we have the range we can identify and specify the correct data type.

Inspecting Object columns:

Apart from numeric (int or float) you will see another most important datatype known as ‘Object’. This data type represents the categorical data which has some unique values repeated over and over again. We can convert ‘object’ datatype to ‘category’ with the help of astype method.

I generally use the below logic to optimize the categorical column.

If (unique_value (of Column) / Total Value (of column) < 0.5):
            data[col] = data[col].astype('category')

Look at the above table, the name column has more unique values than Cabin or Embarked, therefore as per our criteria if the ratio between unique values and total values is less than 50% then it will be optimized. In the above case, the name column does not meet the criteria (which is obvious!) and hence optimization doesn’t happen.

Parallelize your operations

Have you ever thought of parallelizing the program execution? Your laptop or system has multiple cores. Let’s not keep them idle. Let’s put them to work by using the multicore functionality with the help of pandarallel.

Note :- Pandarallel is alternative to Dask

First thing first, you need to install the library called — Pandarallel.

pip install pandarallel

Let me show you the difference in execution → With and Without Parallelization!

This is without parallelization:

This is with parallelization:

Note:- By modifying only one line of code, it offers a quick solution to parallelize your pandas operations over all of your CPUs. Additionally, progress bars are shown.

Let me show you how what changes you need to make to utilize the capabilities of the Pandarallel.

Read in chunks!

There are times when you face high-dimensional data. All other techniques mentioned above will be useless if you cannot read the data in the first place in the memory.

One interesting thing about Pandas is serialization. Pandas read data sequentially row-by-row at a time. We can leverage this and command pandas to read data by chunk size.

We can easily achieve this with the following python code -

Note that, every chunk is dataframe. You can add type(chunk_data) to check the type of the dataframe.

Footnote

These are the ways I generally employ in projects to optimize memory. This has a very much impact on the process when you are dealing with high-dimensional data.

Above mentioned subtle ways to optimize memory are not exhaustive in nature. There could be many ways to enhance memory optimization in pandas. I will be adding those in future updates.

Stay updated with all the content I post -

• Follow me on → Medium
• Follow me on → Linkedin

More content is in the draft and will be ready to post. Until then, stay tuned. :)

Let’s check the agenda for this blog post.

1. Collinearity Vs Multicollinearity
2. How does the weight vector of features get corrupted in the presence of collinearity?
3. Why Multicollinearity could be dangerous to your regression model?
4. Ways to detect multicollinearity among the features?
5. How to remove Multicollinearity?

Let’s start with understanding the difference between collinearity and multicollinearity.

Collinearity Vs Multicollinearity

These are the two most overlapping concepts from statistics. Collinearity is when there is a linear relationship or association between two variables or features. The term multicollinearity is formed for the linear association between two or more variables.

How does the weight vector of features get corruptted in the presence of collinearity?

Let’s say you have a 3 features F = {F1, F2, F3} with corresponding weights are W* = {1, 2, 3}.

Let’s say you have a new query point (row vector) Xq = {Xq1, Xq2, Xq3} then the weights of the features will align it this way -

Let’s rewrite the above sentence and label it as Equation (1).

Now as I have explained above, there must be some linear association between features or variables before we term them as multicollinear. To demonstrate this, let’s say,

Now let’s plug this value into our equation (1) and solve the equation.

Since we have two equations ready, let’s compare weight vectors corresponding to these two equations.

Note:-

Due to the presence of collinearity, weight vectors or weight associated with each feature could change. Many algorithms use the weight vector to do feature selection internally. Take the example of Logistic Regression, it uses |Wi| for feature importance. Therefore we must check if there is multicollinearity exists among the features.

Why Multicollinearity could be dangerous to your regression model?

Until now, I have explained how problematic it could be to model-based feature selection. Now let’s discuss the potential problems that may arise when we build regression models by ignoring multicollinearity.

Having multicollinearity in the regression model can change the interpretation of the regression coefficient. The regression coefficient is the value that represents the relationship when a meaningful change in the dependent variable for each unit change in the independent variable by keeping all other independent variables constant. That’s the problem. When you have independent variables correlated then it’s difficult to fulfill the criteria of “keeping other independent variables constant”. As a result, if we change 1 unit in the independent variable there might be an ‘X’ unit change in other independent variables due to collinearity between features. This makes the model difficult to understand the true relationship between dependent and independent variables.

Ways to detect multicollinearity among the features?

There are multiple methods to capture the multicollinearity among the features. In this blog post, let’s try to cover these methods!

1. Correlation Matrix
2. VIF

Let’s explore each of these above.

Correlation Matrix

You can be a Sherlock Holmes and plot the correlation matrix where you can spot multicollinearity very easily. This is the easiest way that I could think of for now.

Let me take you to a practical example. Here is the data that I am using to demonstrate.

The objective of the problem is to predict the sales price based on the mentioned variables. I know, I know, you don’t require much information on the objective of the dataset as we aren’t going to accomplish that in this blog. :)

Once we have the dataset ready, all you need to do is to load and exclude the target variable and run bunch of lines of Python code to print the correlation matrix like below. (Don’t worry, the code will be shared with you by the end of this blog post)

The couple of observations that I would like to add,

1. The variable ‘Interior’ is highly correlated with # of Beds, the # Of Rooms, and # of Bath.
2. There is a high correlation between # of Rooms and # of Beds which is oblivious!
3. There is also a high correlation between ‘Condo Fee’ and ‘Tax’

As we have already seen the problem pertains to the high correlation and how it changes the weight vector. Hence building a model with these variables (without treating the problem) wouldn’t be advisable.

Variance Inflation Factor (VIF)

It’s better to have one metric to understand the multicollinearity than the correlation interpretation. There is one popular metric known as Variance Inflation Factor or shortly known as VIF which does the same thing. But what do you mean by VIF?

It’s ratio of the variance of the model to the variance of the model which include only that one independent variable.

In other words, we can say that the VIF is the measure of the strength of correlation between independent variables. It is predicted by taking a variable and regressing it with every other variable.

Let’s calculate the VIF for our dataset and showcase it here in the form of a table.

Look at the table carefully, the # of Beds and # of Rooms have a high VIF score which indicates that there is strong collinearity between these features. In real life, this is so oblivious and the above table does make sense. But what exactly those scores are?

The score of each independent variable represents how well is the variable explained by other independent variables. In order to accomplish this, each independent variable acted as the dependent variable. The R2 (R-squared) score will be generated. Higher the R2 score more that variable correlated with other variables. This is what VIF returns as a score (See the table above)

The main problem that I found with VIF is that there is no upper limit. It starts with 1 and goes towards infinity. The above table is a mere thumb rule as there are no hard and set rules for their interpretation. For example, some companies accept the VIF values till 7 and some want to have VIF below 4.

How to Fix Multicollinearity?

Since are multiple ways to detect multicollinearity, Likewise, there are multiple ways to fix it. Let me make life easier by jotting down some of these methods to fix the multicollinearity issue.

• Dropping the Correlated Variables
• Combining the Variables
• Life savior → PCA

Let’s start exploring each of the above techniques to understand more about it.

Dropping the Correlated Variables

This would be the easiest method to cut down the multicollinearity of the data. When there is high multicollinearity between two independent variables would mean that each variable is capable of explaining the other variables. Therefore we can keep one and remove other variables from the dataset without affecting much variance.

If you look at the above table, the variables like ‘Interior (Sq Ft) and # of Rooms are well explained by the other variables like # of Bed and # of Bath, etc. Therefore we can remove those variables from the dataset and check whether it brings down the multicollinearity or not.

One thing to remember is that dropping variables and checking VIF is an iterative process. Even though we have the information about two variables that need to be removed, we have not immediately removed them. Here, we have removed the ‘Interior(Sq Ft)’ first as it has the highest VIF value. As soon as we removed it and again checked for VIF values, the VIF values of other variables changed. Therefore we have to remove variables in an iterative fashion.

Combining the Variables

Yes, we can combine multiple variables together and create one variable which has capabilities to represent the other variables. In more technical terms, the representative variable should capture the variance of other variables.

In our case, we can combine ‘# of Beds’ and ‘# of Bath’ and form a new variable as ‘Total Rooms (Space)’ which can serve as a representative of both of these variables.

Let’s validate our assumptions with the VIF table. (We will be calling our VIF function on the modified data)

These two methods are used most in the industry to remove the effect of multicollinearity. There is one other method that we have yet to discuss is on, how PCA would be a life savior when you have a multicollinearity issue. …

Life savior → PCA

Principal Component Analysis (Short, PCA) is a dimensional reduction technique used to reduce the dimensions of the data. In the dimensionality reduction process, we reduce the dimensions of data with the help of % of variance required to build sensible ML models.

Most of us get confused easily about PCA being a feature selection technique. But in reality, it’s not. It rather features extraction techniques where features new features get extracted in the form of the components which capture most of the variance in the data.

The above table shows the number of principal components and variance captured by each of these. You can completely neglect the last 2 and be happy with 94% of variance or you can neglect the last 1 and can have 98% of captured variance by model. In both cases, you are reducing the dimensions. But here, we are not for ‘dimensionality reduction, we are here for ‘multicollinearity reduction’. Thus, for now, let’s keep all the components as it is and run the VIF model to check the VIF score for each of these components.

I don’t recommend using this method when you want to build a model with interpretability. We lose the identity of the variables as soon as we implement PCA. You must forget the interpretability when you use the PCA.

Final words

Having multicollinearity in the data can corrupt the model-building process. The multicollinearity can be problematic to your regression model as well as your classification models. A simple technique like a correlation matrix can be used for the identification of multicollinearity and as simple as dropping or combining the variables can be employed to fix the issue. Selection of the right variables (domain-specific) can make life easier, therefore special attention needs to be given while doing feature selection. We also talked about techniques like PCA to reduce multicollinearity but the red flag is for the interpretability. The moment we use PCA we lose the identity of the variables. As I always say, there are no right or wrong techniques, there are only techniques that can be used based on the situation.

Footnote

I have written numerous articles on Quora & Medium. There are articles lying in draft waiting for me to finish and publish. Follow me on Medium as well as on Linkedin to stay in touch with me. See you around. :)

Credits where it’s due -

https://www.instagram.com/p/ChpUW7cKjKS/

I have created this blog to consolidate best material available on internet to one place. I have given the credits where it’s due in the credit section at the end of this blog posts. Please do check that out.

The Cloud: Foundation

It is very important to understand the basics of cloud computing, why do we even use it, etc.? There are many companies that provide cloud services to end-users, businesses, etc to carry out the daily essential needs.

Here is a list of the top 10 cloud service providers:

• Amazon Web Services (AWS)
• Microsoft Azure.
• Google Cloud.
• Alibaba Cloud.
• IBM Cloud.
• Oracle.
• Salesforce.
• SAP.

Now let’s understand, what exactly cloud computing is -

What Is Cloud Computing?

If you ask me the definition of cloud computing, I would simply term this concept as → It is a delivery of computing services through the internet. Now you must be thinking about what could be those computing services that are being offered by cloud computing providers?

• Servers
• Cloud Storage
• Networking
• Databases
• Softwares

But, why cloud computing is the requirement?

To understand this, we need to go back and look at the traditional IT setup which was the last thing for many users or businesses before adopting the cloud computing technology.

This setup is mostly known as start-up setup where most of the big giants of today had started their office journey through Home or Garage. Later, moved to office space and subsequently maintained the required data centers. But if you look at the problems of maintaining data centers, for a moment, we might feel why should we maintain the data center in the first place?

• Rent → For required space
• Pay → For power supply, cooling equipment, maintenance, etc.
• Limited in Scaling
• Hardware / Software Limitation
• Hire a 24/7 team to monitor the infrastructure
• Lack of disaster management etc.

In the traditional setup, you have to look after everything that is related to data centers. This is risky thing to do. This is where cloud services comes into the picture. They will help you to move your data service into the virtual location than the physical location. Therefore problems pertaining to physical presence of your data centers will go automatically.

Benefits of Cloud Computing?

There are many benefits of cloud computing. Few of them have been listed here.

Pay as you go Pricing Model:

It is possible to pay for the services that you use only. Oftentimes in a traditional setup, we buy expensive equipment thinking that it will suffice our requirements. In business, requirements are not always fixed but they are variable in nature. To avoid this pitfall, with cloud computing services, we pay for what we use.

No Upfront Cost:

There are no upfront costs like setting up the data centers and servers or buying up some hardware etc.

Benefit from massive economies of scale:

Usage from hundreds of thousands of customers aggregated in the cloud. Therefore we are sharing the costs with other customers and hence it will be economical and cheaper than handling the data centers on your own.

Unlimited Storage & Processing Power:

As per requirements businesses can scale up or scale down their storage and processing powers.

Speed and Agility:

Launch resources within a few clicks in minutes instead of days or weeks on data centers.

No maintenance costs:

Since the central authority (Businesses that are providing cloud services) are looking after the maintenance and hence customers need not spend their time, energy and money looking after the maintenance.

High performance and lower latency:

With cloud computing services, we can deploy services around multiple regions in minutes with lower latency and high performance.

After understanding the application, we can define the cloud computing services as -

Cloud computing is the on-demand delivery of compute power, database storage, applications, and other IT resources

Types of Cloud Computing:

After understanding the basics of cloud computing, it’s necessary for users to know various types of cloud computing before adopting it for personal use or business use.

Public Cloud:

• In the public cloud, the cloud resources are managed by cloud service providers (CSP) and delivered to end-users via the internet.
• This type of cloud is created from the IT infrastructure and it is not owned by end-users.
• Some of the largest public cloud providers include Alibaba Cloud, Amazon Web Services (AWS), Google Cloud, IBM Cloud, and Microsoft Azure.
• Public clouds are cheaper than Private cloud services as it comes under shared cloud services.

Private Cloud:

• These cloud services are used by a single organization and not exposed to the public.
• Private cloud is also known as Internal Cloud or Corporate Cloud. It is used by an organization that wants to control their data centers in the cloud.
• Most of the time private cloud is used when security or data protection is most important.
• Private cloud provides a high level of security and privacy to the users.

Hybrid Cloud:

Hybrid Cloud = Public Cloud + Private Cloud

• This cloud service is semi secured. Because it offers two types of services. Services that are running on public cloud can be accessed by anyone while the services that are running on the private clouds can only be accessed by the organization.
• Hybrid cloud offers flexible resources because of the public cloud and secure resources because of the private cloud.
• Managing a hybrid cloud is complex because it is difficult to manage more than one type of deployment model.

Community Cloud:

• Community cloud allows systems and services to be accessible by a group of several organizations to share the information between the organization and a specific community
• It is owned, managed, and operated by one or more organizations in the community, a third party, or a combination of them.

Cloud Platform Models:

There are 3 types of cloud computing service models -

1. Infrastructure as a Service (Iaas)
2. Platform as a Service (Paas)
3. Software as a Service (SaaS)

Let’s look at them one by one.

Infrastructure as a Service (IaaS):

• IaaS is also known as ‘Hardware as a Service (HaaS)’
• Actually, IaaS means a cloud service provider manages infrastructure for you. Like the actual servers, network, virtualization, and data storage.
• The main advantage of the IaaS is it won’t let users or organizations buy expensive servers or other physical equipment but provide them as a service.
• The user has access through an API or dashboard, and essentially rents the infrastructure.
• The user manages things like the operating system, apps, and middleware while the provider takes care of any hardware, networking, hard drives, data storage, and servers.
• Example: DigitalOcean, Linode, Amazon Web Services (AWS), Microsoft Azure, Google Compute Engine (GCE), Rackspace, and Cisco Metacloud

Platform as a Service (Paas):

• In PaaS, hardware, as well as the applications/software platform, is provided and managed by the outside cloud service provider, but the user handles the apps running on top of the platform and the data the app relies on.
• Primarily it is created for the programmer to develop, test, run, and manage the applications.
• Example: AWS Elastic Beanstalk, Windows Azure, Heroku, Force.com, Google App Engine, Apache Stratos, Magento Commerce Cloud, and OpenShift.

Software as a Service (SaaS):

• SaaS is also known as “On-Demand Software”.
• It is a service that delivers a software application — which the cloud service provider manages — to its users.
• Typically, SaaS apps are web applications or mobile apps that users can access via a web browser.
• Software updates, bug fixes, and other general software maintenance are taken care of for the user.
• Think, it is a completed product that is run and managed by the service provider.
• SaaS also eliminates the need to have an app installed locally on each individual user’s computer, allowing greater methods of a group or team access to the software.
• Example: BigCommerce, Google Apps, Salesforce, Dropbox, ZenDesk, Cisco WebEx, ZenDesk, Slack, and GoToMeeting.

Server Virtualization:

The secret sauce that lets cloud providers give their customers on-demand compute resources in such a wide range of configurations is virtualization. When you request a virtual machine (VM) with a particular processor speed, memory capacity, and storage size, AWS doesn’t send some poor engineer running through the halls of its data center looking for an available machine with exactly that profile.

Rather, as you can see illustrated in Figure 1.1, AWS carves the necessary resources from larger existing devices.

Serverless Workloads:

• Build and run applications without thinking about servers
• Now don’t be fooled by the name. You can’t run a compute function without a computer environment (a “server”) somewhere that’ll host it. What “serverless” does allow is for individual developers to run their code for seconds or minutes at a time on someone else’s cloud servers.
• Serverless technologies feature automatic scaling, built-in high availability, and a pay-for-use billing model to increase agility and optimize costs.
• These technologies also eliminate infrastructure management tasks like capacity provisioning and patching, so you can focus on writing code that serves your customers.
• The serverless model — as provided by services like AWS Lambda — makes it possible to design code that reacts to external events.

Scalability and Elasticity: Two Key Principles

The world’s largest public cloud providers can accomplish a great deal by combining the wonders of server virtualization with the power that comes from owning vast data centers filled with racks upon racks of hardware resources. Elasticity and scalability are the two key principles through which a lot of this happens, and understanding exactly what they mean can help you optimize your design choices so you’ll get the most bang for your cloud buck.

Scalability:

• A scalable service will automatically grow in capacity to seamlessly meet any changes in demand.
• A well-designed cloud-based operation will constantly monitor the health of its application stack and respond whenever preset performance metrics might soon go unmet.
• The response might include automatically launching new server instances to add extra computing power to your existing cluster.
• Note that, a large cloud provider like AWS will, for all practical purposes, have endless available capacity so the only practical limit to the maximum size of your application is your organization’s budget

Just how big is AWS? Well, if it can handle the capacity stresses required to keep millions of Netflix customers happy — and if you’ve ever watched a movie on the AWS-hosted Netflix, then you know it can — then AWS will surely be able to keep up with whatever trouble your applications send its way.

Elasticity:

You can stretch an elastic band far beyond its resting state. But part of what makes it truly elastic is the fact that, when you let go of it, it immediately returns to its original size.

The reason the word elastic is used in the names of so many AWS services (Elastic Compute Cloud, Elastic Load Balancing, Elastic Beanstalk, and so on) is that those services are built to be easily and automatically resized.

Generally, you set the maximum and minimum performance levels you want for your application, and the AWS service(s) you’re using will automatically add or remove resources to meet changing user demands.

For an instance, a scalable eCommerce website could be configured to function using just a single server during low-demand periods, but any number of additional servers could be automatically brought online as demand spikes. When demand drops back down, unused servers will be shut down automatically. This is what the elasticity means.

AWS Cloud History

AWS Cloud Number Facts

• In 2019, AWS had $35.02 billion in annual revenue
• AWS accounts for 47% of the market in 2019 (Microsoft is 2nd with 22%)
• Pioneer and Leader of the AWS Cloud Market for the 9th consecutive year
• Over 1,000,000 active users

AWS Global Infrastructure

The AWS Global Cloud Infrastructure is the most secure, extensive, and reliable cloud platform, offering over 200 fully-featured services from data centers globally.

Whether you need to deploy your application workloads across the globe in a single click, or you want to build and deploy specific applications closer to your end-users with single-digit millisecond latency, AWS provides you the cloud infrastructure where and when you need it.

The AWS Cloud spans 84 Availability Zones (AZ) within 26 geographic regions around the world, with announced plans for 24 more Availability Zones and 8 more AWS Regions in Australia, Canada, India, Israel, New Zealand, Spain, Switzerland, and United Arab Emirates (UAE).

The AWS infrastructure consists of following things →

• Region
• Availability Zones (AZ)
• Data Centers
• Edge Locations / Points of Presence

Let’s look at them one-by-one.

Region:

• AWS has the concept of a Region, which is a physical location around the world where they cluster multiple data centers.
• We call each group of logical data centers an Availability Zone. (AZ)
• It is a geographically distinct location with multiple AZs (data centers)
• Every region is physically isolated from and independent of every other region in terms of location, power, and water supply
• Each region has at least 2 AZs
• Not all services are available in all regions
• AWS maintains multiple geographic Regions, including Regions in North America, South America, Europe, China, Asia Pacific, South Africa, and the Middle East.

How to choose an AWS Region?

If you wish to launch a new application, where you should do it? There are multiple factors associated with it. Let’s look at them

• Compliance with data governance: Most of the country with their data never leaves a region without govt. explicit permission. Therefore if you are doing business in India, your data center must be located in India.
• Proximity matters: Proximity matters as it reduces the latency. If you have business in India and you are targeting Indian customers then what’s the point of choosing North America as your region?
• Localization: New services and new features aren’t available in every Region
• Pricing: Pricing varies from region to region and is transparent in the Service Pricing Page

Availability Zones (AZ):

• An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region
• Each region has many AZs ( At least 2, Usually 3, Max 6)
• They’re separate from each other, so that they’re isolated from disasters
• They’re connected with high bandwidth, ultra-low latency networking
• AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other.

Points of Presence (Edge Locations):

• It is used for getting data out or uploading data fast to AWS
• EL is a data center owned by a trusted partner of AWS and has a direct connection to the AWS network.
• These locations serve requests for CloudFront and Route53. Requests going to either of these services will be routed to the nearest EL automatically.
• S3 Transfer Acceleration traffic and API Gateway endpoint traffic also use the AWS Edge Network
• Content is delivered to end-users with lower latency

Tour of the AWS Console

AWS has Global Services:

• Identity and Access Management (IAM)
• Route 53 (DNS service)
• CloudFront (Content Delivery Network)
• WAF (Web Application Firewall)\

Most AWS services are Region-scoped:

• Amazon EC2 (Infrastructure as a Service)
• Elastic Beanstalk (Platform as a Service)
• Lambda (Function as a Service)
• Rekognition (Software as a Service)

Shared Responsibility Model diagram

• CUSTOMER = RESPONSIBILITY FOR THE SECURITY IN THE CLOUD
• AWS = RESPONSIBILITY FOR THE SECURITY OF THE CLOUD

AWS Acceptable Use Policy

• No Illegal, Harmful, or Offensive Use or Content
• No Security Violations
• No Network Abuse
• No E-Mail or Other Message Abuse

Understanding Your AWS Account

Amazon Web Services is a really big place. Each account has access to more than 100 distinct services, which can each be deployed in hundreds or even thousands of permutations.

Anyone with the right account credentials can — legitimately or otherwise — launch new service instances without anyone else knowing.

With all that going on all the time, there are going to be two account-level things you want to be clear about:

Costs:

What’s it all going to cost you? Because cloud services are scalable, there’s virtually no limit to how much of them you can purchase, or to how much they can cost you. If something isn’t going according to plan, you’ll want to know what it is and how to pull its plug as soon as possible.

Administration:

Who’s running the show? In other words, who has the authority to make the big decisions about firing up and shutting down account resources? In addition to the fact that those decisions will have significant financial implications, they’ll also define your organization’s public presence.

In this section, we will be mostly discussing monitoring and keeping a lid on costs. If it is not controlled, it will affect you financially.

The Free Tier

Human tendency is to take trial first before purchasing the goods or services. AWS has taken this into account. AWS figures that the more comfortable you feel working with its services, the more likely you’ll eventually start moving serious workloads to its cloud. So, AWS offers a generous Free Tier for the first 12 months after opening a new account.

Under the Free Tier, you can freely experiment with light versions of most AWS services without being billed.

What you get in Free Tier?

• It’s remarkable how much you can do with the Free Tier. Since, for instance, you’re allowed to run a t2.micro Elastic Cloud Compute (EC2) instance — powered by either Linux or Windows — for up to 750 hours per month
• You can run multiple concurrent EC2 resources for an experimental purpose. You won’t be billed as long as the total monthly hours of run time don’t exceed 750
• You can run low demand website without interruption for 12 months.
• You can run lightweight relational database workload under Amazon’s Relational Database Service (RDS) for 750 hours per month.
• You can also utilize storage services. You can store up to 5GB of data within the Storage Services (S3) bucket.
• We can utilize 25 GB of storage in Amazon DynamoDB (NoSQL Database)
• We can utilize Amazon SageMaker (Tool for ML and Data Scientist) for 2 months.
• AWS Lambda allows you to send 1M requests per month. (AWS Lambda is a serverless compute service that runs your code in response to events and automatically manages the underlying compute resources for you)
• You get 2 months of free trial in Amazon Redshift (Data Warehousing )
• You get 1M publishes for Amazon’s fast push messaging service called Amazon SNS

There are more to this but you get my point. You can have that checked right from AWS website.

Tracking Your Free Tier Usage

Note that, the free tier is much like the free trial which comes with some limitations. Once you have completed checking something or you are done with utilizing the services, it is necessary to stop them. Leaving those services running can cost you. Therefore you definitely want to keep an eye or two on your account.

• AWS sents you email notification as soon as you are near to end of free tier usages. It’s important to keep eye on the emails.
• The second tracking tool will appear toward the bottom of the Billing Dashboard for any account that’s still Free Tier–eligible.

What’s Available Under the Free Tier?

There are actually two kinds of free services:

■ Light implementations of most AWS services that are available through the first 12 months of a new AWS account

■ Light usage of certain AWS services that are always available — even after your initial 12 months are up

Permanently Free Services

There are few services from Amazon AWS are permanently free. It includes following services:

• 10 GB of data retrievals from Amazon Glacier per month. Glacier is a low-cost storage service that’s ideal for long-term storage where you’re not likely to require quick access to your data.
• 62,000 outbound email messages per month using Amazon Simple Email Service (SES). SES is an enterprise-scale email service geared to organizations that regularly send large volumes of emails.
• One million requests and 3.2 million seconds of compute time for AWS Lambda functions. Lambda lets you run code in response to network-based triggers without having to worry about provisioning any underlying server infrastructure.
• 10 custom monitoring metrics and 10 alarms on Amazon CloudWatch active at any given time. You use CloudWatch either directly or in combination with other tools to track resource performance and costs.

Note:- AWS Free Tier benefits do not apply to resources launched in the GovCloud (US) region or through an AWS Organizations member account

Online Calculators for Calculating the Costs

There are various calculator available to visualize the total costs that will incurred if you use the services for particular duration.

• AWS Simple Monthly Calculator
• AWS Total Costs Ownership Calculator (TCO Calculator)

In the monthly calculator, you can add the required services and check for the estimated uses and costs that will be incurred for users.

In the TCO calculator, it will return the services that costs you on Amazon AWS and it will also show you, how much costs would’ve occurred to you if you have selected “on-premises” resources than AWS cloud. (Kind of comparison)

Service Quotas (Service Limits)

The total scope of the AWS cloud is vast. It’s hard to even visualize just how much compute and storage capacity is available for its customers. And the scalability of its design means that there shouldn’t be a practical ceiling on how much infrastructure any customer is able to access on demand

AWS maintains service quotas (formerly called service limits) for each account to help guarantee the availability of AWS resources and prevent accidental provisioning of more resources than needed. Some service quotas are raised automatically over time as you use AWS. However, most AWS services require that you request quota increases manually.

You can use AWS Service Quotas console to view and request increases for most AWS quotas.

Billing and Cost Management

No doubt that AWS provides great services for various applications. Along with performance of the AWS, the organization also needs to keep eye on the billing and usages details. There are various tools that could help any organization or customer to overlook their account.

The AWS Billing Dashboard

• It contains spend summary is visual way (Past & Present Spend Summary)
• It also contains forecast for current or future months so we can plan accordingly
• There’s also a Month-to-Date Spend By Service display from which choosing the Bill Details button will take you to an itemized breakdown of your current spending.
• The Billing Dashboard also includes links to pages where you can set payment methods and enter your organization’s tax information.
• The Payment History link provides records of your previous transactions.

AWS Budgets

An AWS budget is a tool for tracking a specified set of events so that when a preset threshold is approached or passed, an alert — perhaps an email — is triggered.

You can create one of three budget types:

• Cost budget to monitor costs being incurred against your account
• Usage budget to track particular categories of resource consumption
• Reservation budget to help you understand the status of any active EC2, RDS, Redshift, or Elasticache reserve instances you might have

Monitoring Your Costs

We can monitor the costs for services through following tools

Cost Explorer:

• Cost Explorer lets you build graphs to visualize your account’s historical and current costs.
• It allows you to filter account cost events by date, region, Availability Zone, instance type, platform (Linux, Windows, and so on), and others. CSV-formatted files based on customized views can also be downloaded.

Cost and Usage Reports:

• Cost and usage reports are (right now, at least) accessed from the Reports link on the Billing Dashboard.
• You can configure reports to be created that include the full range of activity on your account, including what resources you have running and how much they’re costing you.
• The reports, once generated at preset intervals, are compressed, saved using the CSV format

Cost Allocation Tags:

Tags are metadata identification elements representing a resource and its actions. Tags can be used to organize and track your resources, allowing you to visualize and better understand how resources are being used.

AWS offers two distinct kinds of tags that actually have nothing to do with each other.

• Resource tags: Resource tags are often used in busy accounts to help administrators quickly identify the purpose and owner of a particular running resource.
• Cost allocation tags: Cost tags are only meant to interact with billing tools and won’t show up in the context of any other AWS resource or process. Like resource tags, cost allocation tags help you efficiently identify your resources, but only for the purpose of tracking your account spending.

AWS Organizations

• Formerly known as Consolidated Billing
• AWS Organizations allows you to centralize the administration of multiple AWS accounts owned or controlled by a single company.
• This can make a lot of sense, since many companies will own more than one account — or might share AWS resources with vendors and clients
• Being able to control the allocation of resource permissions, security, and spending from a single pane of glass is a big deal. It’s also a convenient way to manage payments

Getting Support on AWS

AWS provides layers of support designed to help you as you take your first confusing steps on the platform, learn about the many ways you can use AWS services, and struggle to optimize and secure your increasingly complex resources.

In this section we will learn about -

• What kinds of AWS support that are available and how to find them.
• You’ll learn about the four AWS support plans
• what professional services are available, what kinds of online documentation and forums are out there?
• what Amazon’s Trusted Advisor tool is?

Support Plans

When you create a new AWS account, you’re required to select a support plan. Depending on the kinds of projects you plan to run with your account, the $15,000/month and up that you’ll pay for the Enterprise Support plan which might feel a bit excessive, so the free Basic Support plan is always an option while you’re getting started.

Let’s look at the benefits of having AWS supports →

There are total 4 types of support plans and we will look at them one by one.

The Basic Support Plan:

• On the Basic plan, you’re not paying anything beyond the regular costs of consuming AWS resources.
• However, for the most part, you’re given access only to publicly available documentation, including white papers, tutorials, and support forums.
• In addition, you’ll be able to contact customer service at any time of the day or night for account-related issues (such as bill payment).
• You also get limited access to the Trusted Advisor tool.

The Developer Support Plan:

• AWS recommends the Developer Support plan for organizations running non-production workloads. (Things that organization currently developing but not pushed to the production)
• Developer plan is not suitable for critical applications. Because your direct access to AWS cloud support associates is limited to emails during business hours and to asking their advice on general information about AWS use cases.
• The associates you contact when you open a support ticket won’t be able to discuss specific problems you’re having or details about the way your particular application should be deployed.
• So if you are running infrastructure that simply must remain active, then you should seriously consider signing up for a higher support level.

The Business Support Plan:

• The Business Support plan can meet the needs of many organizations by offering relatively fast and detailed answers to your technical questions.
• For help, a cloud support engineer is available by email, call, chat within one hour.
• Less severe issues can take longer — up to 24 hours.
• This level of support can include help troubleshooting interoperability between AWS resources and third-party software and operating systems.

The Enterprise Support Plan:

• As you can easily tell from the price tag (starting at $15,000/month), the Enterprise plan is appropriate only for large operations whose scope is global and for whom downtime is simply unthinkable.
• For example, can you imagine an evening without Netflix (which is, after all, a key AWS customer)?
• It includes the technical account manager (TAM) who is assigned as a dedicated “guide and advocate” for your account.
• Your TAM is more than just a technical resource; the TAM becomes closely involved in your deployment, guiding your team through planning, launches, and proactive reviews — all optimized using best practices.
• Your TAM can provide you innovative solutions as well.
• Besides a TAM, Enterprise customers get 24/7 access to senior cloud support engineers and a 15-minute response time for business-critical troubleshooting

Documentation and Online Help

Even if you decide not to sign up for a paid support plan, there are still many places to look for help — it’s just that, for the most part, it’ll be self-help.

Documentation:

• AWS provides a lot of documentation resources. By “a lot” we mean there’s so much of it that you’ll probably never have enough time to read through the whole thing.
• On the AWS documentation page you’ll find links to more than 100 AWS services along with tutorials and projects, software development kits (SDKs), toolkits, and general resources

Knowledge Center:

• The AWS Knowledge Center is basically a frequently asked questions (FAQ) page.
• The Knowledge Center page contains links to nearly 1,000 questions arranged by service, each representing a fairly common problem encountered by real AWS customers.
• You’ll find questions like, “How can I install the AWS CloudFormation helper scripts on Ubuntu or Red Hat Enterprise Linux?” and “What happens when my reserved instance expires?”

Discussion Forums:

• When things don’t work the way you expected, it’s generally safe to assume you’re not the first person to hit this particular brick wall. Even if this is something only a couple dozen people have experienced, there’s a good chance that at least one of them asked about it on some online forum.
• While many of those questions were probably asked on Stack Overflow or another public site, the AWS discussion forums have also addressed their share of problems.

Trusted Advisor (TA)

You use Trusted Advisor to visually confirm whether your account resource configurations are sound and are compliant with best practices.

Trusted Advisor organizes its compliance alerts across five categories.

The full range of Trusted Advisor alerts is only available for users signed on to either the Business or Enterprise Support service tier. For everyone else — including those on the free Basic Support plan — access is limited to service limits and some of the security alerts.

Securing Your AWS Resources

Everyone in the IT world knows that application and infrastructure security are critically important elements of any deployment. Unfortunately, the constant stream of news about catastrophic data breaches suggests that some of us may not yet realize just how important.

In this section, we will learn how to use the Identity and Access Management (IAM) service to control which people and processes get past the wall guarding your resources (authentication) and what they’re allowed to do once they’re in (authorization).

Identity and Access Management (IAM)

IAM is a global service and not depends upon the regions. In IAM’s management dashboard, we will see all administration tools that we will need to manage the basics of account security.

Protecting The Root User:

• Root user is created by default when you sign-up with the AWS. This account must not be used or shared but always be protected.
• For practical reasons, the root user has the permissions necessary to perform any task on your account. That includes launching any resources in any region and authorizing any expenses.
• The problem is that actively using the root user over the long term with only minimal protections presents a significant security risk.
• Should the root credentials ever be compromised through an attack, there will be no limit to the potential for mischief. All your data could be stolen or deleted or the attacker could run up hundreds of thousands of dollars of charges running a rogue crypto mining operation.
• The recommended best practice is, therefore, to protect the root user by creating a complex password and implementing multifactor authentication and, for most administration activities, use IAM users instead.

IAM Users:

• As we have stated above, once you are root users you can create multiple users to do various tasks.
• If you wanted to create a primary admin to replace your root user for day-to-day administration, you would select the AdministratorAccess policy, which would provide enough permissions to get most admin tasks done without exposing your account to unnecessary risk.
• Users are people within your organization, and can be grouped.

IAM Groups:

• IAM groups only contains IAM users.
• Certain IAM users are grouped together in the organization to form IAM Groups.
• Creating group is important as it is very tedious task to assign policy to each users. Once we created groups, we can assign related policies to group directly and it will be applicable to all users in that group.
• User can belong to one or multiple groups. Also, it is possible for users not be part of any group.

IAM Roles:

• Like users and groups, IAM roles define the limits for what can be done within your AWS account.
• Some AWS service will need to perform actions on your behalf. To do so, we will assign permissions to AWS services with IAM Roles.
• Common roles:

1. EC2 Instance Roles,
2. Lambda Function Roles,
3. Roles for CloudFormation, etc.

IAM Permissions:

• Users or Groups can be assigned JSON documents called policies
• These policies define the permissions of the users.
• • In AWS you apply the least privilege principle: don’t give more permissions than a user needs.

IAM Policies inheritance:

IAM Policies Structure:

Consists of →

• Version: policy language version, always include “2012 -10–17”
• Id: an identifier for the policy (optional)
• Statement: one or more individual statements (required)

Statements consists of →

• Sid: an identifier for the statement (optional)
• Effect: whether the statement allows or denies access (Allow, Deny)
• Principal: account/user/role to which this policy applied to
• Action: list of actions this policy allows or denies
• Resource: list of resources to which the actions applied to
• Condition: conditions for when this policy is in effect (optional)

IAM Password Policy:

The root user logs into the AWS Management Console with the associated email address along with the password set when the account was created. But as you’re no doubt already well aware, not all passwords are created equal. If you want your account to have any chance of surviving an attack, you should avoid passwords that can be easily guessed.

Instead, choose complex and long passwords that include uppercase and lowercase characters, numbers, and symbols. Even better, use only randomly generated strings that aren’t even distantly based on the stuff going on in your life (like your birthday). And never (ever) reuse passwords across multiple accounts.

In AWS, you can set up a password policy:

• Set a minimum password length
• Require specific character types: Including Uppercase letter, lowercase letter, numbers, non-alphanumeric characters etc.
• Allow all IAM users to change their own passwords.
• Require users to change their password after some time (We can set password expiration)
• We can also prevent password re-use. You won’t be able to set same past password as current password.

Multi-Factor Authentication — MFA:

• Users have access to your account and can possibly change configurations or delete resources in your AWS account.
• Multi-factor authentication (MFA) adds a second layer of security to your logins.
• Once it’s enabled, a user will need to provide not only a password to authenticate (“something you know”) but also, as a second factor, a temporary digital token sent through a preset device (“something you have”), like a smartphone running an Authenticator app.
• MFA works by associating a physical device — either a Universal 2nd Factor (U2F) or MFA-compliant device like YubiKey or a smartphone with the Authenticator app installed — with your account.
• When you try to log in, you’ll fi rst enter a password, and then a short-lived code (usually a six-digit number) will be sent to your MFA device.

• Main benefit of MFA: if a password is stolen or hacked, the account is not compromised.

IAM Security Tools:

• IAM Credentials Report (account-level): There is one more IAM resource that we must know and that is Credentials Report. It is accessed from IAM’s dashboard. It consists of ‘simple download button’ upon clicking on it you will download CSV file. It contains lists all your account’s users and the status of their various credentials.
• IAM Access Advisor (user-level): Access advisor shows the service permissions granted to a user and when those services were last accessed.

IAM Guidelines & Best Practices:

1. Don’t use the root account except for AWS account setup.
2. Assign users to groups and assign permissions to groups.
3. Create a strong password policy.
4. Use and enforce the use of Multi-Factor Authentication (MFA)
5. Create and use Roles for giving permissions to AWS services.
6. Use Access Keys for Programmatic Access (CLI / SDK).
7. Audit permissions of your account with the IAM Credentials Report.
8. Never share IAM users & Access Keys! (We will talk about this more in next section)

How can users access AWS ?

To access the AWS, user has 3 main options : →

1. AWS Management Console (protected by password + MFA)
2. AWS Command Line Interface (CLI): protected by access keys
3. AWS Software Developer Kit (SDK) — for code: protected by access keys.

Now let’s look at one by one to understand about them.

The AWS Management Console

The primary way you’ll interact with AWS services is through the AWS Management Console, a web-based application you can use from any compatible web browser. Until you get intimately familiar with the core AWS services, you’ll likely use the AWS Management Console almost exclusively.

The AWS Management Console gives you a user friendly, point-and-click way to navigate your AWS resources, and you’ll start by learning your way around it. We also have AWS Console Mobile Application, which allows you to view and manage some of your AWS resources from an Android or iOS mobile device.

Now let’s look at some important pointers about AWS Management Console →

• The AWS Management Console, also known as the AWS Console or the AWS Web Console, is a web interface you can use to manage all of your AWS cloud resources using a web browser, including compute, storage, and networking.
• The AWS Management Console is what you’ll choose if you want to do things using a point-and-click interface.
• Each AWS service has its own service console. Some services such as CloudWatch and AWS Billing and Cost Management offer visual reports that you can view and download.
• Most settings for your AWS resources can be configured using the AWS Management Console.
• The AWS Management Console is compatible with the these web browsers:

1. Apple Safari
2. Google Chrome
3. Microsoft Edge
4. Microsoft Internet Explorer
5. Mozilla Firefox

Accessing the AWS Management Console:

To access the AWS Management Console, browse to https://console.aws.amazon.com in your web browser.

There are two ways to sign in: as the root user or as an Identity and Access Management (IAM) user.

• To log in as the root user for your account, enter the email address and password associated with your AWS account.

• To log in as an IAM user, enter your account ID or account alias and choose Next.

• The AWS Management Console will take you to another page where you should enter your IAM username and password.

Note:-

• Your login is protected by password + MFA
• Once you’re logged in, your session will remain active for 12 hours. After that, it’ll expire and log you out to protect your account.

The AWS Console Mobile Application:

The AWS Console Mobile Application is a smartphone application that lets you manage your AWS account and resources on the go

The application features a dashboard showing key information about your AWS account and resources, including the following:

• Service Health — View any current health issues with AWS services across different regions.
• CloudWatch Alarms — View alarm graphs and current alarm status.
• Billing — View your current billing balance and a graph of usage charges.

The AWS Console Mobile Application supports Apple iOS 7.0+ and Android 4.0+. You can download it from the Amazon Appstore, Google Play, or iTunes.

When you first launch the application, it’ll prompt you to authenticate to an existing AWS account by adding an identity.

The AWS Command Line Interface (AWS CLI)

The AWS Command Line Interface (AWS CLI) is a unified command-line tool to manage your AWS resources. AWS gives access to the public application programming interfaces for all AWS services within 180 days of service launch.

A tool that enables you to interact with AWS services using commands in your command-line shell

Anything you can do in the AWS Management Console, you can do from your terminal using the AWS CLI. The AWS CLI is useful for performing repetitive tasks, such as launching EC2 instances, taking EBS snapshots, or attaching policies to IAM users.

You can enter AWS CLI commands manually for convenience, or you can incorporate them into a script for automation.

The AWS CLI is a versatile tool that can save you a lot of time versus the AWS Management Console.

Login with Access Keys:

AWS CLI requires you to log in using the access keys. The Access Keys are generated through the AWS Console. User is responsible for managing their access keys.

Access Keys are secret, just like a password. Don’t share them. Just like we need username to login in console, with CLI, we need Access Key ID and as password is your Secret Access Key

• Username → Access Key ID
• Password → Secret Access Key

Requirements:

• The AWS CLI is compatible with Windows, macOS, and Linux. To connect to AWS using the CLI, your network should allow outbound access to the internet on TCP port 443.
• You’ll also need an IAM access key ID and secret key to authenticate to AWS using the CLI.

Installation:

• Standalone installation available for Windows, Mac, and Linux.
• For Windows, use the MSI installer that includes setup packages for Windows 64-bit and 32-bit architectures.
• The stand-alone installer for Linux and macOS is a bundled installer that includes the AWS CLI, dependencies, and a script to install it

Note:- You need Python to be installed in your system.

Software Development Kits (SDK)

AWS software development kits simplify the use of AWS services in custom applications. Application developers can use an SDK to integrate their applications with AWS services easily and reliably.

SDKs save application developers from having to write low-level code to interact directly with the AWS service API endpoints. Instead, the developer just uses the SDK’s well-documented methods to learn how to incorporate it into their application.

AWS offers SDKs for the following popular programming languages:

• Java
• .NET
• Node.js
• PHP
• Python
• Ruby
• JavaScript
• Go
• C++

Mobile Software Development Kits:

AWS also offers mobile SDKs for the development of applications for smartphones and tablets. Available mobile SDKs include the following:

• AWS Mobile SDK for Android
• AWS Mobile SDK for iOS.

Internet of Things Device Software Development Kits:

You can use the AWS IoT SDKs to create applications that run on Internet of Things (IoT) devices, such as sensors, microcontrollers, smart appliances, smart lightbulbs, and AWS IoT buttons.

AWS IoT is a collection of services that allow IoT devices to interact with AWS services, other applications, and even other devices. The AWS IoT platform lets you centrally onboard, manage, and monitor IoT devices.

The AWS IoT SDKs allow developers to optimize memory, network, and power usage; reduce the size of applications; and ensure secure, fast communication with AWS.

IoT device SDKs are available for the following languages and platforms:

• Embedded C
• JavaScript
• Arduino Yún
• Python
• Java
• C++

The AWS EC2 (Elastic Compute Cloud)

The AWS EC2 (Elastic Compute Cloud) is one of the most popular of AWS’ offering from Amazon and it is comes under the “Infrastructure as a Service (IaaS)”

It consists of →

• Renting virtual machines (EC2)
• Storing data on virtual drives (EBS)
• Distributing load across machines (ELB)
• Scaling the services using an auto-scaling group (ASG)

To get your virtual machine (VM) instance running, you’ll first define the elements one at a time. Rather than installing an operating system and a software stack from scratch the traditional way, you need to choose Aamzon Machine Image (AMI).

Instead of choosing the right CPU, memory modules, and network adapters and adding them to your physical motherboard, you’ll choose the instance type matching your application needs.

And rather than purchasing storage drives and sliding them into your server chassis, you’ll define virtual storage volumes available through the Elastic Block Store (EBS)

Amazon Machine Images (AMI)

An image is a software bundle that was built from a template definition and made available within a single AWS Region.

The bundle can be copied to a freshly created storage volume that, once the image is extracted, will become a bootable drive that’ll turn the VM it’s attached to into a fully operational server. (Think like it is bootable image which contains software’s and OS)

The nice thing about AMIs is that they’re available in so many flavors. You can select AMIs that will give you clean, cloud-optimized operating systems like official versions of Red Hat Enterprise Linux (RHEL), Ubuntu, Windows Server, or Amazon’s own Amazon Linux.

It also contains OS you need preloaded with one of hundreds of popular software stacks like OpenVPN (secure remote connectivity), TensorFlow (neural networks), or a Juniper firewall already installed and ready to go.

AMIs are organized into four collections:

• Quick Start AMIs
• Custom AMI’s
• AWS Marketplace
• Community AMIs

Quick Start AMIs:

• AWS makes the most popular AMIs easily available through the QuickStart tab on the Choose An Amazon Machine Image page.
• They’re Free Tier–eligible.

• In Free Tier → We can use open-source operating systems like shown above.
• But other choices — like Windows Server and RHEL — will carry the normal licensing charges. Once you launch an instance running one of those images, the charges will be billed through your AWS account.

AWS Marketplace AMIs:

• AWS Marketplace is a software store managed by Amazon where vendors make their products available to AWS customers as ready-to-run AMIs
• Companies like SAP, Oracle, and NVIDIA will package their software solutions into AMIs, often running on one or another Linux distribution
• Marketplace AMIs will often offer a limited-time free trial for their software and reduced annual subscription rates.
• When you select a Marketplace AMI, you’ll be shown its total billing cost, broken down into separate software license and EC2 infrastructure amounts.

Community AMIs:

• Looking for a more specialized image built with a software stack that’s not available in the QuickStart or Marketplace tab? There are more than 100,000 AMIs to choose from in the Community AMIs tab.
• Some are supported by recognized vendors such as Canonical, but others are provided as is, often by end-users like you.
• Given the informal sources of some Community AMIs, you must take responsibility for the security and reliability of AMIs you launch.

Custom AMIs:

It’s possible to convert any existing EC2 instance into an AMI by creating a snapshot from the EBS volume used with an instance and then creating an image from the snapshot.

Understanding EC2 Instance Types

An EC2 instance type is simply a description of the kind of hardware resources your instance will be using.

When we try to configure EC2 instance, we may need to consider following questions →

• What would be the OS? …Linux, Window, macOS
• How much compute power & cores (vCPU)
• How much random-access memory (RAM)
• How much storage space?: Network-attached (EBS & EFS), hardware (EC2 Instance Store)
• Which would be the Network card?: Speed of the card, Public IP address
• What would be Firewall rules?: Security Groups
• What would be the Bootstrap script (configure at first launch): EC2 User Data

A vCPU is a metric used by AWS to describe the compute power you’ll get from a given instance. It’s meant to make you think in terms of the multiprocessor CPUs on consumer and server motherboards, where more is generally better.

While, in general, the more vCPUs and memory you get, the better your instance will perform, that’s definitely not the whole story. EC2 offers instance type families that are optimized for very different computing tasks. The T2, T3, and M5 types are included in the general-purpose family because of their ability to perform well for a wide range of uses.

The following are the various instance type that EC2 offers depending upon the usage or use case.

The bottom line is that choosing the right instance type family will sometimes let you get away with fewer vCPUs and less memory — and a lower cost.

If regulatory restrictions or company policy requires you to use only physically isolated instances, you can request either a dedicated instance or a dedicated host.

But wait, what’s the difference between two?

• A dedicated instance is a regular EC2 instance that, instead of sharing a physical host with other AWS customers, runs on an isolated host that’s set aside exclusively for your account.
• A dedicated host also gives you instance isolation but, in addition, allows a higher level of control over how your instances will be placed and run within the host environment.
• Both levels involve extra per-hour pricing and are available using on-demand, reserved, or spot models

AWS Instance Types Naming Convention:

AWS has the following naming convention,

Example:- m5.2xlarge

• m: Instance class
• 5: Generation (AWS improves them over time)
• 2xlarge: Size within the instance class.

Let’s look at the instance types one by one.

General Purpose:

• Great for a diversity of workloads such as web servers or code repositories
• It is a balance between: Compute, Memory & Networking.
• Most often times you will see t2.micro which is a General Purpose EC2 instance.

Compute Optimized:

• It is great for compute-intensive tasks that require high performance processors.
• Examples: Scientific modeling & machine learning, Dedicated gaming servers, Media transcoding, High performance web servers, High performance computing (HPC), Batch processing workloads etc.

Memory-Optimized:

• When you want fast performance for workloads that process large data sets in memory.
• Examples: High performance, relational/non-relational databases, In-memory databases optimized for BI (business intelligence), processing big unstructured data etc.

Storage Optimized:

• Great for storage-intensive tasks that require high, sequential read and write access to large data sets on local storage
• Examples: High frequency online transaction processing (OLTP) systems, Relational & NoSQL databases, Cache for in-memory databases (for example, Redis), Data warehousing applications, Distributed file systems etc.

Introduction to Security Groups

Security Groups are the fundamental of network security in AWS. They control how traffic is allowed into or out of our EC2 Instances.

The Security groups are acting as a “firewall” on EC2 instances. only contains allow rules.

They regulate:

• Access to Ports
• Authorised IP ranges — IPv4 and IPv6
• Control of inbound network (from other to the instance)
• Control of outbound network (from the instance to other)

Some good to know pointers about Security Groups:

• Security groups can be attached to multiple instances.
• They do live “outside” the EC2 → if traffic is blocked by security group, the EC2 instance won’t see it.
• It’s good to maintain one separate security group for SSH access.
• If your application is not accessible (time out), then it’s a security group issue.
• If your application gives a “connection refused“ error, then it’s an application error, not the security group blocking it.
• By default, security groups blocks all inbound (incoming) traffic.
• By default, All outbound (Outgoing) traffic is authorised by default.

How do we connect to EC2 instance?

• SSH is one of the most important function. It allows you to control a remote machine, all using the command line.
• Windows < 10, we can use PuTTY to connect EC2 instance.
• EC2 instance connect allows you to connect EC2 instance within your browser.

Virtual Server Storage (EC2 Instance Storage Section)

Like everything else in the cloud, the storage volumes holding your instance’s OS and data are going to be virtual.

In most cases, that’ll mean the 20 or 30 (or 2,000) GB drive holding your application is really just a 20 or 30 GB partition cleverly disguised to look like a stand-alone device. In fact, however, it was actually carved out of a much larger drive.

Some instance types support only volumes from the Elastic Block Store (EBS), others get their storage from Instance Store Volumes, and some can happily handle both.

Amazon Elastic Block Store (EBS):

• The physical drive where an EBS volume actually exists may live quite a distance from the physical server that’s giving your instance life.
• Rather than connecting directly to the motherboard via, say, a SATA cable the way a physical drive plugs into a physical computer, EBS volumes speak with your instance over a super low-latency network connection spanning the data center.
• EBS are fast but they are not faster than Instance Store Volumes.
• Unlike the instance store, the data stored on EBS volumes will survive shutdowns and system crashes.
• EBS volumes can be encrypted, which, when you’re working with sensitive data, can make a big difference.
• EBS volumes can be moved around, mounted on other instances, and, as you’ve seen, converted to AMIs.

Amazon EC2 Instance Store Volumes:

• It is faster as data on physical drives attached directly to the physical instance server.
• The downsides of instance store volumes: No encryption, lack of flexibility)

Note:- Think like, EBS is storage away from the physical instance server. Though they are fast as they are connected through super low latency network. The Instance Store Volume in other hand connected to physical instance server.

Understanding EC2 Pricing Models

What you’ll pay for your EC2 instances will depend on the way you consume the service. Here are some pricing models that we need to know.

On-Demand Instances:

• The most expensive pricing tier is on-demand instances.
• In this model, you pay for every hour the instance is running regardless of whether you’re actually using it.
• On-demand is great for workloads that need to run for a limited time without interruption.
• It is like Pay for what you use model.
• It has the highest cost but no upfront payment.
• Since it is pay for what you use model, therefore no long-term commitments.

Reserved Instances:

• If your application needs to run uninterrupted for more than a month at a time, then you’ll usually be better off purchasing a reserved instance.
• Practically, however, you’re unlikely to find an available reservation with a term shorter than 12 months. One- and three-year terms are always available.
• Reserved instances are paid for using an All Upfront, Partial Upfront, or No Upfront payment option.
• When you select reserved instances, there is up to 72% discount compared to on-demand instances.
• As reservation period increases, discount also increases.
• Discount is also depends upon the purchasing options. All upfront gets more discount than partial one.
• Recommended for steady-state usage applications (think database)

EC2 Spot Instances:

• It is most cost-effective AWS instances
• Can get a discount of up to 90% compared to On-demand
• For workloads that don’t need to run constantly and can survive unexpected shutdowns, your cheapest option will probably involve requesting instances on the EC2 spot market.
• It is useful for workloads that are resilient to failure. Such as: Data Analysis, Image processing, and distributed workloads or workload with a flexible start and end time.
• Not suitable for critical jobs or databases.

EC2 Dedicated Hosts:

• An Amazon EC2 Dedicated Host is a physical server with EC2 instance capacity fully dedicated to your use.
• Dedicated Hosts can help you address compliance requirements and reduce costs by allowing you to use your existing server-bound software licenses.
• Allocated for your account for a 3-year period reservation
• It is quite expensive.

EC2 Dedicated Instances:

A dedicated instance is a regular EC2 instance that, instead of sharing a physical host with other AWS customers, runs on an isolated host that’s set aside exclusively for your account.

Which purchasing option is right for me?

• On-demand: Coming and staying in resort whenever we like, we pay the full price
• Reserved: like planning ahead and if we plan to stay for a long time, we may get a good discount.
• Spot instances: the hotel allows people to bid for the empty rooms and the highest bidder keeps the rooms. You can get kicked out at any time
• Dedicated Hosts: We book an entire resort for ourself.

EC2 Instance Storage

Like everything else in the cloud, the storage volumes holding your instance’s OS and data are going to be virtual.

In most cases, that’ll mean the 20 or 30 (or 2,000) GB drive holding your application is really just a 20 or 30 GB partition cleverly disguised to look like a stand-alone device. In fact, however, it was actually carved out of a much larger drive.

Some instance types support only volumes from the Elastic Block Store (EBS), others get their storage from Instance Store Volumes, and some can happily handle both.

Elastic Block Store (EBS)

• An EBS (Elastic Block Store) Volume is a network drive you can attach to your instances while they run.
• Rather than connecting directly to the motherboard via, say, a SATA cable the way a physical drive plugs into a physical computer, EBS volumes speak with your instance over a super low-latency network connection spanning the data center.
• It allows your instances to persist data, even after their termination.
• EBS are fast but they are not faster than Instance Store Volumes.
• Unlike the instance store, the data stored on EBS volumes will survive shutdowns and system crashes.
• EBS volumes can be encrypted, which, when you’re working with sensitive data, can make a big difference.
• EBS volumes can be moved around, mounted on other instances, and, as you’ve seen, converted to AMIs.
• They can only be mounted to one instance at a time (at the CCP level)
• They are bound to a specific availability zone (AZs). Meaning an EBS Volume in us-east-1a cannot be attached to us-east-1b.

Analogy: Think of them as a “network USB stick”

• Note:- Free tier: 30 GB of free EBS storage of type General Purpose (SSD)
• It can be detached from an EC2 instance and attached to another one quickly (Using Snapshot, we will learn this later)
• We can increase the capacity of the drive over time and we will get billed according to provisional capacity.

• We can controls the EBS behaviour when an EC2 instance terminates. When EC2 gets terminated, by default, the root EBS volume is deleted and not any other attached EBS volume.
• We can preserve the root volume when instance is terminated.

EBS Snapshots:

• We can make a backup of EBS volume at any point of time with the help of EBS Snapshot.
• It is not necessary to detach volume to do snapshot, but recommended.
• With the help of this, we can copy snapshots across AZ or Region.

• We can create AMI through EBS snapshot. The AMIs created by EBS snapshots are faster as all the configuration, settings, software were already installed and snapshotting is done from existing EC2 instances.

EC2 Image Builder:

• It is used to automate the creation of Virtual Machines or container images.
• Automate the creation, maintain, validate and test EC2 AMIs
• It’s Free service (only pay for the underlying resources)

EC2 Instance Store

• EBS volumes are network drives with good but “limited” performance as it’s connected through low-latency network. Though, even it is low latency, it impacts somewhat on the performance.
• If you need a high-performance hardware disk, use EC2 Instance Store.
• It provides better I/O performances.
• It is ephemeral. As EC2 Instance Store lose their storage if they’re stopped.
• It is good for buffer / cache / scratch data / temporary content.
• Risk of data loss if hardware fails.
• Backups and Replication are your responsibility.

Elastic File System (EFS)

AWS Elastic File System (EFS) is one of three main storage services offered by Amazon. It is a scalable, cloud-based file system for Linux-based applications and workloads that can be used in combination with AWS cloud services and on-premise resources.

EFS offers a choice between two storage classes, Infrequent Access and Standard access, depending on your needs.

• Standard access storage is designed for frequently accessed files.
• Infrequent Access is intended for storing long-lived but less used files at a lower cost (EFS-IA)

It is highly scalable and expensive service and often used as pay-per-use model.

EFS uses the NFSv4 protocol for its file system structure, which mirrors a standard on-premise structure and simplifies transferring and accessing your files.

It can be used in combination with Elastic Cloud Compute (EC2) instances or as a stand-alone file system.

EFS does not require storage provisioning and is pay-for-use allowing you to scale services as needed.

EBS vs EFS:

EFS Infrequent Access (EFS-IA):

• It is a storage class that is cost-optimized for files not accessed every day.
• Up to 92% lower cost compared to EFS Standard.
• EFS will automatically move your files to EFS-IA based on the last time they were accessed.
• We can Enable EFS-IA with a Lifecycle Policy. Example: move files that are not accessed for 60 days to EFS-IA

Amazon FSx

Amazon FSx is a fully managed third-party file system solution. It uses SSD storage to provide fast performance with low latency.

There are two available FSx solutions available in AWS:

Amazon FSx for Windows File Server:

• A fully managed, highly reliable, and scalable Windows native shared file system.
• It is built on Windows File Server
• It supports SMB protocol & Windows NTFS.

Amazon FSx for Lustre:

• A fully managed, high-performance, scalable file storage for High Performance Computing (HPC)
• The name Lustre is derived from “Linux” and “cluster”.
• Use cases: Machine Learning, Analytics, Video Processing, Financial Modeling

Elastic Load Balancing & Auto Scaling Groups

Scalability & High Availability

• Scalability means that an application / system can handle greater loads by adapting.
• There are two kinds of scalability: Vertical Scalability, Horizontal Scalability (elasticity)

Let’s deep dive into the distinction, using a call center as an example.

Vertical Scalability:

• Vertical Scalability means increasing the size of the instance
• For example, your application runs on a t2.micro. The scaling that application vertically means running it on a t2.large.

• Vertical scalability is very common for non distributed systems, such as a database.
• There’s usually a limit to how much you can vertically scale (hardware limit)

Horizontal Scalability:

• Horizontal Scalability means increasing the number of instances / systems for your application.
• Horizontal scaling implies distributed systems.

• This is very common for web applications / modern applications.
• It’s easy to horizontally scale thanks the cloud offerings such as Amazon EC2.

High Availability

• High Availability usually goes hand in hand with horizontal scaling
• High availability means running your application / system in at least 2 Availability Zones (AZs)

• The goal of high availability is to survive a data center loss in case of disaster or any emergency.

High Availability & Scalability For EC2:

Vertical Scaling: Increase instance size (= scale up / down)

• From: t2.nano — 0.5G of RAM, 1 vCPU
• To: u-12tb1.metal — 12.3 TB of RAM, 448 vCPUs

Horizontal Scaling: Increase number of instances (= scale out / in)

• Auto Scaling Group
• Load Balancer

High Availability: Run instances for the same application across multi AZs

• Auto Scaling Group multi-AZ
• Load Balancer multi-AZ

Scalability vs Elasticity vs Agility

• Scalability: Ability to accommodate a larger load by making the hardware stronger (scale-up), or by adding nodes (scale-out)
• Elasticity: Once a system is scalable, elasticity means that there will be some “auto-scaling” so that the system can scale based on the load. This is “cloud-friendly”: pay-per-use, match demand, optimize costs
• Agility: (not related to scalability — distractor) new IT resources are only a click away, which means that you reduce the time to make those resources available to your developers from weeks to just minutes. (This is the power of cloud)

What is Load Balancing?

Load balancers are servers that forward internet traffic to multiple servers (EC2 Instances) downstream.

Why use a load balancer?

• It spreads a load across multiple downstream instances.
• Expose a single point of access (DNS) to your application
• Seamlessly handle failures of downstream instances
• Do regular health checks to your instances.
• Provide SSL termination (HTTPS) for your websites.
• High availability across zones.

Why use an Elastic Load Balancer?

An ELB (Elastic Load Balancer) is a managed load balancer. It is managed by AWS hence →

• AWS guarantees that it will be working
• AWS takes care of upgrades, maintenance, high availability
• AWS provides only a few configuration knobs

It costs less to set up your own load balancer but it will be a lot more effort on your end (maintenance, integrations)

There are 3 kinds of load balancers offered by AWS:

• Classic Load Balancer: It is an old elastic load balancer which would be retiring soon. It is used in layer 4 and 7.
• Application Load Balancer: It is meant for HTTP and HTTPS only. It is used in layer 7.
• Network Load Balancer: It has ultra high performance. It allows for TCP. It is used in lower layer (Layer 4)

What’s an Auto Scaling Group (ASG)?

In real-life, the load on your websites and application can change time to time. It is important to handle increased load by your website. That’s the reason, we have something called ASG.

The goal of an Auto Scaling Group (ASG) is to:

• Scale-out (add EC2 instances) to match an increased load.
• Scale in (remove EC2 instances) to match a decreased load.
• Ensure we have a minimum and a maximum number of machines running.
• Automatically register new instances to a load balancer.
• Replace unhealthy instances.

Note:- It does cost savings. As it only run at an optimal capacity.

Let’s understand how the Auto Scaling group works with load balancer.

Auto Scaling Groups Scaling Strategies:

There are 3 ASG strategies that followed in the AWS.

Manual Scaling:

• In this case we can update the scaling manually in the dashboard.

Dynamic Scaling:

• Simple / Step Scaling: When a CloudWatch alarm is triggered (for example CPU > 70%), then add 2 units. When a CloudWatch alarm is triggered (for example CPU < 30%) remove 1 unit.
• Target Tracking Scaling: We can set the average ASG CPU to stay at around 40%.
• Scheduled Scaling: Anticipate a scaling based on known usage patterns. Example: increase the min. capacity to 10 at 5 pm on Fridays. (Big Billion Day)

Predictive Scaling:

• Uses Machine Learning to predict future traffic ahead of time.
• Automatically provisions the right number of EC2 instances in advance.
• Useful when your load has predictable time—based patterns.

Amazon S3 (The Core Storage Service)

For many organizations, their most valuable asset is their data. But organizations often have their data split between the cloud and on-premises locations such as offices and data centers. (Hybrid)

It’s therefore critical to understand the different options AWS provides for storing data in the cloud, transferring data between the cloud and on-premises, and doing both in a cost-effective way without sacrificing performance.

Amazon S3 is one of the main building blocks of AWS. It’s advertised as “infinitely scaling” storage for a reason. Nowadays many websites use Amazon S3 as a backbone and many AWS services use Amazon S3 as an integration as well.

Note:- The CCP exam requires “deeper” knowledge about S3

Let’s see some pointers on Amazon S3 (Simple Storage Service)

• Amazon’s Simple Storage Service(S3) lets you store and retrieve unlimited amounts of data from anywhere in the world at any time. You can use S3 to store any kind of file.
• Although AWS calls it “storage for the internet” you can implement access controls and encryption to restrict access to your files to specific individuals and IP addresses.
• S3 opens up a variety of uses, both inside and outside of AWS.

Let’s start with the S3 use cases

1. Backup and storage
2. Disaster Recovery
3. Archive
4. Hybrid Cloud storage (Storage on cloud and on-premises)
5. Application and Media hosting
6. Data lakes & big data analytics
7. Static Websites

Amazon S3 Overview — Objects & Buckets

• Amazon S3 allows people to store objects (files) in “buckets” (directories)
• You can store any kind of file, including text, images, videos, database files, and so on.
• Each object can be up to 5 TB in size.
• S3 stores objects in a container called a bucket that’s essentially a flat-file system. A bucket can store an unlimited number of objects.
• When you create a bucket, you must assign it a name between 3 and 63 characters long, and the name must be globally unique across AWS.
• Remember, Buckets are defined at the region level.
• S3 looks like a global service but buckets are created in a region.

Naming convention:

• No uppercase
• No underscore
• 3–63 characters long
• Not an IP
• Must start with lowercase letter or number

Every object (files) has a key. The key is nothing but the full path as shown below →

• s3://my-bucket/my_file.txt
• s3://my-bucket/my_folder1/another_folder/my_file.txt

If you look at the key which is composed of prefix + object name. Just keys with very long names that contain slashes (“/”)

Example:

Let’s look at the URL:

https://benpiper.s3.amazonaws.com/text.txt

Here the object text.txt is stored in a bucket named benpiper.

As you might expect, S3 comes with a cost. You’re not charged for uploading data to S3, but you may be charged when you download data. For example, in the US East region, you can download up to 1 GB of data per month at no charge. Beyond that, the rate is $0.09 or less per gigabyte. S3 also charges you a monthly fee based on how much data you store and the storage class you use.

Access Permissions (S3 Security)

S3 is storage for the internet, but that doesn’t mean everyone on the internet can read your data. By default, objects you put in S3 are inaccessible to anyone outside of your AWS account.

S3 offers the following three methods of controlling who may read, write, or delete objects stored in your S3 buckets:

1. Bucket policies
2. User policies
3. Bucket and object access control lists

Bucket Policies:

• A bucket policy is a resource-based policy that you apply to a bucket.
• You can use bucket policies to grant access to all objects within a bucket or just specific objects.
• You can also control which principals and accounts can read, write, or delete objects.
• You can also grant anonymous read access to make an object, such as a webpage or image, available to everyone on the internet.

User Policies:

You can use IAM policies to grant IAM principals access to S3 objects. Unlike bucket policies that you apply to a bucket, you can apply user policies only to an IAM principal.

Bucket and Object Access Control Lists(ACL):

Bucket and object access control lists (ACLs) are legacy access control methods that have mostly been superseded by bucket and user policies.

Nevertheless, you can still use bucket and object ACLs to grant other AWS accounts and anonymous users access to your S3 resources.

S3 Bucket Policies:

• It is JSON based policies which will instruct whom to give access and whom not to.
• It is used to Grant public access to the bucket.

Bucket settings for Block Public Access:

• These settings were created to prevent company data leaks.
• If you know your bucket should never be public, leave these on.
• Can be set at the account level.

Encryption

S3 doesn’t change the contents of an object when you upload it. That means if you upload a document containing personal information, that document is stored unencrypted.

Using appropriate access permissions can protect your data from unauthorized access, but to add an additional layer of security, you have the option of encrypting objects before storing them in S3.

This is called encryption at rest . S3 gives you the following two options for encrypting objects at rest:

Server-side encryption:

• When you create an object, S3 encrypts the object and saves only the encrypted content.
• When you retrieve the object, S3 decrypts it and delivers the unencrypted object.
• Server-side encryption is the easiest to implement and doesn’t require you to keep track of encryption keys. Amazon manages the keys and therefore has access to your objects.

Client-side encryption:

• You encrypt the data prior to uploading it to S3.
• You must decrypt the object when you retrieve it from S3.
• This option is more complicated, as you’re responsible for encryption and decryption.
• If you lose the key used to encrypt an object, you won’t be able to decrypt it.
• Organizations with strict security requirements may choose this option to ensure Amazon doesn’t have the ability to read their encrypted objects.

S3 Versioning

To further protect against accidentally deleting or overwriting the contents of your important files, you can use versioning. To understand how versioning works, consider this example:

Without versioning, if you upload an object with the same name as an existing object in the same bucket, the contents of the original object will get overwritten. But if you enable versioning on the bucket and then upload an object with the same name as an existing object, S3 will simply create a new version of that object. The original version will remain intact and available.

If you delete an object in a bucket on which versioning is disabled, the contents of the object aren’t deleted. Instead, S3 adds a delete marker to the object and hides it from the S3 service console view.

Versioning is disabled by default when you create a bucket. You must explicitly enable versioning on a bucket to use it, and it applies to all objects in the bucket.

There’s no limit to the number of versions of an object you can store. You can delete versions manually or automatically using object life cycle configurations.

S3 Access Logs

• For audit purposes, you may want to log all access to S3 buckets, in that case, Access Logs were used.
• Any request made to S3, from any account, authorized or denied, will be logged into another S3 bucket. And that data can be analyzed using data analysis tools

• Very helpful to come down to the root cause of an issue, or audit usage, view suspicious patterns, etc

S3 Replication (CRR & SRR)

• In order to perform replication, user must enable versioning in source and destination.
• There are basically two types of replications: Cross-Region Replication (CRR) & Same Region Replication (SRR)
• Copy process is asynchronous.

• CRR — Use cases: compliance, lower latency access, replication across accounts
• SRR — Use cases: log aggregation, live replication between production and test accounts

S3 Storage Classes

All data is not created equal. Depending on its importance, certain fi les may require more or less availability or protection against loss than others.

Some data you store in S3 is irreplaceable, and losing it would be catastrophic. Examples of this might include digital photos, encryption keys, and medical records.

Durability and Availability

Objects that need to remain intact and free from inadvertent deletion or corruption are said to need high durability, which is the percent likelihood that an object will not be lost over the course of a year.

The greater the durability of the storage medium, the less likely you are to lose an object. To understand how durability works, consider the following two examples.

• First, suppose you store 100,000,000,000 objects on storage with 99.999999999 percent durability. That means you could expect to lose only one (0.000000001 percent) of those objects over the course of a year!
• Now consider a different example. Suppose you store the same number of objects on storage with only 99.99 percent durability. You’d stand to lose 10,000,000 objects per year!

Different data also has differing availability requirements. Availability is the percent of time an object will be available for retrieval.

For instance, a patient’s medical records may need to be available around the clock, 365 days a year. Such records would need a high degree of both durability and availability.

Note that, the level of durability and availability of an object depends on its storage class. S3 offers six different storage classes at different price points.

S3 charges you a monthly storage cost based on the amount of data you store and the storage class you use.

Your choice of storage class then hinges upon your desired level of availability, which depends on how frequently you’ll need to access your files. You may find it helpful to categorize your files in terms of the following three access patterns:

• Frequently accessed objects
• Infrequently accessed objects
• A mixture of frequently and infrequently accessed objects

Storage Classes for Frequently Accessed Objects

If you need to access objects frequently and with minimal latency, the following two storage classes fit the bill:

S3 Standard — General Purposes

• This is the default storage class.
• It offers the highest levels of durability and availability, and your objects are always replicated across at least three Availability Zones in a region.
• Used for frequently accessed data.
• Low latency and high throughput.
• Use Cases: Big Data analytics, mobile & gaming applications, content distribution.

Reduced Redundancy

• The REDUCED_REDUNDANCY (RRS) storage class is meant for data that can be easily replaced, if it needs to be replaced at all.
• It has the lowest durability of all the classes, but it has the same availability as STANDARD.
• AWS recommends against using this storage class but keeps it available for people who have processes that still depend on it.

Storage Classes for Infrequently Accessed Objects

Two of the storage classes designed for infrequently accessed objects are suffixed with the “IA” initialism for “infrequent access.”

These IA classes offer millisecond-latency access and high durability but the lowest availability of all the classes.

They’re designed for objects that are at least 128 KB in size. You can store smaller objects, but each will be billed as if it were 128 KB

S3 Standard — Infrequent Access (IA):

• Suitable for data that is less frequently accessed, but requires rapid access when needed.
• 99.9% Availability.
• Lower cost compared to Amazon S3 Standard, but retrieval fees.
• Use Cases: As a data store for disaster recovery, backups.

S3 One Zone — Infrequent Access (IA):

• Same as IA but data is stored in a single AZ
• 99.5% Availability
• Low latency and high throughput performance.
• Lower cost compared to S3-IA (by 20%)
• Use Cases: Storing secondary backup copies of on-premises data, or storing data you can recreate.

Amazon Glacier & Glacier Deep Archive:

• The GLACIER class is designed for long-term archiving of objects that rarely need to be retrieved.
• Objects in this storage class are stored using the S3 Glacier service.
• Unlike the other storage classes, you can’t retrieve an object in real-time. Instead, you must initiate a restore request for the object and wait until the restore is complete.
• The time it takes to complete a restore depends on the retrieval option you choose and can range from 1 minute to 12 hours.
• Actually it has a low cost object storage (in GB/month) meant for archiving / backup
• Data is retained for the longer term (years).
• Amazon Glacier Deep Archive is the most cost-effective option if you want to archive data and do not have a retrieval time requirement. You can retrieve data in 12 or 48 hours.

Amazon Glacier — cheap:

• Expedited (1 to 5 minutes)
• Standard (3 to 5 hours)
• Bulk (5 to 12 hours)

Amazon Glacier Deep Archive — cheapest:

• Standard (12 hours)
• Bulk (48 hours)

Storage Class for Both Frequently and Infrequently Accessed Objects

S3 currently offers only one storage class designed for both frequently and infrequently accessed objects: INTELLIGENT_TIERING

• This storage class automatically moves objects to the most cost-effective storage tier based on past access patterns.
• An object that hasn’t been accessed for 30 consecutive days is moved to the lower-cost infrequent access tier.
• Once the object is accessed, it’s moved back to the frequent access tier.

Moving between storage classes

• You can transition objects between storage classes
• For infrequently accessed object, move them to STANDARD_IA
• For archive objects you don’t need in real-time, GLACIER or DEEP_ARCHIVE
• Moving objects can be automated using a lifecycle configuration

S3 Object Lock & Glacier Vault Lock

S3 Object Lock:

• Adopt a WORM (Write Once Read Many) model.
• Block an object version deletion for a specified amount of time.

Glacier Vault Lock:

• Adopt a WORM (Write Once Read Many) model.
• Lock the policy for future edits (can no longer be changed)
• Helpful for compliance and data retention

Object Life Cycle Configurations

Because S3 can store practically unlimited amounts of data, it’s possible to run up quite a bill if you continually upload objects but never delete any. This could happen if you have an application that frequently uploads log fi les to a bucket.

You may also spend more than necessary by keeping objects in a more expensive storage class when a cheaper one would meet your needs.

Object life cycle confi gurations can help you control costs by automatically moving objects to different storage classes or deleting them after a time.

Object life cycle configuration rules are applied to a bucket and consist of one or both of the following types of actions:

Transition actions:

• Transition actions move objects to a different storage class once they’ve reached a certain age.
• For example, you can create a rule to move objects from the STANDARD storage class to the STANDARD_IA storage class 90 days after creation.

Expiration actions:

• These can automatically delete objects after they reach a certain age.
• For example, you can create a rule to delete an object older than 365 days.
• If you have versioning enabled on a bucket, you can create expiration actions to delete object versions of a certain age.
• This allows you to take advantage of versioning without having to store endless versions of an object.

Note:-

It’s common to use both types of actions together.

For example, if you’re storing web server log files in a bucket, you may want to initially keep each log file in STANDARD storage. Once the file reaches 90 days, you can have S3 transition it to STANDARD_IA storage. Then, after 365 days in STANDARD_IA storage, the file is deleted

AWS Snow Family

Let’s talk about buzzword, the AWS Snow Family.

It is highly-secure, portable devices to collect and process data at the edge, and migrate data into and out of AWS

Right now if you start migration of your data over the network, depending upon the size of the data you may face following challenges.

• Limited connectivity
• Limited bandwidth
• High network cost
• Connection stability

Basically…

AWS Snow Family: It is an offline devices to perform data migrations If it takes more than a week to transfer over the network, use Snowball devices!

Direct Upload Vs Snow Family Upload

Now let’s talk more about Snow Family Hardware devices.

Snowball Edge (for data transfers):

• It is a Physical data transport solution.
• It can move TBs or PBs of data in or out of AWS.
• Alternative to moving data over the network (and paying network fees)
• It is Pay per data transfer job.

• Snowball Edge Storage Optimized: 80 TB of HDD capacity for block volume and S3 compatible object storage
• Snowball Edge Compute Optimized: 42 TB of HDD capacity for block volume and S3 compatible object storage
• Use cases: large data cloud migrations, DC decommission, disaster recovery.

AWS Snowcone:

• It is a small, portable computing, anywhere, rugged & secure, withstands harsh environments.
• Light (4.5 pounds, 2.1 kg)
• It is used for edge computing, storage, and data transfer.
• 8 TBs of usable storage

• Use Snowcone where Snowball does not fit (space -constrained environment)
• Must provide your own battery / cables.
• Can be sent back to AWS offline, or connect it to internet and use AWS DataSync to send data

AWS Snowmobile:

• Transfer exabytes of data (1 EB = 1,000 PB = 1,000,000 TBs)
• Each Snowmobile has 100 PB of capacity (use multiple in parallel)
• High security: temperature controlled, GPS, 24/7 video surveillance
• Better than Snowball if you transfer more than 10 PB

AWS Snow Family for Data Migrations:

How to use AWS Snow Family?

• Request Snowball devices from the AWS console for delivery.
• Install the snowball client / AWS OpsHub on your desktop or servers.
• Connect the snowball to your servers and copy files using the client.
• Ship back the device when you’re done (goes to the right AWS facility).
• Data will be loaded into an S3 bucket.
• After that, Snowball is completely wiped.

What is Edge Computing?

• Process data while it’s being created on an edge location. But what is an edge location?

• Edge location is a location where data gets generated but can’t be collected with the help of regular data collection methodology.
• Example: A truck on the road, a ship on the sea, a mining station underground
• These locations may have:- Limited / no internet access or Limited / no easy access to computing power.
• We set up a Snowball Edge / Snowcone device to do edge computing.
• Use cases of Edge Computing: Preprocess data, Machine learning at the edge, Transcoding media streams etc.
• Eventually (if need be) we can ship back the device to AWS (for transferring data for example)

Required Hardware for Edge Computing?

Snowcone (smaller):

• 2 CPUs, 4 GB of memory, wired or wireless access
• USB-C power using a cord or the optional battery

Snowball Edge — Compute Optimized

• 52 vCPUs, 208 GiB of RAM
• Optional GPU (useful for video processing or machine learning)
• 42 TB usable storage

Snowball Edge — Storage Optimized

• Up to 40 vCPUs, 80 GiB of RAM
• Object storage clustering available

All above hardware can run EC2 Instances & AWS Lambda functions (using AWS IoT Greengrass)

AWS OpsHub

• Historically, to use Snow Family devices, you needed a CLI (Command Line Interface tool)
• Today, you can use AWS OpsHub (a software you install on your computer / laptop) to manage your Snow Family Device

Hybrid Cloud for Storage

AWS is pushing for ”hybrid cloud”. But what exactly is the Hybrid Cloud? Take a look at below.

• Part of your infrastructure is on-premises
• Part of your infrastructure is on the cloud

This can be due to variety of reasons. Few included below:-

• Long cloud migrations
• Security requirements
• Compliance requirements
• IT strategy

But…

S3 is a proprietary storage technology (unlike EFS / NFS), so how do you expose the S3 data on-premise?

The AWS storage getaway is the solution.

AWS Storage Gateway

AWS Storage Gateway makes it easy to connect your existing on-premises servers to storage in the AWS cloud. Because it uses industry-standard storage protocols, there’s no need to install special software on your existing servers.

It bridge between on-premises data and cloud data in S3.

It is a Hybrid storage service to allow on- premises to seamlessly use the AWS Cloud

Use cases: disaster recovery, backup & restore, tiered storage etc.

There are 3 types of storage gateways.

• File Gateway
• Volume Gateway
• Tape Gateway

AWS Storage Cloud-Native Options

The Core Database Services

Many applications use databases to store, retrieve, and organize data, so the type of database you choose and how you configure it have a big impact on the performance and availability of such a database-backed application.

In traditional infrastructure, organizations typically installed and configured their own database servers. For example, a business might use a Microsoft SQL Server or Oracle database to store customer information.

It’s possible to build and run your own database servers in the cloud, but AWS offers another — and for many, better — option: managed database services.

With a managed database service, you use the AWS Management Console or AWS command-line interface (CLI) to provision a database. AWS handles the installation and maintenance of the database software, as well as database backups and replication.

Database Models

A database can use a relational or nonrelational model. The model you choose depends on how your application needs to store, organize, and retrieve data. It’s important to understand that the needs of the application determine the database model you choose, not the other way around.

• Relational Database
• Nonrelational Database

Let’s find more information about the relational and non-relational databases.

Relational Databases:

• Relational databases have been around a long time. It looks just like Excel spreadsheets, with links between them.

A relational database is a type of database that stores and provides access to data points that are related to one another.

• A relational database is analogous to a spreadsheet that contains columns and rows.
• In a relational database, columns are called attributes , and rows are called records.
• Both are stored in a table, and a database can contain multiple tables.

• An advantage of storing structured data in a relational database is that you can quickly search an entire database for specific values.
• Relational databases are ideal for performing complex analytics and generating reports against large datasets.
• We use the SQL language to perform queries / lookups.

Nonrelational (No-SQL) Databases:

• NoSQL = non-SQL = non relational databases
• Relational databases are wildly popular, but because of their restrictions, they’re also overly complex for applications that don’t need to store structured data.
• Also, relational databases often perform poorly when put under the strain of handling thousands of reads or writes per second.
• Nonrelational databases were developed to provide a fast alternative for applications that need to perform tens of thousands of reads or writes per second.

In addition to being able to handle these high transaction rates, nonrelational databases let you store data that doesn’t have a well-defined, predictable structure. Such data is often called unstructured data, in contrast to the structure imposed on data by a relational database.

• Because of their unstructured nature, nonrelational or no-SQL databases are said to be schemaless.
• Nonrelational databases also store information in tables; tables are sometimes called collections, and each row or record is called an item.
• Examples: Key-value, document, graph, in-memory, search databases

NoSQL data example: JSON

• JSON = JavaScript Object Notation
• JSON is a common form of data that fits into a NoSQL model
• Data can be nested, fields can be changed over time

Amazon Relational Database Service (RDS)

The Amazon Relational Database Service (RDS) is Amazon’s managed relational database service.

RDS lets you provision a number of popular relational database management systems (RDBMSs) including

• Microsoft SQL Server,
• Oracle,
• MySQL,
• PostgreSQL,
• Aurora (AWS Proprietary database).

You can always install and configure your own database server on an EC2 instance. But RDS offers several advantages over this →

• Automated provisioning, OS patching.
• Continuous backups and restore to specific timestamp (Point in Time Restore)!
• Monitoring dashboards
• Read replicas for improved read performance
• Multi AZ setup for DR (Disaster Recovery)
• Scaling capability (vertical and horizontal)

Note:- You can’t SSH into your instances.

RDS Solution Architecture:

Amazon Aurora:

• Aurora is a proprietary technology from AWS (not open sourced)
• PostgreSQL and MySQL are both supported as Aurora DB

Aurora is “AWS cloud optimized” and claims 5x performance improvement over MySQL on RDS, over 3x the performance of Postgres on RDS

• Aurora storage automatically grows in increments of 10GB, up to 64 TB.
• Aurora costs more than RDS (20% more) — but is more efficient.
• It’s not available in the free tier.

RDS Deployments:

1. Read Replicas
2. Multi-AZ
3. Multi-Region

Let’s learn about these two deployments.

Read Replicas:

• Scale the read workload of your DB
• Can create up to 5 Read Replicas

• When you have multiple applications then it is useful to creates Read Replica so that your applications read operation will be faster and much more efficient.
• In case of writing, data is only written to the main DB.

Multi-AZ:

• The concept of Multi-AZ is used when the event of Failover in case of AZ outage.
• Data is only read/written to the main database.
• We can only use one AZ as failover.

• The failover DB is only gets activated in case of failure of main DB.

Multi-Region:

• Disaster recovery in case of region issue
• Local performance for global reads

Amazon ElastiCache

• Amazon ElastiCache is a cache-as-a-service developed and distributed by Amazon Web Services (AWS).
• These caches are in-memory databases with high performance, low latency.
• Amazon ElastiCache relies on two different caching engines — Memcached and Redis.
• It helps reduce load off databases for read-intensive workloads.

DynamoDB

• DynamoDB is Amazon’s managed non-relational database service. It’s designed for highly transactional applications that need to read from or write to a database tens of thousands of times a second.
• It is a NoSQL database — not a relational database.
• It is fully Managed Highly available with replication across 3 AZ.
• It is distributed “serverless” database.
• Fast and consistent in performance.
• Single-digit millisecond latency — low latency retrieval
• It can be Integrated with IAM for security, authorization and administration
• Low cost and auto-scaling capabilities
• DynamoDB is a key/value database.

DAX(DynamoDB Accelerator):

• It is a fully managed in-memory cache for DynamoDB.
• 10x performance improvement: single-digit millisecond latency to microseconds latency — when accessing your DynamoDB tables.
• Secure, highly scalable & highly available.

ElastiCache Vs DAX:

DAX is only used for and is integrated with DynamoDB, while ElastiCache can be used for other databases.

Global Tables:

• Global table is nothing but to make a DynamoDB table accessible with low latency in multiple-regions.
• Active-Active replication (read/write to any AWS Region)

Redshift

• Redshift is a database that is based on PostgreSQL, but it’s not used for OLTP.
• It’s OLAP — Online Analytical Processing (analytics and data warehousing)
• Load data once every hour, not every second
• 10x better performance than other data warehouses, scale to PBs of data.
• Columnar storage of data (instead of row-based)
• Massively Parallel Query Execution (MPP), highly available
• Pay as you go based on the instances provisioned
• It has a SQL interface for performing the queries
• We can integrate BI tools such as s AWS Quicksight or Tableau with it.

So anytime in the exam you are seeing the database needs to be a warehouse and to do analytics on it, then Redshift is going to be your answer.

Amazon EMR

• EMR stands for “Elastic MapReduce”
• EMR helps creating Hadoop clusters (Big Data) to analyze and process vast amount of data.
• The clusters can be made of hundreds of EC2 instances
• It also supports Apache Spark, HBase, Presto, Flink, etc.
• EMR takes care of all the provisioning and configuration
• Use cases: data processing, machine learning, web indexing, big data

So from an exam perspective, any time you see Hadoop cluster, think no more, it’s going to be Amazon EMR.

Amazon Athena

• It is a serverless query service to analyze data stored in Amazon S3.
• It uses standard SQL language to query the files.
• Supports CSV, JSON, ORC, Avro, and Parquet (built on Presto)

• Pricing: $5.00 per TB of data scanned
• Use compressed or columnar data for cost-savings (less scan)
• Use cases: Business intelligence / analytics / reporting, analyze & query VPC Flow Logs, ELB Logs, CloudTrail trails, etc’

So from an exam perspective, whenever you see serverless analyze data in S3 use SQL, then think Amazon Athena.

Amazon QuickSight

• It’s Serverless machine learning-powered business intelligence service to create interactive dashboards.
• Fast, automatically scalable, embeddable, with per-session pricing
• Use cases: Business analytics, Building visualizations, Perform ad-hoc analysis, Get business insights using data etc.
• It can be integrated with RDS, Aurora, Athena, Redshift, S3

DocumentDB

• Aurora is an “AWS-implementation” of PostgreSQL / MySQL. Likewise DocumentDB is the same for MongoDB (which is a NoSQL database)
• MongoDB is used to store, query, and index JSON data.
• It uses the similar “deployment concepts” as Aurora.
• Fully Managed, highly available with replication across 3 AZ
• Automatically scales to workloads with millions of requests per seconds

So at the exam, if you see anything related to MongoDB, think DocumentDB, or if you see anything relate to NoSQL databases, think DocumentDB and also DynamoDB.

Amazon Neptune

• It is fully managed graph database.
• A popular graph dataset would be a social network: Users have friends, Posts have comments, Comments have likes from users, Users share and like posts.

• Highly available across 3 AZ, with up to 15 read replicas.
• Build and run applications working with highly connected datasets — optimized for these complex and hard queries
• Can store up to billions of relations and query the graph with milliseconds latency.
• Great for knowledge graphs (Wikipedia), fraud detection, recommendation engines, social networking.

So, coming from an exam perspective, anytime you see anything related to graph databases, think no more than Neptune.

Amazon QLDB

• QLDB stands for ”Quantum Ledger Database”
• A ledger is a book recording financial transactions.
• Fully Managed, Serverless, High available, Replication across 3 AZ.
• It is used to review history of all the changes made to your application data over time.
• It is an immutable system: no entry can be removed or modified, cryptographically verifiable.
• Difference with Amazon Managed Blockchain: no decentralization component, in accordance with financial regulation rules.

Anytime you see financial transactions and ledger, think QLDB.

Amazon Managed Blockchain

• Blockchain makes it possible to build applications where multiple parties can execute transactions without the need for a trusted, central authority.
• Amazon Managed Blockchain is a managed service to join public blockchain networks or create your own scalable private network.
• Compatible with the frameworks Hyperledger Fabric & Ethereum

So from an exam perspective, if you see anything related to Blockchains or Hyperledger fabric or Ethereum, you have to think Amazon managed Blockchain.

AWS Glue

• It is managed extract, transform, and load (ETL) service.
• It is useful to prepare and transform data for analytics.
• It is a fully serverless service.

DMS (Database Migration Service)

• Quickly and securely migrate databases to AWS, resilient, self healing
• The source database remains available during the migration

It supports:

• Homogeneous migrations: example, Oracle to Oracle.
• Heterogeneous migrations: example, Microsoft SQL Server to Aurora

So any time in the exam you see migration of a database, DMS is going to be the answer for it.

AWS Other Compute Services

In this section we will learn about the other compute services (Apart from core compute services like EC2) that are managed by AWS.

Let’s start with understanding about docker.

Docker

Let me define a docker for you -

Docker is a software development platform to deploy apps. These Apps are packaged in containers that can be run on literally any OS

Benefits of Dockers:

• Apps runs on any machine
• No compatibility issues
• Predictable behavior
• Easier to maintain and deploy
• Works with any language, any OS, any technology
• Scale containers up and down very quickly (seconds)

Where Docker images are stored?

• Dockers generally works on something called the docker images. These images are stored in Docker Repositories.
• There is a public docker repository → https://hub.docker.com/
• Here you will find the docker images with any technology or os.
• Private Docker Repository → Amazon ECR (Elastic Container Registry) where docker images will be stored.

Docker Vs Virtual Machines

• Docker is ”sort of” a virtualization technology, but not exactly

• In docker, we don’t need to have Guest OS. Many containers will run on Docker Daemon.

Amazon ECS (Elastic Container Service)

• ECS = Elastic Container Service
• It is used to launch Docker Containers on AWS
• You must provision & maintain the infrastructure (the EC2 instances)
• It has an integrations with the Application Load Balancer

Fargate

• It is also used to launch Docker Containers on AWS.
• The main difference between ECS and farmgate is, here do not provision the infrastructure (no EC2 instances to manage) — simpler.
• Therefore it is a serverless offering.
• AWS just runs containers for you based on the CPU / RAM you need.

Amazon ECR (Elastic Container Repository)

• It is a private Docker Registry on AWS
• This is where you store your Docker images so they can be run by ECS or Fargate

What’s serverless?

• Serverless is a new paradigm in which the developers don’t have to manage servers anymore.
• They just need to deploy code or functions.
• Serverless == FaaS (Function as a Service)
• Serverless was pioneered by AWS Lambda but now also includes anything that’s managed: “databases, messaging, storage, etc.”

Serverless does not mean there are no servers… it means you just don’t manage / provision / see them

AWS Lambda

AWS Lambda is an event-driven, serverless computing platform provided by Amazon as a part of Amazon Web Services.

Therefore you don’t need to worry about which AWS resources to launch, or how well you manage them. Instead, you need to put the code on Lambda, and it runs.

In AWS Lambda the code is executed based on the response of events in AWS services such as add/delete files in S3 bucket, HTTP request from Amazon API gateway, etc. However, Amazon Lambda can only be used to execute background tasks.

Why AWS Lambda (Serverless Computing Service)?

Amazon EC2:

• It’s not serverless. Basically virtual servers are in cloud.
• Limited by RAM and CPU
• It needs to run continuously
• Here scaling means intervention to add / remove servers.

Amazon Lambda:

• It has a Virtual functions — no servers to manage.
• Limited by time — It is used for short executions.
• It doesn’t run continuously but it runs on-demand.
• Scaling in AWS lambda is automated.

Benefits of AWS Lambda:

• Easy Pricing: Pay per request and compute time.
• Free tier gets 1,000,000 AWS Lambda requests and 400,000 GBs of compute time.
• It can be integrated with the whole AWS suite of services.
• It’s Event-Driven which means functions get invoked by AWS when needed.
• It’s integrated with many programming languages.
• We can do easy monitoring through AWS CloudWatch.
• Easy to get more resources per functions (up to 10GB of RAM!)
• Increasing RAM will also improve CPU and network!

AWS Lambda language support:

• Node.js (JavaScript)
• Python
• Java
• C# (.NET Core)
• Golang
• C# / Powershell
• Ruby
• Custom Runtime API (community supported, example Rust)

Lambda Container Image:

There is something called lambda container images. But it is not used widely. The container image must implement the Lambda Runtime API. Therefore ECS / Fargate is preferred for running arbitrary Docker images.

Lambda Examples:

CRON allows you to define a schedule. For example, every hour, every day or every Monday and based on that schedule to run a script. And by default, a CRON job is run on an Linux AMI.

AWS Lambda Pricing:

• Pay per calls.
• First 1,000,000 requests are free
• $0.20 per 1 million requests thereafter ($0.0000002 per request)
• It also can be priced based on Pay per duration.
• It is usually very cheap to run AWS Lambda so it’s very popular

Amazon API Gateway

• The Amazon API Gateway is used to build the serverless API.

• Fully managed service for developers to easily create, publish, maintain, monitor, and secure APIs
• It is Serverless and scalable
• It supports Supports RESTful APIs and WebSocket APIs
• Support for security, user authentication, API throttling, API keys, monitoring.

AWS Batch

• Fully managed batch processing at any scale
• Efficiently run 100,000s of computing batch jobs on AWS
• A “batch” job is a job with a start and an end (opposed to continuous)
• Batch will dynamically launch EC2 instances or Spot Instances
• AWS Batch provisions the right amount of compute / memory
• You submit or schedule batch jobs and AWS Batch does the rest!
• Batch jobs are defined as Docker images and run on ECS
• Helpful for cost optimizations and focusing less on the infrastructure

Batch vs Lambda

Amazon Lightsail

• Amazon Lightsail is a standalone service from AWS
• It provides virtual servers, storage, databases, and networking
• Low & predictable pricing
• Simpler alternative to using EC2, RDS, ELB, EBS, Route 53
• Great for people with little cloud experience!
• Can set up notifications and monitoring of your Lightsail resources
• Use cases: Simple web applications, Websites (templates for WordPress, Magento, Plesk, Joomla), Dev / Test environment.
• It has a high availability but no auto-scaling, limited AWS integrations.

From the exam perspective, if you see someone that has no cloud experience and need to get started quickly with a low and predictable pricing without configuring things much that will be Lightsail, otherwise Lightsail will almost always be a wrong answer. (It’s distractor)

Amazon Lightsail is designed to be the easiest way to launch and manage a virtual private server with AWS. Lightsail plans include everything you need to jumpstart your project — a virtual machine, SSD- based storage, data transfer, DNS management, and a static IP address — for a low, predictable price. It can be used to create a simple web application, a website or a dev/test environment.

Automating Your AWS Workloads

Automation is a best practice when it comes to designing architectures in the cloud. Automation allows common tasks to be repeated faster than doing them manually. These tasks can be simple and routine, such as installing the latest security patches on an EC2 instance running Linux. Or they can be complex, perhaps creating an entirely new and pristine AWS environment containing dozens of EC2 instances running a multitier, database-backed web application behind an application load balancer.

Whatever the task, whether simple or complex, automation confers several benefits, including the following:

• Rapid testing and experimentation
• Reducing expenses
• Minimizing human error
• Rolling back changes safely and completely

Some have a misconception of automation as replacing a human with a machine. But the truth is that automation requires a tremendous amount of human intelligence and ongoing effort.

Ultimately, automation is about increasing the capacity of the people doing the work so that they can spend more time being productive and less time fixing problems.

You can deploy automation at different levels of your AWS environment, picking and choosing where it makes the most sense for you.

You may find it helpful to approach automation in two different ways: imperative and declarative.

The Imperative Approach

• The most common example of automation using an imperative approach is scripting, wherein you use a scripting language such as Bash or PowerShell, or even a tool such as the AWS Command Line Interface (CLI), to provide the explicit steps necessary to carry out a task.
• For example, you may write a Bash script to install a web server on an instance and then copy HTML files from an S3 bucket to a particular directory on the instance. The imperative approach focuses on the specific step-by-step operations required to carry out a task.

The Declarative Approach

• Using a declarative approach, you write code that declares the desired result of the task, rather than how to carry it out.
• For example, rather than using the AWS CLI to create a new virtual private cloud (VPC) and subnets, you could write declarative code that simply defines the configuration for the VPC and subnets.
• This approach naturally requires some intelligent software to figure out the operations required to achieve the desired resultS.
• Example: CloudFormation

Infrastructure as Code

• Using code to define your infrastructure and configurations is commonly called the infrastructure as code (IaC) approach.
• Defining IaC is a fundamental element of automation in the cloud.
• Because automation relies on code being executed by a machine, it reduces the risk of human error inherent in carrying out the same work manually.

CloudFormation

CloudFormation automatically creates and configures your AWS infrastructure from code that defines the resources you want it to create and how you want those resources configured.

Basically, you write code containing the specifics of the AWS resources you want and present that code to CloudFormation, and it builds and configures those resources on your behalf in a matter of seconds.

CloudFormation use something called Templates to carry out the cloud formation tasks.

Templates:

• Templates contain a description of the AWS resources you want CloudFormation to create, so they simultaneously function as infrastructure documentation.
• Templates use the proprietary CloudFormation language, which can be written in JavaScript Object Notation (JSON) or YAML format.
• You can use the same template to build AWS infrastructure repeatedly, such as for development or testing
• You can also use a single template to build many similar environments. For example, you can use one template to create identical resources for both production and test environments.

Stack:

• To provision resources from a template, you must specify a stack name that’s unique within your account.
• A stack is a container that organizes the resources described in the template.
• The purpose of a stack is to collectively manage related resources.
• If you delete a stack, CloudFormation automatically deletes all of the resources in it.
• This makes CloudFormation perfect for test and development environments that need to be provisioned as pristine and then thrown away when no longer needed.
• Deleting a stack helps ensure that all resources are deleted and that no forgotten resources are left lingering, running up charges.

One of the fundamental principles of good cloud design is to make your test environments mirror your production environments as closely as possible. CloudFormation makes this not only possible but almost trivially easy!

Stack Updates:

• You can have CloudFormation change individual resources in a stack. Just modify the template to delete, modify, or add resources, and then instruct CloudFormation to perform a stack update using the template.
• CloudFormation will automatically update the resources accordingly.
• Alternatively, you can create a changeset. Make the desired changes to your template, and then have CloudFormation generate a changeset based on the template.
• CloudFormation will let you review the specific changes it will make, and then you can decide whether to execute those changes or leave everything as is.

Tracking Stack Changes:

• Each stack contains a timestamped record of events that occur within the stack, including when resources were created, updated, or deleted.
• This makes it easy to see all changes made to your stack.

CloudFormation vs. the AWS CLI:

• You can script AWS CLI commands to create resources fast and automatically, but those resources won’t be kept in a stack, so you won’t get the same advantages that stacks provide.
• When you’re using the AWS CLI, you can’t update your resources as easily as you can with CloudFormation.
• With CloudFormation, you simply adjust the template or parameters to change your resources, and CloudFormation figures out how to perform the changes.

CloudFormation template, you say:

• I want a security group.
• I want two EC2 instances using this security group.
• I want an S3 bucket.
• I want a load balancer (ELB) in front of these machines.

Benefits of AWS CloudFormation:

Infrastructure as code:

No resources are manually created, which is excellent for control. Changes to the infrastructure are reviewed through code.

Costs:

• Each resources within the stack is tagged with an identifier so you can easily see how much a stack costs you.
• You can estimate the costs of your resources using the CloudFormation template.

Productivity:

• Ability to destroy and re-create an infrastructure on the cloud on the fly
• Automated generation of Diagram for your templates!

Don’t re-invent the wheel:

• Leverage existing templates on the web
• Leverage the documentation

Supports (almost) all AWS resources:

• Everything we’ll see in this course is supported.
• You can use “custom resources” for resources that are not supported.

AWS Cloud Development Kit (CDK)

• With the help of CDK, we can define your cloud infrastructure using a familiar language.
• Example: JavaScript/TypeScript, Python, Java, and .NET
• The code is “compiled” into a CloudFormation template (JSON/YAML)

AWS Elastic Beanstalk

• Elastic Beanstalk is a developer-centric view of deploying an application on AWS.

Elastic Beanstalk is a platform within AWS that is used for deploying and scaling web applications.

• It uses all the component’s we’ve seen before: EC2, ASG, ELB, RDS, etc
• But it’s all in one view that’s easy to make sense of. We still have full control over the configuration
• Beanstalk = Platform as a Service (PaaS)
• Beanstalk is free but you pay for the underlying instances.
• AWS Elastic Beanstalk allows you to quickly deploy applications and services without having to worry about configuring underlying resources, services, operating systems or web servers.
• Elastic Beanstalk takes care of the hosting infrastructure, coding language interpreter, operating system, security, HTTPS service and application layer. All you need to worry about is writing your code.
• You can develop code in a number of languages which is then zipped up and the zip file is used when instantiating a new elastic beanstalk instance.
• Supported language platforms include: Java, Node.Js, Python, JS, C#, Ruby etc.

Health Monitoring with Elastic Beanstalk:

• Health agent pushes metrics to CloudWatch
• Checks for app health, publishes health events

AWS Developer Tools

The AWS Developer Tools are a collection of tools designed to help application developers develop, build, test, and deploy their applications onto EC2 and on-premises instances.

These tools facilitate and automate the tasks that must take place to get a new application revision released into production.

However, the AWS Developer Tools enable more than just application development. You can use them as part of any IaC approach to automate the deployment and configuration of your AWS infrastructure.

Tools Includes:-

• CodeCommit
• CodeBuild
• CodeDeploy
• CodePipeline

Lets learn more about these tools.

CodeCommit:

• Before pushing the application code to servers, it needs to be stored somewhere.
• Developers usually store code in a repository, using the Git technology.
• A famous public offering is GitHub, AWS’ competing product is CodeCommit.

CodeBuild:

• Code building service in the cloud (name is obvious)
• CodeBuild is a fully managed build service. A build is a set of actions performed on source code to get it ready for deployment.

• A build could include testing the code for errors or transforming it into a machine-readable language, but the specifi c build actions depend on the application.
• It is fully managed, serverless service.
• Pay-as-you-go pricing — only pay for the build time

CodeDeploy:

• CodeDeploy can automatically deploy applications to EC2 instances, the Elastic Container Service (ECS), Lambda, and even on-premises servers.
• This service is used when we want to deploy our application automatically.
• It works with EC2 Instances and also works with On-Premises Servers (Hybrid service)

CodePipeline:

• CodePipeline helps orchestrate and automate every task required to move software from development to production.
• It works by taking your source code and processing it through a series of stages, up to and including a deployment stage. In between the required source and deployment stages, you can add other stages also, such as a build stage that runs tests, or an approval stage that requires manual approval before deployment.
• Code => Build => Test => Provision => Deploy
• It served basis for CICD (Continuous Integration & Continuous Delivery)
• It’s fully managed, compatible with CodeCommit, CodeBuild, CodeDeploy, Elastic Beanstalk, CloudFormation, GitHub, 3rd-party services (GitHub…) & custom plugin
• Fast delivery & rapid updates

CodeArtifact:

• Software packages depend on each other to be built (also called code dependencies)
• Storing and retrieving these dependencies is called artifact management.
• Traditionally you need to setup your own artifact management system but AWS provides something called CodeArtifact which is a secure, scalable, and cost-effective artifact management for software development.
• It works well with common dependency management tools such as Maven, Gradle, npm, yarn, twine, pip, and NuGet.
• Developers and CodeBuild can then retrieve dependencies straight from CodeArtifact

So from an exam perspective, CodeArtifact is going to be very helpful, if your team needs a artifact management system or a place to store their code dependencies.

CodeStar:

• Unified UI to easily manage software development activities in one place.
• “Quick way” to get started to correctly set-up CodeCommit, CodePipeline, CodeBuild, CodeDeploy, Elastic Beanstalk, EC2, etc
• Can edit the code ”in-the-cloud” using AWS Cloud9 (IDE for Cloud)

AWS Cloud9:

• AWS Cloud9 is a cloud IDE (Integrated Development Environment) for writing, running and debugging code
• “Classic” IDE (like IntelliJ, Visual Studio Code…) are downloaded on a computer before being used.
• A cloud IDE can be used within a web browser, meaning you can work on your projects from your office, home, or anywhere with internet with no setup necessary.
• AWS Cloud9 also allows for code collaboration in real -time (pair programming)

Configuration Management

Configuration management is an approach to ensuring accurate and consistent configuration of your systems.

While automation is concerned with carrying out tasks, confi guration management is primarily concerned with enforcing and monitoring the internal configuration state of your instances to ensure they’re what you expect.

As with automation in general, configuration management tools use either imperative or declarative approaches.

AWS offers both approaches using two tools to help you achieve configuration management of your EC2 and on-premises instances:

1. System Manager
2. OpsWorks

AWS Systems Manager (SSM):

• Systems Manager uses the imperative approach (step-by-step) to get your instances and AWS environment into the state that you want.
• It helps you you manage your EC2 and On-Premises systems at scale. Therefore it would another Hybrid AWS service.
• It sends you operational insights about the state of your infrastructure.
• It is suite of 10+ products.
• Works for both Windows and Linux OS

Features of SSM:

• Patching automation for enhanced compliance
• Run commands across an entire fleet of servers
• Store parameter configuration with the SSM Parameter Store

How Systems Manager works?

• We need to install the SSM agent onto the systems we control
• It’s installed by default on Amazon Linux AMI & some Ubuntu AMI
• If an instance can’t be controlled with SSM, it’s probably an issue with the SSM agent!
• Thanks to the SSM agent, we can run commands, patch & configure our servers

So, from an exam perspective, anytime you see a way to patch your fleet of EC2 instances or On-Premises servers, you have to think about SSM, or if you wanted to run a command consistently

AWS OpsWorks:

OpsWorks is a set of three different services that let you take a declarative approach to configuration management.

As explained earlier, using a declarative approach requires some intelligence to translate declarative code into imperative operations.

OpsWorks uses two popular configuration management platforms that fulfill this requirement:

1. Chef
2. Puppet

Puppet and Chef are popular configuration management platforms that can configure operating systems, deploy applications, create databases, and perform just about any configuration task you can dream of, all using code.

Both Puppet and Chef are widely used for on-premises deployments, but OpsWorks brings their power to AWS as the OpsWorks managed service.

In the exam: Chef or Puppet needed => AWS OpsWorks

Global Infrastructure

Why make a global application?

A global application is an application deployed in multiple geographies. On AWS: this could be Regions and / or Edge Locations.

Decreased Latency:

• With the global application, we can observe the decreased latency. (Latency is the time it takes for a network packet to reach a server)
• It takes time for a packet from Asia to reach the US.
• Deploy your applications closer to your users to decrease latency, better experience.

Disaster Recovery (DR):

• If an AWS region goes down (earthquake, storms, power shutdown, politics)
• You can fail-over to another region and have your application still working
• A DR plan is important to increase the availability of your application

Attack protection: The distributed global infrastructure is harder to attack

Global AWS Infrastructure

• Regions: For deploying applications and infrastructure
• Availability Zones: Made of multiple data centers.
• Edge Locations (Points of Presence): For content delivery as close as possible to users

Global Applications in AWS

Global DNS: Route 53

• Great to route users to the closest deployment with least latency
• Great for disaster recovery strategies

Global Content Delivery Network (CDN): CloudFront

• Replicate part of your application to AWS Edge Locations — decrease latency
• Cache common requests — improved user experience and decreased latency

S3 Transfer Acceleration: Accelerate global uploads & downloads into Amazon S3

AWS Global Accelerator: Improve global application availability and performance using the AWS global network

Now let’s look at these one by one in more details.

Amazon Route 53

• Route 53 is Amazon’s global Domain Name System (DNS) service.
• The primary purpose of DNS is to translate human-readable domain names (such as example.com) into IP addresses.
• DNS is a collection of rules and records which helps clients understand how to reach a server through URLs.

How it works?

when you enter the domain name example.com into your web browser, your computer sends out a query to its configured DNS server asking for the IP address of that domain. The DNS server then sends the query to the domain’s authoritative DNS server — the one that’s in charge of the example.com domain name. The authoritative DNS server responds with the IP address for example.com. This process of translating a domain name to an IP address is called name resolution.

Resource Records:

• Name resolution goes beyond just mapping domain names to IP addresses. DNS can store mappings for different types of data, including IPv6 addresses, mail servers, and even arbitrary text.
• When you send an email to someone, DNS provides the lookup mechanism to ensure it gets routed to the correct mail server for that domain.
• For DNS to work, someone must first define some resource records for a domain. A resource record consists of several fields, but the most important are the name, type, and value.

Domain Name Registration:

• To ensure that no two entities try to use the same domain name, anyone who wants to have a public domain name must register it with a domain registrar.
• When you register a domain name, you must do so under a top-level domain (TLD) such as .com, .net, or .org.
• Route 53 is a domain registrar for hundreds of TLDs.
• If you have an existing domain name with another registrar, you can transfer it to Route 53. Transferring a domain entails extending the registration by at least one year.
• It’s important to understand that domain name registration and DNS hosting are two different functions.
• The domain registrar and DNS hosting provider don’t have to be the same company, but they often are. Route 53 is both a registrar and a DNS hosting provider.

Hosted Zones:

• To have Route 53 host the DNS for a public domain name, you create a public hosted zone and specify the domain name
• If you use Route 53 to register a domain name, it automatically takes care of creating a public hosted zone for the domain.

Routing Policies:

In some cases, you just need a domain name to resolve to a particular IP address. But there are other times when you want the value of a resource record to change dynamically to work around failures or ensure users get pointed to the least busy server.

Route 53 lets you accomplish this with a variety of routing policies.

Simple:

• The Simple routing policy is the default for new resource records. It simply lets you map a domain name to a single static value, such as an IP address.
• It doesn’t check whether the resource the record points to is available.

Weighted:

• A Weighted policy distributes traffic across multiple resources according to a ratio.
• For example, when introducing a new web server, you may want to route only 10 percent of the traffic to the new server while evenly distributing the load across the rest.

Latency:

A Latency policy sends users to resources in the AWS Region that’s closest to them.

Failover:

A Failover policy lets you route traffic to a primary resource unless it’s unavailable. In that case, traffic will be redirected to a secondary resource.

AWS CloudFront

• Amazon CloudFront is a content delivery network (CDN) that helps deliver static and dynamic web content to users faster than just serving it out of an AWS Region.
• For example, if you’re hosting a website from a single AWS Region, as a general rule, the farther a user is away from that region, the more network latency they’ll encounter when accessing it. CloudFront solves this problem by caching your content in a number of data centers called edge locations.
• There are more than 216 edge locations spread out around the world on six continents.
• CloudFront works by sending users to the edge location that will give them the best performance.
• Typically, this is the edge location that’s physically closest to them.
• CloudFront also increases the availability of your content because copies of it are stored in multiple edge locations.
• The more edge locations you use, the more redundancy you have and the better performance you can expect.
• As you might expect, the price of CloudFront increases as you utilize more edge locations.
• You can’t select individual edge locations. Rather, you must choose from the following three options:

1. United States, Canada, and Europe
2. United States, Canada, Europe, Asia, and Africa
3. All edge locations

• To make your content available via CloudFront, you must create a distribution. A distribution defines the type of content you want CloudFront to cache, as well as the content’s origin — where CloudFront should retrieve the content from.
• There are two types of distributions: Web and Real-Time Messaging Protocol (RTMP).

Web Distribution:

• A Web distribution is the most common type. It’s used for static and dynamic content such as web pages, graphic files, and live or on-demand streaming video.
• Users can access Web distributions via HTTP or HTTPS.
• When creating a Web distribution, you must specify an origin to act as the authoritative source for your content.

RTMP Distribution:

• Real-Time Messaging Protocol (RTMP)
• The Real-Time Messaging Protocol (RTMP) delivers streaming video or audio content to end-users.
• To set up an RTMP distribution, you must provide both a media player and media files to stream, and these must be stored in S3 buckets.

S3 Transfer Acceleration

Increase transfer speed by transferring file to an AWS edge location which will forward the data to the S3 bucket in the target region

AWS Global Accelerator

• Improves global application availability and performance using the AWS global network
• Leverage the AWS internal network to optimize the route to your application (60% improvement)
• 2 Anycast IP are created for your application and traffic is sent through Edge Locations
• The Edge locations send the traffic to your application

AWS Global Accelerator vs CloudFront

• They both use the AWS global network and its edge locations around the world
• Both services integrate with AWS Shield for DDoS protection.

CloudFront:

• Improves performance for your cacheable content (such as images and videos)
• Content is served at the edge

Global Accelerator:

• No caching, proxying packets at the edge to applications running in one or more AWS Regions.
• Improves performance for a wide range of applications over TCP or UD

AWS Outposts

• It is useful for businesses that keep an on-premises infrastructure alongside a cloud infrastructure (Hybrid Cloud)
• Therefore, two ways of dealing with IT systems: One for the AWS cloud and one for their on -premises infrastructure.
• Therefore AWS solves this issue of having multiple ITs by AWS Outposts.
• AWS Outposts are “server racks” that offers the same AWS infrastructure, services, APIs & tools to build your own applications on-premises just as in the cloud.
• AWS will set up and manage “Outposts Racks” within your on-premises infrastructure and you can start leveraging AWS services on-premises
• You are responsible for the Outposts Rack physical security

Benefits of AWS Outposts:

• Low-latency access to on-premises systems
• Local data processing
• Data residency
• Easier migration from on-premises to the cloud

AWS WaveLength

• WaveLength Zones are infrastructure deployments embedded within the telecommunications providers’ datacenters at the edge of the 5G networks
• Brings AWS services to the edge of the 5G networks
• This is only used for ultra-low latency applications through 5G networks
• Traffic doesn’t leave the Communication Service Provider’s (CSP) network
• High-bandwidth and secure connection to the parent AWS Region
• Use cases: Smart Cities, ML-assisted diagnostics, Connected Vehicles, Interactive Live Video Streams, AR/VR, Real-time Gaming

The idea is that whenever you see 5G in your questions, this is most likely going to be WaveLength

AWS Local Zones

• Places AWS compute, storage, database, and other selected AWS services closer to end-users to run latency-sensitive applications
• Extend your VPC to more locations — “Extension of an AWS Region”

Example:

• AWS Region: N. Virginia (us-east-1)
• AWS Local Zones: Boston, Chicago, Dallas, Houston, Miami,

Global Applications Architecture

Cloud Integration

When we start deploying multiple applications, they will inevitably need to communicate with one another. There are generally two types of patterns to make applications communicate:

1. Synchronous Communication
2. Asynchronous Communication

The first one is easy to understand. It’s called a synchronous communication, in which an application talks to another application.

For example →

You have created a service to buy something, then you need to talk to a service that ships what has been bought, and so therefore you want to integrate the buying service and the shipping service synchronously because they talk directly to one another.

The second communication type is asynchronous, or event based communication. Here ‘Queue’ will be implemented to act as mediator between two applications.

For Example →

Our buying service this time, If we bought something, that will put an order in a queue, and the shipping service will be reading from the queue to get the orders.

As you can see in this example, your buying service and your shipping service are not directly integrated with one another, they are something called decoupled, because there is a queue in between to talk to. And this allows us to get some nice integration patterns.

Synchronous between applications can be problematic if there are sudden spikes of traffic. For an example, let’s say, there are 10000 orders, instead of 10.

In this case, the service you are talking to may get overwhelmed, and you may get an issue. Therefore In that case, it’s better to decouple your applications with the help of the following models.

1. Using SQS: Queue Based Model
2. Using SNS: Pub/Sub Based Model

Amazon SQS(Simple Queue Service)

• It’s Simple Queue Service
• It allows us to decouple the application with the help of SQS Queue

• Here producers sends messages into the queue. Remember, producers can be single or multiple. Once the messages are stored in a queue, that can be read by consumers who will be polling the message from queue, that means requesting messages from the queue.
• Oldest AWS offering (over 10 years old)
• Fully managed service (~serverless), use to decouple applications.
• Scales from 1 message per second to 10,000s per second.
• Default retention of messages: 4 days, maximum of 14 days.
• There is no limit to how many messages can be in the queue.
• Messages are deleted after they’re read by consumers
• Low latency

Amazon SNS(Simple Notification Service)

• What if you want to send one message to many receivers or subscribers?
• Think like you have large subscriber base and you want to convey your message to them. You can’t send message to all individuals; rather send it to one mediator and that mediator will be sharing your message to all the subscribers.
• Here that mediator is nothing but the Pub/Sub.

• The “event publishers” only send a message to one SNS topic
• As many “event subscribers” as we want to listen to the SNS topic notifications
• Each subscriber to the topic will get all the messages
• Up to 10,000,000 subscriptions per topic, 100,000 topics limit
• Emails, SMS messages, Mobile Notifications etc.

Amazon Kinesis

• Kinesis = real-time big data streaming
• Managed service to collect, process, and analyze real-time streaming data at any scale

Amazon MQ

• SQS, SNS are “cloud-native” services, and they’re using proprietary protocols from AWS.
• Traditional applications running from on-premise may use open protocols such as MQTT, AMQP, STOMP, Openwire, WSS
• When migrating to the cloud, instead of re-engineering the application to use SQS and SNS, we can use Amazon MQ
• Amazon MQ = managed Apache ActiveMQ
• Amazon MQ doesn’t “scale” as much as SQS / SNS
• Amazon MQ runs on a dedicated machine (not serverless)
• Amazon MQ has both queue feature (~SQS) and topic features (~SNS)

Cloud Monitoring

CloudWatch

Amazon CloudWatch is a key service that helps you plan, monitor, and fine-tune your AWS infrastructure and applications.

It lets you collect, search, and visualize data from your applications and AWS resources in the form of logs, metrics, and events.

Let’s discuss the common CloudWatch use cases.

Infrastructure monitoring and troubleshooting: Visualize performance metrics to discover trends over time and spot outliers that might indicate a problem.

Resource optimization: Save money and help with resource planning by identifying overused or underused resources.

Application monitoring: Create CloudWatch alarms to alert you and take corrective action when a resource’s utilization, performance, or health falls outside of a threshold that you define.

Log analytics: Search, visualize and correlate logs from multiple sources to help with troubleshooting and identify areas for improvement.

Now let’s discuss more on CloudWatch Metrics

CloudWatch Metrics:

• CloudWatch provides metrics for every service in AWS
• CloudWatch Metrics is a feature that collects numeric performance metrics from both AWS and non-AWS resources such as on-premises servers.
• A metric is a variable that contains a time-ordered set of data points. Each data point contains a timestamp, a value, and optionally a unit of measure.
• We can create CloudWatch dashboards of metrics.
• All AWS resources automatically send their metrics to CloudWatch. These metrics include things such as EC2 instance CPU utilization, S3 bucket sizes, and DynamoDB consumed read and write capacity units.
• CloudWatch stores metrics for up to 15 months. You can graph metrics to view trends and how they change over time.

CloudWatch Alarms:

• A CloudWatch alarm watches over the value of a single metric. If the metric crosses a threshold that you specify (and stays there), the alarm will take an action.
• For example, you might configure an alarm to take an action when the average CPU utilization for an instance exceeds 80% for five minutes.
• These alarms are used to trigger notifications for any metric.

CloudWatch Logs:

• CloudWatch Logs collects and stores log files from AWS and non-AWS sources and make it easy to view, search, and extract custom metrics from them.
• Enables real-time monitoring of logs.
• You configure your applications and AWS services to send log events to CloudWatch Logs.
• A log event is analogous to a line in a log file and always contains a timestamp and an event message.
• By default, no logs from your EC2 instance will go to CloudWatch
• You need to run a CloudWatch agent on EC2 to push the log files you want

• The CloudWatch log agent can be set up on-premises too.

CloudWatch Events:

• The CloudWatch Events feature lets you continuously monitor for specific events that represent a change in your AWS resources action when they occur.
• For example, an EC2 instance going from the running state to the stopped state would be an event. An IAM user logging into the AWS Management Console would also be an event. CloudWatch Events can then automatically and immediately take actions in response to those events.
• CloudWatch responds to events as they occur, in real-time. Unlike CloudWatch alarms, which take action when a metric crosses and remains crossing a numeric threshold, CloudWatch events trigger immediately.

Amazon EventBridge

• EventBridge is the next evolution of CloudWatch Events
• Default Event Bus: Generated by AWS services (CloudWatch Events)
• Partner Event Bus: receive events from SaaS service or applications (Zendesk, DataDog, Segment, Auth0…)
• Custom Event buses: for your own applications
• EventBridge has a different name to mark the new capabilities.
• Soon the CloudWatch Events name will be replaced with EventBridge

AWS CloudTrail

• CloudTrail keeps detailed event logs of every action that occurs against your AWS resources.
• It is used to provides governance, compliance and audit for your AWS Account
• CloudTrail is enabled by default!
• It tells us n history of events / API calls made within your AWS Account by: Console, SDK, CLI, etc.
• We can put logs from CloudTrail into CloudWatch Logs or S3.
• A trail can be applied to All Regions (default) or a single Region.
• If a resource is deleted in AWS, investigate CloudTrail first.

CloudTrail Events:

Management Events:

• Any operations that are performed on resources in your AWS account.
• Examples: Configuring security, Configuring rules for routing data, Setting up logging, etc.
• By default, trails are configured to log management events.
• Can separate Read Events (that don’t modify resources) from Write Events (that may modify resources)

Data Events:

• By default, data events are not logged (because high volume operations)
• Example: Amazon S3 object-level activity, AWS Lambda function execution activity

CloudTrail Insights:

We need to enable CloudTrail Insights to detect unusual activity in your account. It includes:-

• inaccurate resource provisioning
• hitting service limits
• Bursts of AWS IAM actions
• Gaps in periodic maintenance activity

CloudTrail Events Retention:

• Events are stored for 90 days in CloudTrail
• To keep events beyond this period, log them to S3 and use Athena

AWS X-Ray

when your application is actually deployed, the good old way, you would test locally, then you would add log statements anywhere. Maybe look into how it logs, then you would re-deploy in production, and then see if you can find the problem this way.

The problem is that we have logs from different services and different applications. Doing log analysis is very hard because we have to combine everything.

Remember, debugging is easy if we have one big applications. But it is relatively difficult when we have distributed services. Therefore to solve this problem, AWS has introduced something called AWS X-Ray.

• So, with X-Ray, you’re going to be able to do a tracing and get visual analysis of your application.
• Once you enable the X-Ray on your services, then you’ll get a full picture of what is happening for each service.
• With X-Ray, we can see which services are failing, their performance, and in case one request goes wrong, you will be able to visualize it directly into the X-Ray console.

Benefits of X-Ray:

• Troubleshooting performance (bottlenecks)
• Understand dependencies in a microservice architecture
• Pinpoint service issues
• Find errors and exceptions
• Identify users that are impacted
• It is also known as debugging in Production. It enables us to → Test things locally, Add log statements everywhere, redeploy them in productions.
• Debugging is easy if we have one big applications. But it is relatively difficult when we have distributed services.

Amazon CodeGuru

It’s an ML-powered service for automated code reviews and application performance recommendations.

It provides two services:

1. CodeGuru Reviewer: Automated code reviews for static code analysis (development stage)
2. CodeGuru Profiler: Visibility/Recommendations about application performance during runtime (production stage)

Lets’ talk more about CodeGuru Reviewer.

CodeGuru Reviewer:

• It identifies critical issues, security vulnerabilities, and hard-to-find bugs
• Example: common coding best practices, resource leaks, security detection, input validation
• It uses Machine Learning and automated reasoning
• Supports Java and Python
• Integrates with GitHub, Bitbucket, and AWS CodeCommit

CodeGuru Profiler:

• It helps to understand the runtime behavior of your application
• Example: identify if your application is consuming excessive CPU capacity on a logging routine
• Support applications running on AWS or on-premise

AWS Status (Service Health Dashboard)

• It shows all regions, all services health.
• It shows historical information for each day
• It has an RSS feed you can subscribe to.

AWS Personal Health Dashboard

• AWS Personal Health Dashboard provides alerts and remediation guidance when AWS is experiencing events that may impact you.
• While the Service Health Dashboard displays the general status of AWS services, Personal Health Dashboard gives you a personalized view into the performance and availability of the AWS services underlying your AWS resources.

• The dashboard displays relevant and timely information to help you manage events in progress and provides proactive notification to help you plan for scheduled activities.
• It shows you how AWS outages directly impact you & your AWS resources

Virtual Private Cloud (VPC) & Networking

VPC is something you should know in depth for the AWS Certified Solutions Architect Associate & AWS Certified SysOps Administrator. At the AWS Certified Cloud Practitioner Level, you should know the chapters included into this section.

VPC & Subnets Primer

• VPC is a private network to deploy your resources.
• a VPC is linked to a specific region. So if you have multiple regions in AWS and you have multiple VPC.

• Subnets allow you to partition your network inside your VPC.
• A public subnet is a subnet that is accessible from the internet
• A private subnet is a subnet that is not accessible from the internet
• To define access to the internet and between subnets, we use Route Tables.

Internet Gateway & NAT Gateways

• Internet Gateways helps our VPC instances connect with the internet.
• Public Subnets have a route to the internet gateway

• NAT Gateways (AWS-managed) & NAT Instances (self-managed) allow your instances in your Private Subnets to access the internet while remaining private

Network ACL & Security Groups

NACL (Network ACL):

• A firewall that controls traffic from and to the subnet.
• Unlike Security Groups, it can have ALLOW and DENY rules.
• They are attached at the Subnet level
• Rules only include IP addresses

Security Groups:

• A firewall that controls traffic to and from an ENI / an EC2 Instance
• It can have only ALLOW rules.
• Rules include IP addresses and other security groups

Network ACLs vs Security Groups:

VPC Flow Logs

• It captures information about the IP traffic going into your interfaces.
• It helps to monitor & troubleshoot connectivity issues.
• It captures network information from AWS managed interfaces too: Elastic Load Balancers, ElastiCache, RDS, Aurora, etc.
• VPC Flow logs data can go to S3 / CloudWatch Logs

VPC Peering

• It allows us to connect two VPC, privately using AWS’ network.
• Make them behave as if they were in the same network

• Must not have overlapping CIDR (IP address range)
• VPC Peering connection is not transitive (must be established for each VPC that need to communicate with one another)

VPC Endpoints

• Endpoints allow you to connect to AWS Services using a private network instead of the public www network
• This gives you enhanced security and lower latency to access AWS services
• VPC Endpoint Gateway: S3 & DynamoDB

Site to Site VPN & Direct Connect

Site to Site VPN:

• Connect an on-premises VPN to AWS
• The connection is automatically encrypted
• Goes over the public internet

On-premises: must use a Customer Gateway (CGW)

AWS: must use a Virtual Private Gateway (VGW)

Direct Connect (DX):

• Establish a physical connection between on-premises and AWS
• The connection is private, secure and fastThe connection is private, secure and fast.

• Takes at least a month to establish.
• It goes over a private network

Transit Gateway

• Connect thousands of VPC and on-premises networks together
• For having transitive peering between thousands of VPC and on -premises, hub-and-spoke (star) connection.

• One single Gateway to provide this functionality
• Works with Direct Connect Gateway, VPN connections

Security & Compliance

We have seen the AWS shared responsibility models in almost all the sections. It’s time we must formally introduce it in this section.

AWS Shared Responsibility Model

AWS responsibility is the security of the cloud:

That means that all the infrastructure they provide to you, that includes the hardware, the software facilities, networking. They have to protect it. This is because the infrastructure will run all the services that you’re using on AWS.

On top of it, any service that is managed by AWS, such as S3 DynamoDB, RDS, is the responsibility of AWS. But once they provide a service to you, then how you use that service is your responsibility.

Your responsibility is the security in the cloud:

• You are responsible for the management of all the operating system that including patching the operating system and making updates to it.
• You must configure the firewall.
• You need to make sure that your EC2 instance has the correct IAM information through the use of IAM instance role.
• You also need to ensure that you encrypt the application data, according to compliance requirements.

Shared Responsibility:

Patch management, configuration management, awareness, and training are both shared between AWS and you.

Example:

• For batch managements, if using something like RDS, then AWS will do the patch management for us.
• If we are using something like EC2 then we have to patch our operating system
• For awareness and training, AWS has to train their employees to use their facilities correctly, and to make sure they adhere to their security guidelines. Sameway, you have to make sure to train your employees correctly, to use the cloud.

What’s a DDoS* Attack?

A DDoS attack is a Distributed Denial-of-Service attack on our infrastructure. Look at the example below.

Here is the attacker or hacker and he wants to do a DDoS attack against our application server. In this case, they’re going to launch multiple master servers and these servers are going to launch multiple bots. These bots are going to send requests to our application server.

Now our server is not meant to handle this many requests so it will be overwhelmed and it will not be working anymore.

Therefore any normal user trying to connect to our application server, will see that our server is not accessible or not responsive. This is how the attacks gets carried away which makes our application down.

DDOS Protection on AWS

AWS Shield Standard:

• It’s a free and default service from AWS for all.
• It protects against DDOS attack for your website and applications, for all customers at no additional costs
• Provides protection from attacks such as SYN/UDP Floods, Reflection attacks, and another layer 3/layer 4 attacks

AWS Shield Advanced:

• It’s premium service
• Protection is 24*7
• It provides an optional DDoS mitigation service ($3,000 per month per organization)
• It protects against more sophisticated attack on Amazon EC2, Elastic Load Balancing (ELB), Amazon CloudFront, AWS Global Accelerator, and Route 53
• 24/7 access to AWS DDoS response team (DRP)
• It protects against higher fees during usage spikes due to DDoS

AWS WAF:

• It’s Web Application Firewall (WAF) service from Amazon
• It filters specific requests based on rules
• It protects your web applications from common web exploits (Layer 7)
• Layer 7 is HTTP (vs Layer 4 is TCP)
• It can be deployed on Application Load Balancer, API Gateway, CloudFront
• It also allows us to define Web ACL (Access Control List) which can protect us from SQL injections, cross-site scripting, etc.

CloudFront and Route 53:

• Availability protection using global edge network
• Combined with AWS Shield, provides attack mitigation at the edge

Penetration Testing on AWS Cloud

• Penetration testing is when you’re trying to attack your own infrastructure to test your security.
• AWS customers are welcome to carry out security assessments or penetration tests against their AWS infrastructure without prior approval for 8 services which includes: Amazon EC2 instances, NAT Gateways, and Elastic Load Balancers, RDS, CloudFront, Aurora, Lambda, API Gateways etc.

Encryption with KMS and CloudHSM

There are generally two types of encryption that happens in AWS.

1. Encryption when Data at Rest
2. Encryption when Data at Transit

Let’s look at the above concepts in more detail.

• Encryption when Data at Rest means data is stored or archived on the device. It could be on RDS, S3 Glacier etc.
• Encryption when Data at Transit (in motion) means data being moved from one location to another. It could means data transferred on the network. For example, Transfer from on-premises to AWS, EC2 to DynamoDB, etc.

We want to encrypt data in both states to protect it. For this, we leveraged something called encryption keys. These encryption keys are responsible for encryption and descriptions.

Therefore the encryption service at AWS is called KMS (Key Management Service)

AWS KSM:

• Anytime you hear “encryption” for an AWS service, it’s most likely KMS
• In KMS, AWS manages the encryption keys for us.

CloudHSM:

• In KMS, AWS manages the software for encryption.
• In CloudHSM, AWS provisions encryption hardware.
• Here HSM stands for Hardware Security Module.
• In this service, you manage your own encryption keys entirely (not AWS)

Customer Master Keys (CMK)

Customer Managed CMK:

• Create, manage and used by the customer, can enable or disable
• Possibility of rotation policy (new key generated every year, old key preserved)
• Possibility to bring-your-own-key

AWS managed CMK:

• Created, managed and used on the customer’s behalf by AWS
• Used by AWS services (aws/s3, aws/ebs, aws/redshift)

AWS owned CMK:

• Collection of CMKs that an AWS service owns and manages to use in multiple accounts
• AWS can use those to protect resources in your account (but you can’t view the keys)

CloudHSM Keys (custom Keystore):

• Keys generated from your own CloudHSM hardware device
• Cryptographic operations are performed within the CloudHSM cluster

AWS Certificate Manager (ACM)

• It lets you easily provision, manage, and deploy SSL/TLS Certificates.
• These certificate used to provide in-flight encryption for websites by providing HTTPS endpoints.
• It supports both public and private TLS certificates.
• Free of charge for public TLS certificates
• It provides an Automatic TLS certificate renewal service.

So anytime you see what service can help us do in-flight encryption and generates these certificates then think ACM

AWS Secrets Manager:

• It’s newer service from AWS, meant for storing secrets. As the name indicates, Secrets Manager is a great way to store secrets.
• On top of it, you can force the rotation of these secrets, I mean, they have to change every X number of days.

Example: Every 90 days, I want to change my password

• You can automate the generation of these secrets using Lambda.
• It has an integration with Amazon RDS. So using the Secrets Manager, we can create the passwords for Amazon RDS automatically
• The secrets are going to be encrypted using KMS.

From an exam perspective, anytime you see a secret to be managing in RDS and to be rotated, you have to think about Secrets Manager.

• It’s paid service.

AWS Artifact

It’s actually not a service from AWS but it’s a portal that provides customers with on-demand access to AWS compliance documentation and AWS agreements.

Artifact Reports:

Allows you to download AWS security and compliance documents from third-party auditors, like AWS ISO certifications, Payment Card Industry (PCI), and System and Organization Control (SOC) reports

Artifact Agreements:

Allows you to review, accept, and track the status of AWS agreements such as the Business Associate Addendum (BAA) or the Health Insurance Portability and Accountability Act (HIPAA) for an individual account or in your organization

Amazon GuardDuty

• If you wanted to protect your accounts against attacks from the outside and from the inside, GuardDuty is an intelligent threat discovery mechanism that will use Machine Learning Algorithm to detect anomalies within your account.
• It’s One click to enable service (30 days trial), no need to install any software.
• We can set up CloudWatch Event rules to be notified in case of findings.

GaurdDuty needs to read data or logs in this case from various sources. It includes:

• CloudTrail Logs: unusual API calls, unauthorized deployments
• VPC Flow Logs: unusual internal traffic, unusual IP address
• DNS Logs: compromised EC2 instances sending encoded data within DNS queries

GuardDuty will analyze above input logs using some machine learning algorithm, and then it will emit events into CloudWatch Events.If findings are found, then from cloud watch events, we can trigger a lambda function or send ourselves an email through an SNS topic.

Amazon Inspector

• Inspector is used to conduct automated security assessments on your EC2 instances.
• Analyze the running OS against known vulnerabilities
• Analyze against unintended network accessibility
• For this, AWS Inspector Agent must be installed on OS in EC2 instances
• After the assessment, you get a report with a list of vulnerabilities.

AWS Config

• AWS Config helps with auditing and recording the compliance of your resources by recording the configuration and their changes over time.
• This configuration data can be stored in S3.

Questions that can be solved by AWS Config:

• Is there unrestricted SSH access to my security groups?
• Do my buckets have any public access?
• How has my ALB configuration changed over time?

Amazon Macie

• Amazon Macie is a fully managed data security and data privacy service that uses machine learning and pattern matching to discover and protect your sensitive data in AWS.
• Macie helps identify and alert you to sensitive data, such as personally identifiable information (PII)

AWS Security Hub

It’s a central security tool to manage security across several AWS accounts and automate security checks.

Integrated dashboards showing current security and compliance status to quickly take actions

Automatically aggregates alerts in predefined or personal findings formats from various AWS services & AWS partner tools:

• GuardDuty
• Inspector
• Macie
• IAM Access Analyzer
• AWS Systems Manager
• AWS Firewall Manager
• AWS Partner Network Solutions

Note:- You must first enable the AWS Config Service to use AWS Security Hub.

Amazon Detective

• GuardDuty, Macie, and Security Hub are used to identify potential security issues or findings
• Sometimes security findings require deeper analysis to isolate the root cause and take action — it’s a complex process
• Amazon Detective analyzes, investigates, and quickly identifies the root cause of security issues or suspicious activities (using ML and graphs)
• Automatically collects and processes events from VPC Flow Logs, CloudTrail, GuardDuty and create a unified view
• Produces visualizations with details and context to get to the root cause

AWS Abuse

It report suspected AWS resources used for abusive or illegal purposes. Abusive & prohibited behaviors are:

• Spam — receiving undesired emails from AWS-owned IP addresses, websites & forums spammed by AWS resources
• Port scanning — sending packets to your ports to discover the unsecured ones
• DoS or DDoS attacks — AWS-owned IP addresses attempting to overwhelm or crash your servers/software
• Intrusion attempts — logging in on your resources
• Hosting objectionable or copyrighted content — distributing illegal or copyrighted content without consent
• Distributing malware — AWS resources distributing software to harm computers or machines

Root User Privileges

• Root user = Account Owner (created when the account is created)
• He/She has complete access to all AWS services and resources
• Do not use the root account for everyday tasks, even administrative tasks

But there are certain actions that can be performed only by the root user. These includes:

• Change account settings (account name, email address, root user password, root user access keys)
• Close your AWS account
• Change or cancel your AWS Support plan
• Register as a seller in the Reserved Instance Marketplace
• View certain tax invoices
• Restore IAM user permissions
• Configure an Amazon S3 bucket to enable MFA
• Edit or delete an Amazon S3 bucket policy that includes an invalid VPC ID or VPC endpoint ID
• Sign up for GovCloud

Account Management, Billing & Support Section

AWS Organizations

• It’s a global service.
• It allows us to manage multiple AWS accounts.
• The main account is the master account.
• API is available to automate AWS account creation.
• Restrict account privileges using Service Control Policies (SCP)

It provides us various costs benefits (If we use AWS organization):

• Consolidated Billing across all accounts — single payment method
• Pricing benefits from aggregated usage (volume discount for EC2, S3)
• Pooling of Reserved EC2 instances for optimal savings.

Multi-Account Strategies:

• Create accounts per department, per cost center, per dev/test / prod, based on regulatory restrictions (using SCP), for better resource isolation (ex: VPC), to have separate per-account service limits, isolated account for logging
• Use tagging standards for billing purposes
• Enable CloudTrail on all accounts, send logs to central S3 account

Organizational Units (OU):

We can create above mentioned hierarchy when we create AWS Organization. (Think like you are creating a complete organization from scratch through AWS)

Service Control Policies (SCP)

• Service Control Policies (SCPs) are IAM-like policies to manage permissions in AWS Organizations.
• SCPs are policies and as we know most of the policies are defined in JSON formats.
• With SCPs, we can whitelist or blacklist IAM actions.
• It can be applied at the OU or Account level. But does not gets applied to the Master Account.
• The SCP does not affect service-linked roles. The Service-linked roles enable other AWS services to integrate with AWS Organizations and can’t be restricted by SCPs.
• Use Case:

1. Restrict access to certain services (for example: can’t use EMR)
2. Enforce PCI compliance by explicitly disabling services

Consolidated Billing (AWS Organization)

The consolidated billing is today's need. When we enable the consolidated billings provides the following benefits to us.

• Combined Usage: It combines the usage across all AWS accounts in the AWS Organization to share the volume pricing, Reserved Instances and Savings Plans discounts
• One Bill: We get one bill for all AWS Accounts in the AWS Organization

AWS Control Tower

• Easy way to set up and govern a secure and compliant multi-account AWS environment based on best practices.
• So instead of doing everything manually creating an organization and so on and then applying security practices, we have Control Tower where you can, with a few clicks can create a multi-account, AWS environment.
• It automates the setup of your environment in a few clicks.
• It can automate ongoing policy management using guardrails.
• It can detect policy violations and remediate them.
• Monitor compliance through an interactive dashboard

AWS Control Tower runs on top of AWS Organizations. It automatically sets up AWS Organizations to organize accounts and implement SCPs (Service Control Policies)

Pricing Models in AWS

AWS has total of 4 pricing models.

Pay As You Go: pay for what you use, remain agile, responsive, meet scale demands.

Save When You Reserve:

• It minimizes risks, predictably manages budgets, complies with long-term requirements.
• Reservations are available for EC2 Reserved Instances, DynamoDB Reserved Capacity, ElastiCache Reserved Nodes, RDS Reserved Instance, Redshift Reserved Nodes
• It like reserving things in advance so we get some pricing benefits.

Payless by using more:

• This is Volume Based Discount.
• We tend to get discounts when we order things or use things more.

Pay less as AWS grows:

AWS grows their infrastructure, they will have some cost savings, and they will pass them on to you.

So AWS is very famous for doing cost reductions every month or every year, which is really good.

As their infrastructure grow, and more people use AWS, they have volume and scale, and therefore, they will pass on that economy of scale onto you to get discounts.

Savings Plan

• It’s a new way of saving on your cost in AWS, which is called a Savings Plan. So Savings Plan are a much simpler way to look at your infrastructure on AWS.
• Instead of reserving instances, you just commit to spending a certain dollar amount per hour for the next one or three years.
• It’s easier than the rest because you just think in terms of dollars and you don’t think in terms of specific resources and types and so on.
• Easiest way to setup long-term commitments on AWS

EC2 Savings Plan:

• Up to 72% discount compared to On-Demand
• Commit to usage of individual instance families in a region (e.g. C5 or M5)
• Regardless of AZ, size (m5.xl to m5.4xl), OS (Linux/Windows) or tenancy.
• All upfront, partial upfront, no upfront

Compute Savings Plan:

• Up to 66% discount compared to On-Demand
• Regardless of Family, Region, size, OS, tenancy, compute options
• Compute Options: EC2, Fargate, Lambda

We can set up a saving plan from the AWS Cost Explorer console.

AWS Compute Optimizer

• It reduce costs and improves performance by recommending optimal AWS resources for your workloads.
• It helps you choose optimal configurations and right-size your workloads (over/under-provisioned)
• It uses Machine Learning to analyze your resources’ configurations and their utilization CloudWatch metrics.
• Lower your costs by up to 25% :D

Billing and Costing Tools

In AWS there are tools for almost everything.

Estimating costs in the cloud:

• TCO Calculator
• Simple Monthly Calculator / Pricing Calculator

Tracking costs in the cloud:

• Billing Dashboard
• Cost Allocation Tags
• Cost and Usage Reports
• Cost Explorer

Monitoring against costs plans:

• Billing Alarms
• Budget

Now let’s see one by one in more details.

AWS Total Cost of Ownership (TCO) Calculators

• AWS helps you reduce Total Cost of Ownership (TCO) by reducing the need to invest in large capital expenditures and providing a pay-as-you-go, model
• The TCO calculators allow you to estimate the cost savings when using AWS and provide a detailed set of reports that can be used in executive presentations.
• It compares the cost of your applications in an on-premises or traditional hosting environment to AWS: Server, Storage, Network, IT Labor
• How much cost can incur when moving from on-premises to AWS.

Simple Monthly Calculator / Pricing Calculator

• Simple Monthly Calculator deprecated from service since June 30th, 2020.
• Now it’s replaced by AWS Pricing Calculator.
• It estimates the cost for your architecture solution
• It shows the cost of services on AWS

AWS Billing Dashboard

There are ways you can track your cost in the cloud. The first way is to use the Billing Dashboard, which will show you all the costs that occurred in the month, the forecast, and the month to date.

This is a very high-level tool, but it will be helpful to just get an overview of what’s happening.

It’s the dashboard. You will see billing-related things nicely and visually appealing.

Cost Allocation Tags

• They will allow us to track our costs on a detailed level and group them together.
• In this, we group resources based on tags to create a detailed report.

AWS generated tags:

• Automatically applied to the resource you create
• Starts with Prefix aws: (e.g. AWS: createdBy)

User-defined tags:

• Defined by the user
• Starts with Prefix user

Tags can be used to create Resource Groups:

• Create, maintain, and view a collection of resources that share common tags
• Manage these tags using the Tag Editor

Cost and Usage Reports

• Dive deeper into your AWS costs and usage
• The AWS Cost & Usage Report contains the most comprehensive set of AWS cost and usage data available, including additional metadata about AWS services, pricing, and reservations (e.g., Amazon EC2 Reserved Instances (RIs)).
• The AWS Cost & Usage Report lists AWS usage for each service category used by an account and its IAM users in hourly or daily line items, as well as any tags that you have activated for cost allocation purposes.
• It can be integrated with Athena, Redshift, or QuickSight for analysis purposes. (After all it contains information at a more granular level in excel file)

Cost Explorer

• It’s more of a visual tool
• In cost explorer, we visualize, understand, and manage the cost and usage over time.
• With the help of Cost Explorer, most importantly, we can forecast usage up to 12 months based on the previous usage.
• View current usage (detailed) and forecast usage

Billing Alarms in CloudWatch

• It’s a simple alarm, it’s not as powerful as budgets
• But this will be helpful to send you an email notification.

For example, in case you go over a certain threshold. Let’s say, I set my threshold to $70 and so as soon as I crossed the $70, I received an email notification.

AWS Budget

• Budgets are very helpful to send alarms when the cost exceeds the budget or when a forecast also exceeds the budget.
• Create a budget and send alarms when costs exceed the budget
• There are basically 3 types of budgets: Usages, Cost, Reservation

Trusted Advisor

No need to install anything — It just gives you a high-level AWS account assessment.

It analyses your AWS accounts and provides recommendations on 5 categories (Below mentioned)

1. Cost optimization
2. Performance
3. Security
4. Fault tolerance
5. Service limits

Advanced Identity

AWS STS (SecurityToken Service)

• Enables you to create temporary, limited- privileges credentials to access your AWS resources
• These are short-term credentials: you configure the expiration period.

Use cases:

1. Identity Federation: manage user identities in external systems, and provide them with STS tokens to access AWS resources
2. IAM Roles for cross/same account access
3. IAM Roles for Amazon EC2: provide temporary credentials for EC2 instances to access AWS resources

It’s basically temporary, limited-privileges credentials to access AWS resources

Amazon Cognito

• Identity for your Web and Mobile applications users (potentially millions)
• Instead of creating them an IAM user, you create a user in Cognito
• It just creates a database of users for your mobile & web applications
• Example: Login using Google, Facebook, etc.

Microsoft Active Directory

• With the Microsoft Active Directory, we can use accounts on any organization's laptop or computer.
• With the help of a domain controller, we can use our credentials on any machine.
• We can integrate this into AWS with the help of Active Directory(AD)

AWS Single Sign-On (SSO)

• Centrally manage Single SignOn to access multiple accounts and 3rd-party business applications.
• One login for multiple AWS accounts & applications

Other AWS Services

Amazon WorkSpaces

• It’s managed Desktop as a Service (DaaS) solution to easily provision Windows or Linux desktops.
• Great to eliminate management of on-premise VDI (Virtual Desktop Infrastructure)
• Fast and quickly scalable to thousands of users
• So the idea is that if someone wants you to have a windows laptop in the cloud, they can do so using WorkSpaces.
• It’s secured because it’s integrated with KMS, and it’s pay as you go service, so you only pay for the usage of these actual desktops.

So you’re a user, you’re maybe at home, and you want to access a secure windows desktop from within AWS, you would provision one using workspaces, and then you would get access to your cloud or your corporate data center directly from this virtual desktop that is going to be secure and maybe within your VPC.

Amazon AppStream 2.0

• It’s Desktop Application Streaming Service.
• You want to stream an application to any computer without acquiring and provisioning infrastructure. And the application itself is going to be accessible and delivered from within a web browser.
• The application is delivered from within a web browser

Example:

Say you want to have the Blender application to create the 3D models directly from within a web browser, you can with Amazon AppStream. So it is application-focused, and they’re delivered through your web browser.

Amazon AppStream 2.0 vs WorkSpaces

• WorkSpaces is to give you a full VDI and a full Windows desktop or Linux desktop. So you can launch as many applications as you want within your desktop, and you have to connect directly using your remote desktop application.
• Workspaces are on-demand or always on
• Stream a desktop application to web browsers (no need to connect to a VDI)
• Works with any device (that has a web browser)
• Allow to configure an instance type per application type (CPU, RAM, GPU) Example: You can specify if your apps required more RAMs to run.

Amazon Sumerian

• Sumerian is used to create virtual reality or augmented reality, so VR and AR, and 3D applications.
• It can be used to quickly create 3D models with animations.
• It is Ready-to-use templates based, hence no programming expertise is required.
• Accessible via a web-browser URL or on popular hardware for AR/VR

AWS IoT Core

• IoT stands for “Internet of Things” — the network of internet-connected devices that are able to collect and transfer data
• AWS IoT Core allows you to easily connect IoT devices to the AWS Cloud
• Serverless, secure & scalable to billions of devices and trillions of messages

• Your applications can communicate with your devices even when they aren’t connected
• Integrates with a lot of AWS services (Lambda, S3, SageMaker, etc.)
• Build IoT applications that gather, process, analyze, and act on data

Amazon Elastic Transcoder

• Elastic Transcoder is used to convert media files stored in S3 into media files in the formats required by consumer playback devices (phones etc..)
• Easy to use

• Highly scalable — can handle large volumes of media files and large file sizes
• Cost-effective — duration-based pricing model

AWS Device Farm

• AWS Device Farm is an application testing service that lets you improve the quality of your web and mobile apps by testing them across an extensive range of desktop browsers and real mobile devices; without having to provision and manage any testing infrastructure.
• Device Farm is a real farm of devices
• Run tests concurrently on multiple devices (speed up execution)
• Ability to configure device settings (GPS, language, Wi-Fi, Bluetooth, …)

AWS Backup

• Fully-managed service to centrally manage and automate backups across AWS services
• On-demand and scheduled backups
• Supports PITR (Point-in-time Recovery)
• We can create the the frequency, as well as the retention policy of our backups.

Anytime you see backups at the exam, think AWS Backup.

Disaster Recovery Strategies

CloudEndure Disaster Recovery

• CloudEndure is a company that was acquired by AWS and so, their services are now embedded within these services of AWS.
• Disaster Recovery, as the name indicates, is a way for you to quickly and easily recover your physical, virtual, and cloud-based servers into AWS using a Disaster Recovery Strategy
• You may want to use CloudEndure to protect your most critical databases, such as Oracle, MySQL and SQL Server, your enterprise applications such as SAP, also protect your data in case you’re being attacked.
• The way CloudEndure works is that it does very good backups at the block-level of your server and is continuous.

AWS DataSync

• Move large amount of data from on-premises to AWS
• Replication tasks can be scheduled hourly, daily, weekly
• The replication tasks are incremental after the first full load

So look for that word incremental at the exam. This is going to be a hint.

AWS Fault Injection Simulator (FIS)

• A fully managed service for running fault injection experiments on AWS workloads
• Based on Chaos Engineering — stressing an application by creating disruptive events (e.g., a sudden increase in CPU or memory), observing how the system responds, and implementing improvements
• It’s kind of stress testing in AWS
• It helps you uncover hidden bugs and performance bottlenecks

AWS Architecting & Ecosystem

Well-Architected Framework General Guiding Principles

• In the cloud, when you want to have good architecture, you need to stop guessing your capacity needs. Instead, you should use auto-scaling and scale based on what the actual demand on your system is going to be.
• It’s also very important to automate to make architectural experimentation easier. So for this, cloud formation is very important because if you get infrastructure as code, then you can easily create an architecture on different accounts and different regions.
• You should also allow making your architecture to evolve. So you need to design based on changing requirements.
• Also, improve through game days, so that means that you simulate applications for flash sale days and this will stress your system by stressing your system, you will know if you are doing well.

Well-Architected Framework 5 Pillars

1. Operational Excellence
2. Security
3. Reliability
4. Performance Efficiency
5. Cost Optimization

1st Piller → Operational Excellence

The first pillar in the well-architected framework is going to be operational excellence.

Operational Excellence includes the ability to run and monitor systems to deliver business values and to continuously improve supporting processes and procedures.

It’s about having operations that are great. There are design principles that you need to know about.

Perform operations as code:

All your operations should be code, infrastructure as code is going to be the cornerstone of operational excellence.

Annotate documentation:

You need to annotate documentation, so when you create documentation, make sure it’s annotated, make sure it’s commented, make sure it’s clean and generated after every build of your application.

Make frequent, small, reversible changes:

You need to make frequent and small reversible changes, so that, in case of failures, you can reverse them.

Refine operations procedures frequently:

You need to refine operations procedures frequently, so that the team gets more familiar with the operations, refine it, automate more, and make improvements.

Anticipate failure:

Finally, anticipate failure, that will always happen, and you need to learn from all these failures.

2nd Piller → Security

Security is the second pillar of the Well-Architected Framework.

Security is the ability to protect information, systems and assets while delivering business value through risk assessment and mitigation strategies.

Now let’s look at the design principles for security

Implement a strong identity foundation:

Centralize privilege management and reduce (or even eliminate) reliance on long-term credentials — Principle of least privilege — IAM

Enable traceability:

Integrate logs and metrics with systems to automatically respond and take action

Apply security at all layers:

You need to secure every single layer, such as if one fails maybe the next one will take over. So edge network, VPC, subnet, load balancer, every institute instance you have, the OS, patching it, the application, making sure it’s up to date, all these things

Automate security:

Security is not something you do manually, it’s mostly done well when it’s automated.

Protect data in transit and at rest:

You need to protect data in transit and at rest. That means always enable encryption, always do SSL, always use tokenization and do access control.

Keep people away from data: Reduce or eliminate the need for direct access or manual processing of data

Prepare for security events:

Security events must happen some day in every company, I think, and so run response simulations, use tools to automate the speed of detection, investigation, and recovery

3rd Piller → Reliability

The third pillar of the Well-Architected Framework is reliability.

Ability of a system to recover from infrastructure or service disruptions, dynamically acquire computing resources to meet demand, and mitigate disruptions such as misconfigurations or transient network issues

So it’s about making sure your application runs no matter what.

Now let’s look at the design principles for Reliability

Test recovery procedures:

We need to test recovery procedures so you need to use automation to simulate different failures or to recreate scenarios that led to failures before.

Automatically recover from failure:

We need to automatically recover from failure. That means that you need to anticipate and remediate failures before they occur

Scale horizontally to increase aggregate system availability:

Distribute requests across multiple, smaller resources to ensure that they don’t share a common point of failure

Stop guessing capacity:

Maintain the optimal level to satisfy demand without over or under-provisioning — Use Auto Scaling

Manage change in automation:

You need to basically change everything through automation, and this is to ensure that your application will be reliable or you can roll back, or whatever.

4th Piller → Performance Efficiency

The fourth pillar in the Well-Architecture Framework is performance efficiency.

It includes the ability to use computing resources efficiently to meet system requirements, and to maintain that efficiency as demand changes and technologies evolve

So it’s all about adapting and providing the best performance.

Now let’s look at the design principles for Performance Efficiency

Democratize advanced technologies:

Advance technologies become services and hence you can focus more on product development.

Go global in minutes:

Easy deployment in multiple regions

Use serverless architectures:

Use serverless infrastructure. That’s the golden state. So that means you don’t manage any servers, and everything scales for you, which is really awesome.

Experiment more often:

Easy to carry out comparative testing

Mechanical sympathy:

Be aware of all AWS services

5th Piller → Cost Optimization

The fifth and last pillar in the Well-Architecture Framework is cost optimization.

It includes the ability to run systems to deliver business value at the lowest price point possible

Now let’s look at the design principles for Reliability

Adopt a consumption mode:

Pay only for what you use.

For example, AWS Lambda is one of these services, if you don’t use AWS Lambda you don’t pay for it, whereas RDS if you don’t use your database you still pay for it because you’ve provisioned your database.

Measure overall efficiency:

To measure overall efficiency, we need to use the CloudWatch service from AWS.

Stop spending money on data center operations:

AWS does the infrastructure part and enables customers to focus on organizational projects.

Analyze and attribute expenditure:

Accurate identification of system usage and costs helps measure return on investment (ROI) — Make sure to use tags

Use managed and application-level services to reduce the cost of ownership: As managed services operate at a cloud scale, they can offer a lower cost per transaction or service.

AWS Well-Architected Tool

It’s a free tool to review your architectures against the 5 pillars Well-Architected Framework and adopt architectural best practices.

AWS Right-Sizing

The idea is that because you have so many EC2 instance types to choose from in the cloud, choosing the most powerful one is not the best choice, because the cloud is elastic and you can change the instance type whenever you want.

Right sizing is the process of matching instance types and sizes to your workload performance and capacity requirements at the lowest possible cost.

In the cloud, scaling up is easy so always start small and later scale up as required.

It’s also the process of continuously looking at all your deployed instances. For example, looking at their metrics to identify which are the opportunities to eliminate or downsize without compromising on quality, capacity, performance, and so on, which will, in turn, results in lower cost.

It’s super important to right-size in two moments in your cloud journey. Number one, before you do a cloud migration because it is very common for a company to just migrate to the cloud, puts everything with the biggest instance size and forgets it. This is not the best way. Please right-size before you do cloud migration.

Once you’ve done the climate migration, you need to do it continuously. Maybe once a month, because requirements change over time and it’s important to see if you need to right-size up or right-size down.

CloudWatch, Cost Explorer, Trusted Advisor, 3rd party tools can help in finding the right size for us.

AWS Professional Services & Partner Network

The AWS Professional Services organization is a global team of experts that can help you with AWS.

They will work alongside your team, so your people handling AWS, and a chosen member of the APN.

APN stands for, AWS Partner Network. It’s a network of people, that it was known are good with the cloud.

APN Technology Partners: providing hardware, connectivity, and software

APN Consulting Partners: professional services firm to help build on AWS

APN Training Partners: Find who can help you learn AWS

AWS Competency Program: AWS Competencies are granted to APN Partners who have demonstrated technical proficiency and proven customer success in specialized solution areas.

AWS Navigate Program: help Partners become better Partners

AWS Knowledge Center

Contains the most frequent & common questions and requests

Machine Learning

Amazon Rekognition

• Image Recognition
• Find objects, people, text, scenes in images and videos using ML
• Facial analysis and facial search to do user verification, people counting
• Create a database of “familiar faces” or compare against celebrities

Use cases:

• Labeling
• Content Moderation
• Text Detection
• Face Detection and Analysis
• Face Search and Verification
• Celebrity Recognition
• Pathing (ex: for sports game analysis)

Amazon Transcribe

• Speech to text
• Automatically convert speech to text
• Uses a deep learning process called automatic speech recognition (ASR) to convert speech to text quickly and accurately

Use cases:

• transcribe customer service calls
• automate closed captioning and subtitling
• generate metadata for media assets to create a fully searchable archive

Amazon Polly

• Text to Speech
• Turn text into lifelike speech using deep learning
• Allowing you to create applications that talk

Amazon Translate

• Language translation service
• Natural and accurate language translation
• Amazon Translate allows you to localize content — such as websites and applications — for international users, and to easily translate large volumes of text efficiently

Amazon Lex & Connect

Amazon Lex

• It’s the same technology that powers Alexa
• Automatic Speech Recognition (ASR) to convert speech to text
• Natural Language Understanding to recognize the intent of text, callers
• Helps build chatbots, call center bots

Amazon Connect:

• Receive calls, create contact flows, cloud-based virtual contact center
• Can integrate with other CRM systems or AWS
• No upfront payments, 80% cheaper than traditional contact center solutions

Amazon Comprehend

• For Natural Language Processing — NLP
• Fully managed and serverless service
• Uses machine learning to find insights and relationships in text

Examples: Language of the text, Extracts key phrases, places, people, brands, or events, Understands how positive or negative the text is, Analyzes text using tokenization and parts of speech, Automatically organizes a collection of text files by topic

Use Cases:

1. analyze customer interactions (emails) to find what leads to a positive or negative experience
2. Create and groups articles by topics that Comprehend will uncover

Amazon SageMaker

• Fully managed service for developers / data scientists to build ML models
• Typically, difficult to do all the processes in one place + provision servers

Amazon Forecast

• Fully managed service that uses ML to deliver highly accurate forecasts
• Example: predict the future sales of a raincoat
• 50% more accurate than looking at the data itself
• Reduce forecasting time from months to hours
• Use cases: Product Demand Planning, Financial Planning, Resource Planning

Amazon Kendra

• Fully managed document search service powered by Machine Learning
• Extract answers from within a document (text, pdf, HTML, PowerPoint, MS Word, FAQs…)
• Natural language search capabilities
• Learn from user interactions/feedback to promote preferred results (Incremental Learning)
• Ability to manually fine-tune search results (importance of data, freshness, custom, …)

Amazon Personalize

• Personalized recommendations service
• Fully managed ML-service to build apps with real-time personalized recommendations
• Example: personalized product recommendations/re-ranking, customized direct marketing
• Same technology used by Amazon.com
• Integrates into existing websites, applications, SMS, email marketing systems
• Use cases: retail stores, media and entertainment

Credits:

As I said in the beginning, the reason that I have created this post to bring the best materials that available on internet to one consolidated place. I have referred following materials to create this blog posts for you.

Stephane Maarek’s AWS CCP Course on Udemy

This is the best thing happened with me that I have taken his course to prepare this exam. His knowledge and ability to explain and demonstrate things about AWS is simply great. I am sure you will enjoy his friendly teaching. I recommend you to buy his course on Udemy

Link: https://www.udemy.com/course/aws-certified-cloud-practitioner-new/

AWS Certified Cloud Practitioner Study Guide Book By Ben Piper & David Clinton

If you wish to read in-depth services that AWS offers, this is for you.

AWS Offcial Documentation

There are tons of knowledge lying already on AWS documentation page. You just need to do simple Google Search.

Follow Akash Dugam for more data science related blog posts.

1. What is Hypothesis Testing?
2. Steps involved in Hypothesis Testing
3. Defining Null & Alternate Hypothesis
4. Choosing Test Statistics
5. Decision Making Through P-Value
6. Drawing Conclusion About The Population
7. Examples

Let’s understand each of these concepts in a detailed manner.

What is Hypothesis Testing?

The main objective of statistics is to test a hypothesis. Let’s take an example, you are a medical researcher and by doing an experiment you found out that the drug ‘ABC’ is much effective against treating fever. But to take this into account you’ve to perform this experiment multiple times (could be with different samples) If not, no one will consider your results accurate.

What is a Hypothesis?

A hypothesis is a falsifiable claim that requires verification, typically from experimental or observational data, and that allows for predictions about future observations.

Example:

Imagine you work in the loan approval department. A bank has introduced a new credit score-based loan approval process. Before fully rolling out this system, the management wants to know if the new process improves loan repayment rates compared to the current system.

• Null Hypothesis (H₀): The new process does not improve repayment rates.
• Alternative Hypothesis (H₁): The new process improves repayment rates.

You will use hypothesis testing to evaluate if the observed data (repayment rates from a pilot study) provide enough evidence to support implementing the new system bank-wide.

Why are Hypotheses important?

Below reasons sheds some light on why are hypotheses important in the area of statistics.

• Hypotheses improve experiment design, critical thinking, and data analysis.
• It’s not possible to do clear and meaningful data analysis without a correct hypothesis.
• Hypotheses transform loose ideas or speculation into concrete and specific claims.
• Hypotheses are used to develop new and more accurate theories and dissolve bad theories.
• Most progress in science, engineering, medicine, and technology is the result of hypothesis testing.

Types of Hypotheses

Hypotheses can be divided into 2 types. That is →

• Strong Hypothesis
• Weak Hypothesis

There are some characteristics of Strong Hypotheses that need to understand before we categories statements or claims into Strong or Weak Hypothesis.

Characteristics of Strong Hypothesis →

• It should be clear
• It should be specific
• It should be falsifiable means it can be proved wrong
• A good hypothesis is based on prior data and theory.
• A good hypothesis leads to a statistical test. Meaning, there should be a way to test the hypothesis.
• It has to be a statement, not a question.
• A prediction about the direction of an effect.
• A good hypothesis is relevant for unobserved data too.

Now based on the above information let’s classify the below statements into ‘Strong’, ‘Weak’ or ‘Not a hypothesis’ categories.

Statement 1 →

Medical research is important for curing diseases.

The above statement is correct but this is a weak hypothesis as it doesn’t provide information about data, statistics etc.

Statement 2 →

The medication has an effect.

The above statement falls into the ‘Weak Hypothesis’ category. This statement can be used to falsify the claim but it’s not a clear and specific statement to be categorized into the ‘Strong Hypothesis’ category.

Statement 3 →

Will students pass this exam?

This is a question and not a statement. Thus it’s not a hypothesis.

Statement 4 →

Studying improves grades.

This is a hypothesis. But It’s not clear and specific. Thus it will be categorized into a weak hypothesis.

Statement 5 →

A combination of self-study and group study will improve final exam grades by at least 10%

This is a clear and specific statement. It will be categorized into a strong hypothesis.

Statement 6 →

Washing hands for 20 seconds reduces disease spread.

This is a clear and specific claim. It can be tested. Thus we will label it as a strong hypothesis.

I hope you’ve understood the hypotheses and types of hypotheses. Now it’s time to understand what exactly mean by hypothesis testing?

• Hypothesis testing is a statistical method that is used in making statistical decisions using experimental data.
• Hypothesis Testing is an assumption that we make about the population parameter. This assumption may or may not be true. Hypothesis testing refers to the formal procedures used by statisticians to accept or reject statistical hypotheses.
• In other words, we can say, hypothesis testing is nothing but testing whether or not a claim is valid.

Examples: →

(A) Most people get their job from networking.

P > 0.50 …. ‘Most’ keyword used hence it is proportion. Most means, more than half.

(B) The average payload of trucks on the highway is 18000 lbs.

μ = 18000 …. The average was given hence it’s μ.

(C) The pharmaceutical company X recently launched a gender selection drug, where they claim that if consumed, there are 80% of chances that women will have a baby girl. For this girl, 100 couples were tested.

Let’s state the assumption here.

“This drug doesn’t work and the probability of having a boy or a girl is 50%.”

• Here we are not assuming or accepting the drug works. Because in the statistics you cannot prove something is correct.
• To prove something is correct, we first need to take reciprocal of it and then try to prove that reciprocal is wrong which ultimately proves something is correct.

Let’s assume we tested this drug on 2 batches of 100 couples and the result is below,

• 52/100 had a girl…. It’s a very usual scenario
• 97/100 had a girl…. It’s a very unusual scenario

In the first case, we can call it a usual case as the probability is almost 50%. But in case 2 which is a very unusual scenario as the probability of having a baby girl is 97% which is quite huge compared to what we’ve assumed. Thus the second case proves our assumption wrong and we will be able to say that drug works.

Note →

The best way to determine whether a statistical hypothesis is true would be examining the entire population. But examining the entire population is often impractical. Therefore researchers tends to examine the random sample from the population. If the sample data is not consistant with statistical hypothesis, the hypothesis is rejected.

Steps involved in Hypothesis Testing

Defining Null & Alternate Hypothesis

In this step, we need to state the two hypotheses from the statement. There are two types of statistical hypotheses.

• Null Hypothesis / H0
• Alternative Hypothesis / H1

Null Hypothesis (H0)

• The “Null Hypothesis” is the hypothesis that nothing interesting is happening in the data.
• This is the assumption that there is no effect or no difference. It represents the “default” or “status quo” scenario.
• The “Null Hypothesis” is an assumption made about the population which needs to be tested. While we test the assumption about population parameter the null hypothesis considered to be true until evidence found against it.
• Null hypothesis states that the population parameter (μ, P) is equal to some value. H0: μ = 5 (or) H0: P = 0.5
• It is denoted as H0.
• We start by assuming H0 is true then we use the evidence to conclude.

Reject H0 → When we’ve enough evidence to prove H0 is wrong.

Failed to Reject H0 → When we don’t have enough evidence to prove H0 is wrong.

Note → You cannot accept the null hypothesis. You either reject it or fail to reject it depending upon the evidence.

Alternative Hypothesis (H1)

• It is opposite to the assumption made i.e. the null hypothesis. It is automatically accepted when the null hypothesis gets rejected.
• This is what you aim to prove. It suggests that there is an effect, difference, or relationship.
• In the research, you specify the “alternative hypothesis”
• It should be called an “Effect Hypothesis”
• In statistical analysis, you never test the alternative hypothesis. You only test the null hypothesis.
• It is denoted as H1.
• The alternative hypothesis states that the population parameter (μ, P) has a different value than H0…. H1: μ ≠ 5 (or) H1: P > 0.6
• If you ‘reject’ the null hypothesis that means indirectly you accept the alternative hypothesis.
• If you ‘fail to reject the null hypothesis that means you failed to accept the alternative hypothesis.

Examples:

Scenario 1: You work at a bank that recently introduced a loyalty program to increase the average spending of credit card customers. You want to evaluate if the program has had an impact.

• H₀: The loyalty program does not increase the average monthly spending of customers. (No difference)
• H₁: The loyalty program increases the average monthly spending of customers. (There is a difference)

Scenario 2: A bank introduces a new loan approval policy to reduce loan defaults.

• H₀ (Null Hypothesis): The new loan policy does not reduce the default rate.
• H₁ (Alternate Hypothesis): The new loan policy reduces the default rate.

Scenario 3: A bank introduces a new loan approval policy to reduce loan defaults.

• H₀ (Null Hypothesis): The new loan policy does not reduce the default rate.
• H₁ (Alternate Hypothesis): The new loan policy reduces the default rate.

Scenario 4: A bank launches a promotional campaign offering cashback on purchases. It wants to check if the campaign increases monthly credit card spending.

• H₀: The promotion does not increase average monthly credit card spending.
• H₁: The promotion increases average monthly credit card spending.

Scenario 5: The bank introduces a new fraud detection system and needs to check if it reduces the incidence of fraudulent transactions.

• H₀: The fraud detection system does not reduce fraudulent transactions.
• H₁: The fraud detection system reduces fraudulent transactions.

Let’s understand the above two concepts with the help of examples.

• A medical trial is conducted to test whether or not a new medicine reduces cholesterol by 25%.

H0: Medicine reduces cholesterol by 25%. | P = 0.25

H1: Medicine does not reduce cholesterol by 25%. | P ≠ 0.25

• We want to test whether the mean GPA of students in American colleges is different from 2.0 (out of 4.0).

H0: Average GPA of students in American colleges is 2.0 | μ = 2.0

H1: Average GPA of students in American colleges is not equal to 2.0 | μ ≠ 2.0

Choosing Test Statistics

• Test statistics are used in hypothesis testing to make a decision on ‘Null Hypothesis’ → Whether to Reject the Null Hypothesis or Fail to Reject Null Hypothesis based on the evidence.
• It's a number is that is calculated from a statistical test of the hypothesis. It shows how far is your observed data from the H0
• To decide on your hypothesis, we need to calculate test statistics. Since test statistics is a number, it could be p-value or critical value (z-value, t-value, F-value or x²-value )

Scenario: You want to determine if the average loan approval time at your bank has decreased after introducing a new automated system.

• You collect data on loan approval times before and after implementing the system.
• A test statistic is the calculated value that tells you whether the observed difference in times is statistically significant or likely due to random variation.

p-value Interpretation

It’s very important to understand ‘p-value’ if you are studying statistics. The majority of the decision is based on the interpretation of the p-value. And, let me tell you, this is the most confusing topic in the whole of statistics. There are many places where I have seen people misquote the p-value. I will take a simple example to explain to you what exactly is the p-value.

A Coin Toss Example → Find out whether it is a trick coin or not.

H0: It’s a fair Coin.

H1: It’s a trick Coin.

Now we have defined our H0 and H1. The next thing would be to experiment.

• Let’s toss a coin the first time and observe the result. Let’s say, you got a tails on the first attempt.

Based on this first result should we call it a trick coin? We would probably not. Since we consider the null hypothesis is true which state that it's a fair coin and if it is a fair coin then getting tails on a coin toss is random. With this limited information, we cannot just reject the H0.

Therefore, P(1T | H0) = 0.50

• Let’s toss a coin a second time and observe the result. Let’s say, you got a tails on the second attempt.

Based on the first two results, should we call it a trick coin? By considering H0 is true, there is a 25% chance that it could happen due to randomness. Hence we cannot reject the null hypothesis.

Therefore, P(2T | H0) = 0.25

• Let’s toss a coin for the third time and observe the result. Let’s say, you got a tails on the third attempt as well.

Based on the first three results, should we call it a trick coin? By considering H0 is true, there is a 12.5% chance that this could’ve happened due to randomness. We are not yet convinced to call it a trick coin.

Therefore, P(3T | H0) = 0.12

• Let’s toss a coin for the fourth time and observe the result. Let’s say, you got a tails on the fourth attempt too.

Based on the first four results, should we call it a trick coin? Now things get a little suspicious. By considering H0 is true, there is only a 6% (approx.) chance that this would’ve happened due to random chance.

Therefore, P(4T | H0) = 0.06

Let’s toss a coin one more time to confirm it with proof.

• Let’s toss a coin for the fifth time and observe the result. Let’s say, you got a tails on the fifth attempt as well.

Now should we call this a trick coin, after this conducted experiment? By considering H0 is true, there is only a 3% chance that this would’ve happened due to random chance.

Therefore, P(5T | H0) = 0.03

Since the probability of happening this randomly is less, we would reject the ‘Null Hypothesis’ and consider this coin is a trick coin and biased towards the tails.

The resultant probability that we got is 3% which is less than the threshold value that is 5% (We will see how do we decide this threshold later in this blog). This leads us to reject H0 and accept the alternative hypothesis.

This probability is nothing but the p-value in hypothesis testing. When the probability of observation by considering H0 is true falls below the threshold value/alpha/level of significance (i.e. 0.05) we reject the null hypothesis and accept the alternative hypothesis.

In this example, we’ve calculated the p-value easily with the help of various probability concepts. But with increasing sample size it won’t be easy to calculate the p-value with basic probability concepts. In such cases, we mostly use a technique called ‘Permutation Testing’ to calculate the p-value.

Calculating P-Value: Permutation Testing

Let’s understand Permutation Testing with the help of one example.

People who follow strict diet with gym have significant impact on weight loss than people who only goes to gym.

Let’s consider the two sets of samples with sample size 50 that we gathered in A and B. The initial weight of all the participants was 80KG.

A → Contains sample of those who follow gym routine with a strict diet

B → Contains a sample of those who follow only gym routines.

Let me define what is the P-value in the above case.

The P-Value is the p( Mean Difference in Weight ≥ 6KG | H0) which states that the probability of finding a difference (Let’s call it Δ ≥ 6KG) in mean weight when the null hypothesis is considered true. Now let’s look at calculating this probability i.e. p-value with the help of permutation testing.

Thus p-value literally is the percentage of values greater than Δ. In the above example is 3% hence the p-value is 0.03. Now, what does this p-value tell us? Should we reject the null hypothesis right now? …Nooh, we shouldn’t as we haven’t discussed yet how to make a decision on the basis of the p-value. We will learn how to make decisions on the basis of p-value later in this blog post but before that let’s learn something that we haven’t learnt till now. The p-value alternatives that is z-value & t-Value.

z-value and t-value

In hypothesis testing, we can conclude the hypothesis with the help of z-value or t-value too. Not necessarily we require a p-value always.

How to make a decision?

As we’ve stated already, we need to have test statistics (p-value or z-value or t-value) to make a decision about our hypothesis. We will learn about the decision-making process in this section.

In the decision-making process, we need to have the information about the below concepts →

Confidence Interval

• It is a ‘range’ that is used to estimate a population parameter.
• Confidence Interval has something called ‘Confidence Level’ associated with this.

Confidence Level

• The confidence level tells that ‘ How confident are you that the actual value of the population parameter will be lying inside the range or interval.
• This confidence level is expressed as ‘1-α’ where α is the compliment of a confidence level.
• There are mostly 3 confidence levels that we deal with

• 0.95 or 95% CI is mostly used in statistics. (Marked in Green above)

Critical Value

• In hypothesis testing, a critical value is a point of scale or graph used to split a graph into sections like ‘rejection region’ or ‘non-rejection region’. If your test statistic falls into that region we reject the null hypothesis.
• It is derived from the level of significance α i.e. z = α / 2
• A z-score that separate the ‘Likely Region’ from the ‘Unlikely Region’
• We do not calculate z-score for when the population parameters value is unknown. Instead, we calculate something called a t-score.

The critical value is the value which seperate the rejection region from the region of acceptance.

The critical value serves the boundary value (or condition) to test statistics. If the test statistic falls in the region of rejection (α) which is test statistics > critical value then we reject the H0.

If the test statistic falls into the region of acceptance (1-α) which is nothing but test statistics < critical then we fail to reject H0. The above example is right-tailed hence the critical value is located on the right side. This could be a position of left as well as on both positions also.

Note: The critical values are highly affected by the left tail test, right tail test or two-tail test. See the above table for more information.

Making Decision based on p-value / z- value / t-value

Note → Let’s consider the most used confidence level 95% or 0.95

Let’s consider the below values:

• z-value = 3.81 | z-critical = 1.96
• p-value = 0.03
• t-value = 0.87 | t-critical = 1.96
• α = 0.05
• 1-α = 0.95

With z-value: The z-value > z-critical, we reject the null hypothesis.

With t-value: The t-value < t-critical, we fail to reject the null hypothesis.

With p-value: The p-value < α, we reject the null hypothesis.

In modern computing environment nowadays it is possible to calculate p-value and majority of tools / softwares return p-value for us. Therefore, p-value as test statistics used by most of the people to make decision about hypothesis testing.

Errors in Statistical Test

In statistical hypothesis testing, no test is ever 100% certain as we rely on p-value which is based on probabilities, there is always a chance of making an incorrect conclusion regarding accepting or rejecting the null hypothesis.

Remember, when we make decisions based on statistics there are a total of 4 outcomes possible.

• True Positive
• True Negative
• False Positive
• False Negative

The chances of committing these two types of errors are inversely proportional: that is, decreasing type I error rate increases type II error rate and vice versa.

TYPE I ERROR: → FALSE POSITIVE

• A type 1 error is also known as a false positive. It occurs when a null hypothesis is rejected, even though it is accurate and should not be rejected.
• The probability of making a type 1 error is represented by your alpha level or confidence level. (α)
• A p-value of 0.05 indicates that we are willing to accept 5% chance that we are wrong when we reject the null hypothesis.
• We can reduce the risk of committing a type 1 error by using a lower value for p. For example, a p-value of 0.01 would mean there is a 1% chance of committing a Type I error.
• But, using a lower value for alpha means you are less likely to detect true difference if one really exists. Thus risking type 2 error.
• In terms of the courtroom example, a type I error corresponds to convicting an innocent defendant.

TYPE II ERROR: → FALSE NEGATIVE

• A type 2 error is also known as a false negative.
• It occurs when the researcher fails to reject the null hypothesis which is really false.
• Here a researcher concludes there is no significant effect when actually there really is.
• The probability of making type 2 error is called Beta (β) which is related to the power of the statistical test. (1-β)
• We can decrease the risk of committing type 2 errors by ensuring the test has enough power. We can do this by ensuring the sample size is large enough to detect practical differences when one truly exists.
• In terms of the courtroom example, a type II error corresponds to acquitting a criminal

CROSSOVER ERROR RATE (CER)

• The crossover error rate is the point at which both the errors that is type 1 error and type 2 error are equal.
• It represents the best way of measuring biometrics effectiveness. A system with a lower CER value provides more accuracy than a system with a higher CER value.

Python Code for Calculating P-Value Using Permutation Testing

treatment = [ 28.44,  29.32,  31.22,  29.58,  30.34,  28.76,  29.21,  30.4 ,
              31.12,  31.78,  27.58,  31.57,  30.73,  30.43,  30.31,  30.32,
              29.18,  29.52,  29.22,  30.56]

control = [ 33.51,  30.63,  32.38,  32.52,  29.41,  30.93,  49.78,  28.96,
            35.77,  31.42,  30.76,  30.6 ,  23.64,  30.54,  47.78,  31.98,
            34.52,  32.42,  31.32,  40.72]

from mlxtend.evaluate import permutation_test

p_value = permutation_test(treatment, control,
                           method='approximate',
                           num_rounds=10000,
                           seed=0)
print(p_value)

if p_value <= 0.05 :
    print("Null Hypothesis is rejected and Accepted the alternate hypothesis")
if p_value > 0.05 :
    print("Failed to reject the null hypothesis")

Output →

0.0066993300669933005
Null Hypothesis is rejected and Accepted the alternate hypothesis