Data orchestration is the backbone of modern data engineering, ensuring workflows run smoothly, dependencies are managed, and errors are handled gracefully. Among the myriad of tools, Apache Airflow and Dagster stand out as two powerful contenders. Both tools automate workflows, but their design philosophies and capabilities differ significantly. In this article, we’ll compare these tools with examples, real-world use cases, and insights from companies to help you make an informed decision.

What Is Data Orchestration?

Data orchestration is the process of automating, scheduling, and managing data workflows. A typical workflow might include tasks like:

• Extracting Data: Pulling data from APIs, databases, or files.
• Loading Data: Storing data in a data warehouse or database for analysis.
• Transforming Data: Cleaning, aggregating, or modeling data.

An orchestration tool ensures these tasks run in the right order, handle failures gracefully, and provide monitoring and logging for debugging.

The Contenders

Apache Airflow

Developed by Airbnb in 2014, Apache Airflow is a powerful, Python-based orchestration tool. It uses Directed Acyclic Graphs (DAGs) to define workflows as Python code. Airflow is widely adopted across industries and is known for its flexibility and ecosystem.

Key Stats (as of December 2024):

• GitHub Stars: ~32.7k
• Forks: ~13.6k
• Contributors: ~2.5k

Notable Users:

Airbnb, Uber, Slack, and Lyft rely on Airflow for orchestrating complex workflows.

Features:

• Extensive Ecosystem: Includes prebuilt operators for major cloud providers and services (e.g., AWS, Google Cloud).
• Dynamic Workflows: Python-powered workflows allow dynamic task generation.
• Mature Scheduler: Handles large-scale, distributed workflows.

Dagster

Introduced in 2019 by Elementl, Dagster is a modern orchestration tool that emphasizes data awareness, type safety, and developer experience. Dagster reimagines workflows as modular, testable units of computation called ops and graphs.

Key Stats (as of December 2024):

• GitHub Stars: ~6.9k
• Forks: ~750
• Contributors: ~250

Notable Users:
Companies like Wayfair, Convoy, and Elementl use Dagster for data orchestration.

Features:

• Data-Aware: Tracks inputs, outputs, and metadata natively.
• Type Safety: Reduces runtime errors by enforcing strong typing.
• Integrated Testing: Simplifies testing and debugging workflows.
• Modern UI: Provides rich visualizations for monitoring workflows.

Hands-On Comparison

To better illustrate the differences, let’s implement a simple data pipeline that:

1. Extracts sales data from an API.

2. Loads the data into PostgreSQL.

3. Generates a summary report.

Apache Airflow Example

Here’s how you might define this pipeline in Airflow:

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime
def extract_data():
  print("Extracting data from API…")
  return {"sales": 1000, "revenue": 50000}
def load_to_postgres(data):
  print(f"Loading data to PostgreSQL: {data}")
def generate_report():
  print("Generating sales report…")
with DAG('sales_pipeline', start_date=datetime(2024, 1, 1), schedule_interval='@daily') as dag:
  extract = PythonOperator(
    task_id='extract',
    python_callable=extract_data )
  load = PythonOperator(
    task_id='load',
    python_callable=lambda: load_to_postgres(extract.output))
  report = PythonOperator(
    task_id='report',
    python_callable=generate_report)
  
extract >> load >> report

Dagster Example

Here’s the same pipeline in Dagster:

from dagster import job, op, Out

@op(out=Out(dict))
def extract_data():
  print("Extracting data from API…")
  return {"sales": 1000, "revenue": 50000}

@op
def load_to_postgres(data):
  print(f"Loading data to PostgreSQL: {data}")

@op
def generate_report():
  print("Generating sales report…")

@job
def sales_pipeline():
  data = extract_data()
  load_to_postgres(data)
  generate_report()

Dagster Advantages:

• Type Safety: The @op decorator enforces that data passed between tasks has the correct type.

• Data Visibility: Dagster’s UI shows data passing through the pipeline, making debugging easier.

Real-World Insights

Apache Airflow in Action

• Lyft uses Airflow to orchestrate ETL pipelines that process massive ride-sharing data. Airflow’s scalability and robust ecosystem were critical for managing thousands of tasks daily.

• Slack employs Airflow to manage workflows for data analytics and customer insights, leveraging its integration with cloud services like AWS and Snowflake.

Dagster in Action

• Wayfair adopted Dagster for its machine learning workflows, valuing the tool’s modularity and ability to track data lineage.

• Convoy, a logistics company, uses Dagster to manage data pipelines for optimizing freight operations. Dagster’s type safety reduced runtime errors, improving reliability.

Feature-by-Feature Comparison

-------------
| Feature           | Apache Airflow                        |Dagster
| Ease of Use       | Steep learning curve, script-heavy    | Developer-friendly, modular design||
| Type Safety       | No native support                     | Strongly enforced
| Data Awareness    | Limited                               | Native tracking of datasets
| Ecosystem         | Extensive integrations and operators  | Growing but smaller
| Flexibility       | Fully dynamic workflows in Python     | Composable, reusable components
| Monitoring        | Logging-based                         | Rich metadata and visualizations
| Community Support | Large, active, and established        | Rapidly growing, smaller community

When to Use Each Tool

Choose Apache Airflow If:

• You need extensive integrations with third-party tools (e.g., AWS, GCP).

• Your team is experienced with Python and can manage dynamic workflows.

• Scalability and community support are priorities.

Example:

A global enterprise like Uber uses Airflow for orchestrating ETL jobs across distributed systems, where the tool’s robust scheduling capabilities and large community are invaluable.

Choose Dagster If:

• You want data-aware orchestration with native type safety.

• Testing, debugging, and observability are critical.

• You’re building modern data pipelines or machine learning workflows.

Example:

A startup developing machine learning models might adopt Dagster for its ability to track data lineage and enforce correctness in pipelines.

Final Verdict

Both Airflow and Dagster are excellent tools, but they cater to different needs:

• Apache Airflow is the industry standard for large-scale data workflows, with a robust ecosystem and a mature community.

• Dagster is a modern alternative focusing on developer experience, data observability, and type safety, ideal for teams building modular, testable pipelines.

Summary

• For traditional ETL workflows: Go with Airflow.

• For modern, composable pipelines: Choose Dagster.

What has your experience been with Airflow or Dagster?

References:

• Apache Airflow GitHub

• Dagster GitHub

• Airbnb Engineering Blog on Airflow

• Dagster Success Stories

A summary of Dr. Stylianos Kampakis’ book on Data Science, covers the history, now, with great insights on how to apply in your business, from data strategy to data management, from hiring to building the data culture.

We will be hosting Dr. Stylianos Kampakis on our Agile Data London meetup this month, and I wanted to give a brief summary on his book as I found it very useful for many practical reasons, join us if you would like to ask your questions live.

Building a data science team can be tricky, from the idea creation to setting up the team, and going to live, and Kampakis’ book can help you to start your journey with good questions. Rather than a summary of chapter by chapter, I will go through the top three topics on his Data Strategy Canvas, which is also available as a free tool on his website. It is different than Louis Dorard’s Machine Learning Canvas, and combining both can also result in a powerful tool…

Data Strategy describes a “set of choices and decisions that together, chart a high-level course of action to achieve high-level goals.”

1. Challenge: What is the problem you are trying to solve? The trick here is to state the challenge as a question, so that everyone in your team can look at from the same lenses, and be encouraged to ask more question about the challenge.
2. Data Sources: It is critical identifying the sources and usefulness of them, either quantitate or qualitative, experiments, surveys, observations. Kampakis highlighted several issues with data collection. First one is the data not being recorded properly, either human error or noisy sensors. Second one is that when data collection is not supervised, monitored, verified, it can result in noise and mistakes. Third issue is data not being clear, without a descriptive documentation about itself, which can create misunderstandings and miscommunication, especially for new entrants. Forth one is about the legal requirements, where data is not stored compliant with the legislations, like GDPR, which can result in huge fine. Fifth one is the ownership of the collection/storage of the data rather a centralised system. And lastly, the consolidation issues where multiple data sources stored in multiple locations can make it tricky to create a unified view of your data.
3. Data Acquisition: Deciding which data we should acquire can be tricky too. Kampakis suggests four dimensions: appropriateness of the data, nature of the data, time requirement, cost of acquisition. They involve asking the questions like what the right dataset look like, how much bias and noise my dataset will have, how long time do we need to acquire the data to get ready to be consumed and how much does it cost for the data to be available when we want serve. The tradeoff should be all considered as part of the process.
4. Methods: You can start asking which methods will be used to hire the right data scientist with the right skillset, or after the hire, to explore the options. Each business problem can be unique and may require simpler algorithms, so a thoroughly discussion can help you to stop you from boiling the ocean, when a simple unsupervised learning algorithm can save you time and cost from deep learning investment.
5. Success criteria: What good looks like? A common understanding and acceptance of the goal by the whole team will empower them, and when it has milestones broken down, it will help the team to navigate through the trenches with confidence.
6. People: There are so many titles, roles, suggested by data community, and Kampakis has another take on this. He does not put Software Engineer to his diagram, but a Hacker, as he believes the quality of the code that data scientists produce does not need to be production level. Data scientists try to solve a problem with just enough tooling. Do you agree?

I hope it created bulb light moment for your plans or will engage your team in good conversations to look at the problems from different angles. I am looking forward to your comments. What do you think about Kampakis Data Strategy Canvas? Which one was your favourite one? What do you think about the hackers?

We want high-quality business analytics, a better “process of discovering meaningful and actionable insight in data” as Prof Dursun Delen describes in his book Prescriptive Analytics, as we want to make better decisions and get better at problem-solving by using mathematical/statistical models or machine learning/deep learning algorithms, technologies, tools and practices.

There have been many attempts to suggest a rational human decision-making process, and the best one was by Herbert Alexander Simon (1916–2001), who was an American economist and political scientist, received the Turing Award in 1975, and Nobel Prize in Economics in 1978.

For a successful outcome, the decision-making process should follow standardized, systematic, and logical steps. H. A. Simon, in 1977 came up with the theory that involves intelligence, design, choice, implementation, and a form of feedback between each phase. Simon’s theory is yet the most concise and complete theory on evidence-based, scientific decision-making.

Humans do willingly settle with something less than the best, satisficy Simon calls, as they behave in bounded rationality, as they have a limited capacity for rational thinking, the best solution they come up with may not be the real-world best solution. This is also another reason why many managers go with their gut feelings. My dear friend Emrah has a great book on this, The Myth of Experience, if you want to read more on this.

Briefly, the intelligence phase is about understanding the goals and objectives, and problem identification, problem classification and problem ownership. The output of this phase is a clear problem statement.

The second phase design is about model abstraction, model formulation, alternative generation, and criteria identification. This phase results in a set of alternatives to be considered. It does validate the problem statement.

The third phase choice is about evaluating alternatives, ranking them, and selecting the best one. This is where sensitivity analysis is done to verify and test the alternative along with what-if analysis statement and planning the implementation.

The fourth phase is implementation, it is doing what it takes, taking action and understanding the impact. The results of the implementation will be taken into account for the next phase of the cycle.

Now, we have a better understanding of human-rationale on decision making, let’s look at the evolution of the tools and the technology over the years to enable us to make better decisions.

Short History of Decision-Making Systems Evolution

Prior to the 1970s, Operational Research (OR) dominated the industry with heuristic methods like simulation models. With the introduction of rule-based Expert Systems (ES) were introduced and “if/then” statements regulated the intelligent decision algorithms.

In the 1980s, Enterprise Companies needed to unify the data from different departments, into a single source of truth with a schema. Database Management Systems and ERP systems emerged helping the data to be integrated, referenced. With easy access to data, a basic level of static, on-demand generated reports were giving an overall view of the system.

In the 1990s, with the introduction of the data warehouses, Prism being pioneer, Inman and Kimball published the go-to books for data warehousing, namely Building the DataWarehouse and The DataWarehouse Toolkit. Business executives were now aware of their key performance indicators with decision support systems and were able to track the results with the visual support of dashboards, scorecards.

The 2000s came with the branding of the Business Intelligence on the DW-driven decision support systems, and the amount of the data was big enough to require “mining” the data to “discover” the meaning behind the numbers. The increase demand/overhead of consistently patching, updating DW infrastructure enabled/benefited from the IaaS movement. Software systems were also getting complicated, and needed to split up and create contracts for each communication, where Service Oriented Architecture emerged to address these requirements.

In the 2010s, a variety of data sources contributed to the data ecosystem to be overwhelmed from smart meters to smart health monitors, and social media feeds, and “Big Data” became a thing everyone was part of it. Massively Parallel Computers left their places to Hadoop better Map Reduce programming, with built-in NoSQL query power.

The 2020s challenged the limitation of batch processing, and real-time analytics is at the core of any transaction, with a requirement of full automation of the pipelines to shorten the feedback loops and provide value sooner.

Types of Analytics

The most basic type of Analytics is Descriptive Analytics. This is about collecting data from the information systems designed to support the decision-making process by running reports from Data Warehouses, Data Lakes, Data Lakehouses. Any reports explain the causalities and correlations attempts to explain “What happened?”. It gives an insight and the dashboards, monitoring systems display the metrics to enable the right decisions.

One level more complicated than just aggregating data and doing number crunching is Predictive Analysis, which answers “What will happen?” and comes up with recommendations. The power of predicting customer behaviour/demand, financial market movements, and stock market prices is priceless. Data mining use exploratory analysis with multi-dimensional cubes on high-performance infrastructure and ML models use unsupervised learning (as in where you don’t know the feature names you collected) to predict or cluster the information via classification, regression, or time-series forecasting. Rolls-Royce has an Intelligent Engine Health Management (EHM) system that tracks engine health worldwide by the generated hundreds of terabytes of data from on-board sensors and lives satellite feeds. It is a form of predictive monitoring that flags potential threats and recommends engine improvements. They have extended the system to achieve 14% more fuel-efficiency in their aviation.

Prescriptive Analytics uses sophisticated optimization, simulation, and heuristics-based decision-modelling techniques to answer “What should I do?” so that the system capable of thinking instead of humans. They execute the reinforcement systems, along with other applied statistics, operations research, machine learning, natural language processing, image processing, speech recognition, and signal processing. Kensho is one of the most significant ones augmenting the ability to analyse the impact of the real-world events on the financial markets and be able to answer complex financial queries, categorise events, which is an average salary of $350k-$500k employee.

Hope it will clarify the confusion of the Analytics, with the historical background on decision making systems, models, as well as examples. With reinforcement systems in place, I reckon we need a new word for the automated version of Prescriptive Analytics, (Automated Analytics sound very cliche) so that we can identify the lines in between where the recommendation is and where the action is taken. As we have trusted our satisficied decisions, maybe we can be trusting the systems learning from their feedback loops…

If you have any questions, please leave a comment, and if you did like, don’t forget to follow me. Until the next one, take care!

Where Advanced Analytics is coming from and where it is heading up to? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to process big data with examples: MapReduce

A simple explanation of how to run parallel workloads to process big data

I decided to write a series on distributed computing, a historical review on where we succeeded in handling big data properly, and how it evolved. The challenge with processing big files was that our computation running on distributed machines had to finish within a reasonable time, and this brought the questions on how we can do embarrassingly parallel computations, how to distribute the data, and how we handle the failures. And in 2004, Google shared the MapReduce framework, after 4 years (or 5) of running an average of one hundred thousand MapReduce jobs, processing about twenty petabytes of data per day.

What they have achieved is to decouple the process with Map tasks where they can process the input in parallel, and Reduce tasks where they can do aggregation on the sorted and grouped data with a master, which tracks the workers' health, and completion of the tasks and re-execution of the tasks on failed machines allows fault tolerance.

With a simple pseudo-code, the map takes a key/value pair of inputs and computes another key/value pair independent of the original input. And reduce takes the key, with all values split out of the map function, sorted/grouped ready to run aggregation functionality on it. The output is generally one output value.

map (k1,v1) → list(k2,v2)

reduce (k2,list(v2)) → list(v2)

Let’s consider a simple use case `word count` where we try to find out the number of occurrences for each word. In this scenario, the user program splits the input file into M pairs. If the input is a corpus, each split would be a document, and if the input is one document, each split would be a line, etc…And also it spins up multiple workers, one of them being the master node.

The master node is responsible for assigning workers to map and reduce tasks and checking the health check (via pinging). If any worker does not respond within the time frame, master marks them as failed, and reschedules the tasks in progress on these workers, and also all map tasks as the output data on the machine are lost.

Workers assigned for map function reads the split pair and saves the intermediary key/value pair to its disk (first to buffer and a partitioning function write to disk, but we skip not to get distracted). It was groundbreaking at the time because map workers doing this in embarrassingly parallel resulted in high performance on large clusters of commodity PCs.

MapReduce provides locality optimisation, as it uses GFS, and master asks GFS for locations of the input file blocks and schedules map tasks on the same machine or the same rack, which allows thousands of machines read input at local disk speed and does not have switch rack and be limited by the read rate

The catch is we are responsible for writing the map and reduce functions, so let’s look at what we would have done if we were writing in Python. The original article/implementation is on C++, I will use Python to keep it simple. There are many ways of implementing and I shared in the references part if you want to explore more:

Now, all the intermediary key/value pairs are saved, a partitioning function puts them onto the disk, and part of reduce tasks the words are sorted and grouped. Reduce function will take the unique key and the list of grouped values to compute the sum of the counts by appending each output for the global output. If we were asked to find the maximum temperature of given a list of cities, we would do a max aggregation on the Reduce function.

If the reduce worker fails, the master does not ask for re-execution as the output is globally shared.

And our complete function (with the help of mrjob module) would look like as below. Mrjob module can run almost any on-prem and on cloud Hadoop clusters. (PS: If you prefer to use go, gossamr repo can be useful)

If we check our complete example, we get the number of the occurrences for each word, and it is saved on a globally accessible location. Google’s paper is based on the 2000 worker machines, 5000 partitions, and 200,000 splits, where they chose each split about 16MB to 64MB.

The input file or key/value pairs are different from the intermediary as well as the output key/value pairs, and we won’t be able to predict when handling for petabytes of data.

Next Steps

It was the first abstraction to enable execution of parallel tasks with taking advantage of a local file system. Next, inspired by the MapReduce programming model, Hadoop in 2006 was launched by Yahoo, and cloud providers did not fail to promote the Hadoop-on-cloud tools, namely Google implemented as Dataproc, AWS as EMR (Elastic Map Reduce), and Azure as HDInsights.

Call to Arms:

So, let’s reflect bak. Question is for this simple problem, how would you improve the solution Googlers found? How would you handle the files spilt at the end of map phase if you wanted to handle repeated executions? How would you make the software developer’s life easier to write map/reduce functions in a more abstracted way?

Until the next one, take care!

References:

• MapReduce: Simplified Data Processing on Large Clusters
• Taming Big Data with MapReduce and Hadoop - Hands On!

Micheal Noll’s blogpost

Series on Distributed Computing 1: MapReduce was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

When you are in charge of a team/teams, you need more attention and focus on what you do and tell as the impact you have on them will have the ripple effect. Your team/direct reports will easily, involuntarily, subconsciously copy your approach, your insight and your mental blocks, even if they know it is not the case. So, I decided to write a series of topics what makes a great leader. Morale has such a great impact on motivation and performance, personally, it is the most important fact to predict a team will make it or not, thus the team lead is a successful one or not. So, today’s topic is positivity: How it can easily slip through your control, and how to get it back.

Put Positivity Hat/Crown on:

You have to make sure the goals are achievable and your team is capable of achieving it. There may be moments where your faith drops, either the questions answered or the reports prepared may not pass your quality standards, but do remember it is a longer journey than a question or a report. Building trust, a team does take longer, and it is a never-ending battle. Every challenge is a test, and to pass you have to make sure you have decluttered your brain. If you struggle, check:

• How much input are you taking? Think about the daily and weekly data you are receiving. Compared to before (last week/last month — when last time you felt OK), how much more are you receiving input? Review the number of your subscriptions, reports, emails, meetings per day. Any increase would easily block your mental space to think freely, if not overwhelmed, it can easily do a projection of how hard the next couple of weeks will be -if you don’t take action now- and lead to negativity.
• How many choices do you have? Too many choices can make the decision-making process to go through a harder lap, and spend more effort and energy it normally does. The more tired brain you have, the more stressed your body will be, and the more steroid hormones it will produce, which are called glucocorticoids, which include cortisol, often referred to as the “stress hormone”
• How much negativity are you exposed to during the day? Similar to your eating habits, as they can impact your energy levels, the people and the conversations around you can impact your reflections, responses, and attitudes. If your resilience system is not built, if your reflexes and context awareness is not ready for the specific cases (which is always in constant growth), they may even slip into your daily life, if not hunt you on your sleep. If you can anticipate the challenging, negative meetings in advance, pick your time in the day (morning vs afternoon) where you have more energy, colour code the meetings so you will be prepared mentally. Even you may try doing a battle card to respond on specific cases where you can feel you will stay on top of the play.
• How many of the priority tasks left at the end of the day? Focusing on the full list, and not taking action from top to bottom priorities may make you feel comfortable, but the real focus items still waiting, if not for days, but hours, will create more pressure towards the end of your day/week. If you are finding yourself still working after 5 pm, on a Friday, you need a review on your todo list and learn to ignore smaller priority tasks (assuming you have done the priorities). It is a traded off your work-life balance.
• How many breaks do you take? Do you block your dairy between meetings, not only to reflect back, write reports, respond emails but also to take a deep breath and even do some meditation? With remote working, it is easier to get your calendar fully booked if you have not allocated self 1–1s. And you may find it more challenging if your calendar is so used to be double, if not tripled booked. That’s where it will be easier to lose yourself, give up the priority from yourself to the rest of the world.

The amount of water and exercise/meditation you have done daily does also ad. All of us are different and each five-item listed above will have a different impact on your positivity, and can easily drag you down the rabbit hole before you notice the journey. How about rewarding yourself for the amount of the strain you have taken and zoom out, looking at the big picture, you find ways to make tomorrow a better one, better self?

Looking forward to hearing your tips and tricks to be aware of the day’s hidden gremlins and how to boost your energy?

Motivation:

Hello! Conda is one of the most popular tools at data science community, and yet, it can be confusing to understand the steps and the cost of implementing that step, as there is hardly a single place explains, so I decided to write one up.

I will focus on three topics, the first one is about conda installer options, Anaconda, miniconda, and miniforge, what you will be missing by not using one. The second topic will be about setting up an environment, you can reliably use for multiple projects, and how to modify when you need more configuration. And the last part is about the relationship of channels with environments and packages, which is also an ignored topic, but very important to show good engineering skills if you want to productionize your work with minimum trouble.

PS: I am on macOS Catalina 10.15.7, and have the Conda version 4.9.0. If you have questions on specific versions, please leave a comment.

TL;DR?

So, what am I promising you may get, by the end of this article would be understanding how to set up your conda environment through the lens of the opportunity cost by choosing between:

• miniconda and miniforge installers
• naming environments as unique and standard
• adding channels globally and specific to your environment
• installing packages from different channels

Hope you enjoy, and I’ll see you at the end!

Conda

conda is an open-source, cross-platform, package, dependency and environment management tool for -in theory- any language (but mostly supported on Data Science and Machine Learning specific languages, such as Python, R, Ruby, C/C++, FORTRAN, …). Anaconda is the company who developed it first, then open-sourced under the BSD license. It has more functionality than both pip and virtual env together can offer. Pip is a package manager on top of Python for a limited number of libraries, i.e. it can’t install Python. Virtualenv is a simple environment manager, can’t install packages at all… I guess you are convinced to use conda at this point, so we can continue!

The top three installers for installing conda are Anaconda, miniconda and miniforge. The first two are developed by Anaconda, and available on their website, whereas miniforge is created by the community recently as miniconda does not have any not support foraarch64 architecture.

If you need specific requirements, like running your models on aarch64(arm64) or ppc64le (POWER8/9) architectures, you should use miniforge installation. It has also support for PyPy, a light version of Python. During the installation, it sets conda-forgeas the default -and the only- channel and does not have the defaults channel. We will talk about channels on the next part more in detail.

Another reason to use miniforge is the commercial use restriction(*) on Anaconda/miniconda is due to a recent change on Term Of Service of Anaconda, where it is declared as a violation to use the Repository for commercial activities, which includes usage of the packages installed from the defaults channel.

Also, there are a couple of open-source projects already moved from mini-conda to miniforge, this and this, which suggests there will be an increase at the community supporting this repo.

If you are happy with ToS and don’t need the architecture requirements, we are left with 2 options, Anaconda or miniconda. If you want the full version which provides a one-off installation, and if you have 5GB disk space, Anaconda will install Python + 250 packages for you. It may* be a good choice for new starters, as it has commonly used packages ready to use, as well as applications such as Anaconda Navigator, where you can launch your JupyterLab, pySpider IDE for your environments. Anaconda also has multiple editions, i.e. Individual edition is the free version, Enterprise edition is for your team if you want to extend and manage the customisation of packages and channels privately.

And our last option is the minimal installer miniconda will be a good choice, as it will set up defaults channel. *It is better than Anaconda installation because you learn more about which packages to download and you don’t have to free up 5 GB.

PS: If you think you are missing out Anaconda Navigator, Spyder you can install with conda, the first one is available in the default channel, and Spyder is available on both.

$ conda install anaconda-navigator
$ conda install spyder

🛑 There are also different methods to run our installers:

• Local: Depending on your environment both installers offer different releases: either as a package (i.e. exe for Windows platforms), as a script (for Linux platforms) or both (pkg** and sh for macOS platforms). ** miniconda only, miniforge does not support it (yet).
• Cloud VM: Only miniconda has an offer of AMI images for AWS if you want to isolate your environment and run on the cloud.
• Container: Docker installers are available for both: miniconda and miniforge
• CI/CD Pipeline: Both miniconda and miniforge have Github actions for multiple packages it support and it can take the hard work off from your shoulders.

Environment Creation

Conda environment is an abstract way of organising multiple packages and their dependency together. Any new environment we create has a directory where all the packages will be downloaded to, and any configuration, history related to this environment will be stored. If you have installed the Anaconda’s installer, it creates an environment called base on the anaconda's installation directory. You can check this out by running conda env list command: (*) refers to the default environment (when we are not using any environments actively).

$ conda env list
# conda environments:
#
base        *  /Users/ebrucucen/opt/anaconda3

It’s all good, but we want our very own environment. There are 2 conventions on how to name your environments.

First one is you can give unique names to your environment. This implementation creates an environment for each version of Python (as it is the main language we are all interested, right?), such as conda-py37 or env-py3.8 for Python 3.7 and Python 3.8 versions respectively. This is great, if you are new to environments, possibly you won’t have many versions of Pythons, with multiple packages with complex dependency trees. You can access your environment from any project folder, and follow the conda documentation when executing the commands involving environments without any issues as the environment will be set up on the standard locations (/user/.../envs/)

To create an environment, we use conda create command, followed by the environment name, and a list of package=version pairs, where versions are optional, with the tradeoff installing the latest versions.

$ conda create --name env-py3.8 python=3.8 numpy=1.19.5

The second option is using common name to all of your environments, and create a new one for each project folder, such as conda-env. This means, for any project folder you are working on, you can reference the environment name the same way, and use in any on your automation scripts in a consistent manner. Please note, the subcommand --prefix is exclusively mutual with the--name pick your poison carefully!

$ conda create  --prefix /<possibly a long path>/conda-env python=3.7

# or if you are already on the same directory:

$ conda create  --prefix ./conda-env python=3.7

Environment Activation

Whichever naming convention you have chosen, now you have an environment, (either env-py3.8 or conda-env or both(!)), and next thing we need to do is to activate, so we can start installing packages to these environments:

$ conda activate env-py3.8

which (by default) displays environment name on the command-line, ready to take next set of instructions….

(env-py3.8) $

or

$ conda activate ./conda-env

results in not so pretty display, and definitely not a great use of your space

(/<possibly a long path>/conda-env) $

To change this behaviour to display simply the environment name, you can modify .condarc file (which is by default on your home directory,~/.condarc, if you are not sure you can find out by conda config — show-sources) :

conda config --set env_prompt '({name})'

Now, if you have managed to follow me to this point, we should have an environment, activated waiting for us to install packages.

If you want to breathe and have a bit deep dive, check out the subdirectories of the env folders, where conda-meta folder contains history file to track each action on this environment, JSON files for each package with its build number, dependencies, and file locations listed… If you found/know anything interesting, please do leave a comment, it will help us to understand the environment mystery better. We need packages, and channels help us to get them with the right version and dependencies, let’s move next!

Channels

Channels are repositories for our packages. Each channel, maintained separately, may have a different version of the packages, different build for each version, and the same version of the packages may have different dependencies in each channel. Checkout the Stackoverflow question for more discussion on this.

A good example is for the most common two packages, NumPy and Tensorflow, where Anaconda’s defaults channel and conda-forge has different versions available.

numpy                         1.19.5  py39he588a01_1  conda-forge
numpy                         1.19.2  py39he57783f_0  pkgs/main

tensorflow                    2.0.0 mkl_py37hda344b4_0  pkgs/main
tensorflow                    1.14.0      hcba10bf_0  conda-forge

To avoid one confusion, this does not mean if we want to install tensorflow 2.0.0 version referencing conda-forge channel, but it means conda will try to reach out to default channel for the tensorflow 2.0 modules available and prioritised conda-forge for each dependency, which in return you will get:

_tflow_select      pkgs/main/osx-64::_tflow_select-2.3.0-mkl
  absl-py            conda-forge/osx-64::absl-py-0.11.0-py37hf985489_0
 ...
  tensorboard        pkgs/main/noarch::tensorboard-2.0.0-pyhb38c66f_1
  tensorflow         pkgs/main/osx-64::tensorflow-2.0.0-mkl_py37hda344b4_0
  tensorflow-base    pkgs/main/osx-64::tensorflow-base-2.0.0-mkl_py37h66b1bf0_0
  tensorflow-estima~ pkgs/main/noarch::tensorflow-estimator-2.0.0-pyh2649769_0
  termcolor          conda-forge/noarch::termcolor-1.1.0-py_2
  werkzeug           conda-forge/noarch::werkzeug-1.0.1-pyh9f0ad1d_0

Add channels

As we mentioned above, the default channel for miniconda and Anaconda installations is defaults channel, whereas for miniforge, the default channel will be conda-forge channel. We can display our channels by looking at the configuration file (independent of the fact that whether you are in an activated environment or you are not):

$ conda config --show-sources

which results in something similar to this (yours will be slightly different):

==> /Users/ebrucucen/.condarc <==
auto_update_conda: False
ssl_verify: True
channels:
  - conda-forge
  - defaults

You can add channels globally, or local to your environment when you are in the activated environment. For global installation, you can call conda config add conda-canary to test the packages to be published to live within 24 hour

$ conda config --add channels conda-canary

We can also create a channel-specific to an environment. Let’s say we want to install genomics related packages to our env-py3.8 environment, then we can activate an environment, passing the--envargument to it, and add the bioconda channel to it:

$ conda activate env-py3.8
(env-py3.8) $conda config --env --add channels bioconda

The result will be similar to this (you may/may not have defaults channels)

(env-py3.8) $conda config --show-sources==> /Users/ebrucucen/.condarc <==
auto_update_conda: False
ssl_verify: True
channels:
  - conda-canary
  - conda-forge
  - defaults

==> /Users/ebrucucen/opt/anaconda3/envs/env-py3.8/.condarc <==
channels:
  - bioconda
  - defaults

Append channel

One thing you may have noticed is that the add command modifies the config file with putting the latest additional channel to the top of the list. conda is opinionated about the order of the channels, as it is a priority list in essence. If our new channel should go to the bottom, rather than to the top, we can use the append argument (or modify the config file, I hear you)

(env-py3.8) $conda config --env --append channels test-channel

if we check out the configuration file, we will see our channel as the last item ( yes, you are right, there is no channel validation happens during the execution of add/append channel commands, but it will error when you want to search/install packages)

(env-py3.8) $conda config --show-sources==> /Users/ebrucucen/.condarc <==
auto_update_conda: False
ssl_verify: True
channels:
  - conda-canary
  - conda-forge
  - defaults

==> /Users/ebrucucen/opt/anaconda3/envs/env-py3.8/.condarc <==
channels:
  - bioconda
  - defaults
  - test-channel

Remove channels

If you want to remove a channel (either a mistake you have done or you don’t need it any more), is quite as simple as running the --remove argument, and with the same principle where you need to specify the --env tag to remove the channel from the activated environment, otherwise, conda will give an error to tell you it can’t find the channel.

(env-py3.8) $conda config --env --remove channels test-channel

Packages

And finally, we can install our packages now. To find out what you need, I recommend Anaconda search, which gives the versions and the downloads for each package, you have a better understanding of your options, plus you can find good jupyternotebooks for your packages.

Since each environment has a prioritised list of channels, any installation will checklist one by one whether the version (if specified) is available or not.

$ conda search spyder

Top 6 ways I use to install packages are: on a specific environment, of a specific version, of a specific build ( which can be gobbledygook), from a specific channel, and also with dependencies or without dependencies. For the first 3, we can use this format:

$ conda install -n <env-name> -c <channel> <package_name>=<version>=<build_string>

$ conda install -n env-py3.9 -c conda-forge numpy=1.19.5=py39he588a01_1

conda install by default installs the dependant packages. We should explicitly tell we don’t want the dependencies (danger Williams, I assume we know what we are doing at this point).

$ conda install -c conda-forge numpy --no-deps

Call To Arms

Thank you for your patience, and reading through my post. Now, you have a conda set up with an environment, and channels ready to install any package (as well as Pip, and other tools) you need as I promised with the reasoning behind each choice. Hope you enjoyed it. It is a simple process, and you can customise as much as you want, such as creating your own channels, packages, will all the support available. Happy condaing!

From soup to nuts guide for setting up a conda environment was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Data is the new oil, and the amount of the data we need to handle is increasing tremendously. AWS has a single customer dealing exabytes of data.

The key takeaway from me was that with the demand of the millions of online users, Data Lake will help to store the structured and unstructured data, ready to be consumed by real-time analytics, machine learning algorithms, or dashboards, and we need to make sure it is implemented in a methodological process to avoid data swamps.

I enjoyed all presentations by AWS on Data FlyWheel, Storage, Analytics, and ML/AI. Also, there were also interesting client presentations by Adthera, EDF, and Expedia which were engaging, and here are my notes from the sessions.

“Data Flywheel”

Marc Trimuschat has done the opening session and explained how Amazon was based on a FlyWheel and has done a reflection on data, i.e. how the Data FlyWheel would look like:

• Move to manage: You can break free from legacy databases, by migrating databases, data warehouses, or streaming data directly into AWS.
• Run Fully Managed Databases: You can not only save time and cost but also remove the undifferentiated heavy lifting of common database administrative tasks. You can improve performance, lower the cost, take advantage of the tools to be introduced in the next x months.
• Build data-driven apps: You will have the agility and globally distributed data which can scale with performance. You can choose from purpose-built databases such as Amazon DynamoDB for relational databases or Amazon Timestream for Time Series depending on your application’s requirements.
• Analyze your data in data lake: You can have better, faster insights, broader access to the insights. AWS Lake Formation builds a secure Data Lake, which helps to move, store, catalogue and clean your data faster. You can enforce security policies across multiple services. Amazon QuickSight helps to visualise the data with rich dashboards.
• Innovate: Having momentum in the flywheel, you can enable better experiences, deeper engagement, efficient processes. Amazon SageMaker helps to build, deploy and train the models quickly at scale.

“Storage”

Maribel Rodriguez presented the talk on storage, how S3 is the backbone of all the lake, they have over 10k data lakes on S3. On top of AWS Lake Formation, he talked about AWS Glue, which is a fully managed ETL tool, has crawlers to scan the data on S3 and populates Glue Data Catalogue, then your data will be ready to be analyzed.

And also, he shared the new announcements happened pre-Invent:

• S3 Intelligent-Tiering: You don’t have to think about the storage access layer. With a small fee for monitoring and automation, it works out if the object is not accessed over 30 days, it moves to Infrequent Access Tier, and if accessed later, it moves to Frequently Access Tier.
• S3 Glacier Deep Archive: If you need long term storage, and don’t want to deal with the tapes, storage, AWS is now offering 12-hour retrieval storage with competitive pricing with off-premises tape archival services.

The overall new products on AWS catalogue this year were FSx for Lustre, Amazon FSx for Windows File Server to support Multi-AZ and integrate with Self-Managed Active Directories, AWS Backup, EFS Infrequent Access.

“Analyze the Data”

Kaz Janiskowski presented the session on how to Analyze Data. The highlighted products were:

• AWS Data Exchange: It is a platform to easily provide and subscribe to a product and reach millions of AWS customers. Data Providers have an easy way to package and publish data products, with built-in security and compliance controls. Data Subscribers can quickly find diverse data in one place, can efficiently access 3rd party data, and easily analyze.
• Data Movement: AWS does provide different options to move the data. For On-premise movements for petabytes of data, you can use SnowBall, SnowMobile. To ingest streaming data, you can use Kinesis Data Firehose, Kinesis Data/Video Streams, Managed Streaming for Kafka.
• AWS SCT (Schema Conversion Tool): It helps with the schema conversion. Not only it creates an assessment report, compatibility of source databases with OSS, but also it recommends the best target engine and provides the effort required. The attempt to convert the schema and the code does also cover stored procedures and functions. It scans and converts the embedded in-app code.

As a new feature, Kaz shared Athena’s new federated query, which was announced this week, how it solves the issue of analysing multiple data sources including on-premise or on cloud.

On the popular cloud DataWarehouse product AWS Redshift, Kaz shared the Key Innovations this year. You can find out more on the AWS site:

• Amazon Lake Formation Integration
• Spectrum Request Accelerator
• Concurrency Scaling
• New Management Console
• Auto-Vacuum, Auto-Analyze and Auto-Table Sort
• Snapshot Scheduler
• Auto Data Distribution
• Dynamic WLM Concurrency
• Stored Procedures
• Improving Short Query Acceleration
• Faster Cross Regional Copy and Change
• Query Priorities
• Deferred Maintenance
• Elastic Resize

“AI/ML Becoming Commodity”

Dimitri French had a session on ML/AI to explain the landscape on AWS. dividing the services into 3 categories: ML Frameworks + Infrastructure, ML Services, AI Services. The product range proved that most of ML/AI algorithms are ready to be consumed as commodity products, so I am very excited to watch out space and see the impact of the applications on our daily lives.

ML Frameworks + Infrastructure: If you want to run your own TensorFlow, MxNet, PyTorch, it does support with interfaces Gluon and Keras. You can host your models on your EC2 instances or on-prem/portable Kubernetes clusters.

Amazon SageMaker: It is the service API provides multiple features to enable ML, such as Ground Truth, Notebooks, Algorithms + Marketplace, Reinforcement Learning, Training, Optimization, Deployment and Hosting. Depending on your workloads, ML workflows, you can integrate any of these offerings.

AI services: AWS has the products to support all cognitive features.

• For Vision, it has Rekognition Image, Rekognition Video and Textract, where you can detect images, videos and text (one of the use cases is reading receipts) respectively and focus more on the application features, and not waste time to develop models which are becoming a commodity.
• Polly and Transcribe do provide the speech service. Translate, Comprehend & Comprehend Medical products are for Language services. Lex is the product to enable easy use of Chatbots. Forecast is the product to provide Forecasting, and lastly, Personalization helps with the Recommendation services.

As announcements:

Aurora does support ML by calling SageMaker, Comprehend natively, with simple SQL queries. It gives the ability to run the queries interactively from your codebase, or storing the output of the ML model executions, and review later.

This was a summary of the products I could gather, hope it is useful for you too, to digest what AWS does offer today, and to have insight about how we should look at Data Lake space.

Note: If you are interested in the Re:Invent AI/ML section, this is the list you would want to miss!

Happy data days!