Ship Ahoy: Sign Up for Databricks Community Edition:

Set Sail with a Free Account

• Navigate to Databricks Community Edition.
• Sign up with your email (it’s free, so no hidden treasure required).
• Verify your email to launch your shiny new Databricks workspace.

All Aboard the Workspace

• After logging in, you’ll see the Databricks Home. Congratulations — you’re the proud captain of your own data cruise!

Hoist the Clusters: Creating Your Spark Engine:

Head to Compute

• Click on “Compute” (or “Clusters”) in the left-hand navigation bar.

Raise a Cluster

• Click “Create Cluster”. Give it a memorable name, like S.S. TitanicCluster.
• Accept default settings. (Your cluster’s crew is all set to sail with Spark behind the scenes.)

Shove Off!

• Click “Create” and wait a minute for your cluster to be ready. You’ll see a green icon when the cluster is powered up.

Charting the Course: Bringing in the Titanic Data:

Notebook Creation

• In the left panel, click “Workspace”, then “Create” → “Notebook”.
• Give it a name, like Titanic_Analysis (in case you need a reminder of the watery expedition).
• Choose Python as your language.

Attach Your Cluster

• At the top of the notebook, select your cluster from the dropdown.
• This ensures all your code sails smoothly on the cluster’s Spark engine.

Download the Titanic CSV

• In a new cell, paste:

%sh
wget https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv -O /tmp/titanic.csv

• This uses a shell command (%sh) to fetch the Titanic CSV and place it in a temporary local folder.

Move Titanic Onto DBFS

• In another cell:

dbutils.fs.cp("file:/tmp/titanic.csv", "dbfs:/tmp/titanic.csv")

• Now we have the data inside DBFS (Databricks File System).

Read the Data

• Let’s load the CSV as a Spark DataFrame:

df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("dbfs:/tmp/titanic.csv")

df.createOrReplaceTempView("titanic")
df.printSchema()

• df is your data-laden vessel, and "titanic" is a temporary view to run SQL queries against it.

Navigating the Data: Exploring the Titanic:

Peek at the Deck:

display(df.limit(5))

• Shows you the first few passengers (hop on, hop off).

Head Count:

df.count()

• See how many total passengers (rows) are on this data voyage.

Peek at the Deck:

display(df.limit(5))

• Shows you the first few passengers (hop on, hop off).

Data Stats:

df.describe().show()

• Get basic statistics on numerical columns, like min, max, and average Age.

Null Watch:

from pyspark.sql.functions import col, sum as spark_sum
display(df.select([spark_sum(col(c).isNull().cast("int")).alias(c) for c in df.columns]))

• Check how many missing values lurk in each column (hopefully no “iceberg” column is missing!).

Lifeboat Drills: Simple Data Cleaning

Plug Missing Ages:

# Calculate average age
avg_age = df.selectExpr("avg(Age)").collect()[0][0]
# Fill null 'Age' values with average
df_filled = df.na.fill({"Age": avg_age})

Divide into Age Groups:

from pyspark.sql.functions import when
df_transformed = df_filled.withColumn(
    "AgeGroup",
    when(df_filled["Age"] < 18, "Child").otherwise("Adult")
)

• Now you know who’s a “Child” or an “Adult” for further analysis.

Bon Voyage: Visualize with a Simple Query:

Count Survivors by Gender:

display(spark.sql("SELECT Sex, Survived, COUNT(*) AS Count FROM titanic GROUP BY Sex, Survived"))

• A table of how many people survived (Survived=1) vs. didn’t (Survived=0), split by gender (Sex).

Transform into a Chart:

• In the result panel, click the Chart button.
• Choose Bar (or try Pie if you’re feeling fancy).
• This visual reveals the difference in survival rates by gender, giving you a quick snapshot of the data’s story.

Disembark and Explore More!

Congratulations, Captain! You’ve successfully:

• Spun up a free Databricks workspace.
• Loaded the Titanic dataset into a Spark DataFrame.
• Explored, cleaned, and visualized data using both Python and SQL in a Databricks notebook.

Next Steps:

• Tinker with additional Spark transformations (like grouping by Pclass to see survival based on ticket class).
• Experiment with SQL queries.
• Try more visualizations (e.g., a histogram of Age distribution or a stacked bar chart of AgeGroup vs. Survived).
• Dive deeper into advanced analytics or even machine learning using the same environment.

Thanks for cruising on this Titanic data journey! You’ve learned the fundamentals of Databricks keep exploring, and soon you’ll be the master of your own data ocean. Bon voyage!

Dagster is a powerful data orchestration platform that allows you to build, run, and monitor your data pipelines. In many initial setups, developers combine both the core Dagster infrastructure and the user code (data pipelines, jobs, ops) in one repository and deploy them together. This approach can be convenient at first, but it often becomes unwieldy as your codebase grows. Every small change to your pipeline code might require redeploying the entire Dagster stack, including its infrastructure services.

A cleaner and more scalable approach is to separate the Dagster infrastructure from the user code. By running the Dagster webserver, daemon, and database as a persistent infrastructure, and deploying your user code as a separate gRPC server container, you can update your pipelines independently and without downtime.

Benefits of Separation

• Faster Iteration: Update your user code repository (use-case repo) and deploy changes without redeploying the entire Dagster stack.
• Stability: Keep the Dagster infrastructure stable and running at all times.
• Flexibility: Add or swap out multiple user code repositories with minimal fuss.

Prerequisites

• A Linux VM or server environment (e.g., Ubuntu) with Docker and Docker Compose installed.
• SSH access to the VM.
• Two separate Git repositories:

1. Dagster Infra Repository: Contains only the Dagster infrastructure configuration (webserver, daemon, database, and workspace.yaml).
2. User Code Repository: Contains your actual pipeline code (repo.py) and related dependencies.

High-Level Architecture

Infra Repo:

• Runs dagster-webserver
• Runs dagster-daemon
• Runs a Postgres database
• Has a workspace.yaml file pointing to a gRPC server endpoint (which the user code repo provides)

User Code Repo:

• Contains your pipeline definitions in repo.py
• Builds an image that launches a Dagster gRPC server serving your code
• Deployed independently and connected to the same Docker network as the infra

Step-by-Step Tutorial

1. Set Up the Dagster Infrastructure Repo

In your infra repository (e.g., dagster-infra):

Directory Structure:

dagster-infra/
  ├─ docker-compose-infra.yml
  ├─ Dockerfile.dagster
  ├─ dagster.yaml
  ├─ workspace.yaml
  └─ README.md

Dockerfile.dagster:

FROM python:3.10-slim

# Install Dagster and related libraries
RUN pip install dagster dagster-graphql dagster-webserver dagster-postgres dagster-docker

ENV DAGSTER_HOME=/opt/dagster/dagster_home
RUN mkdir -p $DAGSTER_HOME

# Copy Dagster config files
COPY dagster.yaml workspace.yaml $DAGSTER_HOME

WORKDIR $DAGSTER_HOME

docker-compose-infra.yml:

version: "3.8"

services:
  dagster_infra_postgresql:
    image: postgres:11
    container_name: dagster_infra_postgresql
    environment:
      POSTGRES_USER: "postgres_user"
      POSTGRES_PASSWORD: "postgres_password"
      POSTGRES_DB: "postgres_db"
    networks:
      - dagster_infra_network
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres_user -d postgres_db"]
      interval: 10s
      timeout: 8s
      retries: 5

  dagster_infra_webserver:
    image: dagster_infra_dagster_image
    container_name: dagster_infra_webserver
    entrypoint:
      - dagster-webserver
      - -h
      - "0.0.0.0"
      - -p
      - "3000"
      - -w
      - "/opt/dagster/dagster_home/workspace.yaml"
    expose:
      - "3000"
    ports:
      - "3000:3000"
    environment:
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/io_manager_storage:/tmp/io_manager_storage
    networks:
      - dagster_infra_network
    depends_on:
      dagster_infra_postgresql:
        condition: service_healthy

  dagster_infra_daemon:
    image: dagster_infra_dagster_image
    container_name: dagster_infra_daemon
    entrypoint:
      - dagster-daemon
      - run
    restart: on-failure
    environment:
      DAGSTER_POSTGRES_USER: "postgres_user"
      DAGSTER_POSTGRES_PASSWORD: "postgres_password"
      DAGSTER_POSTGRES_DB: "postgres_db"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/io_manager_storage:/tmp/io_manager_storage
    networks:
      - dagster_infra_network
    depends_on:
      dagster_infra_postgresql:
        condition: service_healthy

networks:
  dagster_infra_network:
    driver: bridge
    name: dagster_infra_network

dagster.yaml:

run_storage:
  module: dagster_postgres.run_storage
  config:
    postgres_db: "postgres_db"
    postgres_user: "postgres_user"
    postgres_password: "postgres_password"
    hostname: "dagster_infra_postgresql"
    port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  config:
    postgres_db: "postgres_db"
    postgres_user: "postgres_user"
    postgres_password: "postgres_password"
    hostname: "dagster_infra_postgresql"
    port: 5432

scheduler:
  module: dagster_postgres.scheduler
  config:
    postgres_db: "postgres_db"
    postgres_user: "postgres_user"
    postgres_password: "postgres_password"
    hostname: "dagster_infra_postgresql"
    port: 5432

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator

workspace.yaml:

load_from:
  - grpc_server:
      host: "example_services"  # This will be the user code container name
      port: 4000

2. Build and Run the Infra

From the dagster-infra directory, run:

docker build -f Dockerfile.dagster -t dagster_infra_dagster_image .
docker compose -f docker-compose-infra.yml up -d

This starts:

• Postgres database
• Dagster webserver on port 3000
• Dagster daemon

At this point, your Dagster UI should be accessible at http://<YOUR_VM_IP>:3000 or if local machine http://localhost:3000.

3. Set Up the User Code Repository

In a separate repository (e.g., example-use-case):

Directory Structure:

example-use-case/
  ├─ Dockerfile.use_case
  ├─ repo.py
  └─ README.md

repo.py (an example assets pipeline):

from dagster import (
    asset,
    Definitions,
    DailyPartitionsDefinition,
    get_dagster_logger,
    define_asset_job,
    ScheduleDefinition,
)

# Define a daily partition
daily_partitions = DailyPartitionsDefinition(start_date="2023-01-01")

# Define the first asset with logging and partitioning
@asset(partitions_def=daily_partitions,description="This is the first asset")
def first_asset(context):
    logger = get_dagster_logger()
    logger.info("Running first_asset as test asset")
    
    # This asset produces some data
    data = {"key": "value"}
    logger.info(f"Produced data: {data}")
    return data

# Define the second asset with logging and partitioning
@asset(partitions_def=daily_partitions,description="This is the second asset")
def second_asset(context, first_asset):
    logger = get_dagster_logger()
    logger.info("Running second_asset  from second deployment")
    
    # This asset takes the output of the first asset as input
    processed_data = {k: v.upper() for k, v in first_asset.items()}
    logger.info(f"Processed data: {processed_data}")
    return processed_data

# Define a job that will materialize the assets
daily_job = define_asset_job("daily_job", selection=[first_asset, second_asset])

# Define a schedule to run the job every day at 10 AM
daily_schedule = ScheduleDefinition(
    job=daily_job,
    cron_schedule="0 10 * * *",  # every day at 10 AM
)

# Define the repository with the assets, job, and schedule
defs = Definitions(
    assets=[first_asset, second_asset],
    schedules=[daily_schedule],
)

Dockerfile.use_case:

FROM python:3.10-slim

RUN pip install dagster dagster-postgres dagster-docker

WORKDIR /opt/dagster/app
COPY repo.py /opt/dagster/app

EXPOSE 4000

CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-f", "repo.py"]

4. Deploying the User Code Container

Build and run the user code container, making sure it joins the same network as the infra:

docker build -f Dockerfile.use_case -t example_services_image .

Now, run it in the same network as the infra stack:

docker run -d --name example_services --network dagster_infra_network -p 4000:4000 example_services_image

5. Verify Your Setup

Open your Dagster UI at http://<YOUR_VM_IP>:3000 or local. You should see your code location and your two assets available, sourced from the gRPC server running in the example_services container.

Whenever you update repo.py in your example-use-case repo:

1. Rebuild the example_services_image.
2. Stop the old container and run a new one:

docker stop example_services
docker rm example_services
docker run -d --name example_services --network dagster_infra_network -p 4000:4000 example_services_image

Your Dagster infra stays running, and your updated pipelines are immediately available.

6. Automating with CI/CD (Optional)

You can further automate this process with CI/CD pipelines. For example:

• Infra Repo Pipeline: Runs occasionally or on infra changes, builds the infra image, and docker compose up the stack.
• User Code Repo Pipeline: On every push to main, builds the example_services_image and redeploys only that container.

This setup ensures you have a stable infrastructure and can push new pipeline code to production quickly.

Conclusion

By separating your Dagster infrastructure from your user code, you gain the flexibility to iterate on your pipelines without constantly redeploying the entire stack. This separation leads to a more maintainable, scalable, and resilient data platform architecture.

Congratulations! You now have a cleaner and more efficient Dagster deployment strategy.

1. Value Your Time and Invest in Yourself

Work for What You’re Paid For: It’s easy to overextend yourself, especially when you’re passionate about your work. However, it’s crucial to align your efforts with your compensation. This doesn’t mean doing the bare minimum, but rather setting healthy boundaries to ensure you’re not undervaluing your contributions.

Invest in Continuous Learning: Use your extra time to learn new technologies and skills. The data technology landscape is rapidly evolving, and staying updated is key to remaining competitive. Whether it’s mastering a new programming language or exploring emerging data processing frameworks, continuous learning will open doors to future opportunities.

2. Maintain Professional Boundaries

Leave Personal Problems at Home: Bringing personal issues into the workplace can blur professional boundaries and potentially impact your relationships with colleagues. It’s important to compartmentalize and focus on your work while you’re at the office or collaborating on projects.

Be Cautious with Personal Information: Sharing too much about your personal life can sometimes backfire. While building camaraderie is important, oversharing can lead to misunderstandings or be used against you in professional settings. Keep conversations professional and steer clear of divulging sensitive information.

3. Stay Professional in Challenging Situations

Handle Difficult Personalities with Grace: In a global work environment, you may encounter individuals with different values or even face discrimination. It’s essential to remain professional, not take negative behavior personally, and focus on your objectives. Protect your energy and don’t let others’ actions derail your progress.

Focus on Your Mission: Remember why you’re there. Keeping your goals at the forefront will help you navigate difficult interactions and stay motivated, even when faced with adversity.

4. Keep Compensation Details Confidential

Don’t Discuss Your Salary: Sharing salary information with colleagues can lead to tension and jealousy within the team. To maintain a harmonious work environment, it’s best to keep compensation details private.

5. Be Discreet About Career Moves

Keep Future Plans to Yourself: If you’re considering a new project or a job change, it’s wise to keep those plans confidential until they’re finalized. Prematurely sharing this information can create uncertainty about your commitment and may affect your current engagements.

6. Learn to Say No

Set Boundaries with Requests: As a freelancer or consultant, you might feel obligated to agree to every request to please clients or colleagues. However, overcommitting can lead to burnout and decreased quality of work. It’s important to assess your capacity and be comfortable declining requests that exceed your limits.

Navigating the professional world as an expat data engineer comes with its unique set of challenges, but by adhering to these guidelines, you can enhance your career while maintaining personal well-being. Remember that your journey is as much about professional growth as it is about personal development. Stay focused, keep learning, and protect your energy — these principles will serve you well in any endeavor.

I wanted to share some thoughts on the incredible data engineering journey. If you’re anything like me, you’ve probably witnessed firsthand how rapidly this field is evolving. It’s both exhilarating and, let’s be honest, a bit overwhelming at times. So, let’s dive into what’s been happening in our world and how we can ride this wave together.

From Pipelines to Partnerships: The Expanding Role of Data Engineers

When I first stepped into data engineering, it was all about building pipelines and managing databases. Fast forward to today, and our roles have blossomed into so much more:

• Bridging Data Gaps: We’re the ones who bring together data from all corners of the organization, painting a complete picture that everyone can use.
• Guardians of Quality: Ensuring data is accurate and reliable isn’t just a task — it’s a responsibility we hold dearly.
• Scaling Mountains of Data: With data growing at lightning speed, designing systems that can keep up has become a thrilling challenge.
• Team Collaborators: We’re no longer working in silos. Partnering with data scientists, analysts, and business teams has become the norm, and it’s making our work more impactful than ever.

The Challenges We Face (And How We’re Tackling Them)

1. Drowning in Data?

Let’s face it — the sheer amount of data can be intimidating. Structured, unstructured, you name it. But with flexible architectures and innovative processing techniques, we’re turning this challenge into an opportunity.

2. The Need for Speed

Real-time data processing isn’t just a buzzword; it’s a necessity. Embracing new tools and methodologies helps us keep latency low and efficiency high.

3. Keeping Data Safe

With privacy regulations tightening, data security is front and center. We’re all learning to weave compliance into everything we do, ensuring that the data we handle is protected.

4. Managing Complex Ecosystems

Juggling multiple platforms and tools can feel like a circus act. But by focusing on compatibility and seamless integration, we’re making the impossible possible.

What’s Next? Trends to Watch

• Heading to the Cloud (and Beyond)

More organizations are moving to cloud-based solutions, and hybrid models are becoming popular to balance control with flexibility. It’s opening up new avenues for scalability and cost savings.

• Automation Nation with DataOps

By applying DevOps principles to data, we’re streamlining our workflows. Automation isn’t about replacing us; it’s about freeing us to focus on the creative and complex parts of our jobs.

• AI and Machine Learning are Here

Incorporating AI into our pipelines isn’t just for the tech giants. It’s becoming accessible, allowing us to unlock insights we never thought possible.

• Going Serverless

Serverless architectures mean less time worrying about infrastructure and more time innovating.

Tips from My Journey

• Never Stop Learning

The tech world doesn’t stand still, and neither should we. Whether it’s diving into Apache Spark or exploring a new programming language, continuous learning keeps us at the top of our game.

• Quality Over Quantity

Taking the time to ensure our data is clean and accurate pays off in the long run. It’s not the most glamorous part of the job, but it’s foundational.

• Teamwork Makes the Dream Work

Collaborating with others has been one of the most rewarding aspects of my career. Different perspectives lead to better solutions — it’s as simple as that.

• Think Ahead

Designing with scalability in mind saves headaches down the road. Trust me on this one!

Wrapping Up

Being a data engineer today is about so much more than code and infrastructure. It’s about being adaptable, collaborative, and always ready to learn something new. The landscape is shifting, but that just means there are more opportunities to innovate and make a real impact.

Just to be honest, this article was written by me and corrected by IA .

Thanks for reading! If you enjoyed this article, please share it with others who might find it helpful. Let’s keep the conversation going in the comments below!