Medium

Matthew Powers

Matthew Powers

·

Feb 22, 2022

Speed up a pandas query 10x with these 6 Dask DataFrame tricks

This post demonstrates how to speed up a pandas query to run 10 times faster with Dask using six performance optimizations. You’ll often…

Speed up a pandas query 10x with these 6 Dask DataFrame tricks

Matthew Powers

·

Jul 4, 2019

Creating a Java Spark project with Maven and junit

This blog post shows how to organize a Spark Java project, write some application code and run a simple test.

Matthew Powers

·

Jun 25, 2019

Dependency Injection with Spark

Dependency injection is a great design pattern to write code that’s more flexible and easier to test.

Matthew Powers

·

Apr 3, 2018

Speaking Slack Notifications from Spark

The spark-slack library can be used to speak notifications to Slack from your Spark programs and handle Slack Slash command responses.

Matthew Powers

·

Feb 18, 2018

Documenting Spark Code with Scaladoc

You can use Scaladoc to generate nicely formatted documentation for your Spark projects, just like the official Spark documentation.

Documenting Spark Code with Scaladoc

Matthew Powers

·

Jan 21, 2018

The different type of Spark functions (custom transformations, column functions, UDFs)

Spark code can be organized in custom transformations, column functions, or user defined functions (UDFs).

Matthew Powers

·

Jan 16, 2018

Adding StructType columns to Spark DataFrames

StructType objects define the schema of Spark DataFrames. StructType objects contain a list of StructField objects that define the name…

Matthew Powers

·

Jan 15, 2018

Adding ArrayType columns to Spark DataFrames with concat_ws and split

The concat_ws and split Spark SQL functions can be used to add ArrayType columns to DataFrames.

Matthew Powers

·

Jan 5, 2018

How to write Spark ETL Processes

Spark is a powerful tool for extracting data, running transformations, and loading the results in a data store.

Matthew Powers

·

Dec 27, 2017

Spark User Defined Functions (UDFs)

Spark let’s you define custom SQL functions called user defined functions (UDFs). UDFs are great when built-in SQL functions aren’t…

Matthew Powers

Matthew Powers

Spark coder, live in Colombia / Brazil / US, love Scala / Python / Ruby, working on empowering Latinos and Latinas in tech

Following

Help

Status

About

Careers

Press

Blog

Store

Privacy

Rules

Terms

Text to speech