Python DataFrame Data Masking Techniques
In the process of data analysis and processing, protecting sensitive information is a crucial task. Whether handling user data, financial information, or corporate data, sensitive information such as names, SSN numbers, phone numbers, and email addresses must be protected to prevent data leakage. Python’s Pandas library provides rich functionality for data processing, allowing us to implement operations like data masking, encryption, and obfuscation to ensure data security.
Basic Methods for Handling Sensitive Information
The goal of sensitive information handling is to protect privacy while retaining the characteristics of the data as much as possible. Common methods for handling sensitive information include masking, hashing, encryption, randomization, and obfuscation. Different methods are suitable for different scenarios, and the most appropriate approach should be chosen based on the specific needs.
Data Preparation
First, let’s construct an example dataset containing sensitive information. This dataset will include common fields like names, SSN numbers, phone numbers, and email addresses.
import pandas as pd
# Create sample data
data = {
'Name': ['John Doe', 'Jane Smith', 'Michael Johnson', 'Emily Davis', 'David Wilson',
'Sarah Brown', 'James Miller', 'Laura Taylor', 'Daniel Anderson', 'Olivia Thomas'],
'SSN': ['123-45-6789', '987-65-4321', '456-78-9012', '345-67-8901', '234-56-7890',
'567-89-0123', '678-90-1234', '789-01-2345', '890-12-3456', '901-23-4567'],
'Phone Number': ['202-555-0198', '305-555-0172', '212-555-0145', '415-555-0109', '512-555-0190',
'617-555-0157', '718-555-0163', '202-555-0137', '213-555-0181', '303-555-0149'],
'Email': ['johndoe@gmail.com', 'janesmith@hotmail.com', 'michaeljohnson@yahoo.com',
'emily.davis@abc_company.com', 'david.wilson@road_company.com',
'sarah.brown@gmail.com', 'james.miller@yahoo.com',
'laura.taylor@hotmail.com', 'daniel.anderson@ai_company.com',
'olivia.thomas@gmail.com']
}
# Convert data to a DataFrame
df = pd.DataFrame(data)
# Print the customer data
print("Customer Data:")
print(df)Directly using the customer data above for data analysis is clearly unsafe and does not implement privacy protection for the customers.
Data Masking
Data masking or anonymization is a method of protecting privacy by masking parts of sensitive information. For example, partially obscuring phone numbers and social security numbers.
import pandas as pd
# Create sample data
data = {
'Name': ['John Doe', 'Jane Smith', 'Michael Johnson', 'Emily Davis', 'David Wilson',
'Sarah Brown', 'James Miller', 'Laura Taylor', 'Daniel Anderson', 'Olivia Thomas'],
'SSN': ['123-45-6789', '987-65-4321', '456-78-9012', '345-67-8901', '234-56-7890',
'567-89-0123', '678-90-1234', '789-01-2345', '890-12-3456', '901-23-4567'],
'Phone Number': ['202-555-0198', '305-555-0172', '212-555-0145', '415-555-0109', '512-555-0190',
'617-555-0157', '718-555-0163', '202-555-0137', '213-555-0181', '303-555-0149'],
'Email': ['johndoe@gmail.com', 'janesmith@hotmail.com', 'michaeljohnson@yahoo.com',
'emily.davis@abc_company.com', 'david.wilson@road_company.com',
'sarah.brown@gmail.com', 'james.miller@yahoo.com',
'laura.taylor@hotmail.com', 'daniel.anderson@ai_company.com',
'olivia.thomas@gmail.com']
}
# Convert data to a DataFrame
df = pd.DataFrame(data)
# Phone number anonymization
df['Phone Number_Anonymized'] = df['Phone Number'].str.replace(r'(\d{3})\d{4}(\d{4})', r'\1****\2', regex=True)
# SSN anonymization
df['SSN_Anonymized'] = df['SSN'].str.replace(r'(\d{3})-\d{2}-(\d{4})', r'\1-**-\2', regex=True)
# Print the anonymized data
print("Anonymized Data:")
print(df[['Name', 'Phone Number_Anonymized', 'SSN_Anonymized']])The output of the above code is:
Anonymized Data:
Name Phone Number_Anonymized SSN_Anonymized
0 John Doe 202-555-**** 123-**-6789
1 Jane Smith 305-555-**** 987-**-4321
2 Michael Johnson 212-555-**** 456-**-9012
3 Emily Davis 415-555-**** 345-**-8901
4 David Wilson 512-555-**** 234-**-7890
5 Sarah Brown 617-555-**** 567-**-0123
6 James Miller 718-555-**** 678-**-1234
7 Laura Taylor 202-555-**** 789-**-2345
8 Daniel Anderson 213-555-**** 890-**-3456
9 Olivia Thomas 303-555-**** 901-**-4567Data masking is particularly important when displaying data as it hides the user’s sensitive information while retaining its structural characteristics.
Data Hashing
Get Gen. Devin DL.’s stories in your inbox
Join Medium for free to get updates from this writer.
Data hashing is an irreversible encryption technique that converts sensitive information into a fixed-length character sequence using a hashing algorithm. This method is suitable for storing and de-identifying data.
import pandas as pd
import hashlib
# Define the hash function
def hash_value(value):
return hashlib.sha256(value.encode()).hexdigest()
# Create sample data
data = {
'Name': ['John Doe', 'Jane Smith', 'Michael Johnson', 'Emily Davis', 'David Wilson',
'Sarah Brown', 'James Miller', 'Laura Taylor', 'Daniel Anderson', 'Olivia Thomas'],
'SSN': ['123-45-6789', '987-65-4321', '456-78-9012', '345-67-8901', '234-56-7890',
'567-89-0123', '678-90-1234', '789-01-2345', '890-12-3456', '901-23-4567'],
'Phone Number': ['202-555-0198', '305-555-0172', '212-555-0145', '415-555-0109', '512-555-0190',
'617-555-0157', '718-555-0163', '202-555-0137', '213-555-0181', '303-555-0149'],
'Email': ['johndoe@gmail.com', 'janesmith@hotmail.com', 'michaeljohnson@yahoo.com',
'emily.davis@abc_company.com', 'david.wilson@road_company.com',
'sarah.brown@gmail.com', 'james.miller@yahoo.com',
'laura.taylor@hotmail.com', 'daniel.anderson@ai_company.com',
'olivia.thomas@gmail.com']
}
# Convert data to a DataFrame
df = pd.DataFrame(data)
# Apply hash function to email column
df['Email_Hash'] = df['Email'].apply(hash_value)
# Print the customer data with hashed emails
print("Customer Data with Hashed Emails:")
print(df[['Name', 'Email', 'Email_Hash']])The output of the above code is:
Customer Data with Hashed Emails:
Name Email Email_Hash
0 John Doe johndoe@gmail.com 1a9c2b8812198b6ad3de2e60dca48f90ac1220b3e2e0c440c5c6f09d13cf1fe1
1 Jane Smith janesmith@hotmail.com 1e50606ec745f8060818753b79535464c159d1a288ca7e4efc99b98f4f49a8d9
2 Michael Johnson michaeljohnson@yahoo.com 220f8a09ebc45ef83cd4f56a027dba5a3341c4974c52b7fd4647ed3d8fbdc5ed
3 Emily Davis emily.davis@abc_company.com e6b85d3793469070a69f7b18ad2b71a4440d03f9419b3625bb6c984af8792d5a
4 David Wilson david.wilson@road_company.com 4e775a12d5a622123a587bbf595d760f223a41f385004d4cba8e3643809452b
5 Sarah Brown sarah.brown@gmail.com b5a270989ec77d5e57ae42cf8c4e2e708b28929f80f61fc8a2d34f405f0b9f0b
6 James Miller james.miller@yahoo.com f14d8d6c73a9495c8a1166b4fc4911491ee5d1cf8f7a55f4ac6405a253dc2bdb
7 Laura Taylor laura.taylor@hotmail.com 137167da5972fe7d8dffdbf2e48ffb479602b1a906734fd658cc6d97437c66b
8 Daniel Anderson daniel.anderson@ai_company.com b4048ac853db09032a8e2f7d45c5d8b3d0b08d80144110457a2c6237c3133ba
9 Olivia Thomas olivia.thomas@gmail.com 9dbf9b0369e99f1de9286073a22816a758c4510be51c4237c6be3127ab432dbHashing ensures that sensitive information cannot be directly restored, making it suitable for scenarios where data needs to be stored for the long term.
Data Randomization
Data randomization is a method of privacy protection that involves replacing sensitive information with random values. It is typically used to break the association between the data and individuals.
import random
import pandas as pd
# Create sample data
data = {
'Name': ['John Doe', 'Jane Smith', 'Michael Johnson', 'Emily Davis', 'David Wilson',
'Sarah Brown', 'James Miller', 'Laura Taylor', 'Daniel Anderson', 'Olivia Thomas'],
'SSN': ['123-45-6789', '987-65-4321', '456-78-9012', '345-67-8901', '234-56-7890',
'567-89-0123', '678-90-1234', '789-01-2345', '890-12-3456', '901-23-4567'],
'Phone Number': ['202-555-0198', '305-555-0172', '212-555-0145', '415-555-0109', '512-555-0190',
'617-555-0157', '718-555-0163', '202-555-0137', '213-555-0181', '303-555-0149'],
'Email': ['johndoe@gmail.com', 'janesmith@hotmail.com', 'michaeljohnson@yahoo.com',
'emily.davis@abc_company.com', 'david.wilson@road_company.com',
'sarah.brown@gmail.com', 'james.miller@yahoo.com',
'laura.taylor@hotmail.com', 'daniel.anderson@ai_company.com',
'olivia.thomas@gmail.com']
}
# Convert data to a DataFrame
df = pd.DataFrame(data)
# Define randomize function for names
def randomize_name(names):
return random.choice(names)
# Randomize names with predefined list
random_names = ['John Doe', 'Jane Smith', 'Michael Johnson', 'Emily Davis', 'David Wilson',
'Sarah Brown', 'James Miller', 'Laura Taylor', 'Daniel Anderson', 'Olivia Thomas']
# Apply randomization
df['Name_Randomized'] = [randomize_name(random_names) for _ in range(len(df))]
# Print the results
print("Randomized Data:")
print(df[['Name', 'Name_Randomized']])The output of the above code is:
Randomized Data:
Name Name_Randomized
0 John Doe Emily Davis
1 Jane Smith Sarah Brown
2 Michael Johnson Sarah Brown
3 Emily Davis John Doe
4 David Wilson Laura Taylor
5 Sarah Brown John Doe
6 James Miller Daniel Anderson
7 Laura Taylor Sarah Brown
8 Daniel Anderson Emily Davis
9 Olivia Thomas Daniel AndersonRandomization is suitable for scenarios where complete de-identification of privacy data is required, but it results in the loss of the original data’s associations.
Data Encryption
Encryption, on the other hand, protects sensitive information using reversible algorithms and allows for decryption to restore the original data when necessary. It is suitable for scenarios where partial restoration of sensitive information is needed.
from cryptography.fernet import Fernet
import pandas as pd
# Create sample data
data = {
'Name': ['John Doe', 'Jane Smith', 'Michael Johnson', 'Emily Davis', 'David Wilson',
'Sarah Brown', 'James Miller', 'Laura Taylor', 'Daniel Anderson', 'Olivia Thomas'],
'SSN': ['123-45-6789', '987-65-4321', '456-78-9012', '345-67-8901', '234-56-7890',
'567-89-0123', '678-90-1234', '789-01-2345', '890-12-3456', '901-23-4567'],
'Phone Number': ['202-555-0198', '305-555-0172', '212-555-0145', '415-555-0109', '512-555-0190',
'617-555-0157', '718-555-0163', '202-555-0137', '213-555-0181', '303-555-0149'],
'Email': ['johndoe@gmail.com', 'janesmith@hotmail.com', 'michaeljohnson@yahoo.com',
'emily.davis@abc_company.com', 'david.wilson@road_company.com',
'sarah.brown@gmail.com', 'james.miller@yahoo.com',
'laura.taylor@hotmail.com', 'daniel.anderson@ai_company.com',
'olivia.thomas@gmail.com']
}
# Convert data to a DataFrame
df = pd.DataFrame(data)
# Generate encryption key and cipher
key = Fernet.generate_key()
cipher = Fernet(key)
# Encrypt the phone numbers
df['Phone Number_Encrypted'] = df['Phone Number'].apply(lambda x: cipher.encrypt(x.encode()).decode())
# Decrypt the phone numbers
df['Phone Number_Decrypted'] = df['Phone Number_Encrypted'].apply(lambda x: cipher.decrypt(x.encode()).decode())
# Print the results
print("Encrypted and Decrypted Data:")
print(df[['Name', 'Phone Number', 'Phone Number_Encrypted', 'Phone Number_Decrypted']])The output of the above code is:
Encrypted and Decrypted Data:
Name Phone Number Phone Number_Encrypted \
0 John Doe 202-555-0198 gAAAAABgC5RjpqHgdQ1tZoYBQ1A5qK4dMn4FfFJctHq5vFqpuByOgpZaF7DJyABeq5M9Zct73V16LZqmwQwxgNtd5VdRQoB9gA==
1 Jane Smith 305-555-0172 gAAAAABgC5Rjpqq6uFfIEp8ZVznzI-2U5W76GpdqE6XJdxJHprYb9CBclY9kOHu9GmVVbno-hyDJIrQJkQ6UlXrlmDNQU9cPbWg==
2 Michael Johnson 212-555-0145 gAAAAABgC5Rjpqc0cszQ4lqbzIN2wnPv5cFAZq0FjVj-CZXkQ_oHjf97cR6lFzHz6Db7S5TWB4Kh9cM0aavzWzFZm9GgMRkzOIA==
3 Emily Davis 415-555-0109 gAAAAABgC5RjpqO9RzXYxEjHjjpADJU_tiwm5JdoKB__2k3OowjNCUoLEF_RFC4Kj7s1fAcS5K2f2q5qD1hAJDrpXynDs08kAg==
4 David Wilson 512-555-0190 gAAAAABgC5RjpqY9uW7D7jmptuZTkA_bWeRgpkrrW8p4_FvlR8lGjmP9wW8mIBzYaqVn7L2oVr2dvnU7lbrAkFLlFpqOPr7swrQ==
5 Sarah Brown 617-555-0157 gAAAAABgC5RjpqHghXVV4ng6dH3u-uW5vYwI9YFw_1rHhRa6kHppZFOo62SQ2S4ssX5xhgAKql3XxkWyWnDOFr4pTVy0C1nQWw==
6 James Miller 718-555-0163 gAAAAABgC5RjpqNks9z7A2k0l3d7cYlo2Wpt_i4ALi9xkJ1BLm1HYp7PYkPiwx7v67xmvYJdHE8rfyyg2WpXWYZVBeRZ4WnVrQ==
7 Laura Taylor 202-555-0137 gAAAAABgC5RjpqHhwHT54FZGVvFS2jLDosUKUpEjijM7HNd5_Vb6pAtxmwk-6GEms-FBlUm9V_sGrJecjIc4NdGxykgfPQCqdPA==
8 Daniel Anderson 213-555-0181 gAAAAABgC5RjpqCxxd_r4y7Ffu38iAi-sYtbYt5Ev4UN3F2I9GiOCfcSy88fhNOgqR6oHg7l6duK8ns4OeO8j5VnLo1BKT8PpA==
9 Olivia Thomas 303-555-0149 gAAAAABgC5RjpqGBYbzZdwIUjbMbdRe3n9fowRH7OtEXoHRQF2fW2dz6zQYwssA9rjvdk5Q5soIkeFdgsBoQOTdUpjpA0-l2gSg==
Phone Number_Decrypted
0 202-555-0198
1 305-555-0172
2 212-555-0145
3 415-555-0109
4 512-555-0190
5 617-555-0157
6 718-555-0163
7 202-555-0137
8 213-555-0181
9 303-555-0149Encryption can effectively protect sensitive information and allows decryption when needed to restore the original data.
Summary
In this post we provide a detailed overview of various methods for handling sensitive information using Pandas, including desensitization or anonymization, hashing, randomization, and encryption. Desensitization and masking are suitable for data presentation, encryption is used in scenarios where data needs to be restored, and hashing and randomization are ideal for complete de-identification. In practical applications, the most appropriate method should be chosen based on the data characteristics and application scenario to effectively protect the security of sensitive information.
