Module — Modules are used to organize code into reusable and logical components, making it easier to manage and maintain larger projects. It contains Python definitions and statements.
Create module with ‘.py’ extension with the desired module name
Define functions, classes, and variables inside the file
Use it in other Python scripts by importing it
Can import modules from Python’s standard libraries
# using alias import my_module as mm mm.say_hello() print(mm.favorite_number)
#import specific item from my_module import say_hello, favorite_number say_hello() print(favorite_number)
Package : Package is a way to organize related modules into a directory hierarchy. Packages are essentially directories that contain one or more module files, along with an additional special file called __init__.py. These __init__.py files can be empty or can contain initialization code for the package.
Create a directory that serve as the package’s root directory. Place your module files and an __init__.py file.
Inside the package directory, you can have multiple module files (e.g., module1.py, module2.py). These module files contain the actual code you want to organize and use.
Importing a package works the same way as importing a module, but you include the package name followed by the module name.
from my_package.module1 import function_in_module1 function_in_module1()
Packages can be nested within other packages, creating a hierarchical structure.
Standard libraries
MATH
import math # the smallest integer greater than or equal to x. math.ceil(20.222) # 21 # the largest integer less than or equal to x math.floor(20.222) # 20 # factorial of a number math.factorial(10) # 3628800
RANDOM
import random # select a random element from a list random.choice(['apple', 'pear', 'banana']) #'apple' # select 10 different numbers within a range of numbers random.sample(range(10,50), 10) # [11, 34, 47, 27, 45, 37, 46, 16, 28, 35] # select a random float number random.random() #0.7673947872076888 # select a random float number within a range. random.uniform(0,2) #1.795804895182703 # random integer chosen from range(6) random.randrange(3,6) #3 random.randint(1,10) #1
DATE TIME
Directive Meaning %a Weekday as locale's abbreviated name. %A Weekday as locale’s full name. %d Day of the month as a zero-padded decimal number. %b Month as locale’s abbreviated name. %B Month as locale’s full name. %m Month as a zero-padded decimal number. %y Year without century as a zero-padded decimal number. %Y Year with century as a decimal number. %H Hour (24-hour clock) as a zero-padded decimal number.
# dates are easily constructed and formatted from datetime import date # get the current date TODAY = date.today() print("Date: ", TODAY) # Date: 2023–08–09
print("Formatted date:", TODAY.strftime("%A on the %d day of %B.")) # Formatted date: Wednesday on the 09 day of August.
birthday = date(1964, 7, 31) age = TODAY - birthday print("Age in days:", age.days # Age in days: 20297
OPERATING SYSTEM
The os module provides dozens of functions for interacting with the operating system
import os # Return the current working directory print("Get current working directory:", os.getcwd()) Get current working directory: /home/mbuttar/Documents/Python Course
# Change current working directory print("Change current directory to sample_data") os.system('mkdir sample_data') os.chdir('sample_data/') # Change current directory to sample_data
# Run the command mkdir in the system shell print("Make new directory") os.system('mkdir today') # Make new directory
# Check if new directory got created print("Listing existing directories:")os.system('ls') # Listing existing directories:
FILE HANDLING
Text Files — .txt
File access Modes
Read Only ('r') : It opens the text file for reading. If the file does not exists, raises I/O error. Read and Write ('r+') : It opens the file for reading and writing. Raises I/O error if the file does not exists. Write Only ('w') : It opens the file for writing. For existing file, the data is over-written. Creates the file if the file does not exists. Write and Read ('w+') : It opens the file for reading and writing. For existing file,data is overwritten. Append Only ('a') : It opens the file for writing. The file is created if it does not exist. The data will be inserted at the end. Append and Read ('a+') : It opens the file for reading and writing. The file is created if it does not exist. The data will be inserted at the end.
Reading Text File
# Open the file in read only mode. text_file = open("dataset/file_1.txt", "r")
# read the file and store the text in a variable complete_text = text_file.read() complete_text # will display the text from the above file
Read the file again
# read the file again ; cannot re-read same stuff complete_text = text_file.read() complete_text # no data displayed, close it to re-read
text_file.close() text_file= open("dataset/file_1.txt", "r") complete_text=text_file.read() complete_text # will display the text from the above file again
Seek function — moving the pointer to the starting point again to re-read the file.
text_file.seek(0) complete_text.read() # will display the text again fom starting complete_text.seek(300) # read from 300th character in the file
Miscellaneous
# close the file text_file.close()
# Read the file line by line text_file = open("dataset/file_1.txt", "r") lines = text_file.readlines()
# no of lines print('No of lines in file is ',format(len(lines)) # Display no of lines
# print each line text for line in lines : print(line)
# Another way to read text files with open('dataset/file_1.txt','r') as f: complete_text = f.read() complete_text # Using this, python will automatically closes the file after reading it.
Get Manpreet Buttar’s stories in your inbox
Join Medium for free to get updates from this writer.
WRITING TEXT FILES
# let’s open a file in write only mode my_file = open('dataset/new_file.txt','w') # write function will return the length of text written in the file. my_file.write('Data Science is Awesome!') # 24; it will give no of chars
# open the file write and read mode my_file = open('dataset/new_file.txt','w+')
# read the file my_file.read() #'Data Science is Awesome!'
# close the file my_file.close()
# add some lines to the file again my_file = open('dataset/new_file.txt','w+')
# create a list of strings lines = ['Line 1','Line 2','Line 3','Line 4'] my_file.writelines(lines) # take the pointer to the initial position my_file.seek(0)
# read the data print(my_file.read()) my_file.close()
Line 1Line 2Line 3Line 4
# The content is overwritten when we tried to write again on the same file
APPEND TEXT IN FILE
# open the file in the append and read mode. my_file = open('dataset/new_file.txt','a+') my_file.write('Welcome to the python course')
# take the pointer to the initial position of the file my_file.seek(0) # read the file my_file.read()
'Line 1Line 2Line 3Line 4Welcome to the python course'
# import the pandas library import pandas as pd # check the python version pd.__version__ '1.5.3'
# read the dataset data = pd.read_csv('datasets/big_mart_sales.csv') type(data) # pandas.core.frame.DataFrame
DATAFRAME — DataFrame is a 2-dimensional data structure with columns that can contain different data type. Similar to a spreadsheet or SQL table.It is generally the most commonly used pandas object.
data.head() # to get snippet of data
Press enter or click to view image in full size
data.shape # to check the dimension of the data set; (8523, 12)
CHALLENGES WITH READING CSV FILE
Read the data except the first few rows in the file. To avoid parse error use skiprows parameter.
Read files present in multiple directories. Use glob library
# import the library import glob # list all the files in the folder for directory in glob.glob('datasets/multi-directory/*'): print(directory)
# Concatenate CSV's data_frame_list = [] # list to store data frames # iterate through each folder for directory in glob.glob('datasets/multi-directory/*'): # iterate through each file for files in glob.glob(directory + '/*'): # read and append the data frame data_frame_list.append(pd.read_csv(files)) len(data_frame_list) # concatenate the dataframes final_data = pd.concat(data_frame_list) final_data.head() final_data.shape # Store CSV final_data.to_csv('datasets/merged_big_mart_sales.csv',index=False)
Check and delimiter and read file.
# View the data, see the delimiter data_delimiter = pd.read_csv('datasets/big_mart_sales_delimiter.csv')
# read the file again with delimiter parameter data_delimiter = pd.read_csv('datasets/big_mart_sales_delimiter.csv',delimiter='\t') # view the data data_delimiter.head()
If the dataset is bigger in size, we can read a few rows only.
# Specify the number of rows to read read_sample_from_data = pd.read_csv('datasets/big_mart_sales.csv',nrows=100) # shape of the data read_sample_from_data.shape # view the data read_sample_from_data.head()
If the dataset has large number of columns, read only specific columns
# unique year present in the data data_with_multiple_sheets.Outlet_Establishment_Year.unique() array([1985])
Comments in top rows.
# read the data with skiprows sheet_with_comments = pd.read_excel('datasets/big_mart_sales_comments.xlsx',skiprows=3) # view the top rows of the data sheet_with_comments.head()
# create an array of dataframes sheets = [sheet_1985, sheet_1987, sheet_1997] # concatenate the array final_data = pd.concat(sheets)
# unique years in final data final_data.Outlet_Establishment_Year.unique() array([1985, 1987, 1997])
# To find the files in directories import glob for directory in glob.glob('datasets/multi-directory-excel/*'): for files in glob.glob(directory + '/*'): print(files)
JSON FILES
READING JSON FILES
# import the pandas library import pandas as pd # read the json file data = pd.read_json('datasets/simple.json') # print the top rows of the dataframe data.head()
Challenges with reading JSON files.
Reading JSON files written as records (different lines or nested json).
# read json files with records in differnet lines data_with_records = pd.read_json('datasets/simple_records.json',lines=True) data_with_records.head()
# importing the json module of standard library import json # open and load the data in json file with open('datasets/nested.json') as f : my_json_data = json.load(f) print(my_json_data)
# use pprint or (pretty print) to print the data in the structured format from pprint import pprint pprint(my_json_data)
FILE COMPARISON
import pandas as pd
def compare_and_process_files(file1_path, file2_path): # Read the CSV files into DataFrames df1 = pd.read_csv(file1_path) df2 = pd.read_csv(file2_path)
# Merge the DataFrames on a common key column merged_df = pd.merge(df1, df2, on='common_key_column', how='outer', indicator=True)
# Identify differences using the '_merge' column differences = merged_df[merged_df['_merge'] != 'both']
# Process the differences (for example, print or write to another file) if not differences.empty: print("Differences found:") print(differences) # You can add your custom processing logic here
