将多个csv文件导入到pandas中并串联到一个DataFrame中

本文翻译自：Import multiple csv files into pandas and concatenate into one DataFrame

I would like to read several csv files from a directory into pandas and concatenate them into one big DataFrame. 我想将目录中的多个csv文件读入pandas，并将它们连接成一个大的DataFrame。 I have not been able to figure it out though. 我还无法弄清楚。 Here is what I have so far: 这是我到目前为止的内容：

import glob
import pandas as pd

# get data file names
path =r'C:\DRO\DCL_rawdata_files'
filenames = glob.glob(path + "/*.csv")

dfs = []
for filename in filenames:
    dfs.append(pd.read_csv(filename))

# Concatenate all data into one DataFrame
big_frame = pd.concat(dfs, ignore_index=True)

I guess I need some help within the for loop??? 我想我在for循环中需要一些帮助吗？？？

#1楼

参考：https://stackoom.com/question/1PijC/将多个csv文件导入到pandas中并串联到一个DataFrame中

#2楼

If you have same columns in all your csv files then you can try the code below. 如果所有csv文件中的列均相同，则可以尝试以下代码。 I have added header=0 so that after reading csv first row can be assigned as the column names. 我添加了header=0以便在读取csv之后可以将第一行分配为列名。

import pandas as pd
import glob

path = r'C:\DRO\DCL_rawdata_files' # use your path
all_files = glob.glob(path + "/*.csv")

li = []

for filename in all_files:
    df = pd.read_csv(filename, index_col=None, header=0)
    li.append(df)

frame = pd.concat(li, axis=0, ignore_index=True)

#3楼

Edit: I googled my way into https://stackoverflow.com/a/21232849/186078 . 编辑：我用谷歌搜索https://stackoverflow.com/a/21232849/186078 。 However of late I am finding it faster to do any manipulation using numpy and then assigning it once to dataframe rather than manipulating the dataframe itself on an iterative basis and it seems to work in this solution too. 但是，最近我发现使用numpy进行任何操作，然后将其分配给数据框一次，而不是在迭代的基础上操纵数据框本身，这样更快，并且似乎也可以在此解决方案中工作。

I do sincerely want anyone hitting this page to consider this approach, but don't want to attach this huge piece of code as a comment and making it less readable. 我确实希望任何访问此页面的人都考虑采用这种方法，但又不想将这段巨大的代码作为注释附加到可读性较低的地方。

You can leverage numpy to really speed up the dataframe concatenation. 您可以利用numpy真正加快数据帧的连接速度。

import os
import glob
import pandas as pd
import numpy as np

path = "my_dir_full_path"
allFiles = glob.glob(os.path.join(path,"*.csv"))


np_array_list = []
for file_ in allFiles:
    df = pd.read_csv(file_,index_col=None, header=0)
    np_array_list.append(df.as_matrix())

comb_np_array = np.vstack(np_array_list)
big_frame = pd.DataFrame(comb_np_array)

big_frame.columns = ["col1","col2"....]

Timing stats: 时间统计：

total files :192
avg lines per file :8492
--approach 1 without numpy -- 8.248656988143921 seconds ---
total records old :1630571
--approach 2 with numpy -- 2.289292573928833 seconds ---

#4楼

An alternative to darindaCoder's answer : 替代darindaCoder的答案 ：

path = r'C:\DRO\DCL_rawdata_files'                     # use your path
all_files = glob.glob(os.path.join(path, "*.csv"))     # advisable to use os.path.join as this makes concatenation OS independent

df_from_each_file = (pd.read_csv(f) for f in all_files)
concatenated_df   = pd.concat(df_from_each_file, ignore_index=True)
# doesn't create a list, nor does it append to one

#5楼

If the multiple csv files are zipped, you may use zipfile to read all and concatenate as below: 如果压缩了多个csv文件，则可以使用zipfile读取所有文件并进行如下连接：

import zipfile
import numpy as np
import pandas as pd

ziptrain = zipfile.ZipFile('yourpath/yourfile.zip')

train=[]

for f in range(0,len(ziptrain.namelist())):
    if (f == 0):
        train = pd.read_csv(ziptrain.open(ziptrain.namelist()[f]))
    else:
        my_df = pd.read_csv(ziptrain.open(ziptrain.namelist()[f]))
        train = (pd.DataFrame(np.concatenate((train,my_df),axis=0), 
                          columns=list(my_df.columns.values)))

#6楼

import glob, os    
df = pd.concat(map(pd.read_csv, glob.glob(os.path.join('', "my_files*.csv"))))