更改从 *.csv 文件读取的数据

debugcn 发表于 Dev

启动 Helenius

我有许多 *.csv 文件，我将它们放在一起进行数据分析。

import csv
import glob
import os
import pandas as pd    

### Tells python where to look for the *.csv files we want to combine. 
mycsvdir1 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\LWIR'
mycsvdir2 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\Manta01'
mycsvdir3 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\SWIR'
mycsvdir4 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\LWIR2'
mycsvdir5 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\Manta012'
mycsvdir6 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\SWIR2'

#### Creates lists of all *.csv files to be combined
thelist = glob.glob(os.path.join(mycsvdir1,'*.csv')) + \
          glob.glob(os.path.join(mycsvdir2,'*.csv')) + \
          glob.glob(os.path.join(mycsvdir3,'*.csv')) + \
          glob.glob(os.path.join(mycsvdir4,'*.csv')) + \
          glob.glob(os.path.join(mycsvdir5,'*.csv')) + \
          glob.glob(os.path.join(mycsvdir6,'*.csv'))

#### Reads each *.csv file with a standard header row for each dataframe
#### so they can be concatenated later
dataframe = []
for csvfile in thelist:
    df = pd.read_csv(csvfile,names=['a','b','c','d','e',\
                                    'f','g','h','i','j',\
                                    'k','l','m','n','o',\
                                    'p','q','r','s'], header=0)
    dataframe.append(df)

#### Takes the individual dataframes and concatenates them into one large *.csv
combined = pd.concat(dataframe, ignore_index = True)
combined.to_csv('combined.csv', index = False)

这正如预期的那样工作，但我需要能够跟踪每一行的来源。在单个 *.csv 文件中，每一行在第一列中包含 1、2、3 或 4，但我想将 L、M 或 H 附加到第一列，具体取决于 *.csv 文件的子目录。 .csv 文件来自。因此，在组合文件中，每个数据行的第一列将包含 L1、L2、L3、L4、M1、M2、M3、M4、H1、H2、H3 或 H4 之一。

我过去所做的是将读取命令按子目录分开，并进行相应的编辑。有没有办法使用我的组合读取命令即时完成，或者将读取命令分开是最佳策略？

编辑：

这是我现在基于第一个答案的内容：

import csv
import glob
import os
import pandas as pd

### Tells python where to look for the *.csv files we want to combine. 
mycsvdir1 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\LWIR'
mycsvdir2 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\Manta01'
mycsvdir3 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\SWIR'
mycsvdir4 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\LWIR2'
mycsvdir5 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\Manta012'
mycsvdir6 = 'C:\\Users\\RDEL1LCH\Documents\QuadcamROI\SWIR2'

alldirs = pd.DataFrame({
    'letter': ['L', 'M', 'H','L', 'M', 'H'], # duplicates are OK
    'csv': [glob.glob(os.path.join(d, '*.csv')) for d in [mycsvdir1, \
            mycsvdir2, mycsvdir3, mycsvdir4, mycsvdir5, mycsvdir6]]
})

# build the list of letters and CSV files
letters = np.repeat(alldirs['letter'], alldirs['csv'].apply(len))
thelist = np.concatenate(alldirs['csv'])

### Reads each *.csv file with a standard header row for each dataframe
### so they can be concatenated later
dataframe = []
for letter, csvfile in pd.Series(thelist,letters).iteritems():
    df = pd.read_csv(csvfile,names=['a','b','c','d','e',\
                                    'f','g','h','i','j',\
                                    'k','l','m','n','o',\
                                    'p','q','r','s'], header=0)
    dataframe.append(df)

### Concatenates dataframes into one large *.csv
combined = pd.concat(dataframe, ignore_index = True)
combined.to_csv('combined.csv', index = False)

但是输出没有改变。每行的第一列仍显示 1、2、3 或 4。我认为问题出在我的 pd.read_csv 调用中，但我不确定如何解决。

代码不同

您使用 DataFrame 本身来执行 letter-CSV 文件映射：

alldirs = pd.DataFrame({
    'letter': ['L', 'M', 'L'], # duplicates are OK
    'csv': [glob.glob(os.path.join(d, '*.csv')) for d in [mycsvdir1, mycsvdir2, mycsvdir3]]
})

# build the list of letters and CSV files
letters = np.repeat(alldirs['letter'], alldirs['csv'].apply(len))
thelist = np.concatenate(alldirs['csv'])

# read each CSV file
for letter, csvfile in pd.Series(thelist, letters).iteritems():
    df = pd.read_csv(...)
    df['a'] = letter + df['a').str

    # if pandas report an error: cannot add string and int together, use
    # df['a'] = letter + df['a'].astype(str).str

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。