如何提高熊猫行的运行速度？

debugcn 发表于 Dev

米歇尔·阿维拉（Michele Avella）

我有一个很大的.csv文件，它具有11'000'000行和3列：id，magh和mixid2。我要做的是选择具有相同ID的行，然后检查这些行是否具有相同的mixid2；如果为True，则删除行；如果为False，则使用所选行的信息初始化一个类。那是我的代码：

obs=obs.set_index('id')
obs=obs.sort_index()
#dropping elements with only one mixid2 and filling S
ID=obs.index.unique()
S=[]
good_bye_list = []
for i in tqdm(ID):
    app=obs.loc[i]
    if len(np.unique([app['mixid2'],])) != 1:
        #fill the class list
        S.append(star(app['magh'].values,app['mixid2'].values,z_in))
    else :
    #drop
        good_bye_list.append(i)

obs=obs.drop(good_bye_list)

.csv文件很大，因此计算所有内容需要40分钟。如何提高速度？

感谢您的帮助。

这是.csv文件：

id,mixid2,magh
3447001203296326,557,14.25
3447001203296326,573,14.25
3447001203296326,525,14.25
3447001203296326,541,14.25
3447001203296330,540,15.33199977874756
3447001203296330,573,15.33199977874756
3447001203296333,172,17.476999282836914
3447001203296333,140,17.476999282836914
3447001203296333,188,17.476999282836914
3447001203296333,156,17.476999282836914
3447001203296334,566,15.626999855041506
3447001203296334,534,15.626999855041506
3447001203296334,550,15.626999855041506
3447001203296338,623,14.800999641418455
3447001203296338,639,14.800999641418455
3447001203296338,607,14.800999641418455
3447001203296344,521,12.8149995803833
3447001203296344,537,12.8149995803833
3447001203296344,553,12.8149995803833
3447001203296345,620,12.809000015258787
3447001203296345,543,12.809000015258787
3447001203296345,636,12.809000015258787
3447001203296347,558,12.315999984741213
3447001203296347,542,12.315999984741213
3447001203296347,526,12.315999984741213
3447001203296352,615,12.11299991607666
3447001203296352,631,12.11299991607666
3447001203296352,599,12.11299991607666
3447001203296360,540,16.926000595092773
3447001203296360,556,16.926000595092773
3447001203296360,572,16.926000595092773
3447001203296360,524,16.926000595092773
3447001203296367,490,15.80799961090088
3447001203296367,474,15.80799961090088
3447001203296367,458,15.80799961090088
3447001203296369,639,15.175000190734865
3447001203296369,591,15.175000190734865
3447001203296369,623,15.175000190734865
3447001203296369,607,15.175000190734865
3447001203296371,460,14.975000381469727
3447001203296373,582,14.532999992370605
3447001203296373,614,14.532999992370605
3447001203296373,598,14.532999992370605
3447001203296374,184,14.659000396728516
3447001203296374,203,14.659000396728516
3447001203296374,152,14.659000396728516
3447001203296374,136,14.659000396728516
3447001203296374,168,14.659000396728516
3447001203296375,592,14.723999977111815
3447001203296375,608,14.723999977111815
3447001203296375,624,14.723999977111815
3447001203296375,92,14.723999977111815
3447001203296375,76,14.723999977111815
3447001203296375,108,14.723999977111815
3447001203296375,576,14.723999977111815
3447001203296376,132,14.0649995803833
3447001203296376,164,14.0649995803833
3447001203296376,180,14.0649995803833
3447001203296376,148,14.0649995803833
3447001203296377,168,13.810999870300293
3447001203296377,152,13.810999870300293
3447001203296377,136,13.810999870300293
3447001203296377,184,13.810999870300293
3447001203296378,171,13.161999702453613
3447001203296378,187,13.161999702453613
3447001203296378,155,13.161999702453613
3447001203296378,139,13.161999702453613
3447001203296380,565,13.017999649047852
3447001203296380,517,13.017999649047852
3447001203296380,549,13.017999649047852
3447001203296380,533,13.017999649047852
3447001203296383,621,13.079999923706055
3447001203296383,589,13.079999923706055
3447001203296383,605,13.079999923706055
3447001203296384,541,12.732000350952148
3447001203296384,557,12.732000350952148
3447001203296384,525,12.732000350952148
3447001203296385,462,12.784000396728516
3447001203296386,626,12.663999557495115
3447001203296386,610,12.663999557495115
3447001203296386,577,12.663999557495115
3447001203296389,207,12.416000366210938
3447001203296389,255,12.416000366210938
3447001203296389,223,12.416000366210938
3447001203296389,239,12.416000366210938
3447001203296390,607,12.20199966430664
3447001203296390,591,12.20199966430664
3447001203296397,582,16.635000228881836
3447001203296397,598,16.635000228881836
3447001203296397,614,16.635000228881836
3447001203296399,630,17.229999542236328
3447001203296404,598,15.970000267028807
3447001203296404,631,15.970000267028807
3447001203296404,582,15.970000267028807
3447001203296408,540,16.08799934387207
3447001203296408,556,16.08799934387207
3447001203296408,524,16.08799934387207
3447001203296408,572,16.08799934387207
3447001203296409,632,15.84000015258789
3447001203296409,616,15.84000015258789

您好，欢迎来到StackOverflow。

在熊猫中，经验法则是原始循环总是比专用功能慢。要将函数应用于满足某些条件的行的子DataFrame，可以使用groupby

在您的情况下，该函数有点...令人难以置信，因为的实例化S是一个副作用，并且删除当前正在迭代的行是危险的。例如，在字典中，您永远不要这样做。也就是说，您可以创建如下函数：

In [37]: def my_func(df): 
    ...:     if df['mixid2'].nunique() == 1: 
    ...:         return None 
    ...:     else: 
    ...:         S.append(df['mixid2']) 
    ...:         return df

并通过以下方式将其应用于您的DataFrame

S = []
obs.groupby('id').apply(my_func)

这将迭代所有具有相同子数据帧的子数据帧，id如果中恰好有一个唯一值，则将其丢弃mixid2。否则，它将值附加到列表中S

产生的DataFrame短3行

Out[38]: 
                                   id  mixid2       magh
id                                                      
3447001203296326 0   3447001203296326     557  14.250000
                 1   3447001203296326     573  14.250000
...                               ...     ...        ...
3447001203296409 98  3447001203296409     632  15.840000
                 99  3447001203296409     616  15.840000

[97 rows x 3 columns]

并S包含28个元素。star就像您一样可以将其传递给构造函数。

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。