Deleting rows of data for multiple variables

E. Burrows

I have over 500 files that I cleaned up using a pandas data frame, and read in later as a matrix. I now want to delete missing rows of data from multiple variables for the entirety of my files. Each variable is pretty lengthy for its shape, for example, tc and wspd have the shape (84479, 558) and pressure has the shape (558,). I have tried the following example before and has worked in the past for single dimensional arrays with the same shape, but will no longer work with a two dimensional array.

    bad=[]
    for i in range(len(p)):
        if p[i]==-9999 or tc[i]==-9999:
            bad.append(i)
    p=numpy.delete(p, bad)
    tc=numpy.delete(tc, bad)

I tried using the following code instead but with no success (unfortunately).

import numpy as n 
import pandas as pd

wspd=pd.read_pickle('/home/wspd').as_matrix()
tc=pd.read_pickle('/home/tc').as_matrix()

press=n.load('/home/file1.npz')
p=press['press']
names=press['names']

length=n.arange(0,84479)
for i in range(len(names[0])): #using the first one as a trial to run faster
    print i #used later to see how far we have come in the 558 files
    bad=[]
    for j in range(len(length)):
        if (wspd[j,i]==n.nan or tc[j,i]==n.nan):
            bad.append(j)
        print bad

From there I plan on deleting missing data as I had done previously except indexing which dimension I am deleting from within my first forloop.

     new_tc=n.delete(tc[j,:], bad)

Unfortunately, this has not worked. I have also tried masking the array which also has not worked.

The reason I need to delete the data is my next library does not understand nan values, it requires strictly integers, floats, etc.

I am open to new methods for removing rows of data if anyone has any guidance. I greatly appreciate it.

bunji

I would load your 2 dimensional arrays as pandas DataFrames and then use the dropna function to drop any rows that contain a null value

wspd = pd.read_pickle('/home/wspd').dropna()
tc = pd.read_pickle('/home/tc').dropna()

The documentation for pandas.DataFrame.dropna is here

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Deleting rows with variables

From Dev

Deleting multiple rows in datagridview

From Dev

Deleting multiple rows from a table

From Dev

Deleting multiple rows from a table

From Dev

Deleting multiple rows from TableView

From Dev

Shiny: Deleting rows from data

From Dev

VBA - Deleting matching data and its rows over multiple sheets based on a range of data

From Dev

Reading text file in php with variables and data records as multiple rows

From Dev

Deleting rows based on multiple conditions Python Pandas

From Dev

Deleting multiple rows in Excel using Apache POI

From Dev

Deleting rows from multiple tables with jQuery

From Dev

Deleting multiple rows at once with Doctrine DBAL, is it possible?

From Dev

Deleting Rows Based on Multiple Cell Values

From Dev

Multiple rows deleting in jtable with the same value

From Dev

Deleting multiple rows with array of _id mongodb

From Dev

Deleting a pointer which is used by multiple variables

From Dev

Deleting multiple variables each pass of a loop

From Dev

Deleting data of rows selected from object?

From Dev

Count rows in dataframe for multiple variables

From Dev

Deleting rows and columns of a data frame based on values of another data frame

From Dev

Deleting multiple rows using checkboxes in Struts2

From Dev

Python - NumPy - deleting multiple rows and columns from an array

From Dev

Deleting multiple rows by ids and checking if those ids are not in other tables

From Dev

Deleting multiple rows based on where clause using FluentMigrator

From Dev

Deleting DataFrame rows in Pandas based on column value - multiple values to remove

From Dev

Trying to delete selected row from datagridview but it is deleting multiple rows

From Dev

Deleting multiple rows in R based on a is.na condition

From Dev

Deleting/Highlighting Duplicate rows in Excel across multiple columns

From Dev

Deleting rows based on multiple Joins and where statements in SQL

Related Related

  1. 1

    Deleting rows with variables

  2. 2

    Deleting multiple rows in datagridview

  3. 3

    Deleting multiple rows from a table

  4. 4

    Deleting multiple rows from a table

  5. 5

    Deleting multiple rows from TableView

  6. 6

    Shiny: Deleting rows from data

  7. 7

    VBA - Deleting matching data and its rows over multiple sheets based on a range of data

  8. 8

    Reading text file in php with variables and data records as multiple rows

  9. 9

    Deleting rows based on multiple conditions Python Pandas

  10. 10

    Deleting multiple rows in Excel using Apache POI

  11. 11

    Deleting rows from multiple tables with jQuery

  12. 12

    Deleting multiple rows at once with Doctrine DBAL, is it possible?

  13. 13

    Deleting Rows Based on Multiple Cell Values

  14. 14

    Multiple rows deleting in jtable with the same value

  15. 15

    Deleting multiple rows with array of _id mongodb

  16. 16

    Deleting a pointer which is used by multiple variables

  17. 17

    Deleting multiple variables each pass of a loop

  18. 18

    Deleting data of rows selected from object?

  19. 19

    Count rows in dataframe for multiple variables

  20. 20

    Deleting rows and columns of a data frame based on values of another data frame

  21. 21

    Deleting multiple rows using checkboxes in Struts2

  22. 22

    Python - NumPy - deleting multiple rows and columns from an array

  23. 23

    Deleting multiple rows by ids and checking if those ids are not in other tables

  24. 24

    Deleting multiple rows based on where clause using FluentMigrator

  25. 25

    Deleting DataFrame rows in Pandas based on column value - multiple values to remove

  26. 26

    Trying to delete selected row from datagridview but it is deleting multiple rows

  27. 27

    Deleting multiple rows in R based on a is.na condition

  28. 28

    Deleting/Highlighting Duplicate rows in Excel across multiple columns

  29. 29

    Deleting rows based on multiple Joins and where statements in SQL

HotTag

Archive