Drop rows with a 'question mark' value in any column in a pandas dataframe

Anonymous

I want to remove all rows (or take all rows without) a question mark symbol in any column. I also want to change the elements to float type.

Input:

X Y Z
0 1 ?
1 2 3
? ? 4
4 4 4
? 2 5

Output:

X Y Z
1 2 3
4 4 4

Preferably using pandas dataframe operations.

jezrael

You can try first find string ? in columns, create boolean mask and last filter rows - use boolean indexing. If you need convert columns to float, use astype:

print ~((df['X'] == '?' )  (df['Y'] == '?' ) | (df['Z'] == '?' ))
0    False
1     True
2    False
3     True
4    False
dtype: bool


df1 = df[~((df['X'] == '?' ) | (df['Y'] == '?' ) | (df['Z'] == '?' ))].astype(float)
print df1
   X  Y  Z
1  1  2  3
3  4  4  4

print df1.dtypes
X    float64
Y    float64
Z    float64
dtype: object

Or you can try:

df['X'] = pd.to_numeric(df['X'], errors='coerce')
df['Y'] = pd.to_numeric(df['Y'], errors='coerce')
df['Z'] = pd.to_numeric(df['Z'], errors='coerce')
print df
    X   Y   Z
0   0   1 NaN
1   1   2   3
2 NaN NaN   4
3   4   4   4
4 NaN   2   5
print ((df['X'].notnull() ) & (df['Y'].notnull() ) & (df['Z'].notnull() ))
0    False
1     True
2    False
3     True
4    False
dtype: bool

print df[ ((df['X'].notnull() ) & (df['Y'].notnull() ) & (df['Z'].notnull() )) ].astype(float)
   X  Y  Z
1  1  2  3
3  4  4  4

Better is use:

df = df[(df != '?').all(axis=1)]

Or:

df = df[~(df == '?').any(axis=1)]

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

How to mark DataFrame rows with nan in any column

From Java

How to drop rows of Pandas DataFrame whose value in a certain column is NaN

From Dev

Drop rows if value in a specific column is not an integer in pandas dataframe

From Dev

Pandas dataframe - identify rows with value over threshold in any column

From Dev

Pandas drop rows in one dataframe that share a common value with a rows in a column of another dataframe

From Dev

Drop rows in pandas dataframe based on columns value

From Dev

pandas subset and drop rows based on column value

From Dev

MySQL: DROP a column with a question mark sign?

From Java

Drop rows on multiple conditions (based on 2 column) in pandas dataframe

From Dev

Python/Pandas: Drop duplicate rows in dataframe, concatenate values in one column

From Dev

Repeat rows in a pandas DataFrame based on column value

From Dev

Drop row in pandas dataframe if any value in the row equals zero

From Dev

Drop row if any column value does not a obey a condition in pandas

From Dev

Python PANDAS: Drop All Rows After First Occurrence of Column Value

From Dev

Drop pandas rows if value is not between two other values on the same column

From Dev

How to print rows if values appear in any column of pandas dataframe

From Dev

Drop rows by string in column value

From Dev

Filter pandas dataframe based on a column: keep all rows if a value is that column

From Dev

Select rows in pandas dataframe for which value in the column is in XY0001-XY0879 where X and Y can be any digit

From Java

Dropping rows from pandas dataframe based on value in column(s)

From Dev

Deleting pandas dataframe rows if value in given column not contained in a list

From Dev

Adding rows that have the same column value in a pandas dataframe

From Dev

Deleting DataFrame rows in Pandas based on column value - multiple values to remove

From Dev

pandas dataframe: how to aggregate a subset of rows based on value of a column

From Dev

How to replicate rows based on value of a column in same pandas dataframe

From Dev

Pandas dataframe remove rows based on index and column value

From Dev

Comparing groups of rows in Pandas Dataframe that share a column value

From Dev

pandas dataframe place rows with same column value together

From Dev

Drop Rows by Multiple Column Criteria in DataFrame

Related Related

  1. 1

    How to mark DataFrame rows with nan in any column

  2. 2

    How to drop rows of Pandas DataFrame whose value in a certain column is NaN

  3. 3

    Drop rows if value in a specific column is not an integer in pandas dataframe

  4. 4

    Pandas dataframe - identify rows with value over threshold in any column

  5. 5

    Pandas drop rows in one dataframe that share a common value with a rows in a column of another dataframe

  6. 6

    Drop rows in pandas dataframe based on columns value

  7. 7

    pandas subset and drop rows based on column value

  8. 8

    MySQL: DROP a column with a question mark sign?

  9. 9

    Drop rows on multiple conditions (based on 2 column) in pandas dataframe

  10. 10

    Python/Pandas: Drop duplicate rows in dataframe, concatenate values in one column

  11. 11

    Repeat rows in a pandas DataFrame based on column value

  12. 12

    Drop row in pandas dataframe if any value in the row equals zero

  13. 13

    Drop row if any column value does not a obey a condition in pandas

  14. 14

    Python PANDAS: Drop All Rows After First Occurrence of Column Value

  15. 15

    Drop pandas rows if value is not between two other values on the same column

  16. 16

    How to print rows if values appear in any column of pandas dataframe

  17. 17

    Drop rows by string in column value

  18. 18

    Filter pandas dataframe based on a column: keep all rows if a value is that column

  19. 19

    Select rows in pandas dataframe for which value in the column is in XY0001-XY0879 where X and Y can be any digit

  20. 20

    Dropping rows from pandas dataframe based on value in column(s)

  21. 21

    Deleting pandas dataframe rows if value in given column not contained in a list

  22. 22

    Adding rows that have the same column value in a pandas dataframe

  23. 23

    Deleting DataFrame rows in Pandas based on column value - multiple values to remove

  24. 24

    pandas dataframe: how to aggregate a subset of rows based on value of a column

  25. 25

    How to replicate rows based on value of a column in same pandas dataframe

  26. 26

    Pandas dataframe remove rows based on index and column value

  27. 27

    Comparing groups of rows in Pandas Dataframe that share a column value

  28. 28

    pandas dataframe place rows with same column value together

  29. 29

    Drop Rows by Multiple Column Criteria in DataFrame

HotTag

Archive