I have a word similarity matrix stored as a pandas data-frame where the columns are a "seed set" of ~400 words and the row indexes are a large dictionary of ~50,000 words. The value at any row/column is the similarity from 0 to 1 between the two words.
>>> df_sim_mf.info()
<class 'pandas.core.frame.DataFrame'>
Index: 46265 entries, #angry to wonga
Columns: 451 entries, abandon to wrongs
dtypes: float64(451)
memory usage: 159.5+ MB
>>> df_sim_mf.sample(10).sample(5, axis = 1)
nationality purest unite lawless riot
assaulted 0.114270 -0.140504 0.182024 0.434651 0.510618
peekaboo -0.008734 -0.027742 0.051084 0.260245 0.201117
antibiotic 0.145310 0.270748 -0.126459 -0.083965 0.043086
killin -0.102474 0.123550 0.055935 -0.115381 0.285997
warrior 0.005229 0.281967 0.261230 0.344130 0.359228
actionscript -0.029405 0.077793 0.114047 -0.052599 -0.123401
controversy 0.336688 0.271007 0.373474 0.362565 0.305548
nic 0.164550 -0.159097 0.080056 0.271184 0.231357
healy 0.072831 0.102996 0.286538 0.335697 0.183730
uncovered 0.061310 0.274003 0.328383 0.300315 0.277491
I'm trying to find all words from my large dictionary that are within a certain similarity range from ANY of my "seed set". That is, I'd like to select every row that contains at least one value over 0.75.
Can I do this with a few simple pandas commands?
You could do:
df.loc[(df > 0.75).sum(axis=1) > 0, :]
and get the index attribute if you just want the words.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments