Search
Search

Can I use SELECT from dataframe instead of creating this temp table?

debugcn Published at Dev
10
Aayush Rampal

I am currently using :

+---+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
|id |sen                      |attributes                                                                                                                                                    |
+---+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
|1  |Stanford is good college.|[[Stanford,ORGANIZATION,NNP], [is,O,VBZ], [good,O,JJ], [college,O,NN], [.,O,.], [Stanford,ORGANIZATION,NNP], [is,O,VBZ], [good,O,JJ], [college,O,NN], [.,O,.]]|
+---+-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+

 I want to get above df from :
+----------+--------+--------------------+
|article_id|     sen|           attribute|
+----------+--------+--------------------+
|         1|example1|[Standford,Organi...|
|         1|example1|           [is,O,VP]|
|         1|example1| [good,LOCATION,ADP]|
+----------+--------+--------------------+

using :

df3.registerTempTable("d1")
val df4 = sqlContext.sql("select article_id,sen,collect(attribute) as attributes from d1 group by article_id,sen")

Is there any way that I don't have to register temp table, as while saving dataframe, it is giving lot of garbage!! Something lige df3.Select""??

Sim

The only way Spark currently has to run SQL against a dataframe is via a temporary table. However, you can add implicit methods to DataFrame to automate this, as we have done at Swoop. I can't share all the code as it uses a number of our internal utilities & implicits but the core is in the following gist. The importance of using unique temporary tables is that (at least until Spark 2.0) temporary tables are cluster global.

We use this approach regularly in our work, especially since there are many situations in which SQL is much simpler/easier to write and understand than the Scala DSL.

Hope this helps!

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Can I use SELECT from dataframe instead of creating this temp table?

From Dev

how can i get a temp table result from a string?

From Dev

Select from two temp tables into another temp table

From Dev

How can I use melt() to reshape a pandas DataFrame to a list, creating an index from a crosstab column and creating a new variable in its place?

From Dev

How can I use melt() to reshape a pandas DataFrame to a list, creating an index from a crosstab column and creating a new variable in its place?

From Dev

MySQL: Cursor for select from temp table

From Dev

Select * into Temp Table from Dynamic SQL Results

From Dev

Select as in SQL while selecting from Temp table

From Dev

using same select multiple times vs creating temp table

From Dev

How can I use a variable from one table as a SELECT parameter for another table in php

From Dev

Creating Database Table From Temp-Table (by Code)

From Dev

Creating Database Table From Temp-Table (by Code)

From Dev

Can I select from a destination table in a subquery?

From Dev

UPDATE - SELECT - MYSQL #1093 - You can't specify target table 'temp1' for update in FROM clause

From Dev

How do I get the stored procedure result set into temp table without creating temp Table

From Dev

Can I use LEFT OUTER JOIN, and NULL, instead of one table?

From Dev

Can I use pace instead of data-table's processing?

From Dev

T-SQL syntax on creating temp table from union

From Dev

How can i use SQL Select Insert to copy rows from one table to another

From Dev

Trying to use "Insert into temp2" instead of this "Select into temp2" code in MS Access

From Dev

instead of xpath what can i select from the below code

From Dev

When creating a SQL view can i have temp tables?

From Dev

Select into temp table in PostgreSQL?

From Dev

How can I select from many table to one table?

From Dev

select data from #Temp table after #temp table create in another query in asp.net c#

From Dev

Get a sum from a temp table into my select query

From Dev

Select into a temp table from within a while loop (cursor)

From Dev

how can i select one in table use watirwebdriver?

From Dev

How can I select data from a dask dataframe by a list of indices?

Related Related

  1. 1

    Can I use SELECT from dataframe instead of creating this temp table?

  2. 2

    how can i get a temp table result from a string?

  3. 3

    Select from two temp tables into another temp table

  4. 4

    How can I use melt() to reshape a pandas DataFrame to a list, creating an index from a crosstab column and creating a new variable in its place?

  5. 5

    How can I use melt() to reshape a pandas DataFrame to a list, creating an index from a crosstab column and creating a new variable in its place?

  6. 6

    MySQL: Cursor for select from temp table

  7. 7

    Select * into Temp Table from Dynamic SQL Results

  8. 8

    Select as in SQL while selecting from Temp table

  9. 9

    using same select multiple times vs creating temp table

  10. 10

    How can I use a variable from one table as a SELECT parameter for another table in php

  11. 11

    Creating Database Table From Temp-Table (by Code)

  12. 12

    Creating Database Table From Temp-Table (by Code)

  13. 13

    Can I select from a destination table in a subquery?

  14. 14

    UPDATE - SELECT - MYSQL #1093 - You can't specify target table 'temp1' for update in FROM clause

  15. 15

    How do I get the stored procedure result set into temp table without creating temp Table

  16. 16

    Can I use LEFT OUTER JOIN, and NULL, instead of one table?

  17. 17

    Can I use pace instead of data-table's processing?

  18. 18

    T-SQL syntax on creating temp table from union

  19. 19

    How can i use SQL Select Insert to copy rows from one table to another

  20. 20

    Trying to use "Insert into temp2" instead of this "Select into temp2" code in MS Access

  21. 21

    instead of xpath what can i select from the below code

  22. 22

    When creating a SQL view can i have temp tables?

  23. 23

    Select into temp table in PostgreSQL?

  24. 24

    How can I select from many table to one table?

  25. 25

    select data from #Temp table after #temp table create in another query in asp.net c#

  26. 26

    Get a sum from a temp table into my select query

  27. 27

    Select into a temp table from within a while loop (cursor)

  28. 28

    how can i select one in table use watirwebdriver?

  29. 29

    How can I select data from a dask dataframe by a list of indices?

HotTag

Archive