Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

How use two columns as a single condition to get results in pyspark

I have:

+-----------+------+
|ColA       |ColB  |
+-----------+------+
|       A   |     B|
|       A   |     D|
|       C   |     U|
|       B   |     B|
|       A   |     B|
+-----------+------+

and I want to get:

+-----------+------+
|ColA       |ColB  |
+-----------+------+
|       A   |     D|
|       C   |     U|
|       B   |     B|
+-----------+------+

I want to "remove" all rows with the combination of "colA == A and colB == B".
When I tried this SQL Statement

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

SELECT * FROM table where (colA != 'A' and colB != 'B')

worked fine.

But when I try to translate to spark (or even to pandas) I got an error.

Py4JError: An error occurred while calling o109.and. Trace:…

#spark
sparkDF.where((sparkDF['colA'] != 'A' & sparkDF['colB'] != 'B')).show() 

#pandas
pandasDF[(pandasDF["colA"]!="A" & pandasDF["colB"]!="B")]

What am I doing wrong here?

>Solution :

Need add parentheses and | for bitwise OR:

pandasDF[(pandasDF["colA"]!="A") | (pandasDF["colB"]!="B")]

sparkDF.where((sparkDF['colA'] != 'A') | (sparkDF['colB'] != 'B')).show() 
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading