Tag: apl

Pyspark toPandas ValueError: Found non-unique column index

I get the following error when I try to convert pyspark dataframe to pandas dataframe with the method toPandas. I don’t understand the reason for the error: ————————————————————————— ValueError Traceback (most recent call last) /tmp/ipykernel_64705/3870041712.py in <module> —-> 1 df_who.limit(10).toPandas() /opt/miniforge/miniforge/envs/jupyterlab/lib/python3.7/site-packages/pyspark/sql/dataframe.py in toPandas(self) 2130 if len(batches) > 0: 2131 table = pyarrow.Table.from_batches(batches) -> 2132 pdf… Read More Pyspark toPandas ValueError: Found non-unique column index

MRLeave a comment

How to create hiearchical multi-index dataframe from multiple dataframes?

I have multiple dataframes with identical shape, say: d1 = {‘time’: [1,2,3,4], ‘A’: [55.5,55.5,55.5,55.5], ‘B’:[55.5,55.5,55.5,55.5]} d2 = {‘time’: [1,2,3,4], ‘A’: [7,6,5,4], ‘B’:[9,8,7,6]} d3 = {‘time’: [1,2,3,4], ‘A’: [1,2,3,4], ‘B’:[2,3,4,5]} df1 = pd.DataFrame(data = d1) df2 = pd.DataFrame(data = d2) df3 = pd.DataFrame(data = d3) They will always have the same column names and the entries… Read More How to create hiearchical multi-index dataframe from multiple dataframes?

MRLeave a comment

Even Fibonacci numbers Sum

Even Fibonacci numbers sum whose values do not exceed four million. I am using multiline function in APL, but not getting the output: result←Euler2 a;b;c;sum;i;limit b←0 c←1 sum←0 i←0 limit←4000000 :For i :In limit a←b b←c c←a+b :If (0=2|c) sum←(sum+c) :EndIf :EndFor result←sum >Solution : Warning: opinions below. I would avoid using manual loops and… Read More Even Fibonacci numbers Sum

MRLeave a comment