Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Polars pl.col(field).name.map_fields applies to all struct columns (not the one specified)

I have this code:

import polars as pl

cols = ['Delta', 'Qty']

metrics = {'CHECK.US': {'Delta': {'ABC': 1, 'DEF': 2}, 'Qty': {'GHIJ': 3, 'TT': 4}},
           'CHECK.NA': {},
           'CHECK.FR': {'Delta': {'QQQ': 7, 'ABC': 6}, 'Qty': {'SS': 9, 'TT': 5}}
          }

df = pl.DataFrame([{col: v.get(col) for col in cols} for v in metrics.values()])\
       .insert_column(0, pl.Series('key', metrics.keys()))\
       .with_columns([pl.col(col).name.map_fields(lambda x: f'{col} ({x})') for col in cols])

Now, df.unnest('Qty') correctly gives all columns formatted as Qty (xxx):

shape: (3, 5)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ key      ┆ Delta      ┆ Qty (GHIJ) ┆ Qty (TT) ┆ Qty (SS) β”‚
β”‚ ---      ┆ ---        ┆ ---        ┆ ---      ┆ ---      β”‚
β”‚ str      ┆ struct[3]  ┆ i64        ┆ i64      ┆ i64      β”‚
β•žβ•β•β•β•β•β•β•β•β•β•β•ͺ════════════β•ͺ════════════β•ͺ══════════β•ͺ══════════║
β”‚ CHECK.US ┆ {1,2,null} ┆ 3          ┆ 4        ┆ null     β”‚
β”‚ CHECK.NA ┆ null       ┆ null       ┆ null     ┆ null     β”‚
β”‚ CHECK.FR ┆ {6,null,7} ┆ null       ┆ 5        ┆ 9        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

However, when I do the same thing for df.unnest('Delta') it incorrectly returns columns with Qty (xxx):

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

shape: (3, 5)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ key      ┆ Qty (ABC) ┆ Qty (DEF) ┆ Qty (QQQ) ┆ Qty        β”‚
β”‚ ---      ┆ ---       ┆ ---       ┆ ---       ┆ ---        β”‚
β”‚ str      ┆ i64       ┆ i64       ┆ i64       ┆ struct[3]  β”‚
β•žβ•β•β•β•β•β•β•β•β•β•β•ͺ═══════════β•ͺ═══════════β•ͺ═══════════β•ͺ════════════║
β”‚ CHECK.US ┆ 1         ┆ 2         ┆ null      ┆ {3,4,null} β”‚
β”‚ CHECK.NA ┆ null      ┆ null      ┆ null      ┆ null       β”‚
β”‚ CHECK.FR ┆ 6         ┆ null      ┆ 7         ┆ {null,5,9} β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The values look correct, just the column names are wrong.

Am I using pl.col(col).name.map_field(...) incorrectly? How can I fix my code so that the output becomes this:

shape: (3, 5)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ key      ┆ Delta (ABC) ┆ Delta (DEF) ┆ Delta (QQQ) ┆ Qty        β”‚
β”‚ ---      ┆ ---         ┆ ---         ┆ ---         ┆ ---        β”‚
β”‚ str      ┆ i64         ┆ i64         ┆ i64         ┆ struct[3]  β”‚
β•žβ•β•β•β•β•β•β•β•β•β•β•ͺ═════════════β•ͺ═════════════β•ͺ═════════════β•ͺ════════════║

?

>Solution :

It’s a general Python "Gotcha" with regards to lambdas inside loops.

col is being set to the last value in the loop.

The workaround is to use named params.

pl.col(col).name.map_fields(lambda x, col=col: f'{col} ({x})') for col in cols  
#                                     ^^^^^^^   

1. https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading