Python csv.DictReader. Read alter dictionary then output to file

February 9, 2022

I need to read in a pipe delimited file change some fields and output. It reads in fine and I am able to change the columns. The writer part writes the header but writerows doesn’t write anything. How can I output the updated contents?

csv.register_dialect('pipe',delimiter='|', quoting=csv,QUOTE_NONE)

with open('test.txt') as csvfile
    cfile=csv.DictReader(cfile,dialect='pipe')
    fieldnames=cfile.fieldnames
    for row in cfile:
        row['address']=scrubrow(row['address']
    

    with open('c.txt','w') as outfile:
        writer=csv.DictWriter(outfile,fieldnames=fieldnames,dialect='pipe')
        writer.writeheader()
        writer.writerows(cfile)

>Solution :

cfile is an empty iterator. And you discarded all but the last row, so doing row['address']=scrubrow(row['address'] didn’t actually do anything.

The simple way to do this is to create a list and use that list:

csv.register_dialect('pipe',delimiter='|', quoting=csv,QUOTE_NONE)

with open('test.txt') as csvfile
    reader = csv.DictReader(cfile, dialect='pipe')
    fieldnames = cfile.fieldnames
    rows = []
    for row in reader:
        rows.append(scrubrow(row['address']))
    
    with open('c.txt','w') as outfile:
        writer = csv.DictWriter(outfile, fieldnames=fieldnames, dialect='pipe')
        writer.writeheader()
        writer.writerows(rows)

But this will be inefficient because it will require O(N) space. Instead, keeping only a single row in memory at a time, you could do:

csv.register_dialect('pipe',delimiter='|', quoting=csv,QUOTE_NONE)

with open('test.txt') as csvfile, open('c.txt','w') as outfile:
    reader = csv.DictReader(cfile,dialect='pipe')

    fieldnames = reader.fieldnames
    writer = csv.DictWriter(outfile,fieldnames=fieldnames,dialect='pipe')

    writer.writeheader()
    for row in reader:
        writer.writerow(scrubrow(row['address']))