Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Python for loop storing only the last value

My code are scraping data from public site and storing in a df and saving as csv, but my code are not working well.

    for ano in lista_ano:
     for distribuidora in lista_distribuidores:
      for mes in lista_mes:
        scraping = pd.read_html('https://www2.aneel.gov.br/aplicacoes/indicadores_de_qualidade/decFecSegMensal.cfm?mes={}&ano={}&regiao=SE&distribuidora={}&tipo=d'.format(mes,ano,distribuidora))
        dfs= pd.DataFrame(scraping[0])
        dfs.drop(dfs.tail(3).index,inplace=True)
        dfs.drop(dfs.head(2).index,inplace=True)
        dfs = dfs.assign(MES = '{}'.format(mes))
        dfs = dfs.assign(ANO = '{}'.format(ano))
        dfs = dfs.assign(DISTRIBUIDORA = '{}'.format(distribuidora))
        all_dfs = pd.DataFrame(dfs)
        all_dfs.to_csv('final_data.csv', encoding= 'utf-8')

My problem here is my all_dfs.to_csv are creating a new csv for each looping and not storing data in the same local.

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

>Solution :

You are overwriting the existing csv on each iteration.

To fix it, simply indicate that you want to append instead of write by

all_dfs.to_csv('final_data.csv', encoding='utf-8', mode='a')

Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading