I encountered a rather unexpected result today with a pandas dataframe.
My script takes genome sequence data (in fasta format) as input and calculates several basic metrics. I store those metrics in a pandas dataframe.
The script starts by defining an empty dataframe with headers:
stats_df = pd.DataFrame(columns=['Assembly','Size','#Contigs','#Contigs > 3000','N50','Longest_contig'])
The main body of the script then loops through all genome files and calculates all the metrics listed in the headers. That all works just fine. I then add the metrics for the genome assembly to the stats_df dataframe using these two lines:
new_df_row = pd.DataFrame({'Assembly':[assembly_name],'Size':[assembly_size_in_Mbp],'#Contigs':[num_of_contigs],'#Contigs > 3000':[greater_than_3000_count],'N50':[n50],'Longest_contig':[longest_contig]})
stats_df = pd.concat([stats_df,new_df_row],ignore_index=True)
The unexpected behaviour is when I view stats_df the columns are ordered:
#Contigs, #Contigs > 3000, Assembly, Longest_contig, N50, Size.
This is different to the order of the entries in the empty dataframe a the start and in each new row added. The metrics are all in the right place, I’m just wondering what causes the columns to move around like that?
>Solution :
I think the problem is that for new_df_row you’re using a dictionary to create the data frame. Dictionaries are unordered (in Python 3.6 and below) , so the keys do not have any fix order. This article and also this question explain the issue nicely.
To fix it, you could upgrade to Python 3.7 which has ordered dictionaries.