Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Calculate correlations between two different tables with different structures

I have two tables, on containing a column of one value ‘num1’ per date, and another containing one value per name ‘num2’ per date. I’m interested in calculating the correlation between each name’s ‘num2’ value and ‘num1’, but I’m unsure of whether or not I need to break the ‘data2’ table up into distinct tables for each name, or if there’s a clean way to do this.

data1 = {'date':  ['2022-01-03', '2022-01-04', '2022-01-05'], 'num1': ['.024', '.035', '.04']}
data2 = {'date':  ['2022-01-03', '2022-01-03', '2022-01-03', '2022-01-04', '2022-01-04', '2022-01-04', '2022-01-05','2022-01-05','2022-01-05'], 'name': ['name1', 'name2', 'name3', 'name1', 'name2', 'name3', 'name1', 'name2', 'name3'], 'num2':['20','200','149','36','174','400','45','100','12']}
data1 = pd.DataFrame(data1).set_index('date')
data2 = pd.DataFrame(data2).set_index('date')
print(data1)
print(data2)

Is there a way to calculate correlations between num1 and num2 for each name without manipulating these tables heavily?

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

>Solution :

You can use groupby.corr after ensuring having numerical values and joining the dataframes:

out = (data2
  .astype({'num2': int})
  .join(data1.astype({'num1': float}))
  .groupby('name').corr()
)

output:

                num2      num1
name                          
name1 num2  1.000000  0.998599
      num1  0.998599  1.000000
name2 num2  1.000000 -0.885346
      num1 -0.885346  1.000000
name3 num2  1.000000 -0.141869
      num1 -0.141869  1.000000
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading