How to join data like this in pandas

I am struggling to get data in this form (in pandas)

>Solution :

Use concat to merge your text files:

import pandas as pd
import pathlib

data_dir = pathlib.Path('.')

data = []
for filename in sorted(data_dir.glob('yob*.txt')):
    year = filename.stem[3:]  # .stem is the filename without extension
    sr = pd.read_csv(filename, header=None, names=['Name', 'Sex', year], 
                     index_col=['Name', 'Sex'], squeeze=True)
    data.append(sr)
df = pd.concat(data, axis=1) \
       .reindex(pd.MultiIndex.from_product([df.index.levels[0], ['F', 'M']])) \
       .fillna(0).astype(int)

Fixed with the help of @not_speshal

Output:

>>> df
             1880  2014  2020
Name                         
Aaban     F     0     0     0
          M     0    16     0
Aabha     F     0     9     5
          M     0     0     0
Aabriella F     0     5     7
...           ...   ...   ...
Zytaveon  M     0     8     0
Zyva      F     0     0     6
          M     0     0     0
Zyyon     F     0     0     0
          M     0     6     0

[74374 rows x 3 columns]

Note: I replaced NaN by 0

Focus on "Mary":

>>> df.loc['Mary']
   1880  2014  2020
F  7065  2634  2188
M    27     5     5