Permutation feature importance with multi-class classification problem

I am wondering if we can do Permutation feature importance for multi-class classification problem?

from sklearn.inspection import permutation_importance
metrics = ['balanced_accuracy', 'recall']
pfi_scores = {}
for metric in metrics:
    print('Computing permutation importance with {0}...'.format(metric))
    pfi_scores[metric] = permutation_importance(xgb, Xtst, ytst, scoring=metric, n_repeats=30, random_state=7)

Cell In[5], line 10
      8 for metric in metrics:
      9     print('Computing permutation importance with {0}...'.format(metric))
---> 10     pfi_scores[metric] = permutation_importance(xgb, Xtst, ytst, scoring=metric, n_repeats=30, random_state=7)

File c:\ProgramData\anaconda_envs\dash2\lib\site-packages\sklearn\utils\_param_validation.py:214, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    208 try:
    209     with config_context(
    210         skip_parameter_validation=(
    211             prefer_skip_nested_validation or global_skip_validation
    212         )
    213     ):
--> 214         return func(*args, **kwargs)
    215 except InvalidParameterError as e:
    216     # When the function is just a wrapper around an estimator, we allow
    217     # the function to delegate validation to the estimator, but we replace
    218     # the name of the estimator by the name of the function in the error
    219     # message to avoid confusion.
    220     msg = re.sub(
    221         r"parameter of \w+ must be",
    222         f"parameter of {func.__qualname__} must be",
    223         str(e),
    224     )
...
   (...)
   1528         UserWarning,
   1529     )

ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

Then I tried to use average=’weighted’, then I still got an error saying average=’weighted’ is not available. so how can I add average=’weighted’ into permutation_importance() for multi-class classification? Thanks

from sklearn.inspection import permutation_importance
metrics = ['balanced_accuracy', 'recall']
pfi_scores = {}
for metric in metrics:
    print('Computing permutation importance with {0}...'.format(metric))
    pfi_scores[metric] = permutation_importance(xgb, Xtst, ytst, scoring=metric, n_repeats=30, random_state=7, average='weighted')

TypeError: got an unexpected keyword argument 'average'

>Solution :

The 'recall' string alias stands for recall_score(average='binary'). The easiest way would be using a suffixed version sklearn provides:

metrics = ['balanced_accuracy', 'recall_weighted']

Alternatively, you may go for

from sklearn.metrics import recall_score, make_scorer

recall = make_scorer(recall_score, average='weighted')
metrics = ['balanced_accuracy', recall]

Worth noting balanced accuracy is basically a 'recall_macro'.