flutter: riverpod widget gets rebuilt even when ref.watch(provider.select) is used

I have a list of buttons and the buttons can be selected and unselected individually. The state of the buttons is controlled in a single list of ButtonState and when a state is altered, the button color changes. Everything is according to plan but the problem is all my button gets rebuilt even though I only change the state of a single button.

Here is my code

enum ButtonState { normal, selected, disabled }

final List<ButtonState> tempButtonStates = List.generate(100, (index) => ButtonState.normal);
final buttonStatesProvider = StateProvider<List<ButtonState>>((ref) => tempButtonStates);

class MainPannel extends ConsumerWidget {
  const MainPannel({super.key});

  @override
  Widget build(BuildContext context, WidgetRef ref) {
    final buttonStates = ref.watch(buttonStatesProvider);
    return Padding(
      padding: const EdgeInsets.symmetric(vertical: 15, horizontal: 20),
      child: GridView.builder(
        gridDelegate: const SliverGridDelegateWithFixedCrossAxisCount(
          crossAxisCount: 10,
          crossAxisSpacing: 10,
          mainAxisSpacing: 10,
        ),
        itemCount: buttonStates.length,
        itemBuilder: (context, index) {
          return CustomMainButton(
            state: ref.watch(buttonStatesProvider.select((states) => states[index])),
            index: index + 1,
          );
        },
      )
    );
  }
}

// somewhere in my project I have this function changing the state of a single button
onTap: (){
  final List<ButtonState> states = ref.read(buttonStatesProvider.notifier).state;
  final ButtonState state = states[index - 1];
  late final ButtonState newState;
  switch (state) {
    case ButtonState.disabled:
      newState = ButtonState.normal;
      break;
    default:
      newState = ButtonState.disabled;
  }
  final newStates = List<ButtonState>.from(states);
  newStates[index - 1] = newState;
  ref.read(buttonStatesProvider.notifier).state = newStates;
},

I only want to rebuild the button that I altered the state.

>Solution :

This line

state: ref.watch(buttonStatesProvider.select((states) => states[index])),

must be in the CustomMainButton widget itself in the build method