I want to obtain a list of all the unique characters in a text. A particularity of the text is that it includes composed characters like s̈, b̃. So when I split the text, the special characters are separated. For example, this character s̈ is separated into two characters s and ¨.

This is an example of the text I want to process.

sentence = "nejon ámas̈hó T̃iqu c̈ab̃op"
print(sentence)
print(list[set(sentence)])

I want to obtain a list with the unique characters. For this sentence, this list should be

expected_list = [‘a’, ‘á’, ‘b̃’, ‘c̈’, ‘e’, ‘h’, ‘i’, ‘j’, ‘m’, ‘n’, ‘o’, ‘ó’, ‘p’, ‘q’, ‘s̈’, ‘T̃’, ‘u’ ]

but it is

actual_list = [‘j’, ‘p’, ‘c’, ‘n’, ‘a’, ‘ ‘, ‘i’, ‘á’, ‘o’, ‘T’, ‘u’, ‘̃’, ‘h’, ‘̈’, ‘q’, ‘s’, ‘e’, ‘m’, ‘b’, ‘ó’]

I was reading that I can normalize the special characters as follows

import unicodedata
# Only for the character s̈
print(ascii(unicodedata.normalize('NFC', '\u0073\u00a8')))  #prints 's\xa8'

But I don’t know how to continue. Any help would be greatly appreciated.

>Solution :

Handling composed characters in Python can be a bit tricky due to the nature of how they are encoded. Try the grapheme library, which specifically deals with grapheme clusters (textual units that are displayed as a single character)

Install the grapheme library using pip:

pip install grapheme

or I prefer this way (to make sure it’s installing to the current python binary dirs)

python3 -m pip install grapheme

Then, you can use it to extract the unique grapheme clusters from the sentence:

import grapheme

sentence = "nejon ámas̈hó T̃iqu c̈ab̃op"
unique_characters = list(grapheme.graphemes(sentence))

print(unique_characters)