Advertisements I have some strings in Python loaded from a file. They look like lists, but are actually strings, for example: example_string = ‘["hello", "there", "w\\u00e5rld"]’ I can easily convert it into an actual list of strings: def string_to_list(string_list:str) -> List[str]: converted = string_list.replace(‘"’, ”).replace(‘[‘, ”).replace(‘]’, ”).split(‘,’) return [s.strip() for s in converted] as_list =… Read More How to make Python treat literal string as UTF-8 encoded string

Advertisements Is there any Unicode codepoint that one of the bytes in its utf-8 representation is the ESC byte (0x1b)? Context: The ESC byte is used in ANSI escape codes (in terminals) and I’d like to know whether that byte can appear as part of a utf-8 byte sequence. >Solution : No, all bytes in… Read More Does UTF-8 use the ESC byte?

Advertisements I’m trying to get a title of html document in python, but getting weird symbols. I guess that’s because of encoding, but the html doc in utf-8 encoding. Is there any way I can get normal letters? Here is code and what am I getting: from bs4 import BeautifulSoup with open("index.html") as file: src… Read More Gettin non-english text from html doc

Advertisements I have raw data that looks like this: 25023,Zwerg+M%C3%BCtze,0,1,986,3780 25871,red+earth,0,1,38,8349 25931,K4m%21k4z3,90,1,1539,2530 It is saved as a .txt file: https://de205.die-staemme.de/map/player.txt The "characters" starting with % are unicode, as far as I can tell. I found the following table about it: https://www.i18nqa.com/debug/utf8-debug.html Here is my code so far: urllib.urlretrieve(url,pfad + "player.txt") f = open(pfad + "player.txt","r",encoding="utf-8")… Read More How to convert utf-8 characters to "normal" characters in string in python3.10?

Advertisements I got some text encoded to bytes using utf-8 encoding. When processing this text I incautiously used str() to make it a Unicode string because I assumed this would automatically decode the bytes object with the right encoding. This, however, is not the case. For example: a = "عجائب" a_bytes = a.encode(encoding="utf-8") b =… Read More Python: What is the string representation of a bytes object really and how to get back the bytes object?