If you are not satisfied with the automatic tools you can try all codecs and see which codec is right manually. # Check that bad characters are still in badboy codepage file

# Read CP1252 with a few undefined chars without barfing Print(L) #requires QtConsole to view, Cmd.exe is cp1252 # (User responsible for making sure that all utf8 filesįout.write(b'hi there. # this case is still treated like codepage cp1252 Return open(file, 'r', encoding=bom, errors='ignore')įout = open('myfile1.txt', 'w', encoding='cp1252')įout = open('myfile2.txt', 'w', encoding='utf8') # If BOM is not provided, then assume its the codepage # write endianess generally determined by endianess of CPU # Python automatically detects endianess if utf-16 bom is present Open(file, encoding=bom, errors='ignore') Returns file encoding string for open() function

# otherwise the file is read as a codepage file with the # using a text editor such as notepad++, and rerun the python script, # the user can manually add a BOM header to the text file # If a text file is encoded with utf8, and does not have a BOM header, # (1) The default operating system code page, Or

An encoding sniffed by the chardet library, if you have it installed.

If an encoding is detected at this stage, it will be one of the UTF-* encodings, EBCDIC, or ASCII.

An encoding sniffed by looking at the first few bytes of the file.

The only exception is if you explicitly specified an encoding, and that encoding actually worked: then it will ignore any encoding it finds in the document. If Beautiful Soup finds this kind of encoding within the document, it parses the document again from the beginning and gives the new encoding a try.

An encoding discovered in the document itself: for instance, in an XML declaration or (for HTML documents) an http-equiv META tag.

chardet is a port of the auto-detection code in Mozilla. There is the chardet library that uses that study to try to detect encoding.

Check for an alternativeĬorrectly detecting the encoding all times is impossible. EDIT: chardet seems to be unmantained but most of the answer applies.

Find text encoding