uchardet is hot garbage

[coladict]

Operating System and Version

Linux. NN version v0.12

Distribution

source

Description

Multiple times now I've opened up files saved with this editor as UTF-8 only to have them garbled on reading.
The encoding detection library is reporting them as IBM852.

Steps to Reproduce

1. Save file with a word like Zürich
2. Close editor
3. Open file again

Debug Information

No response

Additional Details

Solution: Do not attempt to read different encodings until feature for changing encodings is implemented. Default to UTF-8.

Additional suggestion: When implementing encoding conversion, upon change, prompt user whether to reinterpret file as the new encoding or convert to it. This is something Notepad++ currently does wrong. It is a mistake to avoid.

I may get involved in this, but do not expect to have time this month.

[dail8859]

Short answer...I totally agree.

I actually have a PR I'm planning on merging soon (#971) that removes the uchardet auto detection for now due to these types of issues.

I am certainly not an expert when it comes to encodings, how Scintilla handles them, and then add Qt into the mix as well. Certainly, any input or help would be appreciated.

[dail8859]

I'll close this for now as #971 does handle utf8 files better in alot of cases. I know there is still lots of room for improvements. So if you find specific cases that are causing issues or have suggestions please feel free to open another issue.