x264 long path and UTF-8 support discussion

[stax76]

Have you tried a filename containing å or ö or ä? ANSI does not work since long path support was added. What code page are you using?

It should only work when saved as UTF8, this does not solve any problem because plugins that open file paths still will not support Unicode, every other tool expects and creates ANSI.

Here it works but only because I'm a UTF8 user, supported only by Windows 10.


Desktop> Get-Culture; Get-UICulture; Get-WinSystemLocale; Get-WinDefaultInputMethodOverride; [Text.Encoding]::Default | select EncodingName,CodePage

LCID Name DisplayName
---- ---- -----------
1031 de-DE Deutsch (Deutschland)
1033 en-US Englisch (Vereinigte Staaten)
1031 de-DE Deutsch (Deutschland)

InputMethodTip : 0407:00000407
Description : German (Germany) - German


EncodingName : Unicode (UTF-8)
CodePage : 65001

[tph]

Have you tried a filename containing å or ö or ä? ANSI does not work since long path support was added. What code page are you using?


I tried some Unicode filenames, seems to work fine for me: https://i.imgur.com/HzHSMzi.png (standard US Windows 10 install).

[LoRd_MuldeR]

Have you tried a filename containing å or ö or ä? ANSI does not work since long path support was added.

This is almost certainly not related to "long path" support, but to explicitly setting the application's ANSI codepage to UTF-8, thus "overwriting" the system's default ANSI codepage.

Even though those are both things that can be configured via the EXE manifest (https://en.wikipedia.org/wiki/Manifest_file), they are two separate and mostly unrelated settings!

Also: ANSI does work just fine. You only have to be aware that all the "ANSI" APIs within the Win32 API will now use the codepage set via EXE manifest, rather than the default codepage set in control panel (region settings).

https://i.imgur.com/r3TwUWG.png

UTF-8 is a huge advantage, because UTF-8 can encode all Unicode characters that may appear in a file name, whereas any of the "legacy" codepages (e.g. Latin-1) only supports a tiny subset of Unicode.

The only "drawback", if you want to call it that way, is that if you feed text into the application that still is encoded in one of the "legacy" codepages but the application now assumes UTF-8, then problems with non-ASCII characters appear.

It is important to note that this is not really different from the "old" situation at all. The application (Win32 API) has always assumed a specific codepage – and has always failed to interpret anything else correctly!

The one and only difference is that now we can be sure that the application always assumes the "proper" codepage, i.e. UTF-8, instead of falling back to whatever happens to be configured as the systems default codepage.


See here for details:
https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page#set-a-process-code-page-to-utf-8


(As an aside: If you like to restore the "old" behavior, it is sufficient to strip the "activeCodePage" line from the EXE's manfiest, e.g. by using a tool like the Resource Hacker (http://www.angusj.com/resourcehacker/))


I tried some Unicode filenames, seems to work fine for me: https://i.imgur.com/HzHSMzi.png (standard US Windows 10 install).

The default ANSI codepage that you set in the control panel (region settings) does not matter for applications that explicitly set the ANSI codepage via EXE manifest – as latest x264 binaries do.

If an application enforces UTF-8 as its ANSI codepage via manifest – which is only possible on recent versions of Windows 10 – then that is the codepage that will be used.

Anyways, if the application expects the text to be encoded in UTF-8 (and thus will interpret the text as UTF-8), then you have to ensure that the text which you feed into the application actually is encoded in UTF-8.


(Pure ASCII characters are encoded the same in UTF-8 as in most other codepages, but anything beyond pure ASCII is going to give weird results, if the "encoder" and "decoder" assume different codepages)

[stax76]

I had missed some things, thanks for testing and clarification guys.