x264: bug or feature?

[MasterNobody]

7ab4c928 (https://code.videolan.org/videolan/x264/-/commit/7ab4c928ef4511ea5753a36a57c3506d9fd5086b) changed the x264 CLI to use the UTF-8 code page on Windows 10 which also enabled the use of Unicode file names in AviSynth scripts as long as such scripts are saved as UTF-8. On the flip side this caused old AviSynth scripts saved using legacy code page encodings containing non-ASCII characters to stop working. Third party tools that currently generates AviSynth scripts using legacy code pages are also affected. In order to use such scripts they currently need to be opened with a text editor and saved as UTF-8. There is MR to revert this regression (https://code.videolan.org/videolan/x264/-/merge_requests/44).

Pros: AviSynth scripts using legacy code page encodings containing non-ASCII characters will work again.

Cons: AviSynth scripts using Unicode (UTF-8) containing non-ASCII characters will stop working. File names containing Unicode characters outside the active code page can no longer be accessed from AviSynth scripts.

AviSynth scripts containing only ASCII characters are unaffected.

So here is question to x264 users: Should we revert/fix it or not?

[jmartinr]

I say don't revert. UTF-8 is a nice step forward. And much more common.

[MasterNobody]

Side note (clarification). This UTF-8 manipulation is supported/need only if you use new enough Windows 10 (1903+ afaik) + new x264 (r3021+). On older Windows versions it will still need legacy code page encoding (as AviSynth doesn't support Unicode). Even on new Windows 10 versions other applications (like VirtualDub) wouldn't be able to open this UTF-8 converted scripts.

[qyot27]

VDub should be able to if you actually change the entire system locale to UTF-8, rather than trying to rely on individual apps to add support for either switching at runtime or doing the local codepage->UTF16->UTF8 tango. That was added to Win10 a couple years ago (Insider builds from April 2018, or I guess it would have been 1903 for wide-release). I addressed it in the AviSynth+ thread several months back (https://forum.doom9.org/showpost.php?p=1909790&postcount=9).

It would potentially be a problem for previous versions of Windows, but that's more of a 'do we still want to support Win7* even though it's left ESR?' question than specifically about AviSynth. 8.1 is the only other version of Windows that Microsoft still supports.

*to say nothing of XP or Vista.


EDIT: okay, nevermind. VDub (and VDub2) seems to have issues on my machine, but I can't tell if it's related to UTF8 or not.

EDIT the 2nd: Yeah, it's not related to UTF8. An ANSI-encoded Version() script fails too. It's probably the changes in 3.6 screwing it up, because the last official build of VDub2 is from March.

EDIT #3: There was something that screwed up the AviSynth+ install. ffmpeg and mpv were fine with it, but VDub2 was not. Refreshing it with the 3.6.1 installer fixed it. VDub2 actually seems to already switch to UTF8 on Win10, since even when I turned the locale back to the local codepage, UTF8 scripts with Unicode names still worked fine.

[manolito]

It would potentially be a problem for previous versions of Windows, but that's more of a 'do we still want to support Win7* even though it's left ESR?' question than specifically about AviSynth.

This is an easy one for me:
You guys drop Win7 support, and I will stop using your builds immediately. And I am sure that I will not be the only one...

[jmartinr]

This is an easy one for me:
You guys drop Win7 support, and I will stop using your builds immediately. And I am sure that I will not be the only one...

If you only use ASCII characters it'll probably still work.

[stax76]

The original long path request came from the staxrip community, great to see that at least a few projects worked on it.

I've tested the latest x264 build now, there is a major problem.

Win10 has an option to set the default code page to UTF8, I think many people have enabled it without experiencing any problems, as German I don't need it for AviSynth since code page Window-1252 covers it fine, the reason why I have it enabled is I want to be able to use Emoji and in general Unicode in Windows Terminal.

The point is however this UTF8 default code page feature is not enabled by default in Win10 and most people don't know about this feature or don't have Win10 or don't need it strictly because for instance they are not using Windows Terminal.

There are still many Win7 users, my situation with Win7 is following, virtual machines like vmware and virtualbox never worked well for me, too slow, 4K display not working or not working at all, so I gave up using virtual machines altogether, uninstalled and intent to never install again, I'm so tired of this technology. And installing Win7 directly will also not happen because it is way too much work due to issues with UEFI and SSD drivers. Both my systems use UEFI and Win10/Ubuntu dual boot. It means I cannot test or debug on Win 7, but Win7 will remain supported a few more years, I will not give up Win7 support intentionally a few more years in any larger project I'm involved, I just can't test and debug on Win7, if it works then fine, if not then pull requests are welcome.

So I tested the latest x264 build and avisynth with UTF8 as default code page and the unicode filename ������äää did work well as expected with both x264 and avisynth.

Now I've disabled UTF8 changing the default code page back to the default of Windows-1252, tested the ANSI filename Bär using German Umlaute, x264 gives now an error, it was working fine before the long path pull request was merged, this is for my and many other countries a major problem.

There is an issue with the long path patch, looking at the code it looks like the problem is bigger than a wrong flag in the manifest.

[stax76]

https://code.videolan.org/videolan/x264/-/issues/25

[stax76]

I think I did not fully understand the issue, it's probably best to not revert, so the latest builds are probably fine.