Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 AVC / H.264

Reply
 
Thread Tools Search this Thread Display Modes
Old 28th November 2020, 21:57   #1  |  Link
MasterNobody
Registered User
 
Join Date: Jul 2007
Posts: 552
x264: bug or feature?

7ab4c928 changed the x264 CLI to use the UTF-8 code page on Windows 10 which also enabled the use of Unicode file names in AviSynth scripts as long as such scripts are saved as UTF-8. On the flip side this caused old AviSynth scripts saved using legacy code page encodings containing non-ASCII characters to stop working. Third party tools that currently generates AviSynth scripts using legacy code pages are also affected. In order to use such scripts they currently need to be opened with a text editor and saved as UTF-8. There is MR to revert this regression.

Pros: AviSynth scripts using legacy code page encodings containing non-ASCII characters will work again.

Cons: AviSynth scripts using Unicode (UTF-8) containing non-ASCII characters will stop working. File names containing Unicode characters outside the active code page can no longer be accessed from AviSynth scripts.

AviSynth scripts containing only ASCII characters are unaffected.

So here is question to x264 users: Should we revert/fix it or not?
MasterNobody is offline   Reply With Quote
Old 29th November 2020, 13:35   #2  |  Link
jmartinr
Registered User
 
jmartinr's Avatar
 
Join Date: Dec 2007
Location: Enschede, NL
Posts: 301
I say don't revert. UTF-8 is a nice step forward. And much more common.
__________________
Roelofs Coaching
jmartinr is offline   Reply With Quote
Old 29th November 2020, 14:55   #3  |  Link
MasterNobody
Registered User
 
Join Date: Jul 2007
Posts: 552
Side note (clarification). This UTF-8 manipulation is supported/need only if you use new enough Windows 10 (1903+ afaik) + new x264 (r3021+). On older Windows versions it will still need legacy code page encoding (as AviSynth doesn't support Unicode). Even on new Windows 10 versions other applications (like VirtualDub) wouldn't be able to open this UTF-8 converted scripts.
MasterNobody is offline   Reply With Quote
Old 29th November 2020, 16:43   #4  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,416
VDub should be able to if you actually change the entire system locale to UTF-8, rather than trying to rely on individual apps to add support for either switching at runtime or doing the local codepage->UTF16->UTF8 tango. That was added to Win10 a couple years ago (Insider builds from April 2018, or I guess it would have been 1903 for wide-release). I addressed it in the AviSynth+ thread several months back.

It would potentially be a problem for previous versions of Windows, but that's more of a 'do we still want to support Win7* even though it's left ESR?' question than specifically about AviSynth. 8.1 is the only other version of Windows that Microsoft still supports.

*to say nothing of XP or Vista.


EDIT: okay, nevermind. VDub (and VDub2) seems to have issues on my machine, but I can't tell if it's related to UTF8 or not.

EDIT the 2nd: Yeah, it's not related to UTF8. An ANSI-encoded Version() script fails too. It's probably the changes in 3.6 screwing it up, because the last official build of VDub2 is from March.

EDIT #3: There was something that screwed up the AviSynth+ install. ffmpeg and mpv were fine with it, but VDub2 was not. Refreshing it with the 3.6.1 installer fixed it. VDub2 actually seems to already switch to UTF8 on Win10, since even when I turned the locale back to the local codepage, UTF8 scripts with Unicode names still worked fine.

Last edited by qyot27; 29th November 2020 at 19:06.
qyot27 is offline   Reply With Quote
Old 1st December 2020, 08:41   #5  |  Link
manolito
Registered User
 
manolito's Avatar
 
Join Date: Sep 2003
Location: Berlin, Germany
Posts: 3,078
Quote:
Originally Posted by qyot27 View Post
It would potentially be a problem for previous versions of Windows, but that's more of a 'do we still want to support Win7* even though it's left ESR?' question than specifically about AviSynth.
This is an easy one for me:
You guys drop Win7 support, and I will stop using your builds immediately. And I am sure that I will not be the only one...
manolito is offline   Reply With Quote
Old 2nd December 2020, 11:06   #6  |  Link
jmartinr
Registered User
 
jmartinr's Avatar
 
Join Date: Dec 2007
Location: Enschede, NL
Posts: 301
Quote:
Originally Posted by manolito View Post
This is an easy one for me:
You guys drop Win7 support, and I will stop using your builds immediately. And I am sure that I will not be the only one...
If you only use ASCII characters it'll probably still work.
__________________
Roelofs Coaching

Last edited by jmartinr; 2nd December 2020 at 11:07. Reason: refrase
jmartinr is offline   Reply With Quote
Old 21st December 2020, 00:37   #7  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
The original long path request came from the staxrip community, great to see that at least a few projects worked on it.

I've tested the latest x264 build now, there is a major problem.

Win10 has an option to set the default code page to UTF8, I think many people have enabled it without experiencing any problems, as German I don't need it for AviSynth since code page Window-1252 covers it fine, the reason why I have it enabled is I want to be able to use Emoji and in general Unicode in Windows Terminal.

The point is however this UTF8 default code page feature is not enabled by default in Win10 and most people don't know about this feature or don't have Win10 or don't need it strictly because for instance they are not using Windows Terminal.

There are still many Win7 users, my situation with Win7 is following, virtual machines like vmware and virtualbox never worked well for me, too slow, 4K display not working or not working at all, so I gave up using virtual machines altogether, uninstalled and intent to never install again, I'm so tired of this technology. And installing Win7 directly will also not happen because it is way too much work due to issues with UEFI and SSD drivers. Both my systems use UEFI and Win10/Ubuntu dual boot. It means I cannot test or debug on Win 7, but Win7 will remain supported a few more years, I will not give up Win7 support intentionally a few more years in any larger project I'm involved, I just can't test and debug on Win7, if it works then fine, if not then pull requests are welcome.

So I tested the latest x264 build and avisynth with UTF8 as default code page and the unicode filename ������äää did work well as expected with both x264 and avisynth.

Now I've disabled UTF8 changing the default code page back to the default of Windows-1252, tested the ANSI filename Bär using German Umlaute, x264 gives now an error, it was working fine before the long path pull request was merged, this is for my and many other countries a major problem.

There is an issue with the long path patch, looking at the code it looks like the problem is bigger than a wrong flag in the manifest.

Last edited by stax76; 21st December 2020 at 01:03.
stax76 is offline   Reply With Quote
Old 21st December 2020, 13:54   #8  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
https://code.videolan.org/videolan/x264/-/issues/25
stax76 is offline   Reply With Quote
Old 29th December 2020, 22:42   #9  |  Link
stax76
Registered User
 
stax76's Avatar
 
Join Date: Jun 2002
Location: On thin ice
Posts: 6,837
I think I did not fully understand the issue, it's probably best to not revert, so the latest builds are probably fine.
stax76 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 06:37.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.