Foreign Language characters in filenames

[IanB]

Unicode guru's your help please!

A tracker bug report, [ 1407212 ] Problem with spanish unicode characters in filenames (https://sourceforge.net/tracker/?func=detail&atid=482673&aid=1407212&group_id=57023), has me a little perplexed.

Most of AviSynth uses 8 bits character file system calls like fopen and as such relies on the system default conversion of filenames to unicode.

DirectShowSource() is the one exception and it explicitly uses "MultiByteToWideChar(CP_ACP, ..." to do the
filename unicode translation.

To the best of my reseach the system default translation should be equivalent to this code, but for Sven Rieke and naugas and probably many others they are not.

Enlightment required please :confused:

[Richard Berg]

Are string variables in the script language Unicode?

[IanB]

No, the entire world is 8 bit. The text files that hold the script, the file i/o calls that read it, the memory array that hold the data, etc.

[Richard Berg]

8 bit is ok if it's treated as UTF-8 (not plain ASCII). I think you'll have to use platform APIs for that to happen, though, not ANSI stuff like fopen.

[IanB]

Okay who are you and what have you done with the real Richard Berg. :D

No seriously, the fundamental question is what translation does Windows use for ANSI file system calls like fopen?

I assumed it was based on the current active code page, but obviously it is not. Can we help these users without recodeing half the internals.

[foxyshadis]

fopen is overloaded depending on what you defined in the way of multibyte support.


TCHAR.H routine _UNICODE & _MBCS not defined _MBCS defined _UNICODE defined
_tfopen fopen fopen _wfopen


If it still uses the plain ansi version, it uses setlocale() and defaults to the system default code page. So if it isn't defaulting something bizarre is happening, like a setlocale() in a header. reference (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclib/html/_CRT_Code_pages.asp)

I found nothing about fopen being translated any differently from anything else, so I assume there's nothing special with its conversion.

[IanB]

@foxyshadis,

No fopen and like are never overloaded. The TCHAR versions are the ones that are overloaded like _tfopen which will map to either fopen or _wfopen. Other calls like CreateFile map to either CreateFileW(LPCWSTR.... or CreateFileA(LPCSTR.... depending on whether UNICODE is defined.

However thank's for the reference, it pushed me thru another documentation iteration of code page stuff and I may be a little closer to understanding what is happening.

Current guess is that on the systems with a problem they are using the OEM code page instead of the ANSI code page for the unicode translation or maybe vice versa.

It is more likely to be fopen inappropriatly using the OEM code page because DSS works and it explicitly does a manual translation using the system default ANSI code page. (which incidently is probably wrong, the code probably should be this ... AreFileApisANSI() ? CP_ACP :CP_OEMCP ...)

Whatever it is, the path from the persons fingers to final data block representing the filename inside the OS is not consistent.

Short of converting the internals to use WCHAR's everywhere AND forcing the .AVS scripts to be 16bit UNICODE text files, I cannot see how to resolve this. I need someone with practicle experience that actually suffers the problem to step up.

[IanB]

Maybe some of our friends that use non "US English" Windows can ask around for a little guidance and report back.

[Fizick]

I am not Unicode guru, but I am one of your friends that use non "US English" Windows. :)

I have no any problem with Russian (Cyrillic) names of avi files, log files, or avs scripts in Win2k or WinXP.
AVS file must be created in ANSI code, of course, by Notepad, etc.
But if I try create AVS file in console application editor like Far Manager in OEM codepage, then Russian names of AVI files in AVS script are not good ("not found" error), of course.

Some not-related info. Some time ago I consider try to do language wrapper of Avisynth functions to Russian.
For example, instead of BlankClip I can create function:

Function ПустойКлип (clip ..., ...)
{
BlankClip(......)
}
ПустойКлип()

This produce Error: expected a function name

But I can use some non-capital Russian letters, if first symbol of function name is Latin or _:

Function _пустойклип (......)
{
BlankClip(......)
}
_пустойклип()

Some Russian letters produce "unexpected character" error.
Why?
For clip names first symbol may be Russian too.
Sorry for off-topic. :)

[IanB]

Ping! Here is another example :- Avisynth and japanese characters

[IanB]

Ping! And here is yet another example :- Avisynth + Unicode

[dukey]

to fix avisynth with funky file names ..
without looking at the code
just use _wfopen
or rather use _tfopen and define _UNICODE and maybe UNICODE as well. Then replace char with TCHAR.

that still leaves the problem though of the scripts being ascii 7 or 8 bit :p

[IanB]

... the problem though of the scripts being ascii 8 bitThe problem you are looking to fix is currently at core\main.cpp@588env->Invoke("Import", szScriptName);so your your last statement is slightly wrong, it is not the remainder of the problem, it is the whole problem


One possibility is to switch to using CP_UTF8 in place of the current CP_ACP and CP_OEMCP. (Ding! Richard your last comment suddenly makes sense now) Now this will not be transparent, you will need to find text editors and be able to configure them to output UTF-8 text files for scripts.

I could provide ANSItoUTF8, ACPtoUTF8 and OEMtoUTF8 script functions to ease the pain a little. i.e.AviSource(ANSItoUTF8("ПустойКлип"))

I could also flip the system code page to CP_UTF8 (65001) on entry to Avisynth and restore it on exit to assist plugins that do fopen, etc, calls.

Thorts?

[Fizick]

AviSource("ПустойКлип.avi", UTF8=true) would be more compatible with existent scripts,

[Leak]

AviSource("ПустойКлип.avi", UTF8=true) would be more compatible with existent scripts,
Looks a bit weird IMHO, having to tack on a parameter for this.

How about AviSource and AviSourceUTF8? :D

[IanB]

@Leak, The blind following the blind straight over a cliff :P

@Fizick, You seem to have missed the issue, AviSource() is but 1 example of the whole problem. i.e. Import(), DirectShowSource(), Mpeg2Source, WavSource(), Nic*Audio(), etc, etc, etc, ...

@All,

Neither AviSource(UTF8=True) or AviSourceUTF8() type thinking addresses the issue, which is what is the true UNICODE filename string intended by the user in the 8bit text of the script. As I said earlier the path between the users fingers and the UNICODE file system must not be ambiguous.

Currently we use the "fopen" class of ansi i/o library calls, which to the best of my research so far internally do MultiBytetoWideChar(AreFileApisANSI() ? CP_ACP : CP_OEMCP, ...) calls to get a UNICODE string for the tail end kernel calls. This mostly work for users who can nail down their Code Page environment. It comes unstuck as soon as you need more than 1 Code Page.

UTF-8 is an encoding scheme to represent UNICODE characters unambiguously in 8 bit data. By assuming that scripts are UTF-8, it solves all my problems dealing with a UNICODE based file system behind the I/O library calls.

But UTF-8 is an encoding scheme it is not a very friendly replacement for the warm safe language based Ansi Code Page world.

As a native english speaker whose scripts consist entirely of chars [32..126] I will not ever see a problem. Everybody else will have serious issues!

More Thorts?

[foxyshadis]

You can start by looking for a BOM in the parser, that should be a very good indicator of how upper characters should be translated. (Table of BOMs (http://unicode.org/faq/utf_bom.html#22).) All unicode-aware text editors are supposed to use it, but if it's not present, as in many files, it'll be more difficult, but what about starting by assuming it's UTF-8, translating that back to UCS-16 internally, if that fails (or if it succeeds but the file isn't found) assume it's some code page and retry the open with that translation? Further translation attempts up to you.

I just ask because anything extra in the script is going to be a big pain.

That doesn't work for 3rd party plugins, obvs, and it'll cause some more complexity internally, but as you know there isn't really a way you can translate everything to one format (like UCS-16) without some things failing or people not realizing they're working in one encoding and not another.

[[P]ako]

I had a similar problem. I had the antialiasing script (aaa) saved as unicode, maybe because of the accent mark on Didé's nick. Avisynth couldn't load the script at all, the error was pointing an issue with the line importing the script. I solved the issue by converting said script to UTF-8.

[IanB]

@foxyshadis,

Looking for a BOM doesn't help. It's cart before the horse. The scripts are plain 8 bit text. To look for the BOM you have to start with the assumption that the file is UTF, the BOM then tells you which flavour of UTF.

Here's a UTF-8 BOM "" .... oh, it doesn't do anything.

@[P]ako,

Yes trying to feed raw UNICODE in to something expecting ASCII chokes big time, every second byte is a NULL (as in NULL terminated strings). Converting to any 8 bit code page would have masked the problem.


@All,

At last some inspiration! :D

I zap all the fopen/createfileA type calls and replace them with _wfopen/createfileW type calls. Plus I manually use MultiBytetoWideChar(AVSCodePage, ...) to do the char to wchar translations.

I set AVSCodePage to (AreFileApisANSI() ? CP_ACP : CP_OEMCP) after script verb "Import" has opened the script file but before it parses it. i.e. it _wfopen's the script file using the existing AVSCodePage translation, then resets AVSCodePage to it's default value.

I provide a script verb to let the user specify the code page to be used for char to wchar translations in subsequent script verbs that result in unicode system or library calls.

In the AVIFileOpenW implementation code for the Avisynth handler I do a WideChartoMultiByte(CP_UTF8, ...), set AVSCodePage=CP_UTF8 and then do the env->Invoke("Import", ...) [this is why I need to keep resetting AVSCodePage]

Maybe I can allow an .AVSI script to set a default default code page.

Many more thorts please?

[DonQ]

Read two first bytes from any source/script/include file in binary mode. If they present BOM (were they FF FE and FE FF?), consider this file UTF8, otherwise ASCII. Of course you need convert both files to same format afterwards, UTF8 or UCS16 for example.

This corresponds exactly to how Notepad saves [new] files - if it detects any character not from current ANSI code page, it automatically adds BOM and saves file in UTF8.

[IanB]

Hmm, keep the ruminations coming :DBytes Encoding Form
FE FF UTF-16, big-endian
FF FE UTF-16, little-endian
EF BB BF UTF-8If I see FF FE or FE FF as the first 2 bytes of a file I will know it is going to be 16bit unicode and I will NOT be able to do anything with it (maybe not so).

However if I saw the EF BB BF sequence I could set the default AVSCodePage to UTF-8, I am not aware of editors that do this, but I could easily add the processing in anticipation. Effectively treating it as if the script had started
AVSCodePage("UTF-8") ... [Is this transparent?]

As for the "maybe not so" I could easily do a WideChartoMultiByte(CP_UTF8, ...) and set AVSCodePage=CP_UTF8 in the file read code of Import() ... [Is this transparent?]

Consider your wishes carefully.

I will not implement anything that will break the existing 7 bit ascii behaviour of Avisynth.

I will resist implementing anything that will break existing 8 bit codepage behaviour that currently works satisfactorily.

I will not implement anything that will prevent a future pure 16bit unicode implementation of Avisynth.

[foxyshadis]

Why do you want to transform back to UTF-8, instead of dropping internal ascii parsing all together? Convert to UCS-16 based on either the UTF marked by the BOM or by the default code page in use. Convert back to 8-bit for string arguments based on the code page only. If people want to use unicode without a BOM, or something, perhaps use the python method - assume everything is 8-bit unless a string is preceded by u, as in u"ангел смерти".

Define a new character to replace 's' ('u'?) for unicode or utf-8 or whatever you want, and then convert all characters to that, for any filters that want to support it. Tell people that they need two filter definitions, one with s and one with u, and show them how to build callbacks using with WideChartoMultiByte(CP_ACP, ...) to hand to a WCHAR constructor.

Avisynth will prefer the u definition over the s definition.

Supporting legacy ASCII and Unicode behavior is not an easy problem, and any easy workarounds are unfortunately likely to make a later conversion to full unicode impossible without breaking something.

[IanB]

Because converting Avisynth to unicode is going to be very very hard and I am lazy :p

Just saying scripts are UTF-8 period! would be exceedingly easy, but I am not that much of an arsehole, so I look for simple ways to maintain the status quo compatibility wise and give Foreign Language users some way to move forward.

[dukey]

UTF-8 seems like a good standard to go for

under linux i believe you can pass a UTF-8 filename to fopen() and it'll work fine.

Under windows this doesn't work so you would have to convert the filenames to
wchar_t
or whatever the defines for windows are. TCHAR with the _UNICODE defines i think and then use _wfopen()

But that doesn't seem like too much of a big deal.
That way all existing scripts will still work and can easily be extended.

[IanB]

Got this email from sourceforge, interesting idea about using the short (8.3) names as a hack around.

A difficulty to consider might be how windows map long filenames that collide in the short namespace. I believe it is chronological, i.e 1st file created is FRED~1.AVS, 2nd file created is FRED~2.AVSMessage body follows:

Hello there

First things first. I'm probably using AviSynth for
something that it isn't intended. But I quite like the
fanciness of it :-)
What I'm doing is, I have an AVS script with embedded
subtitles in it (SSA script after __END__ works like a
charm). The script itself checks its own filename and loads
whatever audio and video is available around it with the
same name and plays them with the subtitles over it.
I have a Perl script which generates such files from a AVS
template and timed text information.

Anyway, I've run into some Unicode issues which were quite
easy to fix (or at least circumvent a bit for the AVS
filename part).

First patch is for AVS files with Unicode filenames.
If one tries to play a file with non-local-CP characters in
the file or any part of the path the AVS script cannot be
opened at all.
I'm aware that its not really feasible to rewrite AviSynth
to use wide characters, so I used windows' short filename
form feature to get the scripts loaded. It tries to keep the
last part of the path (the filename) in the long form if
possible, because that would help my special case when I try
to load video/audio files with the same name during the
scripts execution.
Works fine on XP and should on all OSs which allow Unicode
filenames. It even works under Wine :-)@
-------------------------------------------------------------------------------------------------------------------
@
--- avisynth_base/src/core/main.cpp Mon Jun 9 23:53:20 2008
+++ avisynth/src/core/main.cpp Thu Jan 8 16:32:15 2009
@@ -320,6 +320,24 @@ STDMETHODIMP CAVIFileSynth::Load(LPCOLES
char filename[MAX_PATH];

WideCharToMultiByte(AreFileApisANSI() ? CP_ACP : CP_OEMCP, 0, lpszFileName, -1, filename, sizeof filename, NULL, NULL);
+ if (strchr(filename, '?'))
+ {
+ // if a ? ended up in the converted filename there is a non-representable unicode character in the path
+ // Change to the short file name form (8.3 characters per path element).
+ char filenameshort[MAX_PATH], filenamelong[MAX_PATH];
+ OLECHAR lpszFileNameShort[MAX_PATH];
+ strncpy(filenamelong, filename, sizeof filenamelong);
+ GetShortPathNameW(lpszFileName, lpszFileNameShort, MAX_PATH);
+ WideCharToMultiByte(AreFileApisANSI() ? CP_ACP : CP_OEMCP, 0, lpszFileNameShort, MAX_PATH, filenameshort, sizeof filenameshort, NULL, NULL);
+ // Depending if the found '?' is before or after (or on both sides) of the last path devider
+ // use either LONG-PATH/SHORT-FILE or SHORT-PATH/LONG-FILE or SHORT-PATH/SHORT-FILE
+ char* pfls = max(strrchr(filenamelong, '/'), strrchr(filenamelong, '\\'));
+ char* pfss = max(strrchr(filenameshort, '/'), strrchr(filenameshort, '\\'));
+ if (!pfss || !pfls) memcpy(filename, filenameshort, sizeof filenameshort);
+ else if (strchr(filenamelong, '?') > pfls) memcpy(filename+(pfls-filenamelong), pfss, sizeof(filenamelong)-(pfls-filenamelong));
+ else if (strchr(pfls, '?')) memcpy(filename, filenameshort, sizeof filenameshort);
+ else { memcpy(filename, filenameshort, pfss-filenameshort); memcpy(filename+(pfss-filenameshort), pfls, sizeof(filenameshort)-(pfss-filenameshort)); }
+ }

_RPT3(0,"%p->CAVIFileSynth::Load(\"%s\", 0x%X)\n", this, filename, grfMode);
@
-------------------------------------------------------------------------------------------------------------------
@The second one is for scripts with an added UTF-8 header
thingy often (mis-)called 'byte order mark'. The thing is, I
need these (or rather VSFilter TextSub does) three bytes at
the beginning to get (embedded) Unicode subtitles up and
running. Other than that, it doesn't really hinder the
execution of the script at all, as of course, UTF-8 is fully
compatible with ANSI ASCII.
The setting of the null-terminator was moved up so it will
be there when memmove is being done.@
-------------------------------------------------------------------------------------------------------------------
@
--- avisynth_bases/src/core/parser/script.cpp Fri Jun 20 05:58:18 2008
+++ avisynth/src/core/parser/script.cpp Wed Jan 14 01:13:52 2009
@@ -268,6 +268,7 @@ AVSValue Import(AVSValue args, void*, IS
if (!ReadFile(h, buf, size, &size, NULL))
env->ThrowError("Import: unable to read \"%s\"", script_name);
CloseHandle(h);
+ ((char*)buf)[size] = 0;

// Give Unicode smartarses a hint they need to use ANSI encoding
if (size >= 2) {
@@ -278,11 +279,9 @@ AVSValue Import(AVSValue args, void*, IS
"re-save script with ANSI encoding! : \"%s\"", script_name);

if (q[0]==0xEF && q[1]==0xBB && q[2]==0xBF)
- env->ThrowError("Import: UTF-8 source files are not supported, "
- "re-save script with ANSI encoding! : \"%s\"", script_name);
+ memmove(buf, buf+3, size-3+1);
}

- ((char*)buf)[size] = 0;
AVSValue eval_args[] = { (char*)buf, script_name };
result = env->Invoke("Eval", AVSValue(eval_args, 2));
}
@
-------------------------------------------------------------------------------------------------------------------
@I'm aware that I'm kinda late now with 2.58 out, but maybe
you can use this to get some kind of new alpha or beta out
with an updated copyright notice :-)

Greetings from Tokyo Japan,
Bernhard
--
This message has been sent to you, a registered SourceForge.net user,
by another site user, through the SourceForge.net site. This message
has been delivered to your SourceForge.net mail alias. You may reply
to this message using the "Reply" feature of your email client, or
using the messaging facility of SourceForge.net at:
https://sourceforge.net/sendmessage.php?touser=314083

[Barna]

Yay after 5 long days I am finally allowed to post :-)

There are no collisions possible in the short form.
It's the operating system which cares for that. For Windows and its two file systems VFAT and NTFS it actually stores the short forms in there. So that means the short forms are generated during the naming of the file and will not change until the (long) name does.
After ~1,~2,~3,~4 the naming (at least under NTFS) changes to some hashing/random type which just keeps the first two ascii letters.

Oh, and I'm not sure how I feel about having my full name and an email address that forwards to my main one being posted on a public forum...

[Wilbert]

Oh, and I'm not sure how I feel about having my full name and an email address that forwards to my main one being posted on a public forum...
Barna is offline Report Post IP Click Here to Strike Barna
I removed them for you. Sorry about that.

[IanB]

...
Oh, and I'm not sure how I feel about having my full name and an email address that forwards to my main one being posted on a public forum...Sorry about that, I need to censor more when I cut and paste. Thanks Wilbert.
There are no collisions possible in the short form.
It's the operating system which cares for that. For Windows and its two file systems VFAT and NTFS it actually stores the short forms in there. So that means the short forms are generated during the naming of the file and will not change until the (long) name does.
After ~1,~2,~3,~4 the naming (at least under NTFS) changes to some hashing/random type which just keeps the first two ascii letters.Yes I guess collisions do not matter here, the appropriate script will get opened.

For the benefit of others, I was worried that you cannot predict the short name for a given long filename, i.e. "Microsoft Sucks.avs" and "Microsoft Blows.avs" translate to "MICROS~1.AVS" and "MICROS~2.AVS", which is which depends on the order of creations.


Given there will never be enough developer cycles to do a Unicode conversion and that such a change would not be transparent to the plugin API (i.e. recompile ALL the plugins in existence) I am currently leaning towards making the internals UTF-8.


This means all wide char system entries would do WideCharToMultiByte(CP_UTF8, ...

And all file system interaction would do MultiByteToWideChar(CP_UTF8, ... followed by using the ...W() version of the Windows API.

In Import for BOM-less ASCII files do a MultiByteToWideChar(AreFileApisANSI() ? CP_ACP : CP_OEMCP, ... ->> WideCharToMultiByte(CP_UTF8, ..., for files with a BOM, i.e. UTF, UNICODE, etc strip the BOM and directly translate to UTF8.

To support plugins that deal with the file system provide a ShortPathName() function that uses similar logic to Bernhards above and maybe UTF8_to_ANSI/UTF8_to_OEM/UTF8_to_ASCII() string functions.

Within the 7 bit ASCII character set the change would be transparent.

Only scripts that contain, Code Page specific characters and feeds those characters to plugins that interact with ANSI file API's, will need modification. Currently this case works and fails by the grace of god anyway. Feeding to internal functions will work as expected.

Thoughts?

[IanB]

Remember, because Avisynth is a library stuffing around with the process/thread codepage settings is not possible, just like we cannot pop messageboxes.

Hence the "MultiByteToWideChar(CP_UTF8, ... followed by using the ...W() version of the Windows API." gymnastics will be needed in place of any ANSI file system API's.

[stax76]

Apparently the script self cannot have a unicode filename like:

...\Hoří, má panenko.avs

Shows: Import: Couldn't open "...\Hoří, má panenko.avs"

Using version 2.58

[IanB]

@stax76, yes this is true. As long as we continue to use ...A() windows system calls there will be problems using unicode characters that are not available in the active codepage for the host application.

[mariush]

No.

The BOM is optional and will most likely appear only if the architecture is different, like little endian vs big endian - that's why there are two BOMs. A file can be BOTH ASCII and UTF8 at the same time, as UTF8 includes the ASCII subset.

The best thing would be to ASSUME file is UTF8 and if you stumble on a errors while parsing it, fall back to ASCII. As errors go, there are some documented edge cases and unallowed character combinations, for example as an error you're not allowed to write the pound sign by itself (£) in UTF8 because its character code (0xA3) is used for something else in UTF8, so you have to map it to two bytes.

There's a library ICU (http://site.icu-project.org/) that's also used by PHP to do normalization between UTF formats and code pages and stuff...

ps... AviSource("something", UTF8=true) is kind of dumb... what's next, you'll have utf16=true? If it comes to this, it would make more sense to have Codepage="UTF8" or something like this.

ps2. DON'T make assumptions that the short file names are there. Some people do registry tweakings to disable the creation of short file names or to disable last access/last modified time updates, for better disk access, less I/O, whatever, so these short file names are not always there. And... if I remember correctly, on some media they don't exist by default - like 8GB micro sd cards or removable drives or some CD/DVD media.

[IanB]

@mariush, What do you mean "No." (are you answering my post 1374763 or is this out of context)

Avisynth currently supports ANSI encoding in the host applications current code page. i.e. what the ...A() windows API calls support. To avoid bogus support issues it currently detects UTF8, UTF16le and UTF16be BOM codes and reports a helpless error.

[mariush]

No... in the sense that I don't agree with what was said in the previous few messages (about short file names), which were posts 25 - 29 when I read the thread (I think I left the page open in a tab and replied after a few hours and that's why it's a bit out of context).

My apologies if it was misunderstood.

[stax76]

Has there been any progress?

[krieger2005]

Read in the Blog of the CppCMS-Project (http://art-blog.no-ip.info/cppcms/blog) , that the developer has some problems developing on Windows Multilanguage-Filenames. As i understand he solved his Problem. So maybe look at his results or talk to him would help implimenting access of Multilanguage Filenames.

PS: Here the ULR of the exact Blog-Entry: http://art-blog.no-ip.info/cppcms/blog/post/62

[stax76]

I have a report that a script name with Russian Unicode chars like предприятий.avs works on Russian systems. Can somebody confirm this? If it works can somebody explain me why? On my German system it's not working.

[kemuri-_9]

I have a report that a script name with Russian Unicode chars like предприятий.avs works on Russian systems. Can somebody confirm this? If it works can somebody explain me why? On my German system it's not working.

IanB answered your questions already:
Avisynth currently supports ANSI encoding in the host applications current code page. i.e. what the ...A() windows API calls support.

so whatever your windows system is using as the native code page is what avisynth can recognize.
like on my japanese codepage (cp932) system i can use scripts that use filenames with japanese characters,
but i can not use scripts that contain filenames with japanese characters on my english codepage (cp1252) system.

[stax76]

The report I got must be wrong then, maybe I could try to rename source files converting Unicode chars to ANSI chars if there is a representation, not sure if it can be done with the WinAPI or .NET API or if I need a table or something.

[Groucho2004]

You seem to be confused about something in Kemuri's reply.

You can easily try it yourself even on your German Windows: Change the code page to cyrillic and restart Windows. Avisynth will accept filenames with cyrillic characters.

Just in case you don't know how:
Control Panel -> Regional and Language Options -> Advanced -> Language for non-Unicode programs -->> Change it to Russian.

[stax76]

ANSII covered Cyrillic chars are not my problem but Unicode Cyrillic chars which I could possibly convert to ANSII, maybe I'm just writing nonsense, I don't know.

[Groucho2004]

ANSII covered Cyrillic chars are not my problem but Unicode Cyrillic chars which I could possibly convert to ANSII, maybe I'm just writing nonsense, I don't know.

Are talking about file names or the encoding of the script? If you mean the content of the script then yes, you can easily convert UTF-16 or UTF-8 to Ansi (CP-1251).

[stax76]

Script file names are my problem, StaxRip generates those based on the name of the source file and the source file will of course also appear in the scripts but that's not a problem it seems.

[Groucho2004]

Script file names are my problem, StaxRip generates those based on the name of the source file and the source file will of course also appear in the scripts but that's not a problem it seems.

So, as far as I understand, the report you got about Russian file names states that there is no problem, right? If that is the case, what exactly is the problem?

[stax76]

StaxRip rejects Unicode file names, it was requested to remove this but I don't believe it can be removed because it cannot work, it can work with Cyrillic ANSII but not with Cyrillic Unicode. Using alternative script names would be a big task as scripts are generated in countless locations with complicated code. At least I can say it's not my fault.

[Groucho2004]

StaxRip rejects Unicode file names, it was requested to remove this but I don't believe it can be removed because it cannot work, it can work with Cyrillic ANSII but not with Cyrillic Unicode. Using alternative script names would be a big task as scripts are generated in countless locations with complicated code.

Windows stores file names internally always as Unicode (UTF-16) in order to provide support for file names for all languages. The system code page determines how the characters of the file names are interpreted by the API (if you don't override the default code page in your file and string operations).

I simply don't understand your problem. Since Avisynth doesn't support UTF-8 or UTF-16 you have to create your scripts with the appropriate MBCS encoding using the correct code page. There are Win32 API functions you can use to ensure the proper encoding. Even dummy programming languages like VB should support this.
I'm doing this in plain C without problems, one would think that .NET (which I assume you're using) should make this very simple.

At least I can say it's not my fault.
I suggest you do some reading on the matter and re-think that statement.

[HayateYuki]

Actually, IMHO, for nowadays programs on Windows platforms, Native Unicode support could be the base requirement.

Such as using only Unicode version of API, which there is "W" at the end of API functions...

It could be huge difficulty, but it could be the only complete way to avoid any this kind of problems...

For AVS script parser, as like foxyshadis said, BOM could be the best way to determine whether the script is UTF-8, UTF-16 or MBCS encoding, then convert them to UTF-16 with MultiByteToWideChar() and then do the things...

Please, throw away ...A() for future version.

[stax76]

then convert them to UTF-16 with MultiByteToWideChar() and then do the things...

And that is the problem as I understand it, the core is 8 bit instead of 16 and it's not easy to port it to 16 bit.

[kemuri-_9]

Please, throw away ...A() for future version.

it's not as simple as just tossing the 'A', from the API functions and it'll magically just work.

avisynth is highly designed around the use of char * which would all have to be changed to wchar_t * if unicode was to be used and supported within scripts.
this is by no means a trivial change.
let's not forget that this core change would break every current filter there is as well, and it's no small undertaking to fix them all either.

the avisynth project is even defined to use MBCS which defines 'A'-less API calls back to 'A' versions through the preprocessor.

[Midzuki]

Just a nit-pick, but, IMHO,

"Foreign Language characters in filenames"

should be replaced by

"Non-ASCII characters in filenames"

(or: "Non-ANSI characters in filenames")

((because "foreign" is just a point-of-view)).

[Groucho2004]

For AVS script parser, as like foxyshadis said, BOM could be the best way to determine whether the script is UTF-8, UTF-16 or MBCS encoding, then convert them to UTF-16 with MultiByteToWideChar() and then do the things...

Why would anyone create a script in UTF-8 or UPF-16 when it's known that Avisynth doesn't support it? Am I missing something?

The only "inconvenience" you have with Avisynth not supporting Unicode is that you have choose the proper encoding of your script according to the system code page it's supposed to be used on. It's very simple.

[krieger2005]

It's important if you want to use pathes in AVS which contain Non ASCII Character.

[Groucho2004]

It's important

What's important?

[krieger2005]

Why would anyone create a script in UTF-8 or UPF-16 when it's known that Avisynth doesn't support it?

Answer:

It's important if you want to use pathes in AVS which contain Non ASCII Character.

[Groucho2004]

@krieger2005
You obviously don't know anything about encoding standards.

First of all, I recommend that you look up what ASCII stands for. Wikipedia is your friend.

ACSII is only a 7 bit subset of a 8 bit encoding scheme. AVISynth however supports 8 bit encodings based on the common Windows code pages.

If you are for example using code page 1252 it covers all Western languages like English, German, French, Norwegian, Italian, etc including all accented characters these languages have.
If you set your Windows to Russian (you don't need Russian Windows for that) it will support all Cyrillic languages and so on.

Sorry, but your argument is nonsense. There is only a problem when people don't know how create a script properly.

[mariush]

And which code page contains the Euro character (€) ? Don't answer, there's several code pages that actually exist and contain it but the point is it's not in ASCII and not all people getting a file with such character may actually have installed that code page.

I won't even get into the new currency symbol for the new Indian Rupee (http://en.wikipedia.org/wiki/File:Indian_Rupee_symbol.svg) for example which is planned to be included in the UTF-8 - the rupee is represented by the Unicode character 20A8 (₨) right now, but it's going to receive another position for the new symbol... the point is code pages is antique technology..

The sensible way is to move all the way to utf-8 which is perfectly fine and can represent if not all, then almost all characters out there. There are free libraries like the IBM's unicode one which would make the job easier but I'm not that good of a programmer to get involved.

ps.. some people work as freelancers therefore receive files with various names from various parts of the world - sometimes included in winrar archives, don't make the assumption that a video is created on one computer and ends its life on the same computer, on the same code page window has running.

[Groucho2004]

And which code page contains the Euro character (€) ?

Hm, I tried but can't figure out why one would use the Euro symbol in an AVISynth script.

ps.. some people work as freelancers therefore receive files with various names from various parts of the world - sometimes included in winrar archives, don't make the assumption that a video is created on one computer and ends its life on the same computer, on the same code page window has running.

I'm not saying it's perfect and I certainly agree that Unicode AVISynth would be the best way to go but we have to work with what's given. I find it hard to believe that anyone would go through the trouble porting AVISynth (and all plugins in existance) to Unicode.

[Gavino]

Hm, I tried but can't figure out why one would use the Euro symbol in an AVISynth script.
In a subtitle - just tried it and it works fine for me.

[krieger2005]

actually i am working as a software developer and know exactly what i'm talking about since i used the libraries behind converting character from codepages to unicode and the way back and so must know the theory behind usage of such libraries. Using Codepages is a possible solution and used decades away. but when you start using Character which are in two different codepages you can't use codepages or must search very long for a codepage which support such character-combinations (for example german: ä and russian я).

Why should someone use such a mix? Just because it's possible!!!! Don't ask why peaple do something - because peaple just do it.

[foxyshadis]

To avoid recompiling plugins, and since AVISYNTH_INTERFACE_VERSION isn't passed into any constructors, you could either attempt to find the value within the plugin, or have two copies of AddFunction, a char * and a wchar_t * version, the plugin gets whichever environment it tries to use. Kind of nasty, but that's baked code for you. Actually, you could even modify the header to point AddFunction (ThrowError, etc) to an internal AddFunctionW when recompiled, since the filter only gets the functionality if recompiled, but never loses any; that way you don't need to hack in a char * check.

No matter what happens, there will have to be char->wchar conversion functions, because not all plugins are ever going to be recompiled.

[IanB]

Okay here is a proposal to chew on. Please think this through and post comment if you find issues in the theory.
Define existing internal ansi char* text to now be UTF-8 encoded!
Change Import() to detect BOM prefix for UTF-8 & Unicode input else by default convert from ANSI input using :- MultiBytetoWideChar((AreFileApisANSI() ? CP_ACP : CP_OEMCP, ...)
WideChartoMultiByte(CP_UTF8, ...)Maybe support a magic #!UTF-8 style first line detection to override default behaviour.
Provide a new IScriptEnvironment API Helper :-const wchar* UTF8toWide(const char*);which wraps a MultiBytetoWideChar(CP_UTF8, ...) with an env->SaveString()
Change all the internal filesystemA(name) calls to filesystemW(env->UTF8toWide(name)).
Add some script helper functions for legacy 2.5 plugins that internally use filesystemA(name) calls with input AVS argument strings:-
ShortFileName(string Name) short 8.3 mapped names are defined to be 7bit ascii, but not all file systems support the short name space.
UTF8toANSI(string Name) using AreFileApisANSI() ? CP_ACP : CP_OEMCP to steer the conversion.
UTF8toACP(string Name) using CP_ACP for the conversion.
UTF8toOEMCP(string Name) using CP_OEMCP for the conversion.
UTF8toCodePage(string Name, int CodePage) using tthe user specified codepage number for the conversion.
Notes :-

This is transparent for 7 bit ascii. Yes!

This breaks existing 8 bit codepage behaviour that currently works satisfactorily for 2.5 plugins. Bummer!

This does not hinder a future pure 16bit unicode implementation of Avisynth. Yes!

[kemuri-_9]

is adding wchar_t * support to AVSValue included in this?

more specifically:
* creating AVSValues from wchar_t *
* AsString() being able to return wchar_t *

[IanB]

@kemuri-_9,

I did look into extending AVSValue's range of types, but I couldn't see a way of supporting the 2.5 API baked code.

I could add code that internally does WideChartoMultiByte(CP_UTF8, ...) in a new AVSValue(const wchar* s) contructor. And do a MultiBytetoWideChar(CP_UTF8, ...) in a new AsWString() method but that would be pretty heavy on the SaveString storage.

I could be more inclined to add a const char* WidetoUTF8(const wchar*); helper for symmetry.

[kemuri-_9]

@kemuri-_9,
I could be more inclined to add a const char* WidetoUTF8(const wchar*); helper for symmetry.

this seems fair except this entire concept seems to be putting fairly large strain on env->SaveString().
In cases where only the temporary use of the other version that what was given is required this would just waste memory.

It seems reasonable to add something along the lines of an "AVSString" class that would make the conversion between char* and wchar_t* transparent but stay outside the use of env->SaveString() be added to avisynth's codebase available for plugin devs' use as well.

This would allow plugin devs the use of conversions without having to know how avisynth does it, and be for temporary usage (to avoid unnecessary abuse of env->SaveString())
we have something like this at work and it makes things fairly convenient.

[mariush]

Just keep in mind that a character in UTF-8 may have up to 4 bytes. Though unusual, I don't think it would be impossible to encounter characters using 3 bytes, like mathematical characters... see this list for all the possible ranges: http://www.unicode.org/Public/UNIDATA/Blocks.txt

I'm saying this because I'm not sure what you mean by "This does not hinder a future pure 16bit unicode implementation of Avisynth. Yes! " and why would you think going the pure 16bit unicode way would be smart - utf-8 seems more sane to me based on what I've read. UTF16 would just force you to use 2 bytes for every char, at least, yet you'd still need up to 4 bytes for one, just like with utf-8.

If you need this for optimizations or something, then utf-32 and 32bit words would seem a better choice... but maybe it's not worth it.

[Groucho2004]

One important thing to consider is that there are text editors out there which don't necessarily write a BOM when saving to UTF-8 or even UTF-16, either by default or user-configured.

Most advanced editors (UltraEdit, EMEditor, Unipad) scan a document upon opening to determine the encoding if they don't find a BOM.

So, the absence of a BOM doesn't necessarily mean that the file has 8-bit (Ansi) encoding.

[IanB]

This seems fair except this entire concept seems to be putting fairly large strain on env->SaveString().
In cases where only the temporary use of the other version that what was given is required this would just waste memory.

It seems reasonable to add something along the lines of an "AVSString" class that would make the conversion between char* and wchar_t* transparent but stay outside the use of env->SaveString() be added to avisynth's codebase available for plugin devs' use as well.

This would allow plugin devs the use of conversions without having to know how avisynth does it, and be for temporary usage (to avoid unnecessary abuse of env->SaveString()) we have something like this at work and it makes things fairly convenient.
I'm not sure what you mean by an "AVSString" class, but it seems like you are suggesting something along the lines of the STL String class or the related IOstream class. If so why reinvent the wheel. STLPort has a real beast of an extended implementation if you need such.

The current SaveString model is a little wasteful but it is a fair trade for simplicity and stability. An alternative would be along the lines of strdup(), where the programmer has to eventually remember to do a free() [and they always forget :devil: ] Given char* strings are already successfully using the SaveString model doing wchar* strings a similar way is not a great stretch.

On my quick pass through the code to scope out the work needed I did not find a need for a general WidetoUTF8() feature, hence it was forgotten.

Just keep in mind that a character in UTF-8 may have up to 4 bytes. Though unusual, I don't think it would be impossible to encounter characters using 3 bytes, like mathematical characters... see this list for all the possible ranges: http://www.unicode.org/Public/UNIDATA/Blocks.txt

I'm saying this because I'm not sure what you mean by "This does not hinder a future pure 16bit unicode implementation of Avisynth. Yes! " and why would you think going the pure 16bit unicode way would be smart - utf-8 seems more sane to me based on what I've read. UTF16 would just force you to use 2 bytes for every char, at least, yet you'd still need up to 4 bytes for one, just like with utf-8.

If you need this for optimizations or something, then utf-32 and 32bit words would seem a better choice... but maybe it's not worth it.
Actually the UTF-8 encoding scheme can have up to 6 bytes for a given code point, this is not really a problem, as all code points >127 are represented by bytes all with the 128 bit set. The only visible issue is with the line position pointer in the parser used for reporting script errors.

Fragment of code from a UTF-8 encoder if (ch < 128) {
bt1 = ch;
}
else if (ch <= 2047) {
bt1 = (BYTE)(192 + (ch / 64));
bt2 = (BYTE)(128 + (ch % 64));
}
else if (ch <= 65535) {
bt1 = (BYTE)(224 + (ch / 4096));
bt2 = (BYTE)(128 + ((ch / 64) % 64));
bt3 = (BYTE)(128 + (ch % 64));
}
else if (ch <= 2097151) {
bt1 = (BYTE)(240 + (ch / 262144));
bt2 = (BYTE)(128 + ((ch / 4096) % 64));
bt3 = (BYTE)(128 + ((ch / 64) % 64));
bt4 = (BYTE)(128 + (ch % 64));
}
else if (ch <=67108863) {
bt1 = (BYTE)(248 + (ch / 16777216));
bt2 = (BYTE)(128 + ((ch / 262144) % 64));
bt3 = (BYTE)(128 + ((ch / 4096) % 64));
bt4 = (BYTE)(128 + ((ch / 64) % 64));
bt5 = (BYTE)(128 + (ch % 64));
}
else if (ch <=2147483647) {
bt1 = (BYTE)(252 + (ch / 1073741824));
bt2 = (BYTE)(128 + ((ch / 16777216) % 64));
bt3 = (BYTE)(128 + ((ch / 262144) % 64));
bt4 = (BYTE)(128 + ((ch / 4096) % 64));
bt5 = (BYTE)(128 + ((ch / 64) % 64));
bt6 = (BYTE)(128 + (ch % 64));
}

[IanB]

One important thing to consider is that there are text editors out there which don't necessarily write a BOM when saving to UTF-8 or even UTF-16, either by default or user-configured.

Most advanced editors (UltraEdit, EMEditor, Unipad) scan a document upon opening to determine the encoding if they don't find a BOM.

So, the absence of a BOM doesn't necessarily mean that the file has 8-bit (Ansi) encoding.Both 16bit Unicode variants {LE, BE} are always easy to recognise by the null bytes in each pair of bytes even without a BOM.

Distinguishing UTF-8 from an 8 bit ansi codepage encoding is very difficult without the BOM. On first glance both are streams of bytes [1..255]. So that doesn't help. However UTF-8 will match one of these patterns :-0xxxxxxx
110xxxxx 10xxxxxx
1110xxxx 10xxxxxx 10xxxxxx
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxxSo a file not matching this cannot be UTF-8, but the opposite is not true. A file matching this can still be an 8 bit ansi codepage encoding. And given we want to assume a BOMless file is an 8 bit ansi codepage encoding we are a little stuck. Strictly speaking even the BOM could be a valid 8 bit ansi codepage encoding but such a file cannot be a valid Avisynth script.

Thus my idea for a magic first line :-Maybe support a magic #!UTF-8 style first line detection to override default behaviour.

[Groucho2004]

Maybe this helps with the encoding detection:
http://www.codeproject.com/KB/recipes/DetectEncoding.aspx

I did some experiments with MLang a long time ago and the results were pretty good.

[IanB]

All Fire Trucks are red!

Your car is red.

Therefore your car is a Fire Truck.
________________________________

All UTF-8 files match said pattern!

Your file matches said pattern.

Therefore your file is UTF-8

:devil:

[Gavino]

Actually the UTF-8 encoding scheme can have up to 6 bytes for a given code point, this is not really a problem, as all code points >127 are represented by bytes all with the 128 bit set. The only visible issue is with the line position pointer in the parser used for reporting script errors.
What about the string handling functions in the script language, especially those that refer to index or length? For example, FindStr, LeftStr, RightStr, MidStr (or even StrLen). Does it all come out in the wash, transparently to the user?

[Groucho2004]

All Fire Trucks are red!

Your car is red.

Therefore your car is a Fire Truck.
________________________________

All UTF-8 files match said pattern!

Your file matches said pattern.

Therefore your file is UTF-8

:devil:

:)

Also a nice read: "Utf8_16.cpp" in the Notepad++ source on SourceForge.

[mariush]

RFC 3629 (http://tools.ietf.org/html/rfc3629) restricts UTF-8 to 4 Bytes.

Theoretical, yes, UTF-8 is up to 6 bytes but just like with the x64 architecture which only uses 44 bits out out 64 bits of memory addressing afaik, it looks like for a few years at least it's safe to assume it will only be up to 4 bytes.

UTF-8 files are not required to have a BOM (which is 0xEF, 0xBB, 0xBF for UTF-8).. see this: http://www.unicode.org/faq/utf_bom.html#bom1

You're supposed to detect it the file is UTF-8 or not and that doesn't work quite well.. as you can see by looking on Youtube at all those clips showing how when you write "bush hid the facts" (http://www.youtube.com/watch?v=FueE3WqHNl4) in Notepad the saved result is garbled when viewed back.

Well, it's just bad unicode detection in windows... these pages explain how the encoding is detected (the blogger works at Microsoft at fonts or something like that):
http://blogs.msdn.com/b/michkap/archive/2006/07/11/662342.aspx
http://blogs.msdn.com/b/michkap/archive/2008/03/25/8334796.aspx
http://blogs.msdn.com/b/michkap/archive/2007/04/22/2239345.aspx


I'm not sure if something like #!UTF8 would be wise, I don't often see something NEGATING...

My first thought would be at something like EnableUTF8 = TRUE at the beginning of the script (which would be by default FALSE to keep compatibility with older plugins and be TRUE by default on a later version, 3.0 for example)

Or maybe just having to use the .AVSU extension instead of AVS (just like Windows uses A and W at the end of functions to make distinction)... There's no problem in Windows having more than 3 characters in extension

Or maybe autodetect if the filename is in the form of.. let's say Script.UTF8.avs or Script.UTF-8.avs

ps.

So a file not matching this cannot be UTF-8, but the opposite is not true. A file matching this can still be an 8 bit ansi codepage encoding. And given we want to assume a BOMless file is an 8 bit ansi codepage encoding we are a little stuck. Strictly speaking even the BOM could be a valid 8 bit ansi codepage encoding but such a file cannot be a valid Avisynth script.


It doesn't work quite like that. The standard says that if you encounter an error or a notation for a yet undefined character, it's up to you... you're supposed to either silently ignore the bad bytes, on display show ? or the ? reversed (the Spanish "?") and copy the bad bytes as they are if it's not shown... I can't find exactly in what document this was said (I remember it from yesterday) but I can paste a quote from the specs:

(page 94, http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf#G28070)


Although a UTF-8 conversion process is required to never consume well-formed subsequences
as part of its error handling for ill-formed subsequences, such a process is not otherwise
constrained in how it deals with any ill-formed subsequence itself. An ill-formed
subsequence consisting of more than one code unit could be treated as a single error or as
multiple errors. For example, in processing the UTF-8 code unit sequence <F0 80 80 41>,
the only formal requirement mandated by Unicode conformance for a converter is that the
<41> be processed and correctly interpreted as <U+0041>. The converter could return
<U+FFFD, U+0041>, handling <F0 80 80> as a single error, or <U+FFFD, U+FFFD,
U+FFFD, U+0041>, handling each byte of <F0 80 80> as a separate error, or could take
other approaches to signalling <F0 80 80> as an ill-formed code unit subsequence.


ps2 ... yeah I found that in Wikipedia actually : http://en.wikipedia.org/wiki/UTF-8#Invalid_byte_sequences


RFC 3629 states "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."[7] The Unicode Standard requires decoders to "...treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence."
[..]
Popular replacements are:

* The replacement character '�' (U+FFFD)
* The '?' or '¿' character (U+003F or U+00BF)
* The invalid Unicode code points U+DC80..U+DCFF where the low 8 bits are the byte's value.
* Interpret the bytes according to another encoding (often ISO-8859-1 or CP1252).

Replacing errors is "lossy": more than one UTF-8 string converts to the same Unicode result. Therefore the original UTF-8 should be stored, and translation should only be used when displaying the text to the user.

[IanB]

What part of "cannot detect UTF-8 in the presence 8 bit ANSI code page encoding" do you not understand. Every possible UTF-8 files is also a possible 8 bit ANSI code page file. All the links and references so far pointed note this dilemma.

Unlike my previous facetious expression, the following logic is correct :-

All Fire Trucks are red!
Your car is NOT red.
Therefore your car is NOT a Fire Truck.

i.e. We can easily prove a file is NOT UTF-8 encoded, but we cannot prove that it is.


#!UTF-8 as in #!/bin/sh and #!/usr/bin/perl from Unix.


Parsable constructs like EnableUTF8 = TRUE are not on. All the processing has to be done as Import() reads the contents of the file into memory and before it is Eval()'d by the parser. I'm happy to drop the exclamation point, '!', and just compare the start of the byte stream with something like "#UTF8\r\n".


String handling functions in the script language are gonna be screwed. Document the existing functions as dealing with 8 bit bytes [1..255]. Add new functions that are multi byte aware.


Magic filenames are generally a bad idea and are not really the business of Avisynth.dll, they are more the business of the host application.

[mariush]

#!UTF-8 as in #!/bin/sh and #!/usr/bin/perl from Unix.


Avisynth is not a Linux application, it's Windows only, so I don't understand why you'd want to import a Unix/Linux concept to a Windows audience. Windows users are not familiarized to that notation.

I agree it's very hard to make distinction between UTF-8 and ANSI pages, that's why it's better to have some sort of tag that tells you it's UTF8. But you shouldn't have a tag that says it's NOT utf-8 ( C/C++ programmers without Unix/Linux may think the ! is NOT).

It should be something introduced as optional at first to mantain compatibility with older plugins until they're updated and then at a later version be enforced.


All Fire Trucks are red!
Your car is NOT red.
Therefore your car is NOT a Fire Truck.

i.e. We can easily prove a file is NOT UTF-8 encoded, but we cannot prove that it is.


Not all fire trucks are red... see http://www.google.ro/images?q=airport%20fire%20truck

http://www.oakhillfire.org/images/aE301.jpg
http://lh5.ggpht.com/_C9zOzLEte2o/Sg6c-N9GMFI/AAAAAAAACI8/RkedMOS4s90/purple-fire-truck.jpg
http://cityofsatsuma.com/images/FireTruckWhiteNov2008.jpg
http://www.indianvalley.net/veterans%20day%202008/hires/ryan/yellow_fire_big-re.JPG

[Gavino]

Change Import() to detect BOM prefix for UTF-8 & Unicode input else by default convert from ANSI input using :- MultiBytetoWideChar((AreFileApisANSI() ? CP_ACP : CP_OEMCP, ...)
WideChartoMultiByte(CP_UTF8, ...)Maybe support a magic #!UTF-8 style first line detection to override default behaviour.
Note that Import is not the sole route for getting text into Avisynth. Applications using the library interface can also use Eval - for example, I believe AvsP uses Eval instead of Import. Does your proposal mean that such applications will in future have to pass a UTF-8 string to Eval instead of an ANSI one?
String handling functions in the script language are gonna be screwed. Document the existing functions as dealing with 8 bit bytes [1..255]. Add new functions that are multi byte aware.
So some existing scripts (using these functions on strings with 8-bit ANSI) will stop working?

[henryho_hk]

If your NTFS did not have "8.3 filename generation" disabled, every file has a "secondary" filename which is non-unicode-local-codepage-safe. Do a "dir /x" under a command prompt and you can find them. This is the interim solution I have been using for years to load such files.

[Groucho2004]

If your NTFS did not have "8.3 filename generation" disabled, every file has a "secondary" filename which is non-unicode-local-codepage-safe. Do a "dir /x" under a command prompt and you can find them. This is the interim solution I have been using for years to load such files.

Disabling "8.3 filename generation" is a popular tweak so a programmer cannot assume that the short file name exists. Apart from that, dealing with 8.3 is very messy and error prone.

[krieger2005]

What do you think about this procedure.

1. Detecting of File-Coding can be done in two ways:
a. Reading the BOM
b.if the BOM does not exist, search for Character which mostly exist in Scripts: ".", "(", ")", "{", "}", "=". Determinate based on this Detection the File-Coding (BYTE-CODING, UTF16BE/LE, UTF32BE/LE).
c. if no file-coding could be detected assume BYTE-CODING.

BYTE-CODING can mean UTF-8, but also Coding in some Codepage. Since the Coding in some Codepage means that the Charset can not be interpreted without knowing the Codepage (so the Loading on an external PC can fail) best solution would be to interpret BYTE-CODING allways as UTF-8. For Scripts which are saven in a Codepage an extra Command could tell the Avisynth-Parser which Codepage is used.

Based on the Filecoding the Content should be converted into UTF-8, so that all internal processing is done on UTF-8 char* streams. This step make sure, that
1. Existing Plugins can be used as before since they still work on char* streams
2. Plugins which should support Widechar Character can be sure which type of Intput Stream they get and convert it into they own needed Format - For example mpeg2source could convert the Path into UTF16 (which is used by Windows XP and newer). Still it would be nice for People who implement the Plugin if Avisynth would support Functions to Convert an UTF8-Stream into "what they need".

This is my propose

[Gavino]

if the BOM does not exist, search for Character which mostly exist in Scripts: ".", "(", ")", "{", "}", "=". Determinate based on this Detection the File-Coding (BYTE-CODING, UTF16BE/LE, UTF32BE/LE).
How practical/reliable is this? Can you give a precise algorithm?
best solution would be to interpret BYTE-CODING allways as UTF-8.
This is not backwards compatible - some existing scripts (using 8-bit ANSI strings) will stop working.
For Scripts which are saven in a Codepage an extra Command could tell the Avisynth-Parser which Codepage is used.
Depends what you mean by 'command'. At best, it would be some simple directive outside the script language.
As IanB said in post #74, "Passable [I think he meant 'parsable'] constructs like EnableUTF8 = TRUE are not on. All the processing has to be done as Import() reads the contents of the file into memory and before it is Eval()'d by the parser."
... the Content should be converted into UTF-8, so that all internal processing is done on UTF-8 char* streams. This step make sure, that
1. Existing Plugins can be used as before since they still work on char* streams
But existing plugins expect to see 8-bit ANSI characters in strings, not UTF-8. Filenames, for, example, would need to be handled differently.
Conversion to UTF-8 is part of IanB's proposal, and as he recognised "This breaks existing 8 bit codepage behaviour that currently works satisfactorily for 2.5 plugins."

[IanB]

Note that Import is not the sole route for getting text into Avisynth. Applications using the library interface can also use Eval - for example, I believe AvsP uses Eval instead of Import. Does your proposal mean that such applications will in future have to pass a UTF-8 string to Eval instead of an ANSI one?

So some existing scripts (using these functions on strings with 8-bit ANSI) will stop working?
And yes, you stepped on the landmines as well.

I have been avoiding spoon feeding this discussion too much because I am hoping someone will come up with some ideas that do not paint one into a corner. Just about every way I look at this something always comes unglued somewhere.

My first thought about the API text was existing calls as ANSI, add new calls for UTF8, but that has problems.

And yes I really, really don't want existing scripts or applications to stop working.

Thoughts?


@krieger2005,

UTF16BE/LE, UTF32BE/LE are dead easy, I do not have any difficulties with 16 or 32 bit file encoding.

The hard part is 8 bit byte encodings.

The state of play has to be 8 bit ansi codepage encoding has priority.

That is what works now and should continue to work in the future.

UTF-8 is probably going to have to be externally identified. Either by a BOM, some reserved opening sequence on the first line of the file or some other devious scheme.

[krieger2005]

How practical/reliable is this? Can you give a precise algorithm?
This algorithm is used when detecting HTML/XML Content Coding, but here the Parser search for "<".

This is not backwards compatible - some existing scripts (using 8-bit ANSI strings) will stop working..
ANSI still have the first 7 Bits in common with ASCII and UTF-8. And all the scripts i know does not use special character above the value of 127. And even for this case: it would be practicable to Add the Codepage-Command, so that the Avisynth Parser can convert it internally to UTF-8.

Depends what you mean by 'command'. At best, it would be some simple directive outside the script language
I thought about an Avisynth Command like SetMemoryMax (for Example: UseCodepage(ANSII). I don't think an external derective is a good solution because when people share scripts and one have set this external derective and the other one not it could result in problems reading Scripts.

But existing plugins expect to see 8-bit ANSI characters in strings, not UTF-8. Filenames, for, example, would need to be handled differently.
Do they expect ANSI? Or do they just take what they get? I mean, plugins, which get Unicodestrings have to be updated in any case. For example mpeg2source must call the Widechar Functions to open the File.

EDIT: Sorry forgot the Algorithm:
1. Detect BOM (i will not describe this)
2. if BOM couldn't be detected:
*char_codes = ".{}()=";
for(int i =0; i < min(filesize,50); i++)
b = read_byte_from_file
if(strchr(char_codes, b) != NULL)
look if previous byte was a 0 // UTF 16LE ??
look if next byte is a 0 // UTF16BE ??
if(UTF16LE)
look if the three previous byte are 0 // UTF32LE
if(UT16BE)
look if the three next bytes are 0


If the script contains on of the searched character then the algorithm can determinate in 100% the correct filecoding. I Assume that 99.9% of all script contain one of the characters above. So this algorithm will work for 99.9% of all Files.

[krieger2005]

The hard part is 8 bit byte encodings.

Ah, i forgot this step... in my program i had the fallback to iso8559-1. I made the Parser for HTML-Files. Additionally i ran the ICU-Charset Detector and used a high relibiality to decide to trust the Detector or not.

But, as i stated above i prefer the Definition of the used Charsed inside the File istead definening somewhere outside something. This way it get just the same as HTML does in their Files.

There is still one difference. I have heard here from people that they use Codepages so the Script works in foreign Languages. Others use ANSI. But: What part of the Script use Characters above the 127 Bounds (and not in comments)? Based on the Answer of this Question, i think you can decide to Support ANSI as Fallback or just suppose the Script to be an UTF-8 Script.

[kemuri-_9]

And yes, you stepped on the landmines as well.

I have been avoiding spoon feeding this discussion too much because I am hoping someone will come up with some ideas that do not paint one into a corner. Just about every way I look at this something always comes unglued somewhere.

My first thought about the API text was existing calls as ANSI, add new calls for UTF8, but that has problems.

And yes I really, really don't want existing scripts or applications to stop working.

Thoughts?

the least evil way of having this work as i see it is to have an avisynth variable control what character encoding Eval() is expected to be given
(outside of Import's currently planned handling, as import will have the file encoding to handle the situation)

something similar to the other global variables like OPT_UseWaveExtensible, except that this new variable can't be set within scripts to avoid issues that would bring.

something like

AVSValue y = AVSValue( true );
env->SetVar( "$OPT_EvalIsUTF8", y );

would trigger Eval to expect UTF8 strings instead of ANSI and the value is defaulted to false.

this would technically complicate Import though, as it would need to handle dealing with this variable...

[Groucho2004]

Maybe I'm missing something but why would we need UTF-8 at all? What about either 8 bit Ansi or UTF-16 as choices?

The advantage would be that both encodings are easily identified, whether there is a BOM in UTF-16 or not.

[mariush]

Notepad and most editors, when you select Unicode at Save as, they'd default at UTF-8. You would have to do 2-3 extra clicks to select on purpose UTF-16.
Once you say your app support Unicode, people would naturally assume you support UTF-8 too, so they'd be constantly surprised if their scripts fail. So you'd have to support UTF-8 anyways.

kemuri - you could also use an environment variable for that, I think.

ps. Would installing two libraries be possible, for example avisynth.dll and avisynthw.dll for unicode ? maybe have some "Stub" or whatever it's called that would then load the appropriate dll depending on how it's initialized?

ps2. Well ok, maybe not Notepad, which has ASCII, then Unicode, then Unicode Big Endian and then UTF-8, at least on my Windows 2003, but other editors have UTF-8 first, like for example the popular Ultraedit:

http://savedonthe.net/image/648/uedit.png

[Groucho2004]

Once you say your app support Unicode, people would naturally assume you support UTF-8 too, so they'd be constantly surprised if their scripts fail.

Well, that 's just a matter of documentation, isn't it? You don't see lots of people surprised that their car's engine is broken because they put diesel in it instead of petrol (that's gas for our American readers), right?

[mariush]

When you have a 15-25 thousand car and chances of losing it for a day, you're more careful about that sort of thing. However, even so it will still happen to screw up.

People won't be as careful with a script as with a car, and for lots of people it would be routine, automatism, and they'd expect for something to work as it works with other programs - most won't even read the documentation, as I'm sure you didn't read the manual for Windows or other programs you have first. People will just make mistakes a lot and will be pissed and will fill the forum and most won't even understand the difference between UTF-8 and UTF-16 and won't care to learn - they just want to convert a video.

I've been reading Raymond Chen's blog (http://blogs.msdn.com/b/oldnewthing/) for years (he's a guy that worked at Microsoft on Windows 95 and still works there) and I highly recommend you browse the posts in the History section... I've learned to appreciate the extend Microsoft went to preserve compatibility with lots of things as operating systems improved and often found some things I thought to be wrong to be actually quite smart and appropriate.

I've also found out that we often make assumptions that are not really true in the real world - open source programmers and small developers don't have access to millions of people using the software to know how they think so just assuming they won't use or that they'll learn not to use UTF-8 is bad and won't work in reality.

[Gavino]

What part of the Script use Characters above the 127 Bounds (and not in comments)?
String literals, of course.
Especially (but not limited to) filenames - which is what I thought was the main point of this thread (or at least the starting point).

[krieger2005]

String literals, of course.
Especially (but not limited to) filenames - which is what I thought was the main point of this thread (or at least the starting point).
Exactly - Main Point of this thread are Filenames. But who use a script with a filename multiple times? I for myself write for every Movie another script. So how many people use such scripts out there (PS: I forgot Filenames in Imports... okey)?

This is the reason why i asked if it is really usefull to be backward compatible.

The reason why i suggest UTF8 was (i does not know the internals of Avisynth), that i thought, that avisynth serve the Data of the script to the Plugins after it has parsed it. So the main point is: how to should save avisynth the data internally in the parsing step? Support two formats and supporting two interfaces is one possiblity. But why should one do this?

The Benefit of UTF8 is, that it is represented character by a bytestream. So, every old Plugin can be still used with the old interface without a recompile (no wchar, no int for UTF16 character, simply char*).

But maybe i misunderstood the point how avisynth works internally.

[kemuri-_9]

kemuri - you could also use an environment variable for that, I think.

no, that's asking to get shot with a handgun:
a user could have happened to set the environment variable to 'true' and then they use a program that isn't up-to-date in this regard expecting to use ASCII strings as things stand now.
But due to the environment variable, avisynth will be expecting UTF8 strings.
this would very likely cause unpredictable results and possibly even crashes.

don't even say 'use the registry' either, that's asking to get blasted with a shotgun.

If this route is actually taken, Import() would be required to do something like
1) backup the current value of the variable, whatever it is.
2) read in the file, detecting the encoding as necessary
3) set the variable to be false for the normal ASCII files, otherwise set to true and use UTF-16 -> UTF-8 conversion before passing to Eval(), if necessary.
4) restore the backed-up variable.

doing this with environment variables is ugly especially since they are string values.
the registry is unusable as it can break when multiple instances of avisynth are being used simultaneously.

this is why i proposed to just keep this handling localized within avisynth.

[Gavino]

But who use a script with a filename multiple times? I for myself write for every Movie another script.
I get your point, but the non-ASCII characters could be in a folder name that is used for many movies. Or (as you say) in an Import of common script code. Also, I have many scripts that I use over and over again for playing (eg a specific combination of clips) rather than encoding.
The reason why i suggest UTF8 was (i does not know the internals of Avisynth), that i thought, that avisynth serve the Data of the script to the Plugins after it has parsed it.
That's correct (assuming by 'data' you mean the parameters passed to the plugin function).
The Benefit of UTF8 is, that it is represented character by a bytestream. So, every old Plugin can be still used with the old interface without a recompile (no wchar, no int for UTF16 character, simply char*).
But if the plugin doesn't 'know' that the string is UTF-8, it won't necessarily do the right thing when using the data. For example, if it is a filename, a different procedure must be used to open it.

[henryho_hk]

Alternate proposal (similar to shell script headers) for file format:

1) Support 8-bit local codepage and UTF-8 only
2) BOM is optional and ignored (it is optional for UTF-8 plain text files anyway)
3) All UTF-8 encoded AVS files must have a special comment like "#! UTF-8" at the very first line. If it is not found, the file is assumed to be in 8-bit local code page encoding (irrespective of BOM).

99.999% full backward compatibility with existing scripts and easy to implement.

[LoRd_MuldeR]

Doesn't look like that thread ever came to final conclusion. And even if it eventually will, Unicode (UTF8) support will only be in new versions, not in the old 2.5.x series, right?

So I guess I will stick with the VfW interface for now, which at least should support Unicode names...

I know that this won't work if Unicode strings are used inside the Avisynth script, but at least I can open the AVS file, if the name of the AVS file itself contains Unicode characters.

(Still I have no clue why VFW on my system apparently opens such files successfully, but then reports that there are no streams)

[kemuri-_9]

indeed, avisynth does not support utf-8 within scripts and API calls, this includes the basic 'Import(file.avs)' call necessary to load existing scripts into avisynth via the API.

when support does come, it is very likely that it will not be backported to earlier releases either.

you could always work around the vfw interface not failing for when the file doesn't exist by doing a _waccess( filepath, 0 ) check beforehand.

[LoRd_MuldeR]

indeed, avisynth does not support utf-8 within scripts and API calls, this includes the basic 'Import(file.avs)' call necessary to load existing scripts into avisynth via the API.

But what happens in the case when we have an AVS file that does not contain any Unicode (non-Latin1) characters, but the AVS file itself has a Unicode (non-Latin1) name?

I currently won't be able to import that script file with Avisynth' native API, but I am able to open the file with AVIFileOpenW(). Does Avisynth' VFW wrapper fail internally in that case?

At least I have not been able to successfully load a simple AVS file with Unicode-characters in the file name this way...

you could always work around the vfw interface not failing for when the file doesn't exist by doing a _waccess( filepath, 0 ) check beforehand.

Currently I check for the existence/accessibility manually with wfopen(), but this way it's more convenient. Thanks for the hint!

[kemuri-_9]

But what happens in the case when we have an AVS file that does not contain any Unicode (non-Latin1) characters, but the AVS file itself has a Unicode name?

I currently won't be able to import that script file with Avisynth' native API, but I am able to open the file with AVIFileOpenW(). Does Avisynth' VFW wrapper fail internally in that case?

At least I have not been able to successfully load a simple AVS file with Unicode-characters in the file name this way...



from what i read in the codebase, the (simplified) VFW wrapper flow goes:

entry -> Load( OLEStr, mode )

unchecked convert OLEStr -> char *filename (native/oem codepage - not utf-8)
copy filename to a member variable
...
IScriptEnvironment->Invoke( "Import", filename )

... same code flows as AVS API.


so the only difference is that the VFW does the conversion from utf-16 filename to a native/oem codepage filename itself.

It's Import() that throws an error if the file doesn't exist, and it primarily uses xxxxxA versions of the windows API (avisynth is compiled with MBCS after all).

So if it is working with utf-16 filenames that do not map to the native/oem codepage, that's quite surprising.

[LoRd_MuldeR]

Yup, that's clearly the reason why it doesn't work with Unicode (non-Latin1) file names, even when AVIFileOpenW() initially succeeds. I think the only workaround at this time is converting the file names to "short" names and passing the short ones AVIFileOpenW(). Short names usually survive the conversion to the local 8-Bit Codepage. It's not nice and it's not guaranteed to work (AFAIK the MSDN doc nowhere says that "short" names can't contain Unicode chars, but it does say that files are not guaranteed to have a short name), but it will probably save us most of time...

[kemuri-_9]

Using shortnames is not likely to work either:

Import uses GetFullPathName or SearchPath (which is used depends on if there's a path delimiter in the filename) to get the fullpath/filename of the script file.
these will error if the filepath contains characters outside of the native/oem codepage.

[LoRd_MuldeR]

Using shortnames is not likely to work either:

Import uses GetFullPathName or SearchPath (which is used depends on if there's a path delimiter in the filename) to get the fullpath/filename of the script file.
these will error if the filepath contains characters outside of the native/oem codepage.

GetFullPathName() is one of the things that can prevent the "pass short path name" workaround from working, I know :rolleyes:

However with Avisynth it works for me here, passing the "short" version of a fully-qualified path:
avs2wav v1.2 [May 21 2011]
by Jory Stone <jcsston@toughguy.net>, updates by LoRd_MuldeR <mulder2@gmx.de>

Input: G:\ViDeOz\Music\Володимирович 菅直人 κλασικής.avs
Output: Dump.wav

Checking Avisynth... Done
Analyzing input file... Done
Opening output file... Done

[Audio Info]
TotalSamples: 11391413
TotalSeconds: 258
SamplesPerSec: 44100
BitsPerSample: 16
Channels: 2
AvgBytesPerSec: 176400

Dumping audio data, please wait:
0/11391413 [0%]
11391413/11391413 [100%]

All samples have been dumped. Exiting.