[NEED HELP] Software and Unicode ! ( japanese, korean, chinese ... guy plz )

[Dark-Cracker]

hi,

i search some people who use an non occidental alphabet like :
chinese, japanese , korean, turkish, russian, arabic ... who use an different alphabet.

i actualy try to add multilanguage support on my software and i am trying to add a function for UTF-8 (unicode) decode/encode string in order to display correctely foreign characters.

unhappyly my OS work with occidental alphabet and i can't made all the tests needed .

and i have decided to open this thread and i hope some people could give me an hand in my tests :)

http://www.eclipsedvd.firstream.net/autodub/UTF8.zip
(decompress it and launch setup.bat)

here u can find a test program to see if the characters are correctely converted in UTF8 and if there are decoded correctely.

it come bundle with the vb6fr.dll and the fm20.dll (dll needed to vb for unicode control) and mso97rt.dll (normaly installed with the office 8/9) used to translate some characters using matrix.

i have also post a picture :)
i have copy/paste a japanese word ("optimist" if i beleve the dictionary) to show how the encode/decode function work.

http://www.eclipsedvd.firstream.net/autodub/UTF8.jpg

if u could made the test following :

1) write a word in your language
2) convert it in UTF8 and copy it
3) clear all textbox
4) past the UTF8 text and decode it

and say me if it show u the original word. and in which language u have made the test :) of course japaneses, korean, chinese testers are welcome.

if all work fine i will post u a little software to check if reading a UTF8 file show u correctely the word. very usefull to add universal multilanguage support to the software :)

Bye.

[Dark-Cracker]

just a little question :

if i use UTF-7 instead of UTF-8 this will only change the lenght of the string encoded (because u use 7 bits instead of 8 bits to code the string) but if i use utf-7 does some characters will be still correctely displayed or this will add a range of 127 (because 127 charracters ascii use 1 octet) characters wrong displayed ?

thank u for your answer :)

Bye.

[gabest]

From rfc2152:

"UTF-7 should normally be used only in the context of 7 bit transports, such as mail. In other contexts, straight Unicode or UTF-8 is preferred."

If you read the rest of the text you will understand why this suggestion is there, nothing uses or interprets utf-7 normally unless it is forced to, like email programs.

[Dark-Cracker]

hum apparently VB have some problem to understand correctely some characters encoded in utf-8 for exemple if i try to encode the char 'ö' in utf8 it coded it correctely (i have tested with another tool to encode in utf-8) but when i try to decode it it failed.

i have tested with CP_UTf7 and all work file , i could rewrite an another function to decode utf-8 but if utf7 work fine i could force vb to encode/decode , it need this function to correctely display multilanguage file.

Bye.

[gabest]

It's not clear to me where or why you would like to use utf-7, ... and there no standard way to store utf-7 in a file.

It the window of this test program unicode? I can't enter the letter û (u") at all into the top edit box.

(edit: neighter here correctly :))

[Dark-Cracker]

@gabest

what does it mean ? : "neighter here correctly " sorry my english was very poor could u rewrite it in simple words ?

yes my english is very crap and it's the first time i play with unicode :) and i perhaps misunderdand some english documentation if have read.

i will try to explain me more correctly.

i try to add multilanguage to my VB softwares, i have create some some .text file (".lng") where the label of each textbox are stored.
some translators have report me VB doesn't correctely display foreign caracters ( 'č' 'ω' .... ) they have write in the ".lng" file.
after a quick search i have find a way to use some new VB controls using unicode textbox. all the caracters (arabic, japanese ...) are correctely displayed (if u have unicode font i supose). and i have deceided to use the UTF-8 encode/decode function to store the translated text in the text file (".lng").
after a quick search i have found 2 solutions :
- first a function with the table of the 67*2 chararcters of vowels and only used to encode/decode in UTF8
- another function using the api WideCharToMultibyte and MultibytetoWideChar to convert text in a selected Code page
(i have test with CP_UTF8=65001)

i choose the second solution for a much clean way.
i use an HTML page to check if the decoding work fine.
for exemple : in UTF8 your char 'û' = 'û' i test using this :
--------
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p><font face="Tahoma">û</p>
</body>
</html>
--------
my tool (upper in the page) made this convertion in utf8 . u enter your foreign text and it convert it and try to decode it again to be sure the encoding process is correct.
but after some tests some caracters are not correctely encoded . 'ö' for exemple are not correctely encoded/decoded using my tool.
and i have try with UTF7 (CP_UTF7=65000) and the caracter are correctely displayed in my tool.
for exemple in utf7 : 'û' = '+APs-' as always u can made the test :
--------
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-7">
</head>
<body>
<p><font face="Tahoma">+APs-</p>
</body>
</html>
--------
and it will show u correctely the char.
so i have think : why not use UTF-7 to code all the foreign sentense in the .lng file.
i hope it's more clear now :)

it's the first time i really interest me in unicode and perhaps i say/write some silly sentense. feel free to correct me if i do some mistake.
PS: have u run the setup.bat file for my tool ? could u write letters ?

thank u for your help :)
Bye.

[gabest]

Originally posted by Dark-Cracker
what does it mean ? : "neighter here correctly " sorry my english was very poor could u rewrite it in simple words ?"Here" = "forum", that letter displayed wrong (u^ instead of u"), just like when I entered it to your program to be converted.

For the langauge files, it doesn't really matter how you store the text, but you should know that windows either wants normal 8 bit chars interpreted with the current OS codepage setting for non-unicode apps (means it is global for every non-unicode app), or under nt you can use the unicode api to send utf-16 le text in functions calls directly to the window controls. This latter needs that you create the windows in unicode too, which I'm not sure how vb can handle.

[Dark-Cracker]

apparently utf7 it the same than utf8 except than it use 1 bit less to avoid problem with some mailbox.
i have update my tool to convert in utf-7 to add a working unicode way to my multilanguage function.

http://www.eclipsedvd.firstream.net/autodub/Utf7.zip

i have added in the file "All_language.txt" 19 sentense of foreign language encoded using UTF7 (such as korean, japanese, chinese, turkish, russian, swedish, arabic, norwegian, hebrew ...) normaly if u have all unicode font (downloaded by your browser) u could see all this language by clicking on the button "Display".

screenshot : http://www.eclipsedvd.firstream.net/autodub/utf7.jpg

apparently there is no restriction on characters for utf7 , i am right ?

i have also added a "standard textbox" , could u try to write some strange characters ('û') it's perhaps because "unicode textbox" doesn't support keyboard layer ( like '"u' ) and could u try to copy/paste character to see if u can bypass this possible restriction .

If a japanese or chinese guy could say me if the characters are correctely if will be very nice.

>which I'm not sure how vb can handle.

no i think it's not possible to do this in vb . but i don't know which exactly api unicode use to send text in function.

in vb to made unicode u need some "unicode textbox" or "unicode label" instead of "strandard textbox/label" who only support western europe alphabet and use some function like strconv(string,vbUnicode).

Bye.