Concatenation failure on 4096 byte boundries

[maxxon]

This is an odd one that I found while profile testing some of my code:

global output__junk = BlankClip(height=0, width=0, length=0)
global output__filename = "output.log"
function output(string s)
{
    output__junk.WriteFileStart(output__filename, "s", append=true)
    return 1
}

a = "        " # 8 
b = a+a+a+a+a+a+a+a # 64
c = b+b+b+b+b+b+b+b # 512
d = c+c+c+c+c+c+c+b+b+b+b+b+b+b+a+a+a+a+a+a+a+"      " # 4094
d.strlen.string.output
e = d + " "
e.strlen.string.output
e.output

I've not put in the code to actually output the byte locations (requires too much of my libraries), but if you check it out with an editor, you'll find that the 3rd line outputted contains crap near the end of the string and the string length has increased by 2-4 characters rather than 1.

Must have something to do with your string allocation methods as this can happen on any 4096 byte boundary, though it doesn't always happen. It appears that certain even multiple cats can keep this from happening.


|\/|x

[IanB]

AVSValue ExpPlus::Evaluate(IScriptEnvironment* env)
...
  else if (x.IsString() && y.IsString())
    return env->Sprintf("%s%s", x.AsString(), y.AsString());
  else {
...

char* ScriptEnvironment::VSprintf(const char* fmt, void* val) {
  char *buf = NULL;
  int size = 0, count = -1;
  while (count == -1)
  {
    if (buf) delete[] buf;
    size += 4096;
    buf = new char[size];
    if (!buf) return 0;
    count = _vsnprintf(buf, size-1, fmt, (va_list)val);
  }
  char *i = ScriptEnvironment::SaveString(buf);
  delete[] buf;
  return i;
}

vsnprintf, _vsnprintf, and _vsnwprintf return the number of characters written if the number of characters to write is less than or equal to count; if the number of characters to write is greater than count, these functions return -1 indicating that output has been truncated. The return value does not include the terminating null, if one is written.

So when _vsnprintf returns 4095 no trailing NULL was written, so the trash after the end of the string is exposed.

Will fix asap.

[maxxon]

Cool.

Where would I get the latest build with the fixes to the bugs that I've uncovered?


|\/|x

[looney]

Could this issue be translated into issue i have with multiple concatenation of OpenVidFiles [1] & [2]

Is it possible to detour this errata by concatenating strings into separate variables up to 4096bytes long, followed by pasting all those LessThan4096BytesStrings contained in Vars into one variable without AviSynth being aware of TotalStrLen but final StrLen itself being only of total length of NamesOfVars and plus chars?

Originally Posted by IanB

Hint, check for ((strlen(a)+strlen(b)+1) % 4096) == 0

Thanks. Well that shouldn't concern me though
It's a pretty weird issue, it's more like that someone searched for bug than fint this an issue. Anyway it's good that it's fixed. Patch is in 2.6alphaX version?

[IanB]

The bug is when you concatenate 2 strings and the resulting length is 4096*k-1, where k is a positive integer. i.e 4095, 8191, 12287, 16383, ....

Hint, check for ((strlen(a)+strlen(b)+1) % 4096) == 0

[StainlessS]

I've just been bitten by this bug and expect to be repeatedly bitten again soon so,
have put up an update for The RT_Stats plug with function
which allows concatenation of 2 or more strings and returns the resulting string.

Useful for 2.58 or until bugfix appears in 2.6a4.

EDIT: Reason for being bitten, RT_Stats has several funcs using EOL [chr(10)] separated strings
and a sort of string indexing function allowing usage of concatenated strings like a single dimension array.

EDIT: Can download RT_Stats v1.03 here:
http://forum.doom9.org/showthread.php?t=165479

Two functions that give an alternative solution to this problem

RT_StrAddStr(String, String s1, ... , String sn)
Non-clip function. 
Function to concatenate (join together) 2 or more strings.
D = RT_StrAddStr("A","B","C") same as D="A"+"B"+"C".
There is a bug in v2.58 and 2.6a3 when joining strings that result in sizes of [(n * 4096) - 1],
this function just gives an alternative method of joining strings until 2.6a4 is released.


RT_TxtAddStr(String, String s1, ... , String sn)
Non-clip function. 
Function to concatenate (join together) 2 or more strings with Chr(10) line separators.
If the first string is an empty string ("") it will not have a newline [Chr(10)] appended.
All other strings even empty strings will have a  newline [Chr(10)] inserted after them
(if they dont already have a trailing newline).
Any source string containing Chr(13) will have them converted to Chr(10). 
X = RT_TxtAddStr("A","B","C") same as X = "A" + Chr(10) + "B" + Chr(10) +"C" + Chr(10)
X = RT_TxtAddStr("","A","B","C") == "A" + Chr(10) + "B" + Chr(10) + "C" + Chr(10)

EDITED: