Writing an encoder... strange problems! (confused brain dump inside)

[jonny]

I'm writing an encoder, something similar to AVS2AVI.
I'm actually testing it over XviD & DivX.
I'm testing the frames (keyframe info, length and raw data of every frame) between my app and VirtualDub.

In some codec configurations, the frames are different!
At the start i was thinking about something missing/wrong with my app, but i've noticed that, for example, even VirtualDub generates different frames (i've obtained difference between the release and the debug build of VDub).

Another strange thing is that i obtain some differences with XviD even in my app... there is a printf on the encoding loop, if i delete it, the frames after the first one and the last frames are different... (this happens only with 2 pass + bframes - of course, i'm skipping correctly the delay frames).

I hope someone can bring the light over this:
1) How different library linking (or perhaps different compiling options) can affect the generate frames?
2) The differences are little, is this normal and can i ignore all this stuff?
3) The printf problem i've mentioned... is it my fault or can it be an XviD problem?


Some notes:
- I'm using the ICSeqCompressFrameStart path (VirtualDub uses ICCompressBegin path... but i've tryed to include the full VDub's VideoSequenceCompressor class, still obtaining differences)
- I've tryed to surround the critical part with emms (frame reading and encoding) but this don't change the situation.
- I've not tested this deeply, but seems that the frames generated by my app are identical to the AVS2AVI frames (and putting a printf inside the AVS2AVI encoding loop give also the strange result)
- If someone need this, i can upload my testing enviroment (a modified version of VDub, my encoder and a modified version of Enc that encode and test for frames differences automatically)

Thanks
jonny

[sysKin]

Nothing like this should ever happen. Can you tell me what xvid you are using? - that's the most important part ;)

Radek

[jonny]

Sure ^^.

- I'm using Koepi's build dated 24-06-2003.
- An avisynth file containing only "version()".
- 2-pass, all default options but maxbframes=3 and targetsize=20KB.

Running this 2 passes with the release build and after with the debug build of VDub, you should be able to replicate the problem and see frames differences over the 2 avis.
(i obtain frame's differences only with some configurations, mainly with 2-pass+bframes)

If you need, i can upload my encoder so you can also take a look at the printf problem.

Also my encoder and my modified VDub both writes out raw data (keyframe info, frame lentgh and frame data of each frame), it's easyer to compare this data instead of parsing the avi.
Let me know what you need, i can upload all.

My system is an XP 2000+ under Win2000SP3, using VC6 SP5 + PP + the platform SDK of feb2003.

jonny

[Nic]

There's no way this could be a MMX issue/lack of emms issue is there Radek?

@jonny: Does this problem still occur if not using B-Frames? What other configurations can cause it?

Bit of a stumper this one, good luck :)

-Nic

[jonny]

Some news

1) I've tested Debug vs Release VDub's build over:

XviD
DivX
WMV9
DivX 3 (Fast & Low)

Only XviD seems affected and only on 2-pass + bframes, tested with maxbf = 1, 2 and 3 (i've tested every codec, where possible, with different modes: 1-pass, 1-pass qb and the various 2-passes and different parameters).

2) I've stripped out all the "vdprotected", "VDCHECKPOINT", "_RPT" on VideoSequenceCompressor.cpp (VideoSequenceCompressor VDub's class), in this way the Debug version seems to generate the same frames as the release one.
Those VDub's macros (not sure about VDCHECKPOINT) probably use printf or some kind of stdoutput/stderrput (^^), and probably gets compiled only on the debug mode.
So i think the problem is still related to printf (or stdout/stderr usage???).
Going to sleep now i'll do a better checking about those macros tomorrow ^^

jonny


EDIT:

There's no way this could be a MMX issue/lack of emms issue is there Radek?

Avery Lee told me that there are no issues with XviD in this direction.


EDIT2:

Nic, what is the meaning of stumper :) ? (altavista doesn't traduce it!)

[DDogg]

... meaning of stumper Head scratcher, conundrum, mind bender, unsolvable puzzle, something hard as hell to figure out. :)

[jonny]

Thanks DDogg :)


I quote Avery:

If I had to take a guess, the problem is that some code
in the system is reading uninitialized memory, either uninitialized
variables on the stack, or memory off the end of a buffer. XviD in
particular is suspect for this. You can try to narrow this down by
doing a big honking alloca() and memset() before calling out to a
codec, and padding image buffers with nulls on both ends....

None of the macros you describe call printf() in a release build.
_RPTx() is debug only, while VDCHECKPOINT simply inserts a static
pointer entry into the per-thread debug queue, which is the reason
it's so fast. The vdprotected() macro simply establishes an exception
context frame on the stack with the format and parameters given -- it
doesn't actually format a text string unless someone iterates down the
exception context list and requests the strings, which only the
crash handler does. What it does do, however, is allocate space on
the stack for the context frame, so my original theory of
uninitialized stack accesses still stands.


EDIT:
I have to specify that i've progressively tested/stripped out the macros and _RPTx was the last one (after removing _RPTx the frames were identical).
So i'm not sure about witch macro (both _RPTx&vdprotected?) give the
effect (i'll test this when i'll return home, i'm @work now)

[Nic]

if I get some time ill look through the vfw code, see if there is anything in there that would be causing it, can't believe that the xvid lib has the problem ??

&
http://www.hyperdictionary.com/dictionary/stumper :)

Good luck,
-Nic

[jonny]

Tested, if i trash only _RPTx macros, VDub Debug vs Release build produce the same frames.

Another thing, my encoder:
If i compile it with /MT keeping printf it crash!
If i compile it with /MT removing printf it doesn't crash.
If i compile it with /MTd keeping printf it doesn't crash (but probably the output is equal to VDub Debug build + _RPTx).

if I get some time ill look through the vfw code, see if there is anything in there that would be causing it

I'm sure you'll be not able to resist too much without doing this :)

http://www.hyperdictionary.com/dictionary/stumper

bookmarked :D


I haven't the skills to go more inside the problem, i pass the ball to everyone that want to play ^^

[sysKin]

There are some bugs in dev-api-3 tree... both in VfW as in the core.
We're removing them from our dev-api-4, while we completely ignore them in devapi3 ;) simply because we don't really care about devapi3 anymore ;)

I *hope* it is one of them. It's possible, as p/b/i decision used to have a bug I never really understood (I just replaced some code and it works now, so I'm happy :devil: )

Radek

[jonny]

Ok, let me know if you need testing about this problem.
(you can eventually send me new binaries - jonny AT leffe.dnsalias.com)

[jonny]

I'm finally obtaining the same frames as VDub! :) (running some threads)

XviD is generating different frames if printf/_RPTx/AVIStreamRead are in the same thread of the encoding loop.

DivX (at least 5.0.5) is generating different frames if ICSeqCompressFrameStart is in the same thread of the encoding loop.

yuppie!
jonny