xvid_encraw - Patched with AviSynth input support

[JoeBG]

Just finished the third complete video. Size fitted, quality is best. Congratulations to the tool - much more satisfying than mencoder xvid.

[Doom9]

So either I've utterly confused everyone, or I'm gonna go with this I think the people who could give proper advice have yet to read this.. perhaps it would be best to just run with it and see where it takes you.

[squid_80]

It was a busted idea anyway. Since mkv requires a timecode to be stored with every frame, I can't store a reference frame without knowing how many b frames are before it; I can't store the b frames as they come because the reference frame needs to be written first. Catch 22.
So now I buffer everything in memory from one reference frame to the next, which seems to work pretty well.
The other thing I was doing wrong was assuming a packed frame would always be PB. If closed gop is turned off it's possible to get IB instead.

[squid_80]

There's a new build available with very experimental matroska output.

[bond]

ftp://squid80.no-ip.com/xvid_encraw.zip

squid_80: i tried playing a file produced by this with the haali splitter and ffdshow and i saw lots of artefacts, which seemed to me like maybe wrongly ordered files? wrong references? i used two b-frames

at the end it caused my pc to restart :D

so i think there might be something not really working correctly ;)
you might want to release the code so people can look at whats going wrong

[squid_80]

Argh! What the hell happened, it was fine a minute ago. Does look pretty funky.

Now I can't seem to recreate it. Still got the command line handy?

[bond]

xvid_encraw.exe -i input.avs -type 2 -bitrate 700 -single -max_key_interval 250 -max_bframes 2 -qpel -gmc -stats -mkv xvid.mkv

[squid_80]

S-VOPS! Why didn't anyone remind me about them!

Edit: Can you try without -gmc, pretty please?

[Doom9]

at the end it caused my pc to restartWell.. squid sure showed you what you should do with your WinMEgacrap ;)

[bond]

works indeed without gmc :)
why did it bork things? arent s-vops, plain p-vops?

btw next to I/P/B and S-vops, there are also N-vops
(you still have packed bitstream on by default creating borked raw streams! disable it! :p )

edit:
Well.. squid sure showed you what you should do with your WinMEgacrap ;)WinME rocks! ;)
btw i am using win2000 for some months now, so that excuse doesnt count :p

[squid_80]

works indeed without gmc :)
why did it bork things? arent s-vops, plain p-vops?
Kinda, but they have a different vop type. 0 = IVOP, 1 = PVOP, 2 = BVOP, 3 = SVOP. I forgot to check for SVOPs.

Edit: If I'd left one line as else instead of else if, it would have worked :)

btw next to I/P/B and S-vops, there are also N-vops
(you still have packed bitstream on by default creating borked raw streams! disable it! :p )
Hmmm I remembered to drop the fake n-vops, but forgot to keep the real ones. Whoops.
If you want unborked raw streams, just use -nopacked. Packed bitstream isn't that hard to handle, any worthwhile muxer should be able to process it correctly.

[bond]

Hmmm I remembered to drop the fake n-vops, but forgot to keep the real ones. Whoops.
If you want unborked raw streams, just use -nopacked. Packed bitstream isn't that hard to handle, any worthwhile muxer should be able to process it correctly. the thing is if someone wants a correct raw .m4v stream he shouldnt need to know about packed bitstream.

packed bitstream is a hack for avi and nothing else.
there is no tool in the world expecting fake n-vops in a raw stream, cause they simply dont belong there

[squid_80]

Alrighty. I won't disable packed by default, but I'll make sure fake n-vops are left out of raw output.

[bond]

Alrighty. I won't disable packed by default, but I'll make sure fake n-vops are left out of raw output.better make sure that packed bitstream is disabled for every output mode except avi. will be easier to add mp4 output that way ;)

[bond]

btw i have checked the timestamps of frames with the following samples:
- 0 b-frames, no n-vops
- 2 b-frames, no n-vops
- 0 b-frames, n-vops (drop 100), everything except the keyframes are n-vops
- 2 b-frames, n-vops (drop 100), ~ 3 n-vops used in the stream

and it seems they are correct! great job :)

[squid_80]

btw i have checked the timestamps of frames with the following samples:
- 0 b-frames, no n-vops
- 2 b-frames, no n-vops
- 0 b-frames, n-vops (drop 100), everything except the keyframes are n-vops
- 2 b-frames, n-vops (drop 100), ~ 3 n-vops used in the stream

and it seems they are correct! great job :)
Have you tried each option with/without -nopacked?
Hmmm I remembered to drop the fake n-vops, but forgot to keep the real ones. Whoops.
No you didn't, you dummy. N-VOPS have the same identifier as P-VOPS.:o

[bond]

Have you tried each option with/without -nopacked?the .mkvs were all produced with packed enabled
the raw streams have been produced with -nopacked

is it necessary to also seperately test mkv without packed set? shouldnt that work exactly the same way as with packed?

No you didn't, you dummy. N-VOPS have the same identifier as P-VOPS.:oand? are you sure that the n-vops have different identifiers?

btw make sure to use latest libmatroska:
http://www.matroska.org/downloads/windows.html

[bond]

here another reason why packed bitstream should be disabled by default:
as packed bitstream is enabled by default the cli writes always a fake divx userdata before the xvid userdata, also with raw and mkv output. this divx userdata signals that the stream is packed (which it obviously isnt with raw and mkv and a hopefully to come mp4 output). some decoders (like the one from divx and some standalones propably) take this flag into account.
obviously signalling this with a non packed stream is incorrect and will potentially bork things, therefore this flag needs to be disabled too with every outputmode, except avi output

people should simply realise that avi is a complete mess and the earlier people stop using it the better :D


btw i compared a 2 b-frame, no drop, mkv with and without packed bitstream: same timestamps

[squid_80]

the .mkvs were all produced with packed enabled
the raw streams have been produced with -nopacked

is it necessary to also seperately test mkv without packed set? shouldnt that work exactly the same way as with packed?I just wanted to make sure I'm not throwing away any real n-vops. I rely on the stats returned from the encoder for this, and they can be a bit screwy with -nopacked:
C:\xvid\build\win32\bin>xvid_encraw -i g:\vid\new.avs -frames 10 -start 2000 -no
packed -drop 100
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
0: key=2, time= 15, len= 19838
1: key=0, time= 0, len= 0
2: key=0, time= 0, len= 0
3: key=0, time= 31, len= 9295 | type=I, quant= 4, len= 19838
4: key=0, time= 15, len= 624 | type=B, quant= 7, len= 624
5: key=0, time= 16, len= 624 | type=B, quant= 7, len= 624
6: key=0, time= 31, len= 6 | type=P, quant= 4, len= 6
7: key=0, time= 31, len= 9203 | type=P, quant= 4, len= 6
8: key=0, time= 16, len= 565 | type=B, quant= 7, len= 565
9: key=0, time= 31, len= 1271 | type=P, quant= 4, len= 9203
-1: key=0, time= 16, len= 302 | type=B, quant= 7, len= 302
-1: key=0, time= 0, len= 277 | type=B, quant= 7, len= 277
-1: key=0, time= 0, len= -5 | type=P, quant= 4, len= 1271
Tot: enctime(ms) = 202.00, length(bytes) = 42005
Avg: enctime(ms) = 16.83, fps = 59.41, length(bytes) = 3500If you look at the stats, it looks like there were two n-vops (dropped frames). But in reality the first of those P frames had a length of 9295 bytes. I think this is a bug, but it's a quick way to spot real n-vops vs fake n-vops. I'm still not completely convinced of it's reliability though.
and? are you sure that the n-vops have different identifiers?
No they don't, they're the same as P frames. That's why it works even though I forgot about handling them.
btw make sure to use latest libmatroska:
http://www.matroska.org/downloads/windows.htmlI am. I just copied the line that sets the lib version from test6.cpp and forgot to change it.
here another reason why packed bitstream should be disabled by default:
as packed bitstream is enabled by default the cli writes always a fake divx userdata before the xvid userdata, also with raw and mkv output. this divx userdata signals that the stream is packed (which it obviously isnt with raw and mkv and a hopefully to come mp4 output). some decoders (like the one from divx and some standalones propably) take this flag into account.
obviously signalling this with a non packed stream is incorrect and will potentially bork things, therefore this flag needs to be disabled too with every outputmode, except avi outputDo any of these players actually support anything other than avi? Anyway, I'll just drop the p off the divx userdata for mkv and mp4 (if it happens). I'll do the same for ES stream output (-o) and apply the checks for fake n-vops. That way mkv and raw will give the same output whether or not packed is used. But the -save option (one file for each frame) stays as it is - it's too invaluable as a diagnostic tool to mess around with.

(hehe: I just searched xvidcore's code for divx and found a rude word.)

[squid_80]

btw i compared a 2 b-frame, no drop, mkv with and without packed bitstream: same timestamps
It's possible timestamps could screw up on long encodes/exotic framerates. The whole idea of having a timestamp for every frame, while being handy for VFR material, is a bit of a headache (why is there a default duration if timestamps are required anyway?) and I'm not sure an overflow can't occur with the formula I'm using to calculate them.
Is there some hidden documentation for libmatroska that clearly explains when and which timecode scale applies? This is the main reason cues aren't written - I couldn't get the damn timecodes to work for them, even though they work fine for the blocks/clusters.

[bond]

I just wanted to make sure I'm not throwing away any real n-vops. I rely on the stats returned from the encoder for thisin what way do you rely on the stats? just by checking the "len" values?

i am repeating myself over and over again: why not simply disable packed bitstream by default for every mode except avi? that way you know that for raw and mkv there simply are no fake n-vops and therefore there also isnt anything to drop. is it so hard to add a check in the cmdl parsing which only takes packed into account if avi output is used
in the case of avi with packed bitstream there is no problem with the real n-vops as the fake n-vops dont need to be dropped anyways
in x264cli there are lots of checks which only allow options to be set if another option is set too aso...

why are you trying to make things in a difficult and potentially borking way if its so easy to solve this?


If you look at the stats, it looks like there were two n-vops (dropped frames). But in reality the first of those P frames had a length of 9295 bytes. I think this is a bug, but it's a quick way to spot real n-vops vs fake n-vops. I'm still not completely convinced of it's reliability though.whats the meaning of the two length values?

It's possible timestamps could screw up on long encodes/exotic framerates. The whole idea of having a timestamp for every frame, while being handy for VFR material, is a bit of a headache (why is there a default duration if timestamps are required anyway?) and I'm not sure an overflow can't occur with the formula I'm using to calculate them.
Is there some hidden documentation for libmatroska that clearly explains when and which timecode scale applies? This is the main reason cues aren't written - I couldn't get the damn timecodes to work for them, even though they work fine for the blocks/clusters. i dunno where in the mkv specs its written, but how you have set the timecodes till now is correct

you combine the displaytimecode (or cts in mp4 language) with each frame (the frames themselves are stored in decoding order)

about the timescale: as you atm dont seem to set an own timescale you are using the default one (which is 1000 or multiple of that dunno exactly)
of course you can also use an own timescale (which might be better with not nice framerates like 25fps), but i dunno how to do this with libmatroska

[Doom9]

@bond: I already suggested the commandline override.. but according to squid, it's not that simple:
The headers still need to be stripped from every I frame (there's global flags to turn this off but they're not implemented in xvidcore).

[squid_80]

Re-arranging your words slightly to explain things better...
in what way do you rely on the stats? just by checking the "len" values?
whats the meaning of the two length values?
Here's the code that checks for fake n-vops:
if (m4v_size > 10 || (m4v_size == stats_length))
MKVStreamWrite(myMKVStream, output_num, 1, mp4_buffer, m4v_size, key ? AVIIF_KEYFRAME : 0, NULL, NULL);
In xvid_encraw's output text, m4v_size is the len value on the left. stats_length is the len value on the right. m4v_size is the size of the data returned from the encoder. stats_length is supposedly the encoded length of the input frame. But as demonstrated the stats don't necessarily match the output from the encoder - look at the example I posted, even the stats for the I frame are 3 input frames late. But in the case of real n-vops the stats do always seem to match up, which is how I identify them. I said it was a quick way, maybe I should have said quick n dirty. So far I haven't seen it break.
i am repeating myself over and over again: why not simply disable packed bitstream by default for every mode except avi? that way you know that for raw and mkv there simply are no fake n-vops and therefore there also isnt anything to drop. is it so hard to add a check in the cmdl parsing which only takes packed into account if avi output is used
in the case of avi with packed bitstream there is no problem with the real n-vops as the fake n-vops dont need to be dropped anyways
in x264cli there are lots of checks which only allow options to be set if another option is set too aso...
why are you trying to make things in a difficult and potentially borking way if its so easy to solve this?
If an avi is made without packed bitstream by accident, can it be fixed?
I want to have the least amount of restrictions possible when it comes to parameters. -avi should not imply packed or -nopacked. Neither should -mkv or -o. As I showed above the code to check for real vs. fake n-vops is one line and while that holds up I think it's best to leave packed as default. The idea of allowing multiple simultaneous output files was so they could be compared and make sure everything is behaving correctly - if the mkv is borked check the failsafe avi. So it's better to have a working avi in all cases, getting screwed because I forgot to use the -packed option (ever seen a report of xvid crashing with divide by 0 error? Bet it was unpacked and open gop...) is not something I want to see. -nopacked is always available as an option. If you can break the real vs. fake n-vop check (the divx userdata doesn't count as I haven't fixed it yet :p ) then I'll make -nopacked the default choice.
i dunno where in the mkv specs its written, but how you have set the timecodes till now is correct

you combine the displaytimecode (or cts in mp4 language) with each frame (the frames themselves are stored in decoding order)Ahh, but if displaytimecode has been rounded we get cumulative rounding errors... Leading to audio desync. At the moment I use timecode(ms) = 1000*framenum*framescale/framerate.
about the timescale: as you atm dont seem to set an own timescale you are using the default one (which is 1000 or multiple of that dunno exactly)
of course you can also use an own timescale (which might be better with not nice framerates like 25fps), but i dunno how to do this with libmatroskaThe problem is every section seems to have it's own timescale - segment vs tracks vs clusters. Some seem to multiply the given timecode (segment) while others seem to divide (tracks). Then again some timecodes aren't scaled at all.

P.S. Doom9 rubs it in... I'd forgotten about that. No matter what I do, the encoder still spits out avi sympathetic data :sly: And that extra data is dealt with in a very similar way to packed frames.

[bond]

Here's the code that checks for fake n-vops:
if (m4v_size > 10 || (m4v_size == stats_length))
MKVStreamWrite(myMKVStream, output_num, 1, mp4_buffer, m4v_size, key ? AVIIF_KEYFRAME : 0, NULL, NULL);fake n-vops will hopefully also be dropped for raw stream in the future? or not?

In xvid_encraw's output text, m4v_size is the len value on the left. stats_length is the len value on the right.
m4v_size is the size of the data returned from the encoder. stats_length is supposedly the encoded length of the input frame. hm i still dont really get the difference? :confused:

If an avi is made without packed bitstream by accident, can it be fixed? i think there is an important confusion on your side.
avi files do NOT need to be packed when using b-frames!!! let me say it again: there is no need for packed bitstream when using avi with b-frames
(for how b-frames work without packed in avi read this (http://forum.doom9.org/showthread.php?s=&threadid=80430))

xvid produced for years avi files by default which are not packed with b-frames (and as you see in the xvidvfw gui there is also the option to not use packed bitstream). thats said nopacked works great with xvid and always has been that way (i have NOT been using packed bitstream for years and didnt have a problem)
in fact as good as all of the standalones handling xvid handle no packed bitstreams, as that was simply the way xvid did it for years

apart from that the libavcodec asp encoder also doesnt output packed bitstream in any case (eg if used via mencoder with avi output)

I want to have the least amount of restrictions possible when it comes to parameters. -avi should not imply packed or -nopacked. Neither should -mkv or -o. As I showed above the code to check for real vs. fake n-vops is one line and while that holds up I think it's best to leave packed as default. mkv and raw output should not imply -nopacked??? of course they should imply that, as it simply doesnt make sense to use packed with them
i dont get why simply automatically enabling -nopacked with mkv and raw output is such a tragedy?

-nopacked is always available as an option. If you can break the real vs. fake n-vop check (the divx userdata doesn't count as I haven't fixed it yet :p ) then I'll make -nopacked the default choice.so the user first needs to waste a day of encoding to find out he should have enabled -nopacked with mkv and raw output altough packed bitstream makes no sense with mkv and raw anyways and therefore nopacked should be enabled right away with mkv and raw output always???

The idea of allowing multiple simultaneous output files was so they could be compared and make sure everything is behaving correctly - if the mkv is borked check the failsafe avi.you are serious about the multiple simultaneous outputs? you do know that noone will use this ;)
i would propose to simply drop the idea of simultenous different outputs as noone will use this anyways
i also dont really see the sense in using the different output modes together at the same time as its absolutely no problem to remux from one format to the other afterwards (and thats more likely to be done by users than to output the same file multiple times via the encoder while encoding...)
also i have never seen any encoder offering outputting the same stream in different formats _at the same time_...

/rant

anyways to make it short, i have said my opinion about this and i do know it counts nothing as you are the dev ;)
basically all i want is to have a correctly working cli with correct raw and mkv output (and hopefull mp4 output too) without the user needing to know anything about all the avi shit floating around (userdata, vol, packed bitstream, fake n-vops and whatever else)
i dont care how you get the cli to manage to do this as long as it works correctly :) and as long as its not total hackery ;)

Ahh, but if displaytimecode has been rounded we get cumulative rounding errors... Leading to audio desync. At the moment I use timecode(ms) = 1000*framenum*framescale/framerate.
The problem is every section seems to have it's own timescale - segment vs tracks vs clusters. Some seem to multiply the given timecode (segment) while others seem to divide (tracks). Then again some timecodes aren't scaled at all.sorry i dont know much about the internals, you will need to have to ask a mkv guy about this
all i can say is that mkvinfo tells me that there is no explicit timescale set in your files which means that the default timescale is used
regarding the timecodes i checked them on a ~7000 frames mkv clip with 2 b-frames and drop 100 and they have been correct. that was with a 25fps clip

P.S. Doom9 rubs it in... I'd forgotten about that. No matter what I do, the encoder still spits out avi sympathetic data :sly: And that extra data is dealt with in a very similar way to packed frames.yeah, this avi shit is a total mess and noone should have ever come up with it...
thank god people tend to not use avi for avc anymore

[squid_80]

fake n-vops will hopefully also be dropped for raw stream in the future? or not?Yes. For the -o option, but not -save.
hm i still dont really get the difference? :confused:
Watch the output. The numbers on the left of | refer to the encoder output, they are in the encoded order. The numbers on the right refer to input frames, they are in frame order. By chance or bug or dare I say it by design, when there's a real n-vop the lengths match up. When it's a fake, they don't.
i think there is an important confusion on your side.
avi files do NOT need to be packed when using b-frames!!! let me say it again: there is no need for packed bitstream when using avi with b-frames
(for how b-frames work without packed in avi read this (http://forum.doom9.org/showthread.php?s=&threadid=80430))
encraw doesn't output 0x7f for delays; it doesn't output anything
To be properly seekable, packed is needed. If I'm at frame 50 then that's what I want, not frame 49.xvid produced for years avi files by default which are not packed with b-frames (and as you see in the xvidvfw gui there is also the option to not use packed bitstream). thats said nopacked works great with xvid and always has been that way (i have NOT been using packed bitstream for years and didnt have a problem)With avi output?in fact as good as all of the standalones handling xvid handle no packed bitstreams, as that was simply the way xvid did it for yearsSo why oh why is it enabled by default in the vfw interface?
apart from that the libavcodec asp encoder also doesnt output packed bitstream in any case (eg if used via mencoder with avi output)This is xvid_encraw, not mencoder.
mkv and raw output should not imply -nopacked??? of course they should imply that, as it simply doesnt make sense to use packed with them
i dont get why simply automatically enabling -nopacked with mkv and raw output is such a tragedy?

so the user first needs to waste a day of encoding to find out he should have enabled -nopacked with mkv and raw output altough packed bitstream makes no sense with mkv and raw anyways and therefore nopacked should be enabled right away with mkv and raw output always???
When the next build is done, mkv and raw output will be the same whether packed bitstream is enabled or not. Avi output will be packed by default unless -nopacked is specified. Same goes for the files produced by -save, since it's mainly a diagnostic output and can come in very handy when combined with the output log from encraw.
you are serious about the multiple simultaneous outputs? you do know that noone will use this ;)
i would propose to simply drop the idea of simultenous different outputs as noone will use this anyways
i also dont really see the sense in using the different output modes together at the same time as its absolutely no problem to remux from one format to the other afterwards (and thats more likely to be done by users than to output the same file multiple times via the encoder while encoding...)
also i have never seen any encoder offering outputting the same stream in different formats _at the same time_...
You don't see this as a good option? Having to remux into a different container later on is better than being able to output both at the same time?
Further back in this very thread:an encoder should be able to output something useable right away imho, and raw .m4v isnt imho, cause you need a second tool for muxing the stream and there arent lots of tools being able to handle raw input
So if we're outputting raw to mux into mp4 later on, we can check the avi/mkv output straight away to make sure it's ok. Silly idea or worthwhile endeavour?

/rant

anyways to make it short, i have said my opinion about this and i do know it counts nothing as you are the dev ;)
basically all i want is to have a correctly working cli with correct raw and mkv output (and hopefull mp4 output too) without the user needing to know anything about all the avi shit floating around (userdata, vol, packed bitstream, fake n-vops and whatever else)
i dont care how you get the cli to manage to do this as long as it works correctly :)
It will be done. and as long as its not total hackery ;) Like I said, if you manage to break that fake/real n-vop trick, then I'll change things.

[bond]

Yes. For the -o option, but not -save.good :)

To be properly seekable, packed is needed. If I'm at frame 50 then that's what I want, not frame 49.well you can also seek on not packed avi files with b-frames. try it
in fact the divx encoder outputs in his multiple b-frames mode also NOT packed bitstream.
why? because xvid did it that way long before divx was able to output multiple b-frames and therefore standalone players were written to handle it

With avi output?of course

So why oh why is it enabled by default in the vfw interface?because without packed you have the delay and the delay message shown which newbies always complained about. so packed was enabled by default to shut newbies up
there is no problem or bug with not packed output

It will be done. Like I said, if you manage to break that fake/real n-vop trick, then I'll change things. ok so i couldnt convince you that "outputting the stream packed and unpacking it at the same time" doesnt make any sense, well its your choice :p

edit: dont get me wrong, its great you are working on this. its just that i fear things getting borky (anyways as you said i am working hard at trying to bork your algo ;) )

[squid_80]

well you can also seek on not packed avi files with b-frames. try it
in fact the divx encoder outputs in his multiple b-frames mode also NOT packed bitstream.
why? because xvid did it that way long before divx was able to output multiple b-frames and therefore standalone players were written to handle it
I have tried. If I skip using vdub's keyframe keys, I get the previous keyframe instead of the one I want. The frames shown are nearly always the previously requested one instead of the currently requested. Except if I'm on an I frame and I go backward, I get the next P frame. It's wacky, compared to packed which gives me exactly what I ask for.

because without packed you have the delay and the delay message shown which newbies always complained about. so packed was enabled by default to shut newbies up
there is no problem or bug with not packed output
I thought the lag message was disabled to shut the newbies up and packed was enabled for compatability.
ok so i couldnt convince you that "outputting the stream packed and unpacking it at the same time" doesnt make any sense, well its your choice :p
Not so much that it doesn't make sense, just that it's less work to undo. Especially since I-VOPS always have to have the headers stripped off anyway.

[bond]

Not so much that it doesn't make sense, just that it's less work to undo. Especially since I-VOPS always have to have the headers stripped off anyway.ok if its less work that way its ok, i just thought that automatically enabling the -unpacked flag together with -o and -mkv would be as good as no work at all

about the the header stripping: isnt this an independant issue of the packed bitstream thing as its also necessary with -unpacked?

[bond]

as i couldnt convince you to simply enforce -nopacked with mkv and raw output i thought about your method of differentiating real n-vops from fake n-vops:

as i see it you say:
1) if a frame has a size of smaller than 10 on the coding order side (left side) its a n-vop.

i would say this is a not necessarily always correct workaround as at small bitrates and resolutions it would be imagineable to also have frames with a small size
better would be to check if the not coded flag is set for identifying n-vops, cause than you know it for sure


2) if this p-vop has a different size in coding order than in display order you throw it away, as its a fake n-vop
two things coming to my mind:

a) thinking it through this is (might be ;) ) true for packed bitstream streams, but as your own example about the -nopacked stream also shows this is not true for not packed streams
still its not a problem because of course the whole searching for and dropping fake n-vops isnt necessary with not packed streams, as in this case there are only real n-vops
right?

b) your method is wrong when a real frame (be it a real n-vop or a normal p-vop), that has the same size as the fake n-vop, gets packed together with a b-vop
in this case things would look like this:

len= 1309 | type=B, quant= 11, len= 1295
len= 1217 | type=B, quant= 11, len= 1217
len= 6 | type=P, quant= 7, len= 6the P gets packed together with the first B, but your algo doesnt find the fake n-vop cause it has the same size as the real vop



btw some potential bugs i found in the stats of xvid:
- i also think that there is a bug in this value 6 for the p-vop (it should be 9295) you showed above. i have reproduced this on other streams too, it seems to be a common problem
- also a common bug could be len values in the case of packed frames. shouldnt the size of the packed frames be the same as the sum of the not packed ones?
- the following borked coding order sizes created with -nopacked are also often the case but dont make sense:
6133: key=0, time= 300, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.95, psnr u = 39.68, psnr v = 38.35
6134: key=0, time= 291, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6135: key=2, time= 40, len= 13205 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6136: key=0, time= 230, len= 6 | type=P, quant= 2, len= 6, psnr y = 26.52, psnr u = 44.55, psnr v = 44.81
6137: key=0, time= 271, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.22, psnr u = 37.26, psnr v = 36.73
6138: key=0, time= 290, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6139: key=2, time= 40, len= 10332 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6140: key=0, time= 251, len= 6 | type=P, quant= 2, len= 6, psnr y = 27.14, psnr u = 46.42, psnr v = 45.23
6141: key=0, time= 270, len= 6 | type=P, quant= 2, len= 6, psnr y = 24.35, psnr u = 44.41, psnr v = 43.27maybe you can find the reason for these bugs (?) and fix them.

[squid_80]

1) if a frame has a size of smaller than 10 on the coding order side (left side) its a n-vop.

i would say this is a not necessarily always correct workaround as at small bitrates and resolutions it would be imagineable to also have frames with a small size
better would be to check if the not coded flag is set for identifying n-vops, cause than you know it for sure


2) if this p-vop has a different size in coding order than in display order you throw it away, as its a fake n-vop
two things coming to my mind:

a) thinking it through this is (might be ;) ) true for packed bitstream streams, but as your own example about the -nopacked stream also shows this is not true for not packed streams
still its not a problem because of course the whole searching for and dropping fake n-vops isnt necessary with not packed streams, as in this case there are only real n-vops
right?

b) your method is wrong when a real frame (be it a real n-vop or a normal p-vop), that has the same size as the fake n-vop, gets packed together with a b-vop
in this case things would look like this:

len= 1309 | type=B, quant= 11, len= 1295
len= 1217 | type=B, quant= 11, len= 1217
len= 6 | type=P, quant= 7, len= 6the P gets packed together with the first B, but your algo doesnt find the fake n-vop cause it has the same size as the real vopYou're pretty much right. But are these circumstances actually possible? What is the minimum size of a p-vop without the not coded flag?

btw some potential bugs i found in the stats of xvid:
- i also think that there is a bug in this value 6 for the p-vop (it should be 9295) you showed above. i have reproduced this on other streams too, it seems to be a common problem
- also a common bug could be len values in the case of packed frames. shouldnt the size of the packed frames be the same as the sum of the not packed ones?
When packed bitstream is on, the reported size of the P frames has the bytes for the fake-nvop added in. I think that's what you're seeing.
- the following borked coding order sizes created with -nopacked are also often the case but dont make sense:
6133: key=0, time= 300, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.95, psnr u = 39.68, psnr v = 38.35
6134: key=0, time= 291, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6135: key=2, time= 40, len= 13205 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6136: key=0, time= 230, len= 6 | type=P, quant= 2, len= 6, psnr y = 26.52, psnr u = 44.55, psnr v = 44.81
6137: key=0, time= 271, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.22, psnr u = 37.26, psnr v = 36.73
6138: key=0, time= 290, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6139: key=2, time= 40, len= 10332 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6140: key=0, time= 251, len= 6 | type=P, quant= 2, len= 6, psnr y = 27.14, psnr u = 46.42, psnr v = 45.23
6141: key=0, time= 270, len= 6 | type=P, quant= 2, len= 6, psnr y = 24.35, psnr u = 44.41, psnr v = 43.27maybe you can find the reason for these bugs (?) and fix them.
It's the same bug. I think in reality it should be something like this:
6133: key=0, time= 300, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.95, psnr u = 39.68, psnr v = 38.35
6134: key=0, time= 291, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6135: key=2, time= 40, len= 13205 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6136: key=0, time= 230, len= 6 | type=I, quant= 2, len= 13205, psnr y = 26.52, psnr u = 44.55, psnr v = 44.81
6137: key=0, time= 271, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.22, psnr u = 37.26, psnr v = 36.73
6138: key=0, time= 290, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6139: key=2, time= 40, len= 10332 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6140: key=0, time= 251, len= 6 | type=I, quant= 2, len= 10332, psnr y = 27.14, psnr u = 46.42, psnr v = 45.23
6141: key=0, time= 270, len= 6 | type=P, quant= 2, len= 6, psnr y = 24.35, psnr u = 44.41, psnr v = 43.27Whether it's a bug or by design I don't know. But if the bad lengths are being used for first pass stats (I have a sneaking suspicion they are) the filesize is going to be way off. Can you do that same stream again with -pass1, then examine the stats file for any weirdness?

[bond]

You're pretty much right. But are these circumstances actually possible? What is the minimum size of a p-vop without the not coded flag?i have been able to create smaller than 10 real p-vops with a very small res and very small bitrate. i doubt anyone will ever do something like that, its just that it should be known as an potential issue

about the not coded flag: will you identify n-vops by the flag or by the size?
doing it by the flag will make sure no small p-vop will get dropped and will always be correct

and what are you answer to my question whether you are doing the fake n-vop check also with not packed streams?

When packed bitstream is on, the reported size of the P frames has the bytes for the fake-nvop added in. I think that's what you're seeing.well yeah, still i dont think the size of the packed frame(s) (shown on the left) is correct.
imho it would be correct if it would be the sum of the B and P sizes (shown on right), but it isnt

It's the same bug. I think in reality it should be something like this:what you are saying is more than just a wrong size.
what you are saying is that the frametype is shown wrongly (I vs. P) which would be a more serious bug than just an off size imho

Whether it's a bug or by design I don't know. But if the bad lengths are being used for first pass stats (I have a sneaking suspicion they are) the filesize is going to be way off. Can you do that same stream again with -pass1, then examine the stats file for any weirdness? actually now that you are saying i have got indeed wrong filesizes with no packed output together with n-vops enabled (with one pass encoding)

[squid_80]

i have been able to create smaller than 10 real p-vops with a very small res and very small bitrate. i doubt anyone will ever do something like that, its just that it should be known as an potential issueSmaller than 7 bytes? That's the largest n-vop I've seen. 10 was chosen to allow a few extra bytes, just in case.
about the not coded flag: will you identify n-vops by the flag or by the size?
doing it by the flag will make sure no small p-vop will get dropped and will always be correctSpeed is the issue. The check has to be done for every frame outputted by the encoder, so the simpler it is, the better.
and what are you answer to my question whether you are doing the fake n-vop check also with not packed streams?At the moment yes, because it means one less if statement in the code.
well yeah, still i dont think the size of the packed frame(s) (shown on the left) is correct.
imho it would be correct if it would be the sum of the B and P sizes (shown on right), but it isntTake the size of the P frame from the right. Subtract the size of the encoded n-vop. Add the size of the B frame from the right. The result should match the size of the encoded packed frame. If it doesn't, give an example.
what you are saying is more than just a wrong size.
what you are saying is that the frametype is shown wrongly (I vs. P) which would be a more serious bug than just an off size imhoWell the encoder's saying the encoded frame is a key frame, which means it's an I frame. And the PSNR calculations would seem to back that up. I think that with -nopacked, the stats (right side) are one frame behind i.e. they are actually for the encoded frame from the line above. Except if there's a n-vop, then the stats will match the encoded frame from the same line. This has the side effect of making some stats never appear in the output at all.
actually now that you are saying i have got indeed wrong filesizes with no packed output together with n-vops enabled (with one pass encoding)
Like I said, do a first pass and check the stats file. There's definitely weirdness in there. They even crash Koepi's statsreader.

Edit: hlength in the statsfile - my understanding is that it's the length of the headers in each frame e.g. it won't change no matter what quantizer is used. So a negative value is probably really bad, right?

[squid_80]

Ok, I put on my bug hunting hat and found a few things. Firstly there's this in encoder.c (function FrameCodeP)
/* frame drop code */
if (current->sStat.kblks + current->sStat.mblks <=
(pParam->frame_drop_ratio * mb_width * mb_height) / 100 &&
( (pEnc->bframenum_head >= pEnc->bframenum_tail) || !(pEnc->mbParam.global_flags & XVID_GLOBAL_CLOSED_GOP)) )
{
current->sStat.kblks = current->sStat.mblks = 0;
// Squid's fix
// current->sStat.iTextBits = 0;
current->sStat.ublks = mb_width * mb_height;
I think current->sStat.iTextBits should be set to 0 since an n-vop contains no texture information. I tested and it fixed the bitrate issues for packed bitstreams with frame drops.
For unpacked bitstream there's another problem. These 2 sections of code (again from encoder.c):
/* for unpacked bframes, output the stats for the last encoded frame */
if (!(pEnc->mbParam.global_flags & XVID_GLOBAL_PACKED) && pEnc->mbParam.max_bframes > 0)
{
if (pEnc->current->stamp > 0) {
call_plugins(pEnc, pEnc->reference, &pEnc->sOriginal, XVID_PLG_AFTER, NULL, NULL, stats);
}
else
stats->type = XVID_TYPE_NOTHING;
}
if ( FrameCodeP(pEnc, &bs) == 0 ) {
/* N-VOP, we mustn't code b-frames yet */
// Squid's fix
// if ((pEnc->mbParam.global_flags & XVID_GLOBAL_PACKED) || pEnc->mbParam.max_bframes == 0)
call_plugins(pEnc, pEnc->current, &pEnc->sOriginal, XVID_PLG_AFTER, NULL, NULL, stats);
goto done;
}
The upshot of this is when packed bitstream isn't used and a n-vop is encoded, the plugins are called twice. This explains how I was seeing double entries in the stats file for n-vops. The fix I've come up with seems to solve the problem, but breaks encraw like I thought it would. :(

[squid_80]

I'm confused; need help from a matroska expert.
Haali says ( http://haali.cs.msu.ru/mkv/codecs.pdf ) that the mpeg4 headers up to the first VOP go in the CodecPrivate element and the stream itself should only contain VOPs and no headers, but this borks things if the VOL header changes which seems to happen sometimes (use a constant quant zone at some point after the start). Maybe it's just a first pass quirk, I haven't got that far yet. Keeping the VOL header with the I frames in the mkv seems to work, but is apparently prohibited. Any idea which is correct?

[bond]

mpeg-4 defines that the vol can never change throughout the stream
if a mpeg-4 encoder outputs varrying vols its incompliant to the mpeg-4 standard

[squid_80]

That's what I thought. I'll ignore it and move on to 2nd pass...

[bond]

so will you output varrying vols or will you do it as haali writes?

[squid_80]

Like Haali says. Unless people want busted streams, then I might make an option for it.

[bond]

actually if xvid outputs varrying vols its a serious bug which needs to be fixed :>

[squid_80]

I've only seen it doing first pass with full1pass disabled, which the vfw says: "If you don't discard first pass but keep full quality disabled, the resulting 1st pass stream might not be mpeg-4 compliant."

[bond]

yeah, the results from the fastfirstpass are not useable for anything

[JoeBG]

@ squid_80

Do you plan a new release? Last release works perfect for me with xvidcore 1.2 beta single threat version -> So it´s just a question. :)

[squid_80]

Of course, I'm working on nothing else at the moment (except my real-life job to pay the bills).

@bond: Have you heard anything back about your post to the xvid-devel mailing list? Because I think I pretty much solved the bitrate problems but unfortunately the bug is in xvidcore, I can implement a fix in encraw but it would mean duplicating a lot of code.

[bond]

@bond: Have you heard anything back about your post to the xvid-devel mailing list? Because I think I pretty much solved the bitrate problems but unfortunately the bug is in xvidcore, I can implement a fix in encraw but it would mean duplicating a lot of code.nope i didnt. if you know how to fix it it would be best to do in the core itself. i am sure the xvid devs will apply a working patch from your side to the cvs

[Doom9]

@squid_80: are you making any progress on the remaining commandline options?

[squid_80]

Working on it as we speak, should have something in about 1.5 hours.

[squid_80]

Damnit, not going to get zones finished off properly tonight but there's a new build with rate control options and custom PAR settings. Display AR should now be set for mkv, I/P/B min quants are 2 by default, the trailing p is dropped from the divx userdata for mkv and m4v ES output(not sure if this is done correctly), cartoon mode is no longer accidentally activated, GMC works with mkv, fake vs. real n-vop detection no longer relies on a bug in xvidcore.

BTW I strongly recommend not using frame drops and b-frames together; the encoder is smart enough to not drop the P frame immediately after a B frame series, but not smart enough to know not to use B frames after encoding an n-vop.
159: key=0, time= 0, len= 6 | type=P, quant= 7, len= 12462
160: key=0, time= 125, len= 20950 | type=B, quant= 11, len= 5124
161: key=0, time= 0, len= 6 | type=P, quant= 7, len= 15832
162: key=0, time= 94, len= 6 | type=P, quant= 7, len= 6
163: key=0, time= 125, len= 12282 | type=B, quant= 13, len= 1443
164: key=0, time= 0, len= 6 | type=P, quant= 9, len= 10845
165: key=0, time= 94, len= 5873 | type=B, quant= 15, len= 1705
The highlighted P frame is an n-vop; the following B frame shows artifacts since it references a frame that isn't coded.
I've also seen artifacts when using -noclosed_gop, but haven't had time to look into it.

[squid_80]

Re:zones, I was going to be clever and allocate them dynamically (instead of having a limit of 64 max) but realized this was silly since it would probably be impossible to specify that many via the command line anyway. But now I'm thinking of adding an option to pass a filename (e.g. -zonefile xxx) which would be a text file containing the zones in a specified format. Good idea or bad idea?

[buzzqw]

But now I'm thinking of adding an option to pass a filename (e.g. -zonefile xxx)

Good Idea ! but don't drop support for old command line zones

BHH

[bond]

how are you detecting fake n-vops now?
btw its not possible to deactivate real n-vops placement in xvid

about zones: why not simply do it like other cmdl tools (mp4box, mencoder...):
you specify a zone via -zone and than add specific commands you want right after it with a ":"

like -zone startframe=3000:weight=0.5:cartoonmode:greyscale

that would be straight forward and easy to set and understand