xvid_encraw - Patched with AviSynth input support

[JoeBG]

Just finished the third complete video. Size fitted, quality is best. Congratulations to the tool - much more satisfying than mencoder xvid.

[Doom9]

So either I've utterly confused everyone, or I'm gonna go with this I think the people who could give proper advice have yet to read this.. perhaps it would be best to just run with it and see where it takes you.

[squid_80]

It was a busted idea anyway. Since mkv requires a timecode to be stored with every frame, I can't store a reference frame without knowing how many b frames are before it; I can't store the b frames as they come because the reference frame needs to be written first. Catch 22.
So now I buffer everything in memory from one reference frame to the next, which seems to work pretty well.
The other thing I was doing wrong was assuming a packed frame would always be PB. If closed gop is turned off it's possible to get IB instead.

[squid_80]

There's a new build available with very experimental matroska output.

[bond]

ftp://squid80.no-ip.com/xvid_encraw.zip

squid_80: i tried playing a file produced by this with the haali splitter and ffdshow and i saw lots of artefacts, which seemed to me like maybe wrongly ordered files? wrong references? i used two b-frames

at the end it caused my pc to restart :D

so i think there might be something not really working correctly ;)
you might want to release the code so people can look at whats going wrong

[squid_80]

Argh! What the hell happened, it was fine a minute ago. Does look pretty funky.

Now I can't seem to recreate it. Still got the command line handy?

[bond]

xvid_encraw.exe -i input.avs -type 2 -bitrate 700 -single -max_key_interval 250 -max_bframes 2 -qpel -gmc -stats -mkv xvid.mkv

[squid_80]

S-VOPS! Why didn't anyone remind me about them!

Edit: Can you try without -gmc, pretty please?

[Doom9]

at the end it caused my pc to restartWell.. squid sure showed you what you should do with your WinMEgacrap ;)

[bond]

works indeed without gmc :)
why did it bork things? arent s-vops, plain p-vops?

btw next to I/P/B and S-vops, there are also N-vops
(you still have packed bitstream on by default creating borked raw streams! disable it! :p )

edit:
Well.. squid sure showed you what you should do with your WinMEgacrap ;)WinME rocks! ;)
btw i am using win2000 for some months now, so that excuse doesnt count :p

[squid_80]

works indeed without gmc :)
why did it bork things? arent s-vops, plain p-vops?
Kinda, but they have a different vop type. 0 = IVOP, 1 = PVOP, 2 = BVOP, 3 = SVOP. I forgot to check for SVOPs.

Edit: If I'd left one line as else instead of else if, it would have worked :)

btw next to I/P/B and S-vops, there are also N-vops
(you still have packed bitstream on by default creating borked raw streams! disable it! :p )
Hmmm I remembered to drop the fake n-vops, but forgot to keep the real ones. Whoops.
If you want unborked raw streams, just use -nopacked. Packed bitstream isn't that hard to handle, any worthwhile muxer should be able to process it correctly.

[bond]

Hmmm I remembered to drop the fake n-vops, but forgot to keep the real ones. Whoops.
If you want unborked raw streams, just use -nopacked. Packed bitstream isn't that hard to handle, any worthwhile muxer should be able to process it correctly. the thing is if someone wants a correct raw .m4v stream he shouldnt need to know about packed bitstream.

packed bitstream is a hack for avi and nothing else.
there is no tool in the world expecting fake n-vops in a raw stream, cause they simply dont belong there

[squid_80]

Alrighty. I won't disable packed by default, but I'll make sure fake n-vops are left out of raw output.

[bond]

Alrighty. I won't disable packed by default, but I'll make sure fake n-vops are left out of raw output.better make sure that packed bitstream is disabled for every output mode except avi. will be easier to add mp4 output that way ;)

[bond]

btw i have checked the timestamps of frames with the following samples:
- 0 b-frames, no n-vops
- 2 b-frames, no n-vops
- 0 b-frames, n-vops (drop 100), everything except the keyframes are n-vops
- 2 b-frames, n-vops (drop 100), ~ 3 n-vops used in the stream

and it seems they are correct! great job :)

[squid_80]

btw i have checked the timestamps of frames with the following samples:
- 0 b-frames, no n-vops
- 2 b-frames, no n-vops
- 0 b-frames, n-vops (drop 100), everything except the keyframes are n-vops
- 2 b-frames, n-vops (drop 100), ~ 3 n-vops used in the stream

and it seems they are correct! great job :)
Have you tried each option with/without -nopacked?
Hmmm I remembered to drop the fake n-vops, but forgot to keep the real ones. Whoops.
No you didn't, you dummy. N-VOPS have the same identifier as P-VOPS.:o

[bond]

Have you tried each option with/without -nopacked?the .mkvs were all produced with packed enabled
the raw streams have been produced with -nopacked

is it necessary to also seperately test mkv without packed set? shouldnt that work exactly the same way as with packed?

No you didn't, you dummy. N-VOPS have the same identifier as P-VOPS.:oand? are you sure that the n-vops have different identifiers?

btw make sure to use latest libmatroska:
http://www.matroska.org/downloads/windows.html

[bond]

here another reason why packed bitstream should be disabled by default:
as packed bitstream is enabled by default the cli writes always a fake divx userdata before the xvid userdata, also with raw and mkv output. this divx userdata signals that the stream is packed (which it obviously isnt with raw and mkv and a hopefully to come mp4 output). some decoders (like the one from divx and some standalones propably) take this flag into account.
obviously signalling this with a non packed stream is incorrect and will potentially bork things, therefore this flag needs to be disabled too with every outputmode, except avi output

people should simply realise that avi is a complete mess and the earlier people stop using it the better :D


btw i compared a 2 b-frame, no drop, mkv with and without packed bitstream: same timestamps

[squid_80]

the .mkvs were all produced with packed enabled
the raw streams have been produced with -nopacked

is it necessary to also seperately test mkv without packed set? shouldnt that work exactly the same way as with packed?I just wanted to make sure I'm not throwing away any real n-vops. I rely on the stats returned from the encoder for this, and they can be a bit screwy with -nopacked:
C:\xvid\build\win32\bin>xvid_encraw -i g:\vid\new.avs -frames 10 -start 2000 -no
packed -drop 100
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
0: key=2, time= 15, len= 19838
1: key=0, time= 0, len= 0
2: key=0, time= 0, len= 0
3: key=0, time= 31, len= 9295 | type=I, quant= 4, len= 19838
4: key=0, time= 15, len= 624 | type=B, quant= 7, len= 624
5: key=0, time= 16, len= 624 | type=B, quant= 7, len= 624
6: key=0, time= 31, len= 6 | type=P, quant= 4, len= 6
7: key=0, time= 31, len= 9203 | type=P, quant= 4, len= 6
8: key=0, time= 16, len= 565 | type=B, quant= 7, len= 565
9: key=0, time= 31, len= 1271 | type=P, quant= 4, len= 9203
-1: key=0, time= 16, len= 302 | type=B, quant= 7, len= 302
-1: key=0, time= 0, len= 277 | type=B, quant= 7, len= 277
-1: key=0, time= 0, len= -5 | type=P, quant= 4, len= 1271
Tot: enctime(ms) = 202.00, length(bytes) = 42005
Avg: enctime(ms) = 16.83, fps = 59.41, length(bytes) = 3500If you look at the stats, it looks like there were two n-vops (dropped frames). But in reality the first of those P frames had a length of 9295 bytes. I think this is a bug, but it's a quick way to spot real n-vops vs fake n-vops. I'm still not completely convinced of it's reliability though.
and? are you sure that the n-vops have different identifiers?
No they don't, they're the same as P frames. That's why it works even though I forgot about handling them.
btw make sure to use latest libmatroska:
http://www.matroska.org/downloads/windows.htmlI am. I just copied the line that sets the lib version from test6.cpp and forgot to change it.
here another reason why packed bitstream should be disabled by default:
as packed bitstream is enabled by default the cli writes always a fake divx userdata before the xvid userdata, also with raw and mkv output. this divx userdata signals that the stream is packed (which it obviously isnt with raw and mkv and a hopefully to come mp4 output). some decoders (like the one from divx and some standalones propably) take this flag into account.
obviously signalling this with a non packed stream is incorrect and will potentially bork things, therefore this flag needs to be disabled too with every outputmode, except avi outputDo any of these players actually support anything other than avi? Anyway, I'll just drop the p off the divx userdata for mkv and mp4 (if it happens). I'll do the same for ES stream output (-o) and apply the checks for fake n-vops. That way mkv and raw will give the same output whether or not packed is used. But the -save option (one file for each frame) stays as it is - it's too invaluable as a diagnostic tool to mess around with.

(hehe: I just searched xvidcore's code for divx and found a rude word.)

[squid_80]

btw i compared a 2 b-frame, no drop, mkv with and without packed bitstream: same timestamps
It's possible timestamps could screw up on long encodes/exotic framerates. The whole idea of having a timestamp for every frame, while being handy for VFR material, is a bit of a headache (why is there a default duration if timestamps are required anyway?) and I'm not sure an overflow can't occur with the formula I'm using to calculate them.
Is there some hidden documentation for libmatroska that clearly explains when and which timecode scale applies? This is the main reason cues aren't written - I couldn't get the damn timecodes to work for them, even though they work fine for the blocks/clusters.

[bond]

I just wanted to make sure I'm not throwing away any real n-vops. I rely on the stats returned from the encoder for thisin what way do you rely on the stats? just by checking the "len" values?

i am repeating myself over and over again: why not simply disable packed bitstream by default for every mode except avi? that way you know that for raw and mkv there simply are no fake n-vops and therefore there also isnt anything to drop. is it so hard to add a check in the cmdl parsing which only takes packed into account if avi output is used
in the case of avi with packed bitstream there is no problem with the real n-vops as the fake n-vops dont need to be dropped anyways
in x264cli there are lots of checks which only allow options to be set if another option is set too aso...

why are you trying to make things in a difficult and potentially borking way if its so easy to solve this?


If you look at the stats, it looks like there were two n-vops (dropped frames). But in reality the first of those P frames had a length of 9295 bytes. I think this is a bug, but it's a quick way to spot real n-vops vs fake n-vops. I'm still not completely convinced of it's reliability though.whats the meaning of the two length values?

It's possible timestamps could screw up on long encodes/exotic framerates. The whole idea of having a timestamp for every frame, while being handy for VFR material, is a bit of a headache (why is there a default duration if timestamps are required anyway?) and I'm not sure an overflow can't occur with the formula I'm using to calculate them.
Is there some hidden documentation for libmatroska that clearly explains when and which timecode scale applies? This is the main reason cues aren't written - I couldn't get the damn timecodes to work for them, even though they work fine for the blocks/clusters. i dunno where in the mkv specs its written, but how you have set the timecodes till now is correct

you combine the displaytimecode (or cts in mp4 language) with each frame (the frames themselves are stored in decoding order)

about the timescale: as you atm dont seem to set an own timescale you are using the default one (which is 1000 or multiple of that dunno exactly)
of course you can also use an own timescale (which might be better with not nice framerates like 25fps), but i dunno how to do this with libmatroska

[Doom9]

@bond: I already suggested the commandline override.. but according to squid, it's not that simple:
The headers still need to be stripped from every I frame (there's global flags to turn this off but they're not implemented in xvidcore).

[squid_80]

Re-arranging your words slightly to explain things better...
in what way do you rely on the stats? just by checking the "len" values?
whats the meaning of the two length values?
Here's the code that checks for fake n-vops:
if (m4v_size > 10 || (m4v_size == stats_length))
MKVStreamWrite(myMKVStream, output_num, 1, mp4_buffer, m4v_size, key ? AVIIF_KEYFRAME : 0, NULL, NULL);
In xvid_encraw's output text, m4v_size is the len value on the left. stats_length is the len value on the right. m4v_size is the size of the data returned from the encoder. stats_length is supposedly the encoded length of the input frame. But as demonstrated the stats don't necessarily match the output from the encoder - look at the example I posted, even the stats for the I frame are 3 input frames late. But in the case of real n-vops the stats do always seem to match up, which is how I identify them. I said it was a quick way, maybe I should have said quick n dirty. So far I haven't seen it break.
i am repeating myself over and over again: why not simply disable packed bitstream by default for every mode except avi? that way you know that for raw and mkv there simply are no fake n-vops and therefore there also isnt anything to drop. is it so hard to add a check in the cmdl parsing which only takes packed into account if avi output is used
in the case of avi with packed bitstream there is no problem with the real n-vops as the fake n-vops dont need to be dropped anyways
in x264cli there are lots of checks which only allow options to be set if another option is set too aso...
why are you trying to make things in a difficult and potentially borking way if its so easy to solve this?
If an avi is made without packed bitstream by accident, can it be fixed?
I want to have the least amount of restrictions possible when it comes to parameters. -avi should not imply packed or -nopacked. Neither should -mkv or -o. As I showed above the code to check for real vs. fake n-vops is one line and while that holds up I think it's best to leave packed as default. The idea of allowing multiple simultaneous output files was so they could be compared and make sure everything is behaving correctly - if the mkv is borked check the failsafe avi. So it's better to have a working avi in all cases, getting screwed because I forgot to use the -packed option (ever seen a report of xvid crashing with divide by 0 error? Bet it was unpacked and open gop...) is not something I want to see. -nopacked is always available as an option. If you can break the real vs. fake n-vop check (the divx userdata doesn't count as I haven't fixed it yet :p ) then I'll make -nopacked the default choice.
i dunno where in the mkv specs its written, but how you have set the timecodes till now is correct

you combine the displaytimecode (or cts in mp4 language) with each frame (the frames themselves are stored in decoding order)Ahh, but if displaytimecode has been rounded we get cumulative rounding errors... Leading to audio desync. At the moment I use timecode(ms) = 1000*framenum*framescale/framerate.
about the timescale: as you atm dont seem to set an own timescale you are using the default one (which is 1000 or multiple of that dunno exactly)
of course you can also use an own timescale (which might be better with not nice framerates like 25fps), but i dunno how to do this with libmatroskaThe problem is every section seems to have it's own timescale - segment vs tracks vs clusters. Some seem to multiply the given timecode (segment) while others seem to divide (tracks). Then again some timecodes aren't scaled at all.

P.S. Doom9 rubs it in... I'd forgotten about that. No matter what I do, the encoder still spits out avi sympathetic data :sly: And that extra data is dealt with in a very similar way to packed frames.

[bond]

Here's the code that checks for fake n-vops:
if (m4v_size > 10 || (m4v_size == stats_length))
MKVStreamWrite(myMKVStream, output_num, 1, mp4_buffer, m4v_size, key ? AVIIF_KEYFRAME : 0, NULL, NULL);fake n-vops will hopefully also be dropped for raw stream in the future? or not?

In xvid_encraw's output text, m4v_size is the len value on the left. stats_length is the len value on the right.
m4v_size is the size of the data returned from the encoder. stats_length is supposedly the encoded length of the input frame. hm i still dont really get the difference? :confused:

If an avi is made without packed bitstream by accident, can it be fixed? i think there is an important confusion on your side.
avi files do NOT need to be packed when using b-frames!!! let me say it again: there is no need for packed bitstream when using avi with b-frames
(for how b-frames work without packed in avi read this (http://forum.doom9.org/showthread.php?s=&threadid=80430))

xvid produced for years avi files by default which are not packed with b-frames (and as you see in the xvidvfw gui there is also the option to not use packed bitstream). thats said nopacked works great with xvid and always has been that way (i have NOT been using packed bitstream for years and didnt have a problem)
in fact as good as all of the standalones handling xvid handle no packed bitstreams, as that was simply the way xvid did it for years

apart from that the libavcodec asp encoder also doesnt output packed bitstream in any case (eg if used via mencoder with avi output)

I want to have the least amount of restrictions possible when it comes to parameters. -avi should not imply packed or -nopacked. Neither should -mkv or -o. As I showed above the code to check for real vs. fake n-vops is one line and while that holds up I think it's best to leave packed as default. mkv and raw output should not imply -nopacked??? of course they should imply that, as it simply doesnt make sense to use packed with them
i dont get why simply automatically enabling -nopacked with mkv and raw output is such a tragedy?

-nopacked is always available as an option. If you can break the real vs. fake n-vop check (the divx userdata doesn't count as I haven't fixed it yet :p ) then I'll make -nopacked the default choice.so the user first needs to waste a day of encoding to find out he should have enabled -nopacked with mkv and raw output altough packed bitstream makes no sense with mkv and raw anyways and therefore nopacked should be enabled right away with mkv and raw output always???

The idea of allowing multiple simultaneous output files was so they could be compared and make sure everything is behaving correctly - if the mkv is borked check the failsafe avi.you are serious about the multiple simultaneous outputs? you do know that noone will use this ;)
i would propose to simply drop the idea of simultenous different outputs as noone will use this anyways
i also dont really see the sense in using the different output modes together at the same time as its absolutely no problem to remux from one format to the other afterwards (and thats more likely to be done by users than to output the same file multiple times via the encoder while encoding...)
also i have never seen any encoder offering outputting the same stream in different formats _at the same time_...

/rant

anyways to make it short, i have said my opinion about this and i do know it counts nothing as you are the dev ;)
basically all i want is to have a correctly working cli with correct raw and mkv output (and hopefull mp4 output too) without the user needing to know anything about all the avi shit floating around (userdata, vol, packed bitstream, fake n-vops and whatever else)
i dont care how you get the cli to manage to do this as long as it works correctly :) and as long as its not total hackery ;)

Ahh, but if displaytimecode has been rounded we get cumulative rounding errors... Leading to audio desync. At the moment I use timecode(ms) = 1000*framenum*framescale/framerate.
The problem is every section seems to have it's own timescale - segment vs tracks vs clusters. Some seem to multiply the given timecode (segment) while others seem to divide (tracks). Then again some timecodes aren't scaled at all.sorry i dont know much about the internals, you will need to have to ask a mkv guy about this
all i can say is that mkvinfo tells me that there is no explicit timescale set in your files which means that the default timescale is used
regarding the timecodes i checked them on a ~7000 frames mkv clip with 2 b-frames and drop 100 and they have been correct. that was with a 25fps clip

P.S. Doom9 rubs it in... I'd forgotten about that. No matter what I do, the encoder still spits out avi sympathetic data :sly: And that extra data is dealt with in a very similar way to packed frames.yeah, this avi shit is a total mess and noone should have ever come up with it...
thank god people tend to not use avi for avc anymore

[squid_80]

fake n-vops will hopefully also be dropped for raw stream in the future? or not?Yes. For the -o option, but not -save.
hm i still dont really get the difference? :confused:
Watch the output. The numbers on the left of | refer to the encoder output, they are in the encoded order. The numbers on the right refer to input frames, they are in frame order. By chance or bug or dare I say it by design, when there's a real n-vop the lengths match up. When it's a fake, they don't.
i think there is an important confusion on your side.
avi files do NOT need to be packed when using b-frames!!! let me say it again: there is no need for packed bitstream when using avi with b-frames
(for how b-frames work without packed in avi read this (http://forum.doom9.org/showthread.php?s=&threadid=80430))
encraw doesn't output 0x7f for delays; it doesn't output anything
To be properly seekable, packed is needed. If I'm at frame 50 then that's what I want, not frame 49.xvid produced for years avi files by default which are not packed with b-frames (and as you see in the xvidvfw gui there is also the option to not use packed bitstream). thats said nopacked works great with xvid and always has been that way (i have NOT been using packed bitstream for years and didnt have a problem)With avi output?in fact as good as all of the standalones handling xvid handle no packed bitstreams, as that was simply the way xvid did it for yearsSo why oh why is it enabled by default in the vfw interface?
apart from that the libavcodec asp encoder also doesnt output packed bitstream in any case (eg if used via mencoder with avi output)This is xvid_encraw, not mencoder.
mkv and raw output should not imply -nopacked??? of course they should imply that, as it simply doesnt make sense to use packed with them
i dont get why simply automatically enabling -nopacked with mkv and raw output is such a tragedy?

so the user first needs to waste a day of encoding to find out he should have enabled -nopacked with mkv and raw output altough packed bitstream makes no sense with mkv and raw anyways and therefore nopacked should be enabled right away with mkv and raw output always???
When the next build is done, mkv and raw output will be the same whether packed bitstream is enabled or not. Avi output will be packed by default unless -nopacked is specified. Same goes for the files produced by -save, since it's mainly a diagnostic output and can come in very handy when combined with the output log from encraw.
you are serious about the multiple simultaneous outputs? you do know that noone will use this ;)
i would propose to simply drop the idea of simultenous different outputs as noone will use this anyways
i also dont really see the sense in using the different output modes together at the same time as its absolutely no problem to remux from one format to the other afterwards (and thats more likely to be done by users than to output the same file multiple times via the encoder while encoding...)
also i have never seen any encoder offering outputting the same stream in different formats _at the same time_...
You don't see this as a good option? Having to remux into a different container later on is better than being able to output both at the same time?
Further back in this very thread:an encoder should be able to output something useable right away imho, and raw .m4v isnt imho, cause you need a second tool for muxing the stream and there arent lots of tools being able to handle raw input
So if we're outputting raw to mux into mp4 later on, we can check the avi/mkv output straight away to make sure it's ok. Silly idea or worthwhile endeavour?

/rant

anyways to make it short, i have said my opinion about this and i do know it counts nothing as you are the dev ;)
basically all i want is to have a correctly working cli with correct raw and mkv output (and hopefull mp4 output too) without the user needing to know anything about all the avi shit floating around (userdata, vol, packed bitstream, fake n-vops and whatever else)
i dont care how you get the cli to manage to do this as long as it works correctly :)
It will be done. and as long as its not total hackery ;) Like I said, if you manage to break that fake/real n-vop trick, then I'll change things.

[bond]

Yes. For the -o option, but not -save.good :)

To be properly seekable, packed is needed. If I'm at frame 50 then that's what I want, not frame 49.well you can also seek on not packed avi files with b-frames. try it
in fact the divx encoder outputs in his multiple b-frames mode also NOT packed bitstream.
why? because xvid did it that way long before divx was able to output multiple b-frames and therefore standalone players were written to handle it

With avi output?of course

So why oh why is it enabled by default in the vfw interface?because without packed you have the delay and the delay message shown which newbies always complained about. so packed was enabled by default to shut newbies up
there is no problem or bug with not packed output

It will be done. Like I said, if you manage to break that fake/real n-vop trick, then I'll change things. ok so i couldnt convince you that "outputting the stream packed and unpacking it at the same time" doesnt make any sense, well its your choice :p

edit: dont get me wrong, its great you are working on this. its just that i fear things getting borky (anyways as you said i am working hard at trying to bork your algo ;) )

[squid_80]

well you can also seek on not packed avi files with b-frames. try it
in fact the divx encoder outputs in his multiple b-frames mode also NOT packed bitstream.
why? because xvid did it that way long before divx was able to output multiple b-frames and therefore standalone players were written to handle it
I have tried. If I skip using vdub's keyframe keys, I get the previous keyframe instead of the one I want. The frames shown are nearly always the previously requested one instead of the currently requested. Except if I'm on an I frame and I go backward, I get the next P frame. It's wacky, compared to packed which gives me exactly what I ask for.

because without packed you have the delay and the delay message shown which newbies always complained about. so packed was enabled by default to shut newbies up
there is no problem or bug with not packed output
I thought the lag message was disabled to shut the newbies up and packed was enabled for compatability.
ok so i couldnt convince you that "outputting the stream packed and unpacking it at the same time" doesnt make any sense, well its your choice :p
Not so much that it doesn't make sense, just that it's less work to undo. Especially since I-VOPS always have to have the headers stripped off anyway.

[bond]

Not so much that it doesn't make sense, just that it's less work to undo. Especially since I-VOPS always have to have the headers stripped off anyway.ok if its less work that way its ok, i just thought that automatically enabling the -unpacked flag together with -o and -mkv would be as good as no work at all

about the the header stripping: isnt this an independant issue of the packed bitstream thing as its also necessary with -unpacked?

[bond]

as i couldnt convince you to simply enforce -nopacked with mkv and raw output i thought about your method of differentiating real n-vops from fake n-vops:

as i see it you say:
1) if a frame has a size of smaller than 10 on the coding order side (left side) its a n-vop.

i would say this is a not necessarily always correct workaround as at small bitrates and resolutions it would be imagineable to also have frames with a small size
better would be to check if the not coded flag is set for identifying n-vops, cause than you know it for sure


2) if this p-vop has a different size in coding order than in display order you throw it away, as its a fake n-vop
two things coming to my mind:

a) thinking it through this is (might be ;) ) true for packed bitstream streams, but as your own example about the -nopacked stream also shows this is not true for not packed streams
still its not a problem because of course the whole searching for and dropping fake n-vops isnt necessary with not packed streams, as in this case there are only real n-vops
right?

b) your method is wrong when a real frame (be it a real n-vop or a normal p-vop), that has the same size as the fake n-vop, gets packed together with a b-vop
in this case things would look like this:

len= 1309 | type=B, quant= 11, len= 1295
len= 1217 | type=B, quant= 11, len= 1217
len= 6 | type=P, quant= 7, len= 6the P gets packed together with the first B, but your algo doesnt find the fake n-vop cause it has the same size as the real vop



btw some potential bugs i found in the stats of xvid:
- i also think that there is a bug in this value 6 for the p-vop (it should be 9295) you showed above. i have reproduced this on other streams too, it seems to be a common problem
- also a common bug could be len values in the case of packed frames. shouldnt the size of the packed frames be the same as the sum of the not packed ones?
- the following borked coding order sizes created with -nopacked are also often the case but dont make sense:
6133: key=0, time= 300, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.95, psnr u = 39.68, psnr v = 38.35
6134: key=0, time= 291, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6135: key=2, time= 40, len= 13205 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6136: key=0, time= 230, len= 6 | type=P, quant= 2, len= 6, psnr y = 26.52, psnr u = 44.55, psnr v = 44.81
6137: key=0, time= 271, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.22, psnr u = 37.26, psnr v = 36.73
6138: key=0, time= 290, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6139: key=2, time= 40, len= 10332 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6140: key=0, time= 251, len= 6 | type=P, quant= 2, len= 6, psnr y = 27.14, psnr u = 46.42, psnr v = 45.23
6141: key=0, time= 270, len= 6 | type=P, quant= 2, len= 6, psnr y = 24.35, psnr u = 44.41, psnr v = 43.27maybe you can find the reason for these bugs (?) and fix them.

[squid_80]

1) if a frame has a size of smaller than 10 on the coding order side (left side) its a n-vop.

i would say this is a not necessarily always correct workaround as at small bitrates and resolutions it would be imagineable to also have frames with a small size
better would be to check if the not coded flag is set for identifying n-vops, cause than you know it for sure


2) if this p-vop has a different size in coding order than in display order you throw it away, as its a fake n-vop
two things coming to my mind:

a) thinking it through this is (might be ;) ) true for packed bitstream streams, but as your own example about the -nopacked stream also shows this is not true for not packed streams
still its not a problem because of course the whole searching for and dropping fake n-vops isnt necessary with not packed streams, as in this case there are only real n-vops
right?

b) your method is wrong when a real frame (be it a real n-vop or a normal p-vop), that has the same size as the fake n-vop, gets packed together with a b-vop
in this case things would look like this:

len= 1309 | type=B, quant= 11, len= 1295
len= 1217 | type=B, quant= 11, len= 1217
len= 6 | type=P, quant= 7, len= 6the P gets packed together with the first B, but your algo doesnt find the fake n-vop cause it has the same size as the real vopYou're pretty much right. But are these circumstances actually possible? What is the minimum size of a p-vop without the not coded flag?

btw some potential bugs i found in the stats of xvid:
- i also think that there is a bug in this value 6 for the p-vop (it should be 9295) you showed above. i have reproduced this on other streams too, it seems to be a common problem
- also a common bug could be len values in the case of packed frames. shouldnt the size of the packed frames be the same as the sum of the not packed ones?
When packed bitstream is on, the reported size of the P frames has the bytes for the fake-nvop added in. I think that's what you're seeing.
- the following borked coding order sizes created with -nopacked are also often the case but dont make sense:
6133: key=0, time= 300, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.95, psnr u = 39.68, psnr v = 38.35
6134: key=0, time= 291, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6135: key=2, time= 40, len= 13205 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6136: key=0, time= 230, len= 6 | type=P, quant= 2, len= 6, psnr y = 26.52, psnr u = 44.55, psnr v = 44.81
6137: key=0, time= 271, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.22, psnr u = 37.26, psnr v = 36.73
6138: key=0, time= 290, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6139: key=2, time= 40, len= 10332 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6140: key=0, time= 251, len= 6 | type=P, quant= 2, len= 6, psnr y = 27.14, psnr u = 46.42, psnr v = 45.23
6141: key=0, time= 270, len= 6 | type=P, quant= 2, len= 6, psnr y = 24.35, psnr u = 44.41, psnr v = 43.27maybe you can find the reason for these bugs (?) and fix them.
It's the same bug. I think in reality it should be something like this:
6133: key=0, time= 300, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.95, psnr u = 39.68, psnr v = 38.35
6134: key=0, time= 291, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6135: key=2, time= 40, len= 13205 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6136: key=0, time= 230, len= 6 | type=I, quant= 2, len= 13205, psnr y = 26.52, psnr u = 44.55, psnr v = 44.81
6137: key=0, time= 271, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.22, psnr u = 37.26, psnr v = 36.73
6138: key=0, time= 290, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6139: key=2, time= 40, len= 10332 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6140: key=0, time= 251, len= 6 | type=I, quant= 2, len= 10332, psnr y = 27.14, psnr u = 46.42, psnr v = 45.23
6141: key=0, time= 270, len= 6 | type=P, quant= 2, len= 6, psnr y = 24.35, psnr u = 44.41, psnr v = 43.27Whether it's a bug or by design I don't know. But if the bad lengths are being used for first pass stats (I have a sneaking suspicion they are) the filesize is going to be way off. Can you do that same stream again with -pass1, then examine the stats file for any weirdness?

[bond]

You're pretty much right. But are these circumstances actually possible? What is the minimum size of a p-vop without the not coded flag?i have been able to create smaller than 10 real p-vops with a very small res and very small bitrate. i doubt anyone will ever do something like that, its just that it should be known as an potential issue

about the not coded flag: will you identify n-vops by the flag or by the size?
doing it by the flag will make sure no small p-vop will get dropped and will always be correct

and what are you answer to my question whether you are doing the fake n-vop check also with not packed streams?

When packed bitstream is on, the reported size of the P frames has the bytes for the fake-nvop added in. I think that's what you're seeing.well yeah, still i dont think the size of the packed frame(s) (shown on the left) is correct.
imho it would be correct if it would be the sum of the B and P sizes (shown on right), but it isnt

It's the same bug. I think in reality it should be something like this:what you are saying is more than just a wrong size.
what you are saying is that the frametype is shown wrongly (I vs. P) which would be a more serious bug than just an off size imho

Whether it's a bug or by design I don't know. But if the bad lengths are being used for first pass stats (I have a sneaking suspicion they are) the filesize is going to be way off. Can you do that same stream again with -pass1, then examine the stats file for any weirdness? actually now that you are saying i have got indeed wrong filesizes with no packed output together with n-vops enabled (with one pass encoding)

[squid_80]

i have been able to create smaller than 10 real p-vops with a very small res and very small bitrate. i doubt anyone will ever do something like that, its just that it should be known as an potential issueSmaller than 7 bytes? That's the largest n-vop I've seen. 10 was chosen to allow a few extra bytes, just in case.
about the not coded flag: will you identify n-vops by the flag or by the size?
doing it by the flag will make sure no small p-vop will get dropped and will always be correctSpeed is the issue. The check has to be done for every frame outputted by the encoder, so the simpler it is, the better.
and what are you answer to my question whether you are doing the fake n-vop check also with not packed streams?At the moment yes, because it means one less if statement in the code.
well yeah, still i dont think the size of the packed frame(s) (shown on the left) is correct.
imho it would be correct if it would be the sum of the B and P sizes (shown on right), but it isntTake the size of the P frame from the right. Subtract the size of the encoded n-vop. Add the size of the B frame from the right. The result should match the size of the encoded packed frame. If it doesn't, give an example.
what you are saying is more than just a wrong size.
what you are saying is that the frametype is shown wrongly (I vs. P) which would be a more serious bug than just an off size imhoWell the encoder's saying the encoded frame is a key frame, which means it's an I frame. And the PSNR calculations would seem to back that up. I think that with -nopacked, the stats (right side) are one frame behind i.e. they are actually for the encoded frame from the line above. Except if there's a n-vop, then the stats will match the encoded frame from the same line. This has the side effect of making some stats never appear in the output at all.
actually now that you are saying i have got indeed wrong filesizes with no packed output together with n-vops enabled (with one pass encoding)
Like I said, do a first pass and check the stats file. There's definitely weirdness in there. They even crash Koepi's statsreader.

Edit: hlength in the statsfile - my understanding is that it's the length of the headers in each frame e.g. it won't change no matter what quantizer is used. So a negative value is probably really bad, right?

[squid_80]

Ok, I put on my bug hunting hat and found a few things. Firstly there's this in encoder.c (function FrameCodeP)
/* frame drop code */
if (current->sStat.kblks + current->sStat.mblks <=
(pParam->frame_drop_ratio * mb_width * mb_height) / 100 &&
( (pEnc->bframenum_head >= pEnc->bframenum_tail) || !(pEnc->mbParam.global_flags & XVID_GLOBAL_CLOSED_GOP)) )
{
current->sStat.kblks = current->sStat.mblks = 0;
// Squid's fix
// current->sStat.iTextBits = 0;
current->sStat.ublks = mb_width * mb_height;
I think current->sStat.iTextBits should be set to 0 since an n-vop contains no texture information. I tested and it fixed the bitrate issues for packed bitstreams with frame drops.
For unpacked bitstream there's another problem. These 2 sections of code (again from encoder.c):
/* for unpacked bframes, output the stats for the last encoded frame */
if (!(pEnc->mbParam.global_flags & XVID_GLOBAL_PACKED) && pEnc->mbParam.max_bframes > 0)
{
if (pEnc->current->stamp > 0) {
call_plugins(pEnc, pEnc->reference, &pEnc->sOriginal, XVID_PLG_AFTER, NULL, NULL, stats);
}
else
stats->type = XVID_TYPE_NOTHING;
}
if ( FrameCodeP(pEnc, &bs) == 0 ) {
/* N-VOP, we mustn't code b-frames yet */
// Squid's fix
// if ((pEnc->mbParam.global_flags & XVID_GLOBAL_PACKED) || pEnc->mbParam.max_bframes == 0)
call_plugins(pEnc, pEnc->current, &pEnc->sOriginal, XVID_PLG_AFTER, NULL, NULL, stats);
goto done;
}
The upshot of this is when packed bitstream isn't used and a n-vop is encoded, the plugins are called twice. This explains how I was seeing double entries in the stats file for n-vops. The fix I've come up with seems to solve the problem, but breaks encraw like I thought it would. :(

[squid_80]

I'm confused; need help from a matroska expert.
Haali says ( http://haali.cs.msu.ru/mkv/codecs.pdf ) that the mpeg4 headers up to the first VOP go in the CodecPrivate element and the stream itself should only contain VOPs and no headers, but this borks things if the VOL header changes which seems to happen sometimes (use a constant quant zone at some point after the start). Maybe it's just a first pass quirk, I haven't got that far yet. Keeping the VOL header with the I frames in the mkv seems to work, but is apparently prohibited. Any idea which is correct?

[bond]

mpeg-4 defines that the vol can never change throughout the stream
if a mpeg-4 encoder outputs varrying vols its incompliant to the mpeg-4 standard

[squid_80]

That's what I thought. I'll ignore it and move on to 2nd pass...

[bond]

so will you output varrying vols or will you do it as haali writes?

[squid_80]

Like Haali says. Unless people want busted streams, then I might make an option for it.

[bond]

actually if xvid outputs varrying vols its a serious bug which needs to be fixed :>

[squid_80]

I've only seen it doing first pass with full1pass disabled, which the vfw says: "If you don't discard first pass but keep full quality disabled, the resulting 1st pass stream might not be mpeg-4 compliant."

[bond]

yeah, the results from the fastfirstpass are not useable for anything

[JoeBG]

@ squid_80

Do you plan a new release? Last release works perfect for me with xvidcore 1.2 beta single threat version -> So it´s just a question. :)

[squid_80]

Of course, I'm working on nothing else at the moment (except my real-life job to pay the bills).

@bond: Have you heard anything back about your post to the xvid-devel mailing list? Because I think I pretty much solved the bitrate problems but unfortunately the bug is in xvidcore, I can implement a fix in encraw but it would mean duplicating a lot of code.

[bond]

@bond: Have you heard anything back about your post to the xvid-devel mailing list? Because I think I pretty much solved the bitrate problems but unfortunately the bug is in xvidcore, I can implement a fix in encraw but it would mean duplicating a lot of code.nope i didnt. if you know how to fix it it would be best to do in the core itself. i am sure the xvid devs will apply a working patch from your side to the cvs

[Doom9]

@squid_80: are you making any progress on the remaining commandline options?

[squid_80]

Working on it as we speak, should have something in about 1.5 hours.

[squid_80]

Damnit, not going to get zones finished off properly tonight but there's a new build with rate control options and custom PAR settings. Display AR should now be set for mkv, I/P/B min quants are 2 by default, the trailing p is dropped from the divx userdata for mkv and m4v ES output(not sure if this is done correctly), cartoon mode is no longer accidentally activated, GMC works with mkv, fake vs. real n-vop detection no longer relies on a bug in xvidcore.

BTW I strongly recommend not using frame drops and b-frames together; the encoder is smart enough to not drop the P frame immediately after a B frame series, but not smart enough to know not to use B frames after encoding an n-vop.
159: key=0, time= 0, len= 6 | type=P, quant= 7, len= 12462
160: key=0, time= 125, len= 20950 | type=B, quant= 11, len= 5124
161: key=0, time= 0, len= 6 | type=P, quant= 7, len= 15832
162: key=0, time= 94, len= 6 | type=P, quant= 7, len= 6
163: key=0, time= 125, len= 12282 | type=B, quant= 13, len= 1443
164: key=0, time= 0, len= 6 | type=P, quant= 9, len= 10845
165: key=0, time= 94, len= 5873 | type=B, quant= 15, len= 1705
The highlighted P frame is an n-vop; the following B frame shows artifacts since it references a frame that isn't coded.
I've also seen artifacts when using -noclosed_gop, but haven't had time to look into it.

[squid_80]

Re:zones, I was going to be clever and allocate them dynamically (instead of having a limit of 64 max) but realized this was silly since it would probably be impossible to specify that many via the command line anyway. But now I'm thinking of adding an option to pass a filename (e.g. -zonefile xxx) which would be a text file containing the zones in a specified format. Good idea or bad idea?

[buzzqw]

But now I'm thinking of adding an option to pass a filename (e.g. -zonefile xxx)

Good Idea ! but don't drop support for old command line zones

BHH

[bond]

how are you detecting fake n-vops now?
btw its not possible to deactivate real n-vops placement in xvid

about zones: why not simply do it like other cmdl tools (mp4box, mencoder...):
you specify a zone via -zone and than add specific commands you want right after it with a ":"

like -zone startframe=3000:weight=0.5:cartoonmode:greyscale

that would be straight forward and easy to set and understand

[squid_80]

Good Idea ! but don't drop support for old command line zonesI won't. I'm trying hard not to bork any of the original options for backwards compatability.
how are you detecting fake n-vops now?
btw its not possible to deactivate real n-vops placement in xvid
fake n-vop=a n-vop that follows a b-vop when using packed bitstream.
The real ones that show up when frame-drop=0 are identical to the frame before, so the artifacts aren't visible.
about zones: why not simply do it like other cmdl tools (mp4box, mencoder...):
you specify a zone via -zone and than add specific commands you want right after
it with a ":"

like -zone startframe=3000:weight=0.5:cartoonmode:greyscale

that would be straight forward and easy to set and understand
There will be something like that; I was going to go with doom9's suggestion since it's nice and compact ( http://forum.doom9.org/showthread.php?p=702759#post702759 ) but drop the end zone parameter.

Something I forgot: -pass1 and -pass2 default to using xvid.stats as the statsfile if one isn't specified. Also packed bitstream is disabled if avi output isn't used - this may cause slight oversize in the second pass, we'll see what happens.

[bond]

There will be something like that; I was going to go with doom9's suggestion since it's nice and compact ( http://forum.doom9.org/showthread.php?p=702759#post702759 ) but drop the end zone parameter. nice, tough i prefer a = instead of a , for signalling what belongs together, but thats a matter of taste

Also packed bitstream is disabled if avi output isn't used - this may cause slight oversize in the second pass, we'll see what happens. great!
tough disabling packed bitstream or not should not make any difference on the stream (except the n-vops of course) as packed bitstream is only a "packing mode" on how frames are stored. if with or without packed doesnt give the same results there is a bug (maybe its the same bug i mentioned on the maillist)

[squid_80]

tough disabling packed bitstream or not should not make any difference on the stream (except the n-vops of course) as packed bitstream is only a "packing mode" on how frames are stored. if with or without packed doesnt give the same results there is a bug (maybe its the same bug i mentioned on the maillist)

I wouldn't really label it a bug; just that the statsfile is different depending if packed or nopacked is used, due to each fake n-vops taking up a few bytes. I *think* rate-control will compensate for it, if it does the stream will be encoded differently (when comparing packed/unpacked) and if not there might be a slight oversize. I'll have to do more tests.

[bond]

as i understand it the stats file should be exactly the same no matter if packed bitstream is enabled or not

[squid_80]

BUG: when using raw input and not specifying a maximum number of frames, EVERY frame is encoded as a P frame. Whoops!

[squid_80]

New build with -zones parameter, should work as described except there's no end frame parameter; a zone lasts until the start of the next one just like vfw. There will probably be a bug or two, please report any problems.

Edit: sorry, as described here: http://forum.doom9.org/showthread.php?p=702759#post702759

[JoeBG]

New build with -zones parameter, should work as described except there's no end frame parameter; a zone lasts until the start of the next one just like vfw. There will probably be a bug or two, please report any problems.

Edit: sorry, as described here: http://forum.doom9.org/showthread.php?p=702759#post702759


Where is the download? :) :thanks:

[squid_80]

Same as before; ftp://squid80.no-ip.com/xvid_encraw.zip or http://okejl.dk/dunstan/xvid_encraw.zip

[woah!]

hi again.

so i see that the 29fps bug id faxed now as i get 29.97 fps :)

now to the next question, is pass2 borked? i cant get a readable output file from any encode i do pass2 :(

same setting in either quant or bitrate mode is fine, but 2pass crashes mpc.

pity as its nearly there isnt it ..


heres my cmd line:

xvid_encraw.exe -i file.avs -type 2 -pass1 file.stats -framerate 29.970000 -qtype 1 -lumimasking -bitrate 1830

xvid_encraw.exe -i file.avs -type 2 -pass2 file.stats -o file.avi -framerate 29.970000 -qtype 1 -lumimasking -bitrate 1830

no avi output file for 1st pass only for the second

nothing to fancy as you see.

[squid_80]

<edited out rubbish>

-o is for saving an elementary stream. -avi is for saving an avi file. Use -avi instead.

[woah!]

trim(30000, 30230)

file is small as i am trying to get this to work before i go with a full encode.

i am encoding about 44mins of 1920x1080i HD source to 704x400 and i have no issues doing the same avs script using mencoder.

[woah!]

noooooo say it aint so, how did i miss that heh... couldnt see the forest for the trees.. thx for this it is now working great :)

[Doom9]

@squid: are the new zone options actually in and just the cli help is not up-to-date? Also, now that it seems the missing options are supported, any chance of a -threads parameter?

[squid_80]

Yeah I haven't got around to updating the help screen. But they should be working. (except the -zonefile idea I had, haven't done that yet.)
-threads I can put in, but it won't actually do anything until we get a fixed SMP version of xvidcore which doesn't have the number of threads hardcoded.

[JoeBG]

Can I have a list of all options? I just thought zones and mkv are new.

[squid_80]

c:\>xvid_encraw -help
xvid_encraw built at 22:29:08 on Feb 9 2006
Usage : xvid_encraw [OPTIONS]

Input options:
-i string : input filename (stdin)
-type integer: input data type (yuv=0, pgm=1, avi/avs=2)
-w integer: frame width ([1.2048])
-h integer: frame height ([1.2048])
-frames integer: number of frames to encode

Output options:
-dump : save decoder output
-save : save an Elementary Stream file per frame
-o string : save an Elementary Stream for the complete sequence
-avi string: save an AVI file for the complete sequence
-mkv string: save a MKV file for the complete sequence

BFrames options:
-max_bframes integer: max bframes (2)
-bquant_ratio integer: bframe quantizer ratio (150)
-bquant_offset integer: bframe quantizer offset (100)

Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-max_key_interval integer : maximum keyframe interval (300)

Single Pass options:
-reaction integer : reaction delay factor (16)
-averaging integer : averaging period (100)
-smoother integer : smoothing buffer (100)

Second Pass options:
-kboost integer : I frame boost (10)
-kthresh integer : I frame reduction threshold (1)
-kreduction integer : I frame reduction amount (20)
-ostrength integer : overflow control strength (5)
-oimprove integer : max overflow improvement (5)
-odegrade integer : max overflow degradation (5)
-chigh integer : high bitrate scenes degradation (0)
-clow integer : low bitrate scenes improvement (0)
-overhead integer : container frame overhead (24)
-vbvsize integer : use vbv buffer size
-vbvmax integer : vbv max bitrate
-vbvpeak integer : vbv peak bitrate over 1 second

Other options
-noasm : do not use assembly optmized code
-turbo : use turbo presets for higher encoding speed
-quality integer : quality ([0..6]) (6)
-vhqmode integer : level of R-D optimizations ([0..4]) (1)
-bvhq : use R-D optimizations for B-frames
-qpel : use quarter pixel ME
-gmc : use global motion compensation
-qtype integer : quantization type (H263:0, MPEG4:1) (0)
-qmatrix filename : use custom MPEG4 quantization matrix
-interlaced : use interlaced encoding
-nopacked : Disable packed mode
-noclosed_gop : Disable closed GOP mode
-lumimasking : use lumimasking algorithm
-stats : print stats about encoded frames
-debug : activates xvidcore internal debugging output
-vop_debug : print some info directly into encoded frames
-nochromame : Disable chroma motion estimation
-notrellis : Disable trellis quantization
-imin integer : Minimum I Quantizer (1..31) (2)
-imax integer : Maximum I quantizer (1..31) (31)
-bmin integer : Minimum B Quantizer (1..31) (2)
-bmax integer : Maximum B quantizer (1..31) (31)
-pmin integer : Minimum P Quantizer (1..31) (2)
-pmax integer : Maximum P quantizer (1..31) (31)
-drop integer : Frame Drop Ratio (0..100) (0)
-start integer : Starting frame number
-par integer[:integer] : Set Pixel Aspect Ratio.
1 = 1:1
2 = 12:11 (4:3 PAL)
3 = 10:11 (4:3 NTSC)
4 = 16:11 (16:9 PAL)
5 = 40:33 (16:9 NTSC)
other = custom (width:height)
-help : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as needed.
-zones integer,q/w,float[,flags]
integer = start frame of zone
q/w = q for quantizer zone or w for weight zone
float = quant or weight (floating point value)
optional flags = K for start with keyframe, C for cartoon mode, O for chroma optimizer, G for greyscale or an integer for BVOP sensitivity. Order isn't important.
No spaces anywhere in the zones options.
Separate multiple zones with / . Or I guess you could use multiple -zones but I haven't tested it since it wasn't designed this way.
Also there's a -packed option to force packed bitstream on since it's off by default if there's no .avi output. Kinda weird I know but as we've discussed the bitstream setting affects what is written to the statsfile and that in turn has a substantial effect on the final encoded stream.

[JoeBG]

c:\>xvid_encraw -help
xvid_encraw built at 22:29:08 on Feb 9 2006
Usage : xvid_encraw [OPTIONS]

Input options:
-i string : input filename (stdin)
-type integer: input data type (yuv=0, pgm=1, avi/avs=2)
-w integer: frame width ([1.2048])
-h integer: frame height ([1.2048])
-frames integer: number of frames to encode

Output options:
-dump : save decoder output
-save : save an Elementary Stream file per frame
-o string : save an Elementary Stream for the complete sequence
-avi string: save an AVI file for the complete sequence
-mkv string: save a MKV file for the complete sequence

BFrames options:
-max_bframes integer: max bframes (2)
-bquant_ratio integer: bframe quantizer ratio (150)
-bquant_offset integer: bframe quantizer offset (100)

Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-max_key_interval integer : maximum keyframe interval (300)

Single Pass options:
-reaction integer : reaction delay factor (16)
-averaging integer : averaging period (100)
-smoother integer : smoothing buffer (100)

Second Pass options:
-kboost integer : I frame boost (10)
-kthresh integer : I frame reduction threshold (1)
-kreduction integer : I frame reduction amount (20)
-ostrength integer : overflow control strength (5)
-oimprove integer : max overflow improvement (5)
-odegrade integer : max overflow degradation (5)
-chigh integer : high bitrate scenes degradation (0)
-clow integer : low bitrate scenes improvement (0)
-overhead integer : container frame overhead (24)
-vbvsize integer : use vbv buffer size
-vbvmax integer : vbv max bitrate
-vbvpeak integer : vbv peak bitrate over 1 second

Other options
-noasm : do not use assembly optmized code
-turbo : use turbo presets for higher encoding speed
-quality integer : quality ([0..6]) (6)
-vhqmode integer : level of R-D optimizations ([0..4]) (1)
-bvhq : use R-D optimizations for B-frames
-qpel : use quarter pixel ME
-gmc : use global motion compensation
-qtype integer : quantization type (H263:0, MPEG4:1) (0)
-qmatrix filename : use custom MPEG4 quantization matrix
-interlaced : use interlaced encoding
-nopacked : Disable packed mode
-noclosed_gop : Disable closed GOP mode
-lumimasking : use lumimasking algorithm
-stats : print stats about encoded frames
-debug : activates xvidcore internal debugging output
-vop_debug : print some info directly into encoded frames
-nochromame : Disable chroma motion estimation
-notrellis : Disable trellis quantization
-imin integer : Minimum I Quantizer (1..31) (2)
-imax integer : Maximum I quantizer (1..31) (31)
-bmin integer : Minimum B Quantizer (1..31) (2)
-bmax integer : Maximum B quantizer (1..31) (31)
-pmin integer : Minimum P Quantizer (1..31) (2)
-pmax integer : Maximum P quantizer (1..31) (31)
-drop integer : Frame Drop Ratio (0..100) (0)
-start integer : Starting frame number
-par integer[:integer] : Set Pixel Aspect Ratio.
1 = 1:1
2 = 12:11 (4:3 PAL)
3 = 10:11 (4:3 NTSC)
4 = 16:11 (16:9 PAL)
5 = 40:33 (16:9 NTSC)
other = custom (width:height)
-help : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as needed.
-zones integer,q/w,float[,flags]
integer = start frame of zone
q/w = q for quantizer zone or w for weight zone
float = quant or weight (floating point value)
optional flags = K for start with keyframe, C for cartoon mode, O for chroma optimizer, G for greyscale or an integer for BVOP sensitivity. Order isn't important.
No spaces anywhere in the zones options.
Separate multiple zones with / . Or I guess you could use multiple -zones but I haven't tested it since it wasn't designed this way.
Also there's a -packed option to force packed bitstream on since it's off by default if there's no .avi output. Kinda weird I know but as we've discussed the bitstream setting affects what is written to the statsfile and that in turn has a substantial effect on the final encoded stream.


Thanks :)
:stupid:
weight is in percent? Are the following examples ok?

-z 58000, q, 8 -> will start a zone beginning with frame 58000 with q=8
-z 58000, w, 20 -> will start a zone beginning with frame 58000 with 20 % of quality of the main video.

[squid_80]

weight is in percent? Are the following examples ok?

-z 58000, q, 8 -> will start a zone beginning with frame 58000 with q=8
-z 58000, w, 20 -> will start a zone beginning with frame 58000 with 20 % of quality of the main video.
Weight is a percentage but it's in decimal form. 1 = 100%, 0.1 = 10%, 0.001 = 1% etc.

-zones 58000,q,8/58001,w,0.2
A zone starting at frame 58000 with a quantizer of 8 and another zone starting at 58001 with a weight of 20%. Very important that there's no spaces anywhere.

[JoeBG]

Thanks. I updated this info in the german boards. We are all testing :) :)

[buzzqw]

@squid_80

for your bandwith safe here the link at actual (13/02/2006) build www.64k.it/andres/xvid_encraw-13022006.zip

BHH

[dimzon]

@squid_80
Great Job! Can you add cqmintra && cqminter command-line switches to be capable use CQM without cqm file
sample:
-cqmintra 16,16,...,127 -cqminter 32,32,...,32

[bond]

squid_80, someone has posted a patch on the xvid maillist which seems to fix my issues with high framedrop values, maybe you are interested in it (maybe it was you who wrote the patch? :D )

[squid_80]

Yeah it was.

[squid_80]

@squid_80
Great Job! Can you add cqmintra && cqminter command-line switches to be capable use CQM without cqm file
sample:
-cqmintra 16,16,...,127 -cqminter 32,32,...,32
Should be easy enough. I assume the order of the values is just reading the matrix left to right, one row at a time e.g. for a matrix like this:
8, 11, 13
12, 11, 12
12, 12, 13
You would put 8,11,13,12,11,12,12,12,13?
(I know q-matrices are 16x16, I'm just using 3x3 for a small example)
Just wanted to be sure in case someone expects the values to be in the other order that sometimes gets used (which would be 8,12,11,13,11,12,12,13).

[Sagittaire]

Possible to make that ... ???

- pass1 : write always stat files
- pass2 : read stat files

but with pass1 function always active even if pass2 command is enabled:

xvid_encraw.exe -i file.avs -type 2 -pass1 file1.stats ....
xvid_encraw.exe -i file.avs -type 2 -pass2 file1.stats -pass1 file2.stats ...
xvid_encraw.exe -i file.avs -type 2 -pass2 file2.stats -pass1 file3.stats ...

or perhaps other function like -writestats [filename] with exactly the same file structure than 1pass files. With this function it will be possible to make multipass encoding (better target bitrate, better predictibility for frame size, better quality ...)

[squid_80]

BTW, avi output is broken for files > 2gb in size (thanks to avifile). They might be recoverable using vdub with the rederive keyframe flags option, I haven't let it run long enough to find out. mkv and raw streams seem to be ok.
Possible workarounds: a) Get an avi writer capable of producing odml files and use that instead (probably from virtualdub) or b) output multiple files, splitting at 2gb boundaries.
Of course there's option c) ignore it and hope no-one notices.

[squid_80]

but with pass1 function always active even if pass2 command is enabled:

xvid_encraw.exe -i file.avs -type 2 -pass1 file1.stats ....
xvid_encraw.exe -i file.avs -type 2 -pass2 file1.stats -pass1 file2.stats ...
xvid_encraw.exe -i file.avs -type 2 -pass2 file2.stats -pass1 file3.stats ...

At the moment -pass1 will disable -pass2 and vice versa. Whichever option comes last in the commandline is the one that will be active.
I'll run some tests and see if it's worth implementing multi-pass. I'm not sure if they will actually work together to give better results.

[bond]

i think there is a bug in labelling s-vops as the following doesnt make sense for me (framedropratio of 100):
len= 6 | type=P
len= 6 | type=P
len= 6 | type=S
len= 6 | type=S
len= 6 | type=S
len= 6 | type=Sobviously the frames with size 6 are n-vops, still they are labelled as s-vops

[hellfred]

Of course there's option c) ignore it and hope no-one notices.
I saw it, I noticed it, TOO LATE NOW :D

[squid_80]

i think there is a bug in labelling s-vops as the following doesnt make sense for me (framedropratio of 100):
len= 6 | type=P
len= 6 | type=P
len= 6 | type=S
len= 6 | type=S
len= 6 | type=S
len= 6 | type=Sobviously the frames with size 6 are n-vops, still they are labelled as s-vops
Are not-coded S-VOPs against specs? I know why it happens; the decision to drop a frame is made after encoding, based on the number of not-coded macroblocks. If the percentage of not-coded mbs is greater than or equal to the frame drop ratio the whole frame is not coded.
I can definitely see how this could be bad with s-vops. But it's a problem in xvidcore rather than encraw.

[Wilbert]

@squid_80,

I know why it happens; the decision to drop a frame is made after encoding, based on the number of not-coded macroblocks. If the percentage of not-coded mbs is greater than or equal to the frame drop ratio the whole frame is not coded.
Is it possible to make that decision before the encoding?

[squid_80]

It wouldn't really save much since some sort of motion estimation would still need to be done. If you have a source with lots of close to identical frames and expect lots of N-VOPs it would be better IMO to run a filter that removes dupes and dumps a timecode file to be used later. (Assuming there's a filter to do this and it actually works with MPEG4 ASP in mkv; I haven't got around to finding that out, but I plan to.)

[bond]

It wouldn't really save much since some sort of motion estimation would still need to be done. If you have a source with lots of close to identical frames and expect lots of N-VOPs it would be better IMO to run a filter that removes dupes and dumps a timecode file to be used later. (Assuming there's a filter to do this and it actually works with MPEG4 ASP in mkv; I haven't got around to finding that out, but I plan to.)hehe i didnt want to ask for vfr with mkv ;)

[bond]

squid_80, someone has posted a patch on the xvid maillist which seems to fix my issues with high framedrop values, maybe you are interested in it (maybe it was you who wrote the patch? :D )was applied to cvs

squid: is xvidcli supposed to work with the 1.2 line of xvid?

[dimzon]

Should be easy enough. I assume the order of the values is just reading the matrix left to right, one row at a time e.g. for a matrix like this:
8, 11, 13
12, 11, 12
12, 12, 13
You would put 8,11,13,12,11,12,12,12,13?
(I know q-matrices are 16x16, I'm just using 3x3 for a small example)
Just wanted to be sure in case someone expects the values to be in the other order that sometimes gets used (which would be 8,12,11,13,11,12,12,13).
YES EXATLY

[squid_80]

squid: is xvidcli supposed to work with the 1.2 line of xvid?
It should, that's what I'm using. It uses the current version of xvid.h so it should support all versions of xvidcore that are backwards compatible with that.

YES EXATLY
I'll try it but I don't know if specifying 256 values for each matrix will fit in the command line.

What I've been twiddling over the last few days is a -mkvvfr <filename> option, which produces a vfr mkv file by dropping n-vops completely and setting the duration and timecode of each frame appropriately.

[dimzon]

I'll try it but I don't know if specifying 256 values for each matrix will fit in the command line.
I'm not shure but seem's like there are no restrictions in modern OS (Win2K/WinXP)

[bond]

whats the advantage of specifying a cqm via cmdl and not using an external file?

edit:
What I've been twiddling over the last few days is a -mkvvfr <filename> option, which produces a vfr mkv file by dropping n-vops completely and setting the duration and timecode of each frame appropriately.is it necessary to make this vfr option so mkv centric? i mean regarding an hopefully coming mp4 output which also would be able to handle vfr its maybe better to keep as much vfr things as container agnostic as possible

[dimzon]

whats the advantage of specifying a cqm via cmdl and not using an external file?
to avoid unnececary dependencies when creating profiles in MeGUI ;)

[squid_80]

is it necessary to make this vfr option so mkv centric? i mean regarding an hopefully coming mp4 output which also would be able to handle vfr its maybe better to keep as much vfr things as container agnostic as possible
The main reason I'm using -mkvvfr is so -mkv can still be used at the same time in case of errors. If (when) mp4 output is added I can add a -vfr option which will make vfr mkv+mp4 output streams.

[bond]

why not use -vfr right away?

[ImmortAlex]

Hi! I have some questions:

1. In GUI there is option to specify target file size for 2nd pass. In xvid_encraw I see only target bitrate. Is this equivalent? For example, if bitrate calculator tells me that size is 617820 kbytes and bitrate is 821 kbps, is specifing "-bitrate 821" in encraw equals to specifing 617820 in GUI?

2. Are units for VBV options equals to GUI ? Are "-vbvsize 3145728 -vbvmax 4854 -vbvpeak 8000000" right options, equals to "Cinema Plus" profile?

[squid_80]

Hi! I have some questions:

1. In GUI there is option to specify target file size for 2nd pass. In xvid_encraw I see only target bitrate. Is this equivalent? For example, if bitrate calculator tells me that size is 617820 kbytes and bitrate is 821 kbps, is specifing "-bitrate 821" in encraw equals to specifing 617820 in GUI?Yes. I plan to add a -size parameter later.
2. Are units for VBV options equals to GUI ? Are "-vbvsize 3145728 -vbvmax 4854 -vbvpeak 8000000" right options, equals to "Cinema Plus" profile?
Yes they match the VFW options.

[ImmortAlex]

Thanks, but at least -vbvmax is not in kbps.
"-vbvmax 4854" leads to quant=31 for all frames. Maybe I need "-vbvmax 4854000" ? BTW, it shows like "4854.000" in GUI.

Also I don't understand how to set zones. For example, "-frames 2000 -zones 0,w,1.0/1000,w,0.01" doesn't produce a clip with worse quality in its 2nd half. What is my mistake?

[squid_80]

Thanks, but at least -vbvmax is not in kbps.
"-vbvmax 4854" leads to quant=31 for all frames. Maybe I need "-vbvmax 4854000" ? BTW, it shows like "4854.000" in GUI.
I can't check the source code ATM but you're probably right. The value internally is bps, the vfw gui just chooses to show it as kbps and I missed the decimal point when I checked before.
Also I don't understand how to set zones. For example, "-frames 2000 -zones 0,w,1.0/1000,w,0.01" doesn't produce a clip with worse quality in its 2nd half. What is my mistake?
Probably my fault - I made a mistake by not sorting the zones before passing them to the encoder. Try switching the order of the zones e.g. -zones 1000,w,0.01/0,w,1.0 and see if that helps.

[squid_80]

why not use -vfr right away?
Actually I now think it's a dumb idea anyway, because:
- possibly breaks specs; no frame found matching expected timecode (not sure about this, need a mpeg4asp expert)
- Only saves about 6/7/8 bytes per dropped n-vop, not really worth the effort.

[bond]

Actually I now think it's a dumb idea anyway, because:
- possibly breaks specs; no frame found matching expected timecode (not sure about this, need a mpeg4asp expert)sounds more like a container-centric problem?

- Only saves about 6/7/8 bytes per dropped n-vop, not really worth the effort. yeah, but vfr support is a funky feature ;)

[squid_80]

sounds more like a container-centric problem?That's why I need an expert. Each VOP does have a timecode of sorts based on dwrate (I dunno the real terminology, that's the avi term for it, where framerate = dwrate/dwscale; xvid source code uses fbase/fincr), how it is used I'm not sure. It looks like it can be used for determining if a n-vop is real or fake - at least the way xvid does it. Might just be something to do with error correction.
It would be the same as replacing n-vops in avi (real ones) with actual dropped frames - more or less pointless.

[bond]

i would assume that a vfr mkv file simply doesnt have a framerate (dwrate) anymore?

maybe you want to send Mosu a message on how its handled

[foxyshadis]

It has a default duration; frames that deviate from it have their durations embedded in the frame header. I believe mp4 works the same way.

[squid_80]

I'm not talking about the container, I'm talking about what's in the raw m4v stream - specifically the first 3 or 4 bytes after the vop start code. For example here's a frame I picked at random:
00 00 01 B6 51 60 54 1D....
00 00 01 B6 = vop start code. Now we break the stream down into bits. 51 60 54 =
01 0 1 00010 1 10000001010100
01 = vop type (P)
0 = seconds. Not really seconds, kinda got something to do with fbase. The value is calulated by counting the number of preceding 1s before the 0.
1 = marker bit.
00010 = ticks, measured in fbase units. So for each successive frame, ticks increases by one. It rolls back to 0 when it reaches fbase, and the seconds field is incremented. But then the seconds field goes back to 0 after a few frames... the exact point at which it does seems to be max_bframes dependant.
1 = vop coded, the rest of the bits are to do with the encoded frame. (If it's not coded there's just padding up to the next byte.)

The width of the ticks field varies, it is equal to the base 2 log of fbase-1 (my maths is rusty, can't remember how to write that correctly as text). That's why the size of n-vops can vary depending on the framerate (the seconds field also varies).
What I would like to know is what a decoder is allowed to do with these values and what effect (if any) there would be if there was a sudden jump forwards/backwards in ticks.

[Zero1]

I tried encraw again since the last time I had problems (when I posted the screenshot). It works fine now, no visual errors at all ;) The command line and switches are pretty straight forward too.

Great work and thanks to everyone involved, it's great to be able to totally bypass VfW and produce native, spec compliant encodes.

Just a couple of questions, are there options within the CLI that are, or could be accessible, which would help improve the quality (over the VfW counterpart). I might be wrong, but aren't there some other search modes or anything in there that aren't accessible by the VfW?
Also, is it much hassle to update it's internals to something like 1.1.0 final, or maybe even the current 1.2? Even if the multithreading does not work, maybe the trellis improvments would be handy ;)

Cheers

[squid_80]

Just a couple of questions, are there options within the CLI that are, or could be accessible, which would help improve the quality (over the VfW counterpart). I might be wrong, but aren't there some other search modes or anything in there that aren't accessible by the VfW?
The way the quality and vhqmode options work is the best way to do it IMO; they group together search modes of the same approximate quality. I don't think it would be an improvement to allow individual modes to be (de)selected. If anyone used S_O's build and thought otherwise, feel free to mention it.
Also, is it much hassle to update it's internals to something like 1.1.0 final, or maybe even the current 1.2? Even if the multithreading does not work, maybe the trellis improvments would be handy ;)
It uses whatever version of xvidcore.dll you have installed. If that's the current version then you're already using the improvements.

(note to self: add output message to show the version of xvidcore.dll being used, if possible.)

[squid_80]

New build up. Actually there was a newer build up over 24 hours ago, I just updated it again. Nothing real groundbreaking:
-size option (help text is formatted wrong :( )
-pass1 and -pass2 can be used at the same time even though I couldn't get any improvement from nth pass
-interlaced option has support for TFF
-threads option is implemented and should work when support is added to xvidcore
- information about detected xvidcore version is displayed
-progress option, which I added in about 2 minutes because I was sick of watching the scrolling text output. It's a bit buggy.
-vfr option (not in help text) which should drop n-vops from mkv output streams and make the stream vfr. Haven't played with it much but seems to work.

Edit: Forgot that I also fixed zone handling to sort zones properly.

[shon3i]

Where i can download this lastest version.

[buzzqw]

link is same as usual ftp://squid80.no-ip.com/xvid_encraw.zip

or my (tiny) mirror www.64k.it/andres/xvid_encraw-23022006.zip

BHH

[shon3i]

link is same as usual ftp://squid80.no-ip.com/xvid_encraw.zip

or my (tiny) mirror www.64k.it/andres/xvid_encraw-23022006.zip

BHH
Thanks

[bond]

pretty much a small revolution having a xvid vfr encoder

thx squid!

[bond]

btw for stressing the vfr option via the drop option you should make sure to use a recent 1.2 xvid compile, cause the older ones (eg official 1.1) have a bug with n-vops

[unplugged]

Does this recent xvidenc_raw is compatible with current MeGUI?

[squid_80]

Not as far as I know since I broke the -bitrate parameter (whoops). Might be possible to make a fix - I could assume any value > 10000 is in bps and any value <= 10000 is in kbps. Does anyone regularly use bitrates less than 10kbps or greater than 10 mbps?

[squid_80]

btw for stressing the vfr option via the drop option you should make sure to use a recent 1.2 xvid compile, cause the older ones (eg official 1.1) have a bug with n-vops
Is that the n-vop with unpacked bitstream bug that I fixed? That shouldn't affect the vfr stuff, I though it just mucks up the stats file and bitrate distribution.

[Doom9]

Does this recent xvidenc_raw is compatible with current MeGUI?Not now, but the next stable megui will be.. it has no release date and will be available when it's done.

@squid: when I saw -overhead integer : container frame overhead (24) I wanted to run away screaming.. did you just reintroduce all the mencoder problems or can I make it so that encraw doesn't give a flying hoot about container knowledge? If I give it a bitrate of 700, I want my output video stream to have a bitrate of exactly 700 kbit/s.. and the file having a larger size than length * 700 kbit/s because of the container overhead. So my raw stream would have exactly the desired size, my avi and mkv would be bigger. It would be a shame if different calculation mechanisms would have to be reintroduced everywhere because xvid is so tied to avi. can I force -overhead 0 and never have to worry about container overhead ever again? (well, megui does.. but I have generic calculations for that.. I don't want codec specific ones)

[squid_80]

If the passed bitrate is what you want the raw stream to be, then use -overhead 0. I only made the default=24 because it's better to get an undersized file rather than oversized.

I actually abuse it and use -overhead 665 -size 143360 to get an output avi that I can mux with matching 128kbps mp3 audio and end up with 140megs :) If I had an automated reliable way of cutting out commercials, backing up DVB captures would almost be too easy...

[bond]

Is that the n-vop with unpacked bitstream bug that I fixed? That shouldn't affect the vfr stuff, I though it just mucks up the stats file and bitrate distribution.that bug lead to wrong target bitrates and different, wrong frametype decisions too

a fixed 1.2 compile is available here (http://rapidshare.de/files/14009418/xvidcore.dll.html)

[Doom9]

fyi, on the way from the office I added support for the latest encraw to megui.. I just have to solve a few crucial issues first before making another release, and then refactor some more.

[Beave]

If the passed bitrate is what you want the raw stream to be, then use -overhead 0. I only made the default=24 because it's better to get an undersized file rather than oversized.

I actually abuse it and use -overhead 665 -size 143360 to get an output avi that I can mux with matching 128kbps mp3 audio and end up with 140megs :) If I had an automated reliable way of cutting out commercials, backing up DVB captures would almost be too easy...

This sounds great. How can I calculate that? Say I want to mux a 384 kbit/s ac3 file?
Thanks for all the great work on xvid_encraw, btw!

[squid_80]

This sounds great. How can I calculate that? Say I want to mux a 384 kbit/s ac3 file?

Just to be clear, there's much better ways of doing this. I'm just lazy.

Divide the audio bitrate by 8 to get bytes/sec (384000/8 = 48000).
Divide that number by the framerate to get bytes/frame (48000/25 = 1920, or 48000/23.976 = 2002, or 48000/29.97 = 1602).
Add avi overhead (24) plus 1 (1920 + 24 + 1 = 1945).
The resulting output file should be pretty close to -size once the audio is muxed in.
It might not fit all audio formats but should work for CBR mp3 and ac3.

[Beave]

Just to be clear, there's much better ways of doing this. I'm just lazy.

Divide the audio bitrate by 8 to get bytes/sec (384000/8 = 48000).
Divide that number by the framerate to get bytes/frame (48000/25 = 1920, or 48000/23.976 = 2002, or 48000/29.97 = 1602).
Add avi overhead (24) plus 1 (1920 + 24 + 1 = 1945).
The resulting output file should be pretty close to -size once the audio is muxed in.
It might not fit all audio formats but should work for CBR mp3 and ac3.

Well, I find your way of specifying the overhead great. This way I don't need to calculate the length of the movie nor getting/calculating the filesize of the ac3. All I need to know is the framerate and the audio bitrate.

Thanks for your great input.

One question:
Why do I need the parameter -framerate? Can't xvid_encraw get that framerate info from the avs/d2v automaticly? If I don't use that -framerate switch, will my 23.976 video be converted to 25 fps?

I did a full encode, but I got a totally undersized file. I can't find the problem though:

Starting 1st Pass...

"xvid_encraw.exe" -i "test.avs" -avi "test.avi" -pass1 "test.stats" -size 1362413 -turbo -max_key_interval 240 -qmatrix "HVS Best Picture.qmatrix" -bvhq -framerate 23.976 -zq 0 22 -zw 151 0.6 -zw 2334 1 -zq 80422 20

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
xvidcore build version: xvid-1.1.0
Bitstream version: 1.1.0
Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
Detected 0 cpus, defaulting to 1 thread
83001 frames(100%) encoded, 8.28 fps, Average Bitrate = 3030kbpss
Tot: enctime(ms) =10022143.00, length(bytes) = 1311303606
Avg: enctime(ms) = 120.75, fps = 8.28, length(bytes) = 15798
I frames: 1178 frames, size = 151918/178959925, quants = 1 / 1.29 / 22
P frames: 32472 frames, size = 1515/49198004, quants = 1 / 1.61 / 22
B frames: 49350 frames, size = 21948/1083145677, quants = 2 / 3.02 / 31

Starting 2nd Pass...

"xvid_encraw.exe" -i "test.avs" -avi "test.avi" -pass2 "test.stats" -size 1362413 -max_key_interval 240 -qmatrix "HVS Best Picture.qmatrix" -bvhq -framerate 23.976 -zq 0 22 -zw 151 0.6 -zw 2334 1 -zq 80422 20

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
xvidcore build version: xvid-1.1.0
Bitstream version: 1.1.0
Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
Detected 0 cpus, defaulting to 1 thread
83001 frames(100%) encoded, 10.91 fps, Average Bitrate = 1160kbps
Tot: enctime(ms) =7608925.00, length(bytes) = 501897660
Avg: enctime(ms) = 91.67, fps = 10.91, length(bytes) = 6046
I frames: 1178 frames, size = 22479/26480984, quants = 2 / 7.17 / 22
P frames: 32472 frames, size = 10904/354106754, quants = 2 / 5.70 / 24
B frames: 49350 frames, size = 2458/121309922, quants = 4 / 8.28 / 31

Can someone detect the problem? The filesize should be about 1330 MB, but I only got 480 MB unfortunatly.

[Doom9]

All I need to know is the framerate and the audio bitrate.
Actually.. you need to know a lot more.. video overhead for the given container, audio overhead for the given container, subtitle overhead for the given container ;)

[dimzon]

@Beave
Seems like your source is clear enought to fit it into 480MB ;)
In this case you will obtain perfect 1CD rip instead of 2-cd

[bond]

squid_80, when will your code be ready for cvs commit? ^^

[Beave]

@Bond
No way, sorry. This is a 720p encode and plenty noise in it. It's an episode of a series which I encoded before with VirtualDubMod. If I encode it in Quant=2 I get around 1800MB.

Now I did the second pass again changing -size 1362413 to -bitrate 3221. This time I got the right size. So there must be something wrong with the -size setting.

[squid_80]

One question:
Why do I need the parameter -framerate? Can't xvid_encraw get that framerate info from the avs/d2v automaticly? If I don't use that -framerate switch, will my 23.976 video be converted to 25 fps?
Who says you always need to specify -framerate? It's only needed when using raw input or if you want to change it.

I did a full encode, but I got a totally undersized file. I can't find the problem though:

Starting 1st Pass...

"xvid_encraw.exe" -i "test.avs" -avi "test.avi" -pass1 "test.stats" -size 1362413 -turbo -max_key_interval 240 -qmatrix "HVS Best Picture.qmatrix" -bvhq -framerate 23.976 -zq 0 22 -zw 151 0.6 -zw 2334 1 -zq 80422 20

83001 frames(100%) encoded, 8.28 fps, Average Bitrate = 3030kbpss
Tot: enctime(ms) =10022143.00, length(bytes) = 1311303606
Avg: enctime(ms) = 120.75, fps = 8.28, length(bytes) = 15798
I frames: 1178 frames, size = 151918/178959925, quants = 1 / 1.29 / 22
P frames: 32472 frames, size = 1515/49198004, quants = 1 / 1.61 / 22
B frames: 49350 frames, size = 21948/1083145677, quants = 2 / 3.02 / 31
That command line doesn't seem to quite match, there's a progress report but no -progress option, and you've got quants < 2 but no -imin/pmin/bmin...

Even with an average quantiser close to 1, it's undersized.
Try in order:
Use -qtype 1 so the custom matrix is actually used (someone found a bug; yay!).
Turn off turbo.
Drop the zones.
Use 1 max bframe or even perhaps 0.
If it's still undersized set the minimum quants to 1. Also maybe update to 1.2.0...
Now I did the second pass again changing -size 1362413 to -bitrate 3221. This time I got the right size. So there must be something wrong with the -size setting.Redo both first and second pass with -qtype 1. Otherwise you're just wasting space.
Looks like you're right about -size though - I think any value > 262144 will bork. Easily fixed.
squid_80, when will your code be ready for cvs commit? ^^
When it's not buggy anymore... Remember when I said it was a bad idea to run 2 copies at once? Last week I ignored my own advice and ended up with a movie that changed PAR about one third of the way through. Still can't recreate the situation either.

[woah!]

@Bond
No way, sorry. This is a 720p encode and plenty noise in it. It's an episode of a series which I encoded before with VirtualDubMod. If I encode it in Quant=2 I get around 1800MB.

Now I did the second pass again changing -size 1362413 to -bitrate 3221. This time I got the right size. So there must be something wrong with the -size setting.


did you try -size 1330

[Beave]

That command line doesn't seem to quite match, there's a progress report but no -progress option, and you've got quants < 2 but no -imin/pmin/bmin...

Yes, you are right about one part. I added the -progress 10 on the cmd window, but did not update it in my batch file. I definately did not use any imin/pmin/bmin though. I'm 100% sure about it, I was wondering myself about the quants=1. Since the filesize was off and fps didn't quite match my normal fps I didn't pay too much attention to this.


Try in order:
Use -qtype 1 so the custom matrix is actually used (someone found a bug; yay!).
Turn off turbo.
Drop the zones.
Use 1 max bframe or even perhaps 0.
If it's still undersized set the minimum quants to 1. Also maybe update to 1.2.0...

So in the future releases -qtype 1 will not be nessesary to use a -qmatrix, right? So for the next try I will put them right after each other:

-qtype 1 -qmatrix "file.qmatrix"

in hopes it will use the CQ and not the MPEG Matrix.
Using -bitrate 3221 gives me a perfect sized result.
Your other comments where only valid with the wrong size in mind, right?

Redo both first and second pass with -qtype 1. Otherwise you're just wasting space.

I don't get this. Why am I wasting space?

Looks like you're right about -size though - I think any value > 262144 will bork. Easily fixed.

That's good to hear it's easily fixed.

did you try -size 1330
Why should I use an even smaller number if the file is already undersized?

[squid_80]

So in the future releases -qtype 1 will not be nessesary to use a -qmatrix, right? So for the next try I will put them right after each other:

-qtype 1 -qmatrix "file.qmatrix"

in hopes it will use the CQ and not the MPEG Matrix.
Using -bitrate 3221 gives me a perfect sized result. Yes, specifying a custom matrix should turn on MPEG quantization but it didn't. Fixed in next build.
Your other comments where only valid with the wrong size in mind, right?They were things to try if it still comes out undersized after the custom matrix is used.
I don't get this. Why am I wasting space?Because it used h263 quantization, using the same parameters with -bitrate 3221 is just going to lower the (already quite low) quantizers for no visible gain. You're telling it to use a bitrate higher than what your 1st pass ended up with.

[squid_80]

Yes, you are right about one part. I added the -progress 10 on the cmd window, but did not update it in my batch file. I definately did not use any imin/pmin/bmin though. I'm 100% sure about it, I was wondering myself about the quants=1. Since the filesize was off and fps didn't quite match my normal fps I didn't pay too much attention to this.
Do you still have the stats file from the 1st pass? I'd like to know where the quant1 frames are, it's probably a zones screw up since any frame not in a quant zone should get quant=2 during first pass.

[Kopernikus]

Hi,

is it possible to get the a copy of the sources?

Have you thought about sending a patch to the mailinglist? BTW, is your 64bit stuff already in CVS?

[squid_80]

Actually.. you need to know a lot more.. video overhead for the given container, audio overhead for the given container, subtitle overhead for the given container ;)
You also need to know it doesn't work for vfr encodes. Wish I'd realized that about 10 hours ago. :angry:

[bond]

any news? ^^

[squid_80]

any news? ^^
Yeah. Instead of doing what I should be (reorganizing the code so it's nice and neat, fixing the >2gb avi output problem, adding mp4 support etc.) I've been playing around with dedup making vfr encodes... Made a new parameter -timecode <file> which takes a mkv timecode file and adds dropped frames to the output avi to simulate vfr. You'd think it would be simple to add it to the mkv output module (since timecodes are pretty simple in mkv), but it's not since the timecodes are in display order and the frames are written in encoded order. Just got to figure out the most sensible way to buffer the times until they're written.

[JoeBG]

Any news for the mp4 output? :)

[Zero1]

Why MP4 output? MP4box handles it fine.

[bond]

Why MP4 output? MP4box handles it fine.because direct .mp4 output (maybe even with vfr) without needing to remux would rock much more :D

[squid_80]

It looks like code from mpeg4ip will work out pretty well. I'm just trying to keep it all neat and tidy and trimmed down by cutting out parts of code that aren't needed.

[dimzon]

It looks like code from mpeg4ip will work out pretty well. I'm just trying to keep it all neat and tidy and trimmed down by cutting out parts of code that aren't needed.
Wow, fine news.

[fight2win]

i tried xvid_encraw in megui, it said that the options i choose in encoding not supported!

[buzzqw]

you can try my application

BHH

[JoeBG]

i tried xvid_encraw in megui, it said that the options i choose in encoding not supported!

Because MeGui does not support the commands from xvid_encraw. MeGui uses mencoder commandlines for xvid_encraw. The support for xvid_encraw is planned, but not finished.

Mkv Magic does the job very good :)

[Doom9]

@JoeBG: not true. Public megui releases support the CSV builds of encraw.. but not squid's custom builds. A yet unreleased version will support squid's build.

[JoeBG]

@ doom9

OK, sorry, misunderstanding

[bond]

It looks like code from mpeg4ip will work out pretty well. I'm just trying to keep it all neat and tidy and trimmed down by cutting out parts of code that aren't needed.great stuff! will vfr also work?

[snherbst]

@bond

because direct .mp4 output (maybe even with vfr) without needing to remux would rock much more :D

I thought that this prog only could encode video content and not audio content, or am I very wrong?

As I see it the benefit in getting a MP4 output file vould be that the frame rate is part of a container content.

[Beave]

@squid_80

I am wondering if you fixed some of the bugs in xvid_encraw yet and could maybe provide some sort of beta version? I'm especially interested in using the -size parameter with bigger file sizes.

[squid_80]

Just posted this in another thread: http://okejl.dk/dunstan/xvid_encraw_megui_test.zip

I *think* I fixed that bug by reordering the size->bitrate equation to prevent overflow. I might be mistaken since I can't remember what it used to look like. Try it and see.
This build may contain unfinished code too, since it's my work-in-progress build. In particular, make sure the input is raw YV12 (you might get away with a compressed avi that decompresses to YV12 by default) because it won't throw an error if it's not - you'll just end up with a) garbage b) a cropped picture or c) a crash :)
For the adventurous, you can try making a packed .avi output file using -timecode <file> where file is a matroska v2 timecode file (like the one dedup spits out, for example). The .avi file should have dropped frames inserted to make up the time differences between frames.

[Elic]

squid_80
> Just posted this in another thread: http://okejl.dk/dunstan/xvid_encraw_megui_test.zip
Oops... Server seems to be offline (ping has time out). Are there another copies, please?

[squid_80]

Not unless anyone is able to mirror it. I had to take my ftp server down about a month ago (ftp://squid80.no-ip.com), the okejl.dk webspace is from a 1024/512 ADSL connection in Denmark, provided by someone who offered a home to my 64-bit stuff (thanks Asbjørn!). He did say it was only temporary so if it's down, it's down - He's been very generous and I'm not going to bug him about it.

[woah!]

i have some gigs to shares if you need a host.

i just need the file first

[Elic]

I had to take my ftp server down about a month agoOK, I have noticed that http://okejl.dk/ is online again. And I dowloaded your file and uploaded it into http://elicit.vn.ua/files/hosted/xvid_encraw_megui_test.zip, just to be on the safe side, OK? :)

[kurt]

I got a zone error with:
-zones 1,w,0.5,25,O/359,w,1,O/161519,w,0.04,25,O,G

Error in zone 161519,w,0.04,25,O,G option ,
Error in zone 1,w,0.5,25,O option ,
without flags there are no errors:
-zones 1,w,0.5,25/359,w,1/161519,w,0.04,25

Any ideas, what's going wrong?

btw: thx for your great work, squid_80!

[Adub]

Squid, in case you missed it in the other topic, I can confirm that your knew test build works in MeGUI. It works quite well infact, I was getting about 46fps on the first pass and about 38 fps on the second. Excellent job man! It didn't crash or anything. Of course I wasn't using every setting so I can't tell for sure, but it work for simple encoding purposes. Again, excellent job.

[squid_80]

I got a zone error with:
-zones 1,w,0.5,25,O/359,w,1,O/161519,w,0.04,25,O,G

Don't put commas between the optional flags (including bvop sensitivity), e.g:
-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG

Squid, in case you missed it in the other topic, I can confirm that your knew test build works in MeGUI.Excellent. Now I don't have to be so hard on myself for breaking it in the first place.

[Adub]

Yeah man don't worry about it. It worked wonderfully. Keep going with everything that you are doing to it to. It is faster than the build off of berrinam's page, and it no longer follows the 9999 rule. It's fast and the quality is perfect. Freakin' sweet job.

[kurt]

Don't put commas between the optional flags (including bvop sensitivity), e.g:
-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG

thx, no errors in first pass. :)
(I have to test it also for second pass)

question about -nopacked:

if I choose matroska output, this writes a native stream. So I didn't have to put -nopacked in the line, right? (I don't want to use packet bitstream)

[squid_80]

question about -nopacked:

if I choose matroska output, this writes a native stream. So I didn't have to put -nopacked in the line, right? (I don't want to use packet bitstream)
Right. -packed/-nopacked change the way the encoder passes back data, but mkv output will (at least it should) be the same either way. Packed bitstream is off by default anyway unless you are outputting to an .avi file or force it on using -packed.

[kurt]

Right. -packed/-nopacked change the way the encoder passes back data, but mkv output will (at least it should) be the same either way. Packed bitstream is off by default anyway unless you are outputting to an .avi file or force it on using -packed.
thx for the info. Maybe you should update the -help parameter
(there is only -nopacked) :)

[JoeBG]

Hi,

I have seen a logfile from MeGUI with the following parameters:

-closed_gop : I thought encraw does not have this command. It only has noclosed_gop
-bitrate 4169000. I thought squid changed this. 4169 should be correct

@ squid

I´m a little bit confused now :) Is it possible to update the helpfunction in encraw. I use your tool nearly daily, would be nice to have the correct commands

[squid_80]

The options are there for compatability. You're not missing anything by not knowing them.

-closed_gop more or less does nothing, since closed gop is on by default. (I have experimented with open gop and strongly suggest it shouldn't be used at all.)
-bitrate can be in bits/sec or kbits/sec - if the number given is less than or equal to 10000 it is assumed to be in kbits/sec, otherwise it is assumed to be bits/sec. This makes it possible to replace the original xvid_encraw with this version and not have to change other programs like MeGUI.

encraw is long overdue for an update, but I've been busy with other things. There is a x64 build available now though (as well as an updated avisynth64 and lame64).

[JoeBG]

Thanks for your answer.


encraw is long overdue for an update, but I've been busy with other things. There is a x64 build available now though (as well as an updated avisynth64 and lame64).

Very good.

[squid_80]

BTW, I had a quick look through the german forum (I don't speak german) and if the log you're referring to is this one, (http://forum.gleitz.info/showpost.php?p=261774&postcount=3) I think the problem might be that -o "G:\HD.avi" will not create a valid avi file; it will create a raw m4v stream in a file called hd.avi. If that commandline is being created by MeGUI, it's broken (although I doubt that's the case).

[bond]

i think people dont really get that -o is not useable for every output filetype

maybe -o should be renamed to -raw, or -o can be made useable with all filetypes (detected by the extension)

[JoeBG]

BTW, I had a quick look through the german forum (I don't speak german) and if the log you're referring to is this one, (http://forum.gleitz.info/showpost.php?p=261774&postcount=3) I think the problem might be that -o "G:\HD.avi" will not create a valid avi file; it will create a raw m4v stream in a file called hd.avi. If that commandline is being created by MeGUI, it's broken (although I doubt that's the case).


Yes, and you solved the problem. I will post your answer there in german.

[stax76]

Would be great having CLI frontend and VFW frontend all in one package meaning the CLI frontend is incl. in Koepi's (and others) builds.

Another thing that would be great is building the CLI on top of COM+ automation, interfaces that also allow to utilize the native XviD GUI. It would be trivial then to use XviD without VFW from *any* programming or scripting language, lot's of great tools could be build on top of that and lot's of already existing application could use it.

[shon3i]

where i can find lastest xvid_encraw. Thanks

[Brother John]

Elic provided a link here (http://forum.doom9.org/showpost.php?p=801447&postcount=400).

As I have quite a bit of unused traffic allowance right now I've put the archive on my site too: http://brother-john.net. Let's see how much traffic that'll produce.

[henryho_hk]

Has the "-cartoon" switch gone?

[squid_80]

What build had a -cartoon switch? It is available via -zones.

[bond]

squid_80, when can we see a new build? and when can we see your sources (just in case you abandon encraw so your changes are not lost...)? :)

[squid_80]

Hmm.... I'll go out on a limb and say tomorrow night (roughly 24 hours). If anyone has any feature requests (apart from mp4 output, that's too much testing to do in such a short time) say them now and if they're easy enough to do you might get lucky.

[bond]

moo? ;)

[IgorC]

Maybe I lost something here but question :
Does 2 pass or multipass work? It seems to be broked since some time.

[squid_80]

bond: The source code is there, but my ftp server's not :(

IgorC: Can you be a bit more specific on why it doesn't work?

[IgorC]

Seems 2 pass didn't work because of RC limits 9999 frames.

[kurt]

Did you use squid's latest build? http://forum.doom9.org/showthread.php?p=800747#post800747

no problems here with 2pass and more than 9999 frames :)

[IgorC]

Kurt
Thank you. I will try it.
And 3 pass mode? How it works? How to set it?

[kurt]

Sorry, I didn't try 3pass yet.... (and can't try it for the next few days either)...

How to set it?
I guess for setting up a 3pass encode use the same commandline as for 2pass. Just change the numbers ...

[bond]

bond: The source code is there, but my ftp server's not :(i am sure someone would be willing to host it :)

[Adub]

yeah man, some one would be sure to post it if you emailed it to them or put it in a zip file on the forum.
I can't wait to play with it! Ha!

[ChronoCross]

I can host it. pm me if you want.

[squid_80]

I don't like having stuff hosted by other people if I don't have direct access to it i.e. username and password to be able to upload new versions.

Anyway, turns out my ftp server wasn't down at all, it was just full up with connections left open. I've since reduced the connection time-out to 5 minutes. Also after 2.5 weeks I've *nearly* finished compressing 124gigs of DVB recordings, getting back some much needed space - so now I can get back to work.

[ChronoCross]

well I can create a user account for you if need be if you need off site mirroring and whatnot. just like I did for soulhunter. He has ftp access to his directory and thus can do what he pleases. Whatever you decide.

[Adub]

Squid,
your server is working again. Which Xvid encraw is the latest? is just the one labled "Xvid Encraw.zip". If so, is it compatible with MeGUI, as the specific MeGUI version is?

[squid_80]

I think the megui_test build is the most recent (apart from the x64 build). If you run xvid_encraw -h it shows the build date and time at the top. The older build won't work with MeGUI because it expects bitrate in kbps.

[DarkZell666]

Heh, for a wierd reason, xvid_encraw_test_megui produces invalid files ?

I used the megui_test build, and the resulting file (which was the correct filesize) wasn't playable. MPC just said : couldn't render the file.

I tried muxing the file into an mp4 using YAMB with no success, same goes for demuxing ...
What can I do to get the file to work ? Is it an actual .avi or a (borked) raw stream ?

Note : Input was an avs script (obviously :p, with ConvertToYV12() at the end), and MeGUI generated the following cmdlines :

Pass 1 :

Starting job job1-1 at 22:55:55
Job is a video job. encoder commandline:
-i "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avs" -o NUL: -pass1 "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.stats" -bitrate 800000 -turbo -max_key_interval 300 -vhqmode 4 -closed_gop -max_bframes 4 -bvhq -bquant_ratio 160 -bquant_offset 200
successfully started encoding
Processing ended at 23:43:40


Pass 2 :

job job1-1 has been processed. This job is linked to the next job: job1-2
Starting job job1-2 at 23:43:40
Job is a video job. encoder commandline:
-i "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avs" -pass2 "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.stats" -bitrate 800000 -max_key_interval 300 -vhqmode 4 -closed_gop -max_bframes 4 -bvhq -bquant_ratio 160 -bquant_offset 200 -o "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avi"
successfully started encoding
Processing ended at 00:44:59


(and don't laugh about the pokemon bit, this DVD has such a shaky picture it's perfect for my encoding tests :p)

[squid_80]

You can't use MeGUI to generate avi files using xvid_encraw. The -o parameter is to output a raw stream - that's what "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avi" will be.

(Why MeGUI uses -o NUL: for the first pass is beyond me; encraw has never required an output file to be explicitly set and it just wastes time dumping data down a sink.)

I was thinking of writing a (mpeg4asp) raw to avi muxer, but bond would probably kill me. :devil:

[DarkZell666]

I tried muxing the file into an mp4 using YAMB with no success,
The -o parameter is to output a raw stream - that's what "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avi" will be.

Ok, after making my brain work a couple of seconds I renamed the .avi to .m4v and threw it at YAMB's face ... it worked ;)
I also tried .raw but it failed miserably ^^

It's wierd that YAMB (or mp4box, no idea) doesn't analyse the file but relies on the extension to determine if it is muxable or not.

Anyway thx for your help :)

[Kurtnoise]

Keep in mind that raw asp streams are stored in m4v files. That's all. So, it's completely useless to name or rename files with .raw extension.

and btw, analyse a file is not tricky because you must seek trough it. Thus, very time consuming in order to retrieve some infos, especially big files.

[squid_80]

Erm, did I just accidently break something or have the output .avi files ALWAYS had dwWidth and dwHeight swapped in their headers?

[bond]

heya squid_80. any news on the source (or a new build?)

[squid_80]

source: ftp://squid80.no-ip.com/xvid_encraw_src.zip
new build that should handle compressed (and raw colorspaces other than yv12) avi input files: ftp://squid80.no-ip.com/xvid_encraw.zip

(Also fixes the bug I mentioned above which no-one took any notice of.)

[Adub]

I was wondering about that bug.
Anyways, thanks for the new build! I can't wait to test this baby!
Laters,
Merlin Archangel

[stax76]

I can't download it: The server at squid80.no-ip.com is taking too long to respond

[buzzqw]

@stax

same problem here... waiting for better news :scared:

BHH

[squid_80]

@#$#cking firewall config has reset itself. See mirror at http://okejl.dk/dunstan

Edit: Main server should be back now.

[buzzqw]

quick direct link www.64k.it/andres/xvid_encraw-02-05-2006.zip

BHH

[stax76]

Are there cmdl examples for the most common uses? I find common use examples very helpful to learn a new CLI.

[Elic]

buzzqw > quick direct link www.64k.it/andres/xvid_encraw-02-05-2006.zip
Also http://elicit.vn.ua/files/hosted/xvid_encraw_src.zip, http://elicit.vn.ua/files/hosted/xvid_encraw-02-05-2006.zip

[squid_80]

Thank you for the mirrors. There's been about 120+ downloads in the past 24 hours, I thought there would only be a handful of people using it.

Command line examples
I decided this was a pretty good idea, since there's a couple of undocumented switches.
Here's my standard first pass command:
c:\>xvid_encraw -i final.avs -pass1 -packed -par 4 -bvhq -vhqmode 4 -progress -uses final.avs as input
-pass1 specifies first pass and since no log name follows, the default of xvid.stats is used.
-packed specifies packed b-frames mode, since my final target will be avi.
-par 4 means pal 16:9 pixel aspect ratio (not strictly necessary for first pass).
-bvhq means use VHQ for b-frames.
-vhqmode 4 means VHQ mode 4 - Wide Search.
-progress replaces the standard encoding status output with a minimal display which updates every 10 frames.

For 2nd pass, it's a bit more complicated:
c:\>xvid_encraw -i final.avs -pass2 -par 4 -bvhq -vhqmode 4 -progress -avi "vid final.avi" -size 179200 -overhead 665 -timecode mkvtimes.txt
-pass2 specifies 2nd pass using the default stats file name of xvid.stats
-avi "vid final.avi" means use vid final.avi as an output file.
-size 179200 specifies a target size of 179200 Kb (170Mb).
-overhead 665 = 665 bytes of overhead per frame. Default for avi is roughly 24, 128kbps audio + 25fps video = 640 bytes of audio per frame, 1 extra byte to be sure = 665 total. Better explanation is elsewhere in this thread.
-timecode mkvtimes.txt specifies mkvtimes.txt as a matroska v2 timecode file. If the output is an avi file, dropped frames will be inserted to sync the encoded frames to their timecodes. If the output is a mkv file, the timestamps will be copied thru (should be re-arranged correctly to suit b-frames, but I haven't tested much).

[stax76]

Thanks, when I've got a couple of hours time I'll make a TreeView based GUI for StaxRip.

[squid_80]

If it helps you any, -o will soon become the only option needed for output i.e. -o file.avi will create an avi file and -o file.mkv will create a mkv file. -o any_other_extension.xyz will still create a raw m4v stream.
-avi and -mkv will still be there, this is just to help out MeGUI users who keep getting stuck with raw m4v files which look like something else thanks to incorrect file extensions.

[Brother John]

Thx for the new version, squid_80. I've one question, though.

In your example above you mention -packed to enable packed bframes. Encraw's online help only mentions -nopacked to disable packed streams. No I'm wondering what is the proper way to en/disabled packed bitstream?

[SeeMoreDigital]

Thanks, when I've got a couple of hours time I'll make a TreeView based GUI for StaxRip.If it becomes your intension to incorporate this into StaxRip, by all means please let me know by PM and I'll be happy to test it for you before going live :)


Cheers

[squid_80]

In your example above you mention -packed to enable packed bframes. Encraw's online help only mentions -nopacked to disable packed streams. No I'm wondering what is the proper way to en/disabled packed bitstream?
Deja vu... (http://forum.doom9.org/showthread.php?p=803819#post803819) Packed bitstream is off by default unless outputting to an .avi file, which is not the case in my first pass. But as my second pass target is a packed .avi file, for best results the stats file figures SHOULD include the fake n-vops produced by packed bitstream. It's not absolutely necessary, filesize and quality will probably be more or less the same either way but I'm pedantic.

Also -packed was an option of the original xvid_encraw, hence it remains for compatability.

[SeeMoreDigital]

If say, I wanted to generate an encode with b-frames straight to MP4 (ie: packed bitstream free). How will the b-frames be ordered, in relation to the I and P frames?


Cheers

[squid_80]

Umm, however mp4creator does it? That's where the code is coming from.

[foxyshadis]

mp4 always has to be in jumbled mpeg order anyway, splitters would probably crash if you somehow shoehorned packed into mp4.


IPBBPBBPBIPPBPP
031264587902134

[SeeMoreDigital]

Umm, however mp4creator does it? That's where the code is coming from.
mp4 always has to be in jumbled mpeg order anyway, splitters would probably crash if you somehow shoehorned packed into mp4.


IPBBPBBPBIPPBPP
031264587902134
So to confirm... the B frame order for MP4 will not be the same as it is when they are hacked into AVI?


Cheers

[squid_80]

No, that is how they're stored in avi. Except in packed mode, the first B frame in a run is stored packed with the preceeding reference frame and an extra P (n-vop) is stored at the tail of the run.

Packed mode only exists for avi, for mkv and mp4 there is only one way to do things.

[SeeMoreDigital]

Thanks Squid.... :D

[Brother John]

Thanks Squid.... :D
Thx from me as well. Your answer saved me some boring testing. ;)

Btw: I've updated the enraw mirror on my site (http://brother-john.net) to the current version.

[bond]

Umm, however mp4creator does it? That's where the code is coming from.is xvid_encraw now supposed to support .mp4 output? if yes how?

[squid_80]

is xvid_encraw now supposed to support .mp4 output? if yes how?
Not yet. "That's where the code is coming from," means it's on its way but not quite here yet. But it would be just like me to release a new version and not say anything. :)

[SeeMoreDigital]

Not yet. "That's where the code is coming from," means it's on its way but not quite here yet. But it would be just like me to release a new version and not say anything. :)Great... I'm looking forward to an .MP4 output support :)

[bond]

:thanks:

[MatMaul]

I think I found a very minimal bug :
C:\Documents and Settings\MatMaul\Bureau\xvid_encraw>xvid_encraw -i "D:\lotr1\vi
deo.avs" -type 2 -pass2 -bitrate 2863 -max_bframes 2 -bquant_ratio 162 -bquant_o
ffset 0 -zones 0,w,1,O -quality 6 -vhqmode 4 -bvhq -qpel -qmatrix "SixOfNine.cqm
" -nopacked -imin 2 -imax 5 -pmin 2 -pmax 5 -bmin 2 -bmax 5 -chigh 10 -clow 3 -m
ax_key_interval 250 -progress -mkv "D:\lotr1\video\lotr1.mkv"
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
Input colorspace is YV12
xvidcore build version: xvid-1.2.0-dev
Bitstream version: 1.2.-127
Detected CPU flags: ASM MMX MMXEXT SSE SSE2 TSC
Detected 1 cpus, using 1 threads.
288651 frames(100%) encoded, 6.68 fps, Average Bitrate = -113kbps
Tot: enctime(ms) =43188075.00, length(bytes) = -163231213
Avg: enctime(ms) = 149.62, fps = 6.68, length(bytes) = -565
I frames: 3977 frames, size = 53748/213756687, quants = 2 / 2.33 / 3
P frames: 119693 frames, size = -12651/-1514325311, quants = 2 / 2.66 / 4
B frames: 164825 frames, size = 6900/1137336171, quants = 3 / 3.64 / 5
N frames: 155 frames, size = 8/ 1240

The avg bitrate reported by the log is negative.
The real bitrate of this encoded video is good, so it's not very important.

[squid_80]

Yes, the variables that hold the sizes are signed int32s so when the output grows larger than 2gb they wrap around to negative values. This throws out all the average calculations. The sensible thing to do would probably be for me to change them to uint64s.
The bitrate should be pretty spot on though, I'm guessing the final size was around 3940M? And the source has a framerate of 25fps?

[MatMaul]

yes the real final bitrate is good.

[berrinam]

Hi there squid_80,

Would it be possible to make the messaging system in xvid_encraw more informative? I have something in mind like how x264.exe does it:
x264 [error/warning/info]: message
where error means a fatal error, warning means a non-fatal error, and info means info. Then it would mean that a parser (like MeGUI) could tell whether the encode error'd out or not.

It could also be a good way of choosing the verbosity of messages: the user can choose which level of message they want in the commandline, and xvid_encraw.exe then only prints messages of that level or of more importance.

[squid_80]

Certainly. I've been meaning to do that for a while, the only reason I haven't changed any of the output messages was in case some programs used the original encraw and modifying the messages would break compatability. I'll go poke around in x264 and figure out what constitutes a fatal/non-fatal/info message so things match up nicely.

[acrespo]

I don't know if you saw the MEGUI bug report thread but I will post here again as I think that is a xvid_encraw problem. I receive this error when try to encode a avisynth script:

comand line:
-i "C:\encodes\ygh1.AVS" -pass1 "C:\encodes\ygh1.stats" -bitrate 606 -kboost 100 -overhead 0 -turbo -vhqmode 1 -closed_gop -imax 31 -max_bframes 2 -bmax 31 -threads 1

Result:
Trying to retrieve width and height from input header
Wrong Framerate 0.000000
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

I don't know what's wrong, because I can open this script in virtualdubmod and media player classic.

[squid_80]

I saw it, but I really need to know exactly what is in the avisynth script.

[acrespo]

Here is the script:


SetMemoryMax(64)
LoadPlugin("c:\encodes\filtros\EEDI2.dll")
LoadPlugin("c:\encodes\filtros\tdeint.dll")
LoadPlugin("c:\encodes\filtros\dgdecode.dll")
LoadPlugin("c:\encodes\filtros\tivtc.dll")
LoadPlugin("c:\encodes\filtros\dup.dll")
LoadPlugin("c:\encodes\filtros\RemoveDirt10SSE2.dll")
LoadPlugin("c:\encodes\filtros\RemoveGrain10SSE2.dll")
LoadPlugin("c:\encodes\filtros\RepairSSE2.dll")
LoadPlugin("c:\encodes\filtros\masktools-v2.0a28.dll")
LoadPlugin("c:\encodes\filtros\gradfun2db.dll")
LoadPlugin("c:\encodes\filtros\frfun7.dll")

import("c:\encodes\filtros\RemoveDust.avs")
import("c:\encodes\filtros\FastLineDarken_1.3.avs")

MPEG2Source("c:\encodes\2006-04-01\ygh2.d2v",cpu=6)

AssumeTFF()
Interp = SeparateFields().EEDI2(field=-2)
Deinted = TDeint(mode=2,edeint=Interp)
TFM(PP=7,Clip2=Deinted)
TDecimate(mode=1)

ColorYUV(levels="PC->TV")
Crop(8,8,-8,-8,align=true)
Spline16Resize(320,240)
RemoveDirt()
frfun7()

FastLineDarken2(thinning=0)
dup(threshold=1,blend=true,maxcopies=8)
gradfun2db()

str = "WAS Hub - dchub://wasbr.com - " + Time("%d") + "/" + Time("%m") + "/" + Time("%Y")
Subtitle (str,15,10,0,240,"Courier New", 14)
Subtitle ("WAS",258,202,0,300000,"Courier New",20)

[berrinam]

squid: Basically, the fatal errors in x264 are the ones in which there is no way to continue, like the video is stuffed or the log file is missing, etc. The warnings tend to be settings that are wrong for some reason, but can be fixed by x264, so the warning just tells you that it isn't encoding quite with the settings you asked for.

[squid_80]

acrespo: I can't find anything wrong. Can you do me a favour, add info() to the bottom of that script, open it in virtualdub(mod) and tell me what it says on the Frames per second line? Specifically what the numbers in the brackets are.

[acrespo]

Here is the exact information extracted from info() filter in avisynth:

Frame: 0
ColorSpace: YV12
Width: 320 pixels, Height: 240 pixels.
Frames per second: 23.9760
FieldBased (Separated) Video: NO
Parity: Assumed Top Field First
Video Pitch: 640 bytes
Has Audio: NO
Audio Channels: 0
Sample Type: NONE
Samples Per Second: 0
CPU detected: x87 MMX ISSE SSE SSE2 SSE3

[squid_80]

The only thing I can suggest is updating avisynth and dgindex/dgdecode; I think this might be related to the way avisynth used to give some framerates large numerator/denominator values.

[bond]

The only thing I can suggest is updating avisynth and dgindex/dgdecode; I think this might be related to the way avisynth used to give some framerates large numerator/denominator values.avisynth 2.5.7alpha doesnt do this anymore

[squid_80]

True. It also displays dwrate and dwscale via info(), which is what I really need because I can't reproduce this error. Closest I can get is a crash in xvidcore when dwscale is >2^31.

[sysKin]

I want to put it in CVS.
However, I was crazy enough to try to compile it first >_< and, well, it doesn't because it's trying to use c++ syntax in c file.

What do I do about IID_IAvisynthClipInfo to make it C? :)

[squid_80]

I just compile it as c++ :D
I did try to convert that code to C, but couldn't get it to work. However I'll try again and see how it goes.

[squid_80]

Looks like this works, but I dunno if it's completely correct. First define the interface inside the #ifdef XVID_AVI_INPUT block:

#ifdef XVID_AVI_INPUT
/* Avisynth related */
const GUID IID_IAvisynthClipInfo // {E6D6B708-124D-11D4-86F3-DB80AFD98778}
= {0xe6d6b708, 0x124d, 0x11d4, {0x86, 0xf3, 0xdb, 0x80, 0xaf, 0xd9, 0x87, 0x78}};

#undef INTERFACE
#define INTERFACE IAvisynthClipInfo
DECLARE_INTERFACE_ (INTERFACE, IUnknown)
{
STDMETHOD (QueryInterface) (THIS_ REFIID, void **) PURE;
STDMETHOD_ (ULONG, AddRef) (THIS) PURE;
STDMETHOD_ (ULONG, Release) (THIS) PURE;
STDMETHOD_ (INT, GetError) (THIS_ char**) PURE;
STDMETHOD_ (LONG, GetParity) (THIS_ INT) PURE;
STDMETHOD_ (LONG, IsFieldBased) (THIS) PURE;
};
#endif

(Bool wouldn't work, so I used LONG for the return type of GetParity and IsFieldBased. Shouldn't be relevant, they're not used here.)
Then change the avisynth checking stuff to match the new interface:

if (FAILED(avi_file->lpVtbl->QueryInterface(avi_file, &IID_IAvisynthClipInfo, (void **)&pAvisynthClipInfo)))
pAvisynthClipInfo = NULL;
else {
const char *s;

if (pAvisynthClipInfo->lpVtbl->GetError(pAvisynthClipInfo, &s)) {
fprintf(stderr, "xvid [error]: Avisynth open failure:\n%s\n", s);
pAvisynthClipInfo->lpVtbl->Release(pAvisynthClipInfo);
AVIFileRelease(avi_file);
AVIFileExit();
return (-1);
}
}


if (pAvisynthClipInfo != NULL)
pAvisynthClipInfo->lpVtbl->Release(pAvisynthClipInfo);

Then there were a few spots where I've declared and assigned variables in the middle of code which plain C doesn't like. Easy enough, just move the declarations up.

[sysKin]

Looks like this works, but I dunno if it's completely correct.
Well, it didn't work :/

I removed the avisynth error catching and commited the file.

I officially declare this is the ugliest piece of code in XviD's CVS. It's technically incorrect (static variables), hackish to the bone, and probably not cross-platform at all.

It needs an urgent rewrite. If doing it properly requires xvid interface changes (wrt packed bitstream and userdata) then so be it. Better that than hackish removal of userdata from encoded frames.

Volunteers?....

[shpitz]

i'm using latest build and for some reason it always says detected 0 cpus and using 0 threads.

if i force -threads 2 or 4 it will still be using only 1 cpu as my cpu usage is stuck on 25%.

i have a dual xeon 3ghz with HT enabled.

Detected 0 cpus, using 4 threads.
701 frames( 31%) encoded, 7.19 fps, Average Bitrate = 14231kbps

that's 1st pass, 2nd pass goes down to 3fps.

i'm using the batch file made by henry_hk in the settings thread by teegedeck.

any ideas about what i'm doing wrong?

[SeeMoreDigital]

any ideas about what i'm doing wrong?It would be handy if you could confirm the resolutions you are encoding to!!!

[shpitz]

oh, sorry, i was encoding a 720p clip to a 720p output resolution.

what was suspicious to me was that encraw reported 'xvidcore build version: xvid-1.1.0'

so i figured it must be using an old version of xvid that is not multithreaded.

i installed a newer build and now it is working great, getting 2x the speed, and # of cpus is detected properly now.

so that's all it was, using an older version of xvidcore.dll .

i will do another test with 2 threads, right now i'm doing with 4 threads.

[mod]

Hi all.
I'm not able to limit the number of frames encoded. Arguments:

xvid_encraw -i pippo.avs -type 2 -frames 200 -custom_par 80 81 +chroma_opt -quality 6 -max_bframes 1 -framerate 25 -bitrate 2000 -pass1 pippo.log -iquants 2 31 -pquants 2 31 -bquants 2 31 +trellis -lumimasking -vhq 4

it simply doesn't stop at frame 200. Am I making some error?
Thanks!

[buzzqw]

Hi mod !

first of all: you are using the old sintax...
second: latest build of xvid_encraw doesn't have -frame parameter... just -start for specifying the START frames...

i can suggest to use zones... to lower quant

BHH

[mod]

Really thanks for the info.
I'm using the version linked a 1st page, and the syntax is the one I got with

xvid_encraw -help ..

EDIT: I found this (http://forum.doom9.org/showthread.php?p=800912#post800912) one and supports -frames. Is this the latest?

I'm really sorry if I missed some link.

[kurt]

latest is this (AFAIK): http://forum.doom9.org/showthread.php?p=822253#post822253

[buzzqw]

please mod ... don't complain !

i don't know older built fupporting this features... but... may i suggest to use the TRIM feature in your avs script ?

BHH

[SeeMoreDigital]

Excuse for my ignorance but I have to ask....

Is one of the goals of XviD EncRAW to create an non VfW, ie: CLI version of XviD?


Cheers

[imcold]

yes /I believe :D

[buzzqw]

yes ! for sure... but what's the matter ? :confused:

BHH

[john27]

small bug:
#include <ctype.h> is missing //toupper(c)

[JoeBG]

Excuse for my ignorance but I have to ask....

Is one of the goals of XviD EncRAW to create an non VfW, ie: CLI version of XviD?


Cheers

Yes...and it works great. I use it for all my xvid encodes. You can create a proper raw or *.mkv stream without packet bitstreams.

[shpitz]

Yes...and it works great. I use it for all my xvid encodes. You can create a proper raw or *.mkv stream without packet bitstreams.

Joe, and your encodes have no issues with fast forwarding (seeking)? if so what do you use to mux the encoded video with the audio?

[SeeMoreDigital]

Yes...and it works great. I use it for all my xvid encodes. You can create a proper raw or *.mkv stream without packet bitstreams.That's encouraging news :)

I wonder if Stax can code it into StaxRip before he abandons it altogether?


Cheers

[foxyshadis]

Is it possible to split this thread from page 10 or so, or better have a mod edit the first post to include some information on getting current versions of encraw?

[mod]

I made a little GUI for xvid_encraw, maybe someone finds it useful.

here (http://forum.doom9.org/showthread.php?t=112808)

I use it for SAP, so not really a lot of options (ie only avi output), but if somene wants I can add what needed, no problem :D

In the next days I'll try to write something in english, but it's really easy, doesn't need any real explanation..