xvid_encraw - Patched with AviSynth input support

[squid_80]

I'm not talking about the container, I'm talking about what's in the raw m4v stream - specifically the first 3 or 4 bytes after the vop start code. For example here's a frame I picked at random:
00 00 01 B6 51 60 54 1D....
00 00 01 B6 = vop start code. Now we break the stream down into bits. 51 60 54 =
01 0 1 00010 1 10000001010100
01 = vop type (P)
0 = seconds. Not really seconds, kinda got something to do with fbase. The value is calulated by counting the number of preceding 1s before the 0.
1 = marker bit.
00010 = ticks, measured in fbase units. So for each successive frame, ticks increases by one. It rolls back to 0 when it reaches fbase, and the seconds field is incremented. But then the seconds field goes back to 0 after a few frames... the exact point at which it does seems to be max_bframes dependant.
1 = vop coded, the rest of the bits are to do with the encoded frame. (If it's not coded there's just padding up to the next byte.)

The width of the ticks field varies, it is equal to the base 2 log of fbase-1 (my maths is rusty, can't remember how to write that correctly as text). That's why the size of n-vops can vary depending on the framerate (the seconds field also varies).
What I would like to know is what a decoder is allowed to do with these values and what effect (if any) there would be if there was a sudden jump forwards/backwards in ticks.

[Zero1]

I tried encraw again since the last time I had problems (when I posted the screenshot). It works fine now, no visual errors at all ;) The command line and switches are pretty straight forward too.

Great work and thanks to everyone involved, it's great to be able to totally bypass VfW and produce native, spec compliant encodes.

Just a couple of questions, are there options within the CLI that are, or could be accessible, which would help improve the quality (over the VfW counterpart). I might be wrong, but aren't there some other search modes or anything in there that aren't accessible by the VfW?
Also, is it much hassle to update it's internals to something like 1.1.0 final, or maybe even the current 1.2? Even if the multithreading does not work, maybe the trellis improvments would be handy ;)

Cheers

[squid_80]

Just a couple of questions, are there options within the CLI that are, or could be accessible, which would help improve the quality (over the VfW counterpart). I might be wrong, but aren't there some other search modes or anything in there that aren't accessible by the VfW?
The way the quality and vhqmode options work is the best way to do it IMO; they group together search modes of the same approximate quality. I don't think it would be an improvement to allow individual modes to be (de)selected. If anyone used S_O's build and thought otherwise, feel free to mention it.
Also, is it much hassle to update it's internals to something like 1.1.0 final, or maybe even the current 1.2? Even if the multithreading does not work, maybe the trellis improvments would be handy ;)
It uses whatever version of xvidcore.dll you have installed. If that's the current version then you're already using the improvements.

(note to self: add output message to show the version of xvidcore.dll being used, if possible.)

[squid_80]

New build up. Actually there was a newer build up over 24 hours ago, I just updated it again. Nothing real groundbreaking:
-size option (help text is formatted wrong :( )
-pass1 and -pass2 can be used at the same time even though I couldn't get any improvement from nth pass
-interlaced option has support for TFF
-threads option is implemented and should work when support is added to xvidcore
- information about detected xvidcore version is displayed
-progress option, which I added in about 2 minutes because I was sick of watching the scrolling text output. It's a bit buggy.
-vfr option (not in help text) which should drop n-vops from mkv output streams and make the stream vfr. Haven't played with it much but seems to work.

Edit: Forgot that I also fixed zone handling to sort zones properly.

[shon3i]

Where i can download this lastest version.

[buzzqw]

link is same as usual ftp://squid80.no-ip.com/xvid_encraw.zip

or my (tiny) mirror www.64k.it/andres/xvid_encraw-23022006.zip

BHH

[shon3i]

link is same as usual ftp://squid80.no-ip.com/xvid_encraw.zip

or my (tiny) mirror www.64k.it/andres/xvid_encraw-23022006.zip

BHH
Thanks

[bond]

pretty much a small revolution having a xvid vfr encoder

thx squid!

[bond]

btw for stressing the vfr option via the drop option you should make sure to use a recent 1.2 xvid compile, cause the older ones (eg official 1.1) have a bug with n-vops

[unplugged]

Does this recent xvidenc_raw is compatible with current MeGUI?

[squid_80]

Not as far as I know since I broke the -bitrate parameter (whoops). Might be possible to make a fix - I could assume any value > 10000 is in bps and any value <= 10000 is in kbps. Does anyone regularly use bitrates less than 10kbps or greater than 10 mbps?

[squid_80]

btw for stressing the vfr option via the drop option you should make sure to use a recent 1.2 xvid compile, cause the older ones (eg official 1.1) have a bug with n-vops
Is that the n-vop with unpacked bitstream bug that I fixed? That shouldn't affect the vfr stuff, I though it just mucks up the stats file and bitrate distribution.

[Doom9]

Does this recent xvidenc_raw is compatible with current MeGUI?Not now, but the next stable megui will be.. it has no release date and will be available when it's done.

@squid: when I saw -overhead integer : container frame overhead (24) I wanted to run away screaming.. did you just reintroduce all the mencoder problems or can I make it so that encraw doesn't give a flying hoot about container knowledge? If I give it a bitrate of 700, I want my output video stream to have a bitrate of exactly 700 kbit/s.. and the file having a larger size than length * 700 kbit/s because of the container overhead. So my raw stream would have exactly the desired size, my avi and mkv would be bigger. It would be a shame if different calculation mechanisms would have to be reintroduced everywhere because xvid is so tied to avi. can I force -overhead 0 and never have to worry about container overhead ever again? (well, megui does.. but I have generic calculations for that.. I don't want codec specific ones)

[squid_80]

If the passed bitrate is what you want the raw stream to be, then use -overhead 0. I only made the default=24 because it's better to get an undersized file rather than oversized.

I actually abuse it and use -overhead 665 -size 143360 to get an output avi that I can mux with matching 128kbps mp3 audio and end up with 140megs :) If I had an automated reliable way of cutting out commercials, backing up DVB captures would almost be too easy...

[bond]

Is that the n-vop with unpacked bitstream bug that I fixed? That shouldn't affect the vfr stuff, I though it just mucks up the stats file and bitrate distribution.that bug lead to wrong target bitrates and different, wrong frametype decisions too

a fixed 1.2 compile is available here (http://rapidshare.de/files/14009418/xvidcore.dll.html)

[Doom9]

fyi, on the way from the office I added support for the latest encraw to megui.. I just have to solve a few crucial issues first before making another release, and then refactor some more.

[Beave]

If the passed bitrate is what you want the raw stream to be, then use -overhead 0. I only made the default=24 because it's better to get an undersized file rather than oversized.

I actually abuse it and use -overhead 665 -size 143360 to get an output avi that I can mux with matching 128kbps mp3 audio and end up with 140megs :) If I had an automated reliable way of cutting out commercials, backing up DVB captures would almost be too easy...

This sounds great. How can I calculate that? Say I want to mux a 384 kbit/s ac3 file?
Thanks for all the great work on xvid_encraw, btw!

[squid_80]

This sounds great. How can I calculate that? Say I want to mux a 384 kbit/s ac3 file?

Just to be clear, there's much better ways of doing this. I'm just lazy.

Divide the audio bitrate by 8 to get bytes/sec (384000/8 = 48000).
Divide that number by the framerate to get bytes/frame (48000/25 = 1920, or 48000/23.976 = 2002, or 48000/29.97 = 1602).
Add avi overhead (24) plus 1 (1920 + 24 + 1 = 1945).
The resulting output file should be pretty close to -size once the audio is muxed in.
It might not fit all audio formats but should work for CBR mp3 and ac3.

[Beave]

Just to be clear, there's much better ways of doing this. I'm just lazy.

Divide the audio bitrate by 8 to get bytes/sec (384000/8 = 48000).
Divide that number by the framerate to get bytes/frame (48000/25 = 1920, or 48000/23.976 = 2002, or 48000/29.97 = 1602).
Add avi overhead (24) plus 1 (1920 + 24 + 1 = 1945).
The resulting output file should be pretty close to -size once the audio is muxed in.
It might not fit all audio formats but should work for CBR mp3 and ac3.

Well, I find your way of specifying the overhead great. This way I don't need to calculate the length of the movie nor getting/calculating the filesize of the ac3. All I need to know is the framerate and the audio bitrate.

Thanks for your great input.

One question:
Why do I need the parameter -framerate? Can't xvid_encraw get that framerate info from the avs/d2v automaticly? If I don't use that -framerate switch, will my 23.976 video be converted to 25 fps?

I did a full encode, but I got a totally undersized file. I can't find the problem though:

Starting 1st Pass...

"xvid_encraw.exe" -i "test.avs" -avi "test.avi" -pass1 "test.stats" -size 1362413 -turbo -max_key_interval 240 -qmatrix "HVS Best Picture.qmatrix" -bvhq -framerate 23.976 -zq 0 22 -zw 151 0.6 -zw 2334 1 -zq 80422 20

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
xvidcore build version: xvid-1.1.0
Bitstream version: 1.1.0
Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
Detected 0 cpus, defaulting to 1 thread
83001 frames(100%) encoded, 8.28 fps, Average Bitrate = 3030kbpss
Tot: enctime(ms) =10022143.00, length(bytes) = 1311303606
Avg: enctime(ms) = 120.75, fps = 8.28, length(bytes) = 15798
I frames: 1178 frames, size = 151918/178959925, quants = 1 / 1.29 / 22
P frames: 32472 frames, size = 1515/49198004, quants = 1 / 1.61 / 22
B frames: 49350 frames, size = 21948/1083145677, quants = 2 / 3.02 / 31

Starting 2nd Pass...

"xvid_encraw.exe" -i "test.avs" -avi "test.avi" -pass2 "test.stats" -size 1362413 -max_key_interval 240 -qmatrix "HVS Best Picture.qmatrix" -bvhq -framerate 23.976 -zq 0 22 -zw 151 0.6 -zw 2334 1 -zq 80422 20

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
xvidcore build version: xvid-1.1.0
Bitstream version: 1.1.0
Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
Detected 0 cpus, defaulting to 1 thread
83001 frames(100%) encoded, 10.91 fps, Average Bitrate = 1160kbps
Tot: enctime(ms) =7608925.00, length(bytes) = 501897660
Avg: enctime(ms) = 91.67, fps = 10.91, length(bytes) = 6046
I frames: 1178 frames, size = 22479/26480984, quants = 2 / 7.17 / 22
P frames: 32472 frames, size = 10904/354106754, quants = 2 / 5.70 / 24
B frames: 49350 frames, size = 2458/121309922, quants = 4 / 8.28 / 31

Can someone detect the problem? The filesize should be about 1330 MB, but I only got 480 MB unfortunatly.

[Doom9]

All I need to know is the framerate and the audio bitrate.
Actually.. you need to know a lot more.. video overhead for the given container, audio overhead for the given container, subtitle overhead for the given container ;)

[dimzon]

@Beave
Seems like your source is clear enought to fit it into 480MB ;)
In this case you will obtain perfect 1CD rip instead of 2-cd

[bond]

squid_80, when will your code be ready for cvs commit? ^^

[Beave]

@Bond
No way, sorry. This is a 720p encode and plenty noise in it. It's an episode of a series which I encoded before with VirtualDubMod. If I encode it in Quant=2 I get around 1800MB.

Now I did the second pass again changing -size 1362413 to -bitrate 3221. This time I got the right size. So there must be something wrong with the -size setting.

[squid_80]

One question:
Why do I need the parameter -framerate? Can't xvid_encraw get that framerate info from the avs/d2v automaticly? If I don't use that -framerate switch, will my 23.976 video be converted to 25 fps?
Who says you always need to specify -framerate? It's only needed when using raw input or if you want to change it.

I did a full encode, but I got a totally undersized file. I can't find the problem though:

Starting 1st Pass...

"xvid_encraw.exe" -i "test.avs" -avi "test.avi" -pass1 "test.stats" -size 1362413 -turbo -max_key_interval 240 -qmatrix "HVS Best Picture.qmatrix" -bvhq -framerate 23.976 -zq 0 22 -zw 151 0.6 -zw 2334 1 -zq 80422 20

83001 frames(100%) encoded, 8.28 fps, Average Bitrate = 3030kbpss
Tot: enctime(ms) =10022143.00, length(bytes) = 1311303606
Avg: enctime(ms) = 120.75, fps = 8.28, length(bytes) = 15798
I frames: 1178 frames, size = 151918/178959925, quants = 1 / 1.29 / 22
P frames: 32472 frames, size = 1515/49198004, quants = 1 / 1.61 / 22
B frames: 49350 frames, size = 21948/1083145677, quants = 2 / 3.02 / 31
That command line doesn't seem to quite match, there's a progress report but no -progress option, and you've got quants < 2 but no -imin/pmin/bmin...

Even with an average quantiser close to 1, it's undersized.
Try in order:
Use -qtype 1 so the custom matrix is actually used (someone found a bug; yay!).
Turn off turbo.
Drop the zones.
Use 1 max bframe or even perhaps 0.
If it's still undersized set the minimum quants to 1. Also maybe update to 1.2.0...
Now I did the second pass again changing -size 1362413 to -bitrate 3221. This time I got the right size. So there must be something wrong with the -size setting.Redo both first and second pass with -qtype 1. Otherwise you're just wasting space.
Looks like you're right about -size though - I think any value > 262144 will bork. Easily fixed.
squid_80, when will your code be ready for cvs commit? ^^
When it's not buggy anymore... Remember when I said it was a bad idea to run 2 copies at once? Last week I ignored my own advice and ended up with a movie that changed PAR about one third of the way through. Still can't recreate the situation either.

[woah!]

@Bond
No way, sorry. This is a 720p encode and plenty noise in it. It's an episode of a series which I encoded before with VirtualDubMod. If I encode it in Quant=2 I get around 1800MB.

Now I did the second pass again changing -size 1362413 to -bitrate 3221. This time I got the right size. So there must be something wrong with the -size setting.


did you try -size 1330

[Beave]

That command line doesn't seem to quite match, there's a progress report but no -progress option, and you've got quants < 2 but no -imin/pmin/bmin...

Yes, you are right about one part. I added the -progress 10 on the cmd window, but did not update it in my batch file. I definately did not use any imin/pmin/bmin though. I'm 100% sure about it, I was wondering myself about the quants=1. Since the filesize was off and fps didn't quite match my normal fps I didn't pay too much attention to this.


Try in order:
Use -qtype 1 so the custom matrix is actually used (someone found a bug; yay!).
Turn off turbo.
Drop the zones.
Use 1 max bframe or even perhaps 0.
If it's still undersized set the minimum quants to 1. Also maybe update to 1.2.0...

So in the future releases -qtype 1 will not be nessesary to use a -qmatrix, right? So for the next try I will put them right after each other:

-qtype 1 -qmatrix "file.qmatrix"

in hopes it will use the CQ and not the MPEG Matrix.
Using -bitrate 3221 gives me a perfect sized result.
Your other comments where only valid with the wrong size in mind, right?

Redo both first and second pass with -qtype 1. Otherwise you're just wasting space.

I don't get this. Why am I wasting space?

Looks like you're right about -size though - I think any value > 262144 will bork. Easily fixed.

That's good to hear it's easily fixed.

did you try -size 1330
Why should I use an even smaller number if the file is already undersized?

[squid_80]

So in the future releases -qtype 1 will not be nessesary to use a -qmatrix, right? So for the next try I will put them right after each other:

-qtype 1 -qmatrix "file.qmatrix"

in hopes it will use the CQ and not the MPEG Matrix.
Using -bitrate 3221 gives me a perfect sized result. Yes, specifying a custom matrix should turn on MPEG quantization but it didn't. Fixed in next build.
Your other comments where only valid with the wrong size in mind, right?They were things to try if it still comes out undersized after the custom matrix is used.
I don't get this. Why am I wasting space?Because it used h263 quantization, using the same parameters with -bitrate 3221 is just going to lower the (already quite low) quantizers for no visible gain. You're telling it to use a bitrate higher than what your 1st pass ended up with.

[squid_80]

Yes, you are right about one part. I added the -progress 10 on the cmd window, but did not update it in my batch file. I definately did not use any imin/pmin/bmin though. I'm 100% sure about it, I was wondering myself about the quants=1. Since the filesize was off and fps didn't quite match my normal fps I didn't pay too much attention to this.
Do you still have the stats file from the 1st pass? I'd like to know where the quant1 frames are, it's probably a zones screw up since any frame not in a quant zone should get quant=2 during first pass.

[Kopernikus]

Hi,

is it possible to get the a copy of the sources?

Have you thought about sending a patch to the mailinglist? BTW, is your 64bit stuff already in CVS?

[squid_80]

Actually.. you need to know a lot more.. video overhead for the given container, audio overhead for the given container, subtitle overhead for the given container ;)
You also need to know it doesn't work for vfr encodes. Wish I'd realized that about 10 hours ago. :angry:

[bond]

any news? ^^

[squid_80]

any news? ^^
Yeah. Instead of doing what I should be (reorganizing the code so it's nice and neat, fixing the >2gb avi output problem, adding mp4 support etc.) I've been playing around with dedup making vfr encodes... Made a new parameter -timecode <file> which takes a mkv timecode file and adds dropped frames to the output avi to simulate vfr. You'd think it would be simple to add it to the mkv output module (since timecodes are pretty simple in mkv), but it's not since the timecodes are in display order and the frames are written in encoded order. Just got to figure out the most sensible way to buffer the times until they're written.

[JoeBG]

Any news for the mp4 output? :)

[Zero1]

Why MP4 output? MP4box handles it fine.

[bond]

Why MP4 output? MP4box handles it fine.because direct .mp4 output (maybe even with vfr) without needing to remux would rock much more :D

[squid_80]

It looks like code from mpeg4ip will work out pretty well. I'm just trying to keep it all neat and tidy and trimmed down by cutting out parts of code that aren't needed.

[dimzon]

It looks like code from mpeg4ip will work out pretty well. I'm just trying to keep it all neat and tidy and trimmed down by cutting out parts of code that aren't needed.
Wow, fine news.

[fight2win]

i tried xvid_encraw in megui, it said that the options i choose in encoding not supported!

[buzzqw]

you can try my application

BHH

[JoeBG]

i tried xvid_encraw in megui, it said that the options i choose in encoding not supported!

Because MeGui does not support the commands from xvid_encraw. MeGui uses mencoder commandlines for xvid_encraw. The support for xvid_encraw is planned, but not finished.

Mkv Magic does the job very good :)

[Doom9]

@JoeBG: not true. Public megui releases support the CSV builds of encraw.. but not squid's custom builds. A yet unreleased version will support squid's build.

[JoeBG]

@ doom9

OK, sorry, misunderstanding

[bond]

It looks like code from mpeg4ip will work out pretty well. I'm just trying to keep it all neat and tidy and trimmed down by cutting out parts of code that aren't needed.great stuff! will vfr also work?

[snherbst]

@bond

because direct .mp4 output (maybe even with vfr) without needing to remux would rock much more :D

I thought that this prog only could encode video content and not audio content, or am I very wrong?

As I see it the benefit in getting a MP4 output file vould be that the frame rate is part of a container content.

[Beave]

@squid_80

I am wondering if you fixed some of the bugs in xvid_encraw yet and could maybe provide some sort of beta version? I'm especially interested in using the -size parameter with bigger file sizes.

[squid_80]

Just posted this in another thread: http://okejl.dk/dunstan/xvid_encraw_megui_test.zip

I *think* I fixed that bug by reordering the size->bitrate equation to prevent overflow. I might be mistaken since I can't remember what it used to look like. Try it and see.
This build may contain unfinished code too, since it's my work-in-progress build. In particular, make sure the input is raw YV12 (you might get away with a compressed avi that decompresses to YV12 by default) because it won't throw an error if it's not - you'll just end up with a) garbage b) a cropped picture or c) a crash :)
For the adventurous, you can try making a packed .avi output file using -timecode <file> where file is a matroska v2 timecode file (like the one dedup spits out, for example). The .avi file should have dropped frames inserted to make up the time differences between frames.

[Elic]

squid_80
> Just posted this in another thread: http://okejl.dk/dunstan/xvid_encraw_megui_test.zip
Oops... Server seems to be offline (ping has time out). Are there another copies, please?

[squid_80]

Not unless anyone is able to mirror it. I had to take my ftp server down about a month ago (ftp://squid80.no-ip.com), the okejl.dk webspace is from a 1024/512 ADSL connection in Denmark, provided by someone who offered a home to my 64-bit stuff (thanks Asbjørn!). He did say it was only temporary so if it's down, it's down - He's been very generous and I'm not going to bug him about it.

[woah!]

i have some gigs to shares if you need a host.

i just need the file first

[Elic]

I had to take my ftp server down about a month agoOK, I have noticed that http://okejl.dk/ is online again. And I dowloaded your file and uploaded it into http://elicit.vn.ua/files/hosted/xvid_encraw_megui_test.zip, just to be on the safe side, OK? :)