xvid_encraw - Patched with AviSynth input support

[squid_80]

You're right; in my builds I made the default framerate 0 so it would report an error instead of assuming 25fps, since I found this to be undesirable behaviour. The code in xvid's CVS has reverted to a default of 25fps.

[henryho_hk]

squid_80, shouldn't it detect the framerate from the source?

[squid_80]

It should, but making 25fps the default breaks it. The intended mode of operation (as I coded it) is like this:

Set framerate=0 (default)
Check command line for -framerate parameter. Adjust framerate to given value if found.
If framerate==0, set it to the framerate of the input source if possible (not possible if input is stdin or pgm).
If framerate==0, show an error and abort.

[henryho_hk]

Yeah, that version crashes if I put -framerate 0

[Brother Darrell]

So far I have managed to create several -avi files with no issue. Still can't get the W/H to be seen in mencoder if original file is a raw stream. BUT!, moving on...Avi is wonderful. I LOVE the fact I can do multiple 1st passes WHILE doing a second pass. :D ..
I am an n-pass junkie, sad but true.
A question though. I don't see an option for B frame sensitivity. Is there such an option & I am just missing it?

[kurt]

it's part of the zones, eg
-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG
--> first zone got a bframe sens. of 25, 2nd 0, 3rd 25 again...

[Brother Darrell]

-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG

I don't know this switch.
so...-zones
1=the zone or starting frame
w(weight) ,(ok to put a delimiter here?).05,250 (I noticed you used an 'oh', instead of 'zero' but it is zero, correct?)
And to use it globally, I simply change the weight to 1 and leave it as is. -zq I guess will work as well as -zw.
Thanks! I'll give it a try.
Oh. 1 more thing...Why is -stats suppressed (if being written to a log file) if the -progress switch is used?
Like this: -stats>> "stats.log" -progress 10

Same result regardless of where the -progress switch is located. ie: before the -stats switch or after the -stats switch..doesn't matter. It will still not output. Take the -progress switch out though & my log file is produced.

[squid_80]

The -zones parameter works like described here: http://forum.doom9.org/showthread.php?p=702759#post702759
except there's no parameter for the end frame; a zone runs until the next zone or the end of the stream. In kurt's example it is in fact the letter "O", for chroma optimizer.
As for the stats/progress thing... -progress turns off all frame-by-frame output, otherwise there's not much point in having it (unless stdout is redirected, like you're doing). If you really want the progress update and stats to happen at the same time I guess I could add an optional filename parameter for -stats, to specify the file to output to.

[Brother Darrell]

The only reason I was wanting a stats output was because of the PSNR values. It's just something for me to play with, not something essential.
If it's not alot of trouble to do so then YES! I would LOVE to have that feature incorporated! I would buy you many cold beers for the effort! That is, if you are open to such blatant bribery.

:thanks:
To you & Kurt for the info about zones. The only switches I know are the ones from -help.
Are there any other switches I can't get from the -help option?

[Brother Darrell]

I found them listed in one place.
For anyone else is having an issue finding the parameters not listed with -help..
look here->http://forum.doom9.org/showthread.php?p=784465#post784465
It includes syntax and options.

[Brother Darrell]

Is there an alternate way to set chapter points on the 1st or 2nd pass without using the -zones parameter to force an I-frame? Or editing the pass file directly?

Lets say I have 40 chapter points. That's ALOT of typing!
So I was hoping there was an easier way, such as feeding (porting?) a text file with the frames needed, into the command line, forcing an I-frame where I think I need it.
I realize MkvToolnix can mux an xml or ogg file into the finished product..but what if the chapter point falls on a b-frame? Heck...what if I just want to split the stream at the chapter point(s)?
For now, I'll either do it with the -zones or directly edit my pass file.
I can't help but think there must be (or should be) an easier/proper way. Too much to hope for?

[Sharktooth]

The chapter start is always set at the beginning of a new scene and the codec will almost likely detect the scene change and place an I-Frame.
MeGUI supports chapters creation and muxing, so it should be ok as soon as this version of xvid will be supported (not until the patches go official...)

[Brother Darrell]

Does anyone recognise this specifically?
.0 0. ax + + 1 - bn

I was running a batch file using Encraw with the following command line:

xvid_encraw -i "Versus.avs" -pass2 "5th.pass" -max_key_interval 0 -qmatrix matrix.txt -framerate 23.976 -max_bframes 120 -bquant_offset 000 -bquant_ratio 162 -imin 2 -imax 3 -pmin 2 -pmax 3 -bmin 3 -bmax 5 -nopacked -quality 6 -vhqmode 4 -avi "VsBframes2.avi" -size 1706540 -pass1 "6th.pass" -stats>>"6thpass.stats" -progress 10.

I started running it in a command window when I left home this morning. When I got home, the file was finished, but the above er..characters appeared immediately afterwards, as though it was part of my original command line, with the exception of course of XP not recognising anything as a batch process or command...The last part however, -bn, did seem to process, at least XP didn't say "not recognized as an internal or external command,operable program or batch file".
Any clues? Did I break something?

Edit: sorry, the matrix file I used was HvsBest.txt under the name Matrix.txt

[squid_80]

If the command window has focus while processing and you type something, it will appear at the c:\> prompt when processing finishes. Is that what happened?

[Brother Darrell]

That's entirely possible. I didn't think about that. So something I was running at the time may have simply dumped garbage. Maybe something that happens all the time but would normally not be seen. That sounds reasonable. I was concerned because it looked a little like code. As far as me typing something while the command window has focus, It certainly would not have been that.

[henryho_hk]

was that something in your batch file?

[Brother Darrell]

Not explicitly. I don't even recognize the -bn as a parameter I use for anything. This all appeared after the batch completed.
I would have blown it off as a catwalk over the keyboard, except my keyboard was on top of my monitor. My cat is FAR too old to jump 5 feet onto the top of my monitor, land on the keyboard, type something for the ax register and jump off without knocking down the keyboard.
(Ok, maybe she can...but there is still the fact she never passed her computer basics class...waste of money, that).
Hell.. I don't know what it was. But no harm seems to have come to my file or computer. So I ain't gonna sweat it. I just wanted to make sure it didn't have to do with the batch file doing something Xvid_encraw didn't like.

[henryho_hk]

Is there an alternate way to set chapter points on the 1st or 2nd pass without using the -zones parameter to force an I-frame?

I have made a batch file implementing Teegedeck's quality presets as well as zone input thru a plain text file. Please refer to http://forum.doom9.org/showthread.php?p=874332#post874332
BTW, there is basically no documentation :p :p :p .

[henryho_hk]

squid_80, I found that zone quant. settings seem to be reseting the b-frame max-consec., ratio and offset settings. I need to put the zone parameters before the b-frame parameters so as to use something like 1/1.62/0. Is it intended?

[Champs]

Hi, first of all thank you squid_80 for taking over and improving this cli program, I really like it a lot.

I've spent a few days going through every single page of this thread and found several posts by squid_80 and others about encoded outputs set to 25fps without setting the fps explicitly regardless of the what the input avisynth script is serving. I read them but I can't really grasp what is really implied.

My question is, is it simply not possible to change the source code so that the default output framerate is the same as what the avs is serving, or will such change be implemented in the future? I guess I have no problem with preparing a set of batch scripts, each with different fps set explicitly and also sorting avs scripts into different folders by framerates, but I just feel xvid_encraw.exe should be able to do what x264.exe can do.
(I know next to nothing about programming so please forgive me if I'm saying something stupid.)

Apart from this I have nothing to say and want to express my deepest gratitude!

[squid_80]

squid_80, I found that zone quant. settings seem to be reseting the b-frame max-consec., ratio and offset settings. I need to put the zone parameters before the b-frame parameters so as to use something like 1/1.62/0. Is it intended?
Not sure if it's encraw related or xvidcore, can you give me a complete example command line and brief description of what output you expect?

I've spent a few days going through every single page of this thread and found several posts by squid_80 and others about encoded outputs set to 25fps without setting the fps explicitly regardless of the what the input avisynth script is serving.
My builds of xvid_encraw set the output framerate based on the input. xvid_encraw builds made from xvid's cvs code are forced to 25fps. My latest available compiles should be available from http://members.optusnet.com.au/squid_80/

I may not be available in this forum for a while, so please have patience if reporting problems.

[Champs]

squid_80, thank you so much for a quick reply! I was using the 12th of July version that I got off this thread and having the afformentioned trouble. Just downloaded your 13th of Sept version now and tried it. Yes it does indeed take the correct fps from the input avs! I'm sincerely sorry for not having been aware of this more recent version, and thanks again!!

[squid_80]

I have made a new build available for testing: http://members.optusnet.com.au/squid_80/xvid_encraw.zip

Changes:

-o can be used for avi and mkv files, it decides what format to write based on file extension.
-stats takes an optional filename parameter, to specify a file to write the stats to. I guess if you want no output at all (no progress display or stats) you can use -stats nul.
The avi writing code is all new (borrowed from virtualdub), this should fix the > 2GB problems people were having. It may contain large bugs, particularly if you run out of space.
MKV writer interface was changed a bit, should be transparent (but may have introduced bugs).
If ctrl-c is used to terminate the program it should exit gracefully (output files should be flushed, statistics shown).


The new avi writing code is a big change, if there are any problems with the avi files it creates please me know ASAP.

Keep aviwriter.dll with xvid_encraw.exe, so if I change it (likely) it can be overwritten without too much hunting around on your hard drive.

If you get errors about missing msvc dlls, get the vs2005 redistributable pack from here:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=32BC1BEE-A3F9-4C13-9C99-220B62A191EE
I know it's big (2.6mb) but most people will probably already have it and if you don't you'll need it sooner or later, unless you never plan on installing any new software.

@henryho_hk: Any news on your problem? I can't see anything wrong with it here. The numbers for -bquant_ratio and -bquant_offset have to be multiplied by 100 i.e. 162 for 1.62, that's about the only thing I can think the problem would be.

[henryho_hk]

@henryho_hk: Any news on your problem? I can't see anything wrong with it here.

I cannot reproduce the problem again. I think I had done something wrong in the parameters.

[henryho_hk]

squid_80, AVIMux is complaining on the output avi files when the framerate is not an integer.

xvid_encraws 20060621 gives:

Version: 1.17.7, Aug 8 2006
-----------------------------------------------
AVI-Type: standard
Writing-App : n/a
Title : n/a
resolution : 640x480
FourCC : xvid
biCompression value : XVID
framerate (strh): 29.97 fps
framerate (avih): 29.9697 fps
number of frames : 501 (0:00:16.717)
thereof keyframes : 8
thereof deltaframes : 493
thereof dropped frames : 0
strh[0].dwLength : 501

MainAVIHeader.dwTotalFrames : 501
real value : 501
suggested buffer size : 7809

Flags set: : 0x00000810
AVIF_HASINDEX
AVIF_TRUSTCKTYPE
size of video stream : 208,705 bytes
video data rate : 12 kByte/s


xvid_encraws 20061005 gives:

Version: 1.17.7, Aug 8 2006
-----------------------------------------------
AVI-Type: standard
Writing-App : n/a
Title : n/a
resolution : 640x480
FourCC : xvid
biCompression value : XVID
framerate (strh): 29.97 fps
framerate (avih): 29.9706 fps
The framerate value has appearently been written by a broken program!
number of frames : 501 (0:00:16.717)
thereof keyframes : 8
thereof deltaframes : 493
thereof dropped frames : 0
strh[0].dwLength : 501

MainAVIHeader.dwTotalFrames : 501
real value : 501
suggested buffer size : 0

Flags set: : 0x00000110
AVIF_HASINDEX
AVIF_ISINTERLEAVED
size of video stream : 208,705 bytes
video data rate : 12 kByte/s

[squid_80]

If you use virtualdub to do the processing, it'll say the same thing about the output file. Basically it's being picky; the framerate in the stream header(strh) is specified as a rational number (rate/scale) while the framerate in the avi file header is given in Microseconds per frame. 29.97fps is normally represented as 30000/1001 in the stream header, which gives 0.0333666.... microseconds per frame. Virtualdub seems to truncate this value rather than rounding it and avimux_gui complains because it's not as accurate as possible. But applications shouldn't be using the avih framerate value anyway since the strh framerate is more accurate.

[Blue_MiSfit]

Hey everyone...

So, I haven't ever trid XviD through MeGUI before, but I tried giving it a whirl last night and came up with some problems.


Isolated it to the CQM. I remember some talk of xvid_encraw not supporting CQMs, is that the case?


I got the latest xvid_encraw from squid80's optusnet site, pointed MeGUI at it, and built a profile that matches my usual high bitrate virtualdub settings:


C:\Program Files\megui\tools\xvid_encraw\xvid_encraw.exe -i "F:\Movies Work\City of God\God.avs" -pass2 ".stats" -bitrate 700 -kboost 100 -overhead 0 -max_key_interval 240 -nopacked -vhqmode 4 -qpel -qmatrix "D:\XviD Matricies\Sharktooth's Matricies\eqm_v3hr (gp hb).xcm" -closed_gop -lumimasking -max_bframes 3 -bvhq -threads 1 -avi "F:\Movies Work\City of God\God.avi"


script:

LoadPlugin("D:\Archives\Video Software\AviSynth Plugins\Common\RemoveGrainSSE3.dll")

DGDecode_mpeg2source("F:\Movies Work\City of God\VTS_01_1.d2v",info=3)
ColorMatrix(hints=true)

RemoveGrain(mode=5)


It's automated 2 pass, so the bitrate should get overridden..

Anyway, when I try to encode my AviSynth script that works perfectly in virtualdub or x264.exe, it crashes... I get the unexpectedly closed box, and it references xvidcore.dll.

Maybe my xvidcore.dll is out of date? It's 856,064b and in the same folder as xvid_encraw... Am I just really n00bing it and totally missing some crucual step here?

Sorry if I'm asking about a totally common thing, I did some search and didn't find anything..

Also tried a 1 pass CQ encode and had the same problem.

Suggestions?
~MiSfit

[henryho_hk]

Why is "-overhead 0"? And I think XviD wants us to do a pass1 and then 1 pass2 manually.

[squid_80]

Isolated it to the CQM. I remember some talk of xvid_encraw not supporting CQMs, is that the case?


I use Sharktooth's cqms a lot with no problems. Does xvid_encraw give any text output at all before it crashes?

[Blue_MiSfit]

Well the megui log says:

Generating jobs. Desired size: 2306867200 bytes
No audio encoding. Calculating desired video bitrate directly.
Encoded audio file is present: F:\Movies Work\City of God\VTS_01_1 T01 3_2ch 448Kbps DELAY 0ms.mp4 It has a size of 228491910 bytes.
Setting video bitrate for the video jobs to 2131 kbit/s
Setting desired size of video to 2074358784 bytes
Starting job job1-1 at 9:03:29 PM
encoder commandline:
-i "F:\Movies Work\City of God\God.avs" -pass1 "F:\Movies Work\City of God\God.stats" -bitrate 2131 -kboost 100 -overhead 0 -max_key_interval 240 -nopacked -vhqmode 4 -qpel -qmatrix "D:\XviD Matricies\Sharktooth's Matricies\eqm_v3hr (gp hb).xcm" -closed_gop -lumimasking -max_bframes 3 -bvhq -threads 1
successfully started encoding


and then I get the generic "xvid_encraw.exe has encountered a problem and needs to close..." error box.

~MiSfit

[Brother John]

You isolated it to the CQM? Then with a standard matrix everything works fine?

Corrupted XCM file? Can you use it with XviD VfW?
Just to rule out path name problems: Try a path with only letters and no spaces like "D:\eqm_v3hr.xcm". Spaces shouldn't be a problem, but for some weird reason the ' might.
Which version of xvid_encraw do you use? Run "xvid_encraw.exe -help" to find out.

[squid_80]

Probably easiest to just post the matrix file, or a link to it. encraw checks to make sure it's 128 bytes long but that's about all it looks for. There's no string manipulation on the filename so spaces and slashes shouldn't be an issue. (Yes it freads in binary mode, in case anyone wondered.)

[Blue_MiSfit]

My CQM definately works in VFW XviD.

I'm using the October 5 2006 build (from 'xvid_encraw.exe -help')

http://rapidshare.de/files/36538364/eqm_v3hr__gp_hb_.xcm.html

Pretty sure it's not the matrix itself, as it works in vfw just fine..

Tried again, with a no space name (called it eqm_v3hr.xcm and put it in the root of my D: drive...

Very strange indeed..

[squid_80]

There's something wrong with that matrix file. I started xvid vfw's config and went to the custom matrix section, when I loaded your matrix it showed a 121 and 0 at the end of the second row of the inter matrix. When OK is selected the VFW config corrects the 0 into a 1, xvid_encraw isn't that smart so it crashes.

[Blue_MiSfit]

How goofy.

I will re-download Sharktooth's CQM pack. Thanks for the help squid_80...


Did that and everything works perfectly. Nice bit of code gentlemen!! No more vdub encoding for me :)


~MiSfit

[MetalPhreak]

I either found a bug or I'm doing something wrong:
Using the latest encraw by squid_80 I cannot specify a negative b-frame sensititvity in -zones. Eg. -zones 1000,q,2,-20 the "-" gets ignored and encraw uses the integer entered as a positive value.
Any help would be greatly appreciated.

--EDIT--
Maybe I should also add this is with XviD v1.1.2.
Also I'm going away tomorrow so I won't be able to reply for about a week (if more info should be needed)

[henryho_hk]

Try inserting e.g., "K" (start with keyframe), O (chroma optimizer), etc. and see if it is parsed correctly.


-zones 0,q,3,KO-3/1,q,3,KO-10

[squid_80]

Using the latest encraw by squid_80 I cannot specify a negative b-frame sensititvity in -zones. Eg. -zones 1000,q,2,-20 the "-" gets ignored and encraw uses the integer entered as a positive value.

I can't reproduce your results (i.e. it works fine here), can you post your full command line in case it's something else?
G:\vid1>xvid_encraw -i sources.avs -frames 200 -progress
I frames: 3 frames, size = 43890/ 131671, quants = 4 / 4.00 / 4
P frames: 76 frames, size = 24035/1826677, quants = 4 / 4.00 / 4
B frames: 121 frames, size = 5087/ 615613, quants = 7 / 7.00 / 7

G:\vid1>xvid_encraw -i sources.avs -frames 200 -progress -zones 0,q,4,-20
I frames: 3 frames, size = 43890/ 131671, quants = 4 / 4.00 / 4
P frames: 124 frames, size = 20997/2603670, quants = 4 / 4.00 / 4
B frames: 73 frames, size = 3940/ 287642, quants = 7 / 7.00 / 7

[MetalPhreak]

Try inserting e.g., "K" (start with keyframe), O (chroma optimizer), etc. and see if it is parsed correctly.

Already tried that, eg. forcing keyframes work but same result with b-frame bias.

I can't reproduce your results (i.e. it works fine here), can you post your full command line in case it's something else?

xvid_encraw -single -cq 2 -zones 50,q,2,K-30 -quality 6 -vhqmode 4 -bvhq -qpel -gmc -qtype 0 -max_bframes 2 -bquant_ratio 100 -bquant_offset 100 -i "C:\GARDEN_STATE\VIDEO_TS\gs.avs" -mkv "F:\xvid_encraw\test.mkv" -max_key_interval 250 -par 64:45 -progress 10

--EDIT--
Nevermind, just tried again and now it works perfectly. I was probably half asleep the last time I tried and messed something up. Thanks henryho_hk and squid_80 for your help anyway.

[Adub]

Yo, Squidy!
Any new developments in the works? Some crazy optimizations or insane quality modes that blow x264 out of the water? ;p

[squid_80]

encraw's only a frontend for xvid. If you want some new options to play with, check this thread: http://forum.doom9.org/showthread.php?t=114811

[henryho_hk]

squid_80, it seems both the stat file output and the AVI output routines are quite stable. Will you consider sending the source patch to xvid-devel?

[SeeMoreDigital]

Hi guys....

Can somebody please confirm where we got to concerning XvidEncRAW and Packed Bit-stream.

Can XvidEncRAW be configured to generate MPEG-4 part-2 streams with PBS. And if so, is it really neccessary to include such an implementation?


Cheers

[squid_80]

Default setting with .avi output is to use packed bitstream, if there is no .avi output no packed bitstream.
It is also possible to force it on/off with -packed/-unpacked. This does not affect matroska output.

[SeeMoreDigital]

Default setting with .avi output is to use packed bitstream, if there is no .avi output no packed bitstream.
It is also possible to force it on/off with -packed/-unpacked. This does not affect matroska output.I see... So when generating and outputting RAW "MPEG-4_ASP.M4V" streams, there is no packed bit-stream... Excellent :)

Pardon my ignorance... But may I ask why it was felt necessary to develop XvidEncRAW to include PBS at all, even when outputting to AVI?

[squid_80]

AVIs without packed bitstream are hard to edit due to the lag between the frame being displayed and the frame being decoded. For example with virtualdub the image that is displayed on screen does not necessarily come from the frame indicated by the slider. Also don't some SAPs actually require packed bitstream?

It's good to have it as an example of how to use the xvid API as well, since that's where the functionality really comes from. xvid_encraw simply passes a flag to the encoder to tell it to deliver a packed bitstream or not.

[708145]

AVIs without packed bitstream are hard to edit due to the lag between the frame being displayed and the frame being decoded.

Why use avi at all then? If it is not even useful for editing?

[SeeMoreDigital]

It's good to have it as an example of how to use the xvid API as well, since that's where the functionality really comes from. xvid_encraw simply passes a flag to the encoder to tell it to deliver a packed bitstream or not.I see....

And as a back-up, for all those people who find their SAP's need PBS, but forgot to include it. There is always Moitah's excellent MPEG4 Modifier application - which was recently upgraded to add PBS to MPEG-4 (in AVI) streams too ;)

[squid_80]

Why use avi at all then? If it is not even useful for editing?
It's fine with packed bitstream. That's all I'm going to say since I don't want this to turn into another avi/vfw war thread.

[708145]

It's fine with packed bitstream. That's all I'm going to say since I don't want this to turn into another avi/vfw war thread.

OK then.