xvid_encraw - Patched with AviSynth input support

[S_O]

I already made a more or less working of xvid_encraw with AviSynth input support (I simply copied the code from avs2yuv) several month ago (when 1.1.0 beta1 was released), but I somehow borked the source and I never published it.
Now I copied the changes into XviD 1.1.0 beta2, but more cleanly now and did some other changes:
-Added VBV-Buffer options (don´t ask me how to use them - I have no idea)
-Disabled the statistic display for every frame by default
-Added progress display
-Added stats at the end that displays how many I/P/B/S-VOPs with avg. quant and size.

How to use it:
For example: xvid_encraw -i script.avs -type 2 -asm -quality 6 -qpel -single -bitrate 1000 -max_bframes 2 -o rawmpeg4.m4v
For mor details use xvid_encraw -h
It only outputs raw streams, no MP4! You can use mp4box to mux the stream into mp4:
mp4box -add rawmpeg4.m4v video.mp4

Known bugs:
-Time/fps in statistics is ..hmm.. not realistic (I don´t know if I introduced the bug or if it has always been there)
-With B-Frames enabled stat says that more than 100% of the source frames were encoded.

If you like to compile it yourself using VC++, copy the xvid_encraw.cpp and the avs2yuv-dir in the examples-dir of xvidcore, open the main project, delete the xvid_encraw.c from the xvid_encraw project and add xvid_encraw.cpp.

I noticed another problem (not in this release): I had many for me unexplainable bugs in stats display, every time I fixed one another display broke. I hunted bugs over 3 hours. I couldn´t explain why a the value of variable which I set to 0 is somewhere else outputted as 875644. But when I add messagebox in between, that displays another value - it works.
I fixed the problem by changing compiler from "Optimize for speed" to "Standard" (this has only affected the frontend, not xvidcore.dll). Now it works exactly as it´s supposed to work. I don´t know if I write crappy code or Microsoft crappy compilers... (or both)

Edit: Release 3 is ready:
for changes goto:
http://forum.doom9.org/showthread.ph...181#post698181
http://forum.doom9.org/showthread.ph...555#post698555
Attachment has been approved by Koepi, please download and test, and post here how it works for you (or if you have found any bugs).

[Doom9]

thank you. Now we just need somebody to pick up the slack and add the missing options and this will be an awsome tool.

[Sirber]

Once more stable, I will add it to RealAnime Keep the good work!

[S_O]

Originally Posted by Doom9

thank you. Now we just need somebody to pick up the slack and add the missing options and this will be an awsome tool.

What options are missing exactly? I´ll try to add them.

Once more stable, I will add it to RealAnime Keep the good work!

Thanks

Edit: I yust looked into xvid.h and the vfw interface, there are lot´s of options missing. I´ll try to add them.

[Sirber]

Can mkvmerge merge M4V stream into MKV?

[Doom9]

What options are missing exactly?

interlaced: bff
quantization type
adaptive quantization
par
1 pass mode: reaction delay factor, averaging period, smoother
2 pass mode, 2nd pass: iframe boost, iframes closer than, reduced by, overflow control strength, max overflow improvement, max overflow degradation, high bitrate scenes degradation, low bitrate scenes improvement
vhq mode
vhq for b-frames
chroma motion
turbo
frame drop ratio
trellis
min/max quantizer for i/p/b frames
and zones also have: begin with keyframe, cartoon mode, grayscale, chroma optimizer and bvop sensitivity
last but not least custom quantization matrices

phew, quite a long list, isn't it?

[S_O]

Can mkvmerge merge M4V stream into MKV?

No, but it shouldn´t be that difficult to add.

It seems that xvid.h offers much more options than the vfw-interface uses (some of them might be grouped toghether). I´ll try to add all.

Edit: @Doom9: The list is even longer.

[Doom9]

I'd have some more ideas but I'll keep them to myself for now, as I've already flooded you.

[Sirber]

Well, thanks a lot for your efforts! I won't bug you longer either

[bond]

really great stuff, thx a lot!

[S_O]

I yust noticed that the quality presets automatically activate features. At quality 6 for example Trellis, Inter4MV, Halfpel and Chroma Motion are always used.

Yust for curiosity: It seems that XviD allows a new Quant Matrix at every I-VOP (if I read the code correctly I can even force a new quant matrix without a new I frame).
In theory it should be possible if the VOL would be written again. But for seeking, the VOL should be repeated at every I-VOP, otherwise the decoder would need to scan the entire file back until it finds a quant matrix.
Is that allowed by the specs? Also I found informtion how inverse quantisation with the different methods work (H.263; MPEG) and how this is indicated in the bitstream, I couldn´t find one sentence saying it is allowed or not (maybe it´s more general, like "VOL is not allowed to change in a stream"). I remember XviD had an option some time ago to automatically switch between H.263 and MPEG quant, but it was removed, because its not ISO compliant. Where in the specs is that written? In theory, where a VOL can be (are they only allowed in front of a I-VOP?), there could be new quantisation.

[sysKin]

Originally Posted by S_O

maybe it´s more general, like "VOL is not allowed to change in a stream"

Yes this is exactly it. I don't remember where it is exactly, but although VOL can be repeated accross the stream, it must not change.

[edit]oh, I almost forgot: great work, I'll commit it to cvs when it's ready

[bond]

as syskin said its not allowed to have varying vols

eg when placing the raw m4v stream in .mp4 the vol is placed only once in the file (as it cant change anyways) and its also not placed together with the i-frames

[IgorC]

it's sumary usefull tool. Many settings were mentioned to apply. And Full quality first pass?
I don't care much about -debug , but it crashes here.

-bitrate (bps) . -bitrate 800000 . NOT -bitrate 800

[multicone]

Can this tool be patched with Haali MKV writing lib from x264, so it will write native MPEG4 MKV ?

[Doom9]

@multicone: sure... are you volunteering your time? We've dumped too many things on poor S_O already.. I have a feeling the more it gets the more likely it'll never get done - it's kinda annoying to have a classroom full of people clamoring I want this and that

[S_O]

Ok, that´s all switches I want to implement, have I missed something (except zones). Note that the description is either taken from vfw or directly from xvid.h, what means I don´t what they are all good for. I also reorderd them in the help display:

Input options:
 -i      string : input filename (default=stdin)
 -type   integer: input data type (yuv=0, pgm=1, avisynth=2)
 -w      integer: frame width ([1.2048])
 -h      integer: frame height ([1.2048])
 -frames integer: number of frames to encode

Output options:
 -dump    : save decoder output
 -save    : save an Elementary Stream file per frame
 -o string: save an Elementary Stream for the complete sequence

Bitstream options:
 -prf  profile      : indicated profile ([S0.S3] or [AS0.AS5])
                                        (default=unrestricted)
 -par  aspect_ratio : pixel aspect ratio ([VGA11; PAL43; NTSC43; PAL169;
                                           NTSC169]) (default=VGA11)
 -custom_par  X Y   : custom pixel aspect ratio
 -divx_ud           : write DivX-userdata
 -vol_ivop          : repeat VOL at every I-VOP

General encoding options:
 -quality    integer: quality ([0..6]) presets
 -hqacp             : high quality ac prediction
 -inter4v           : use 4 motion vectors per MB
 -gmc               : use global motion compensation
 -interlaced        : use interlaced encoding (this is NOT a deinterlacer!)
 -i_tff             : top field first (interlaced encoding)
 -i_alt             : alternate scan (interlaced encoding)
 -reduced_res       : enable reduced resolution
 -force_rrv         : force all frames to be reduced resolution
 -greyscale         : greyscale mode, all chroma is ignored
 -cartoon           : use 'cartoon' mode
 -chroma_opt        : enable chroma-optimizer pre-filter
 -frame_drop integer: frame drop ratio (in percent)

BFrames options:
 -max_bframes   integer : max bframes (default=0)
 -bquant_ratio  integer : bframe quantizer ratio (default=150)
 -bquant_offset integer : bframe quantizer offset (default=100)
 -packed                : packed mode (DO NOT USE! WILL CORRUPT OUTPUT!)
 -closed_gop            : closed GOP mode

Rate control options:
 -framerate         float    : target framerate (>0 | default=25.0)
 -bitrate           integer  : target bitrate
 -single                     : single pass mode
 -pass1             filename : twopass mode (first pass)
 -pass2             filename : twopass mode (2nd pass)
 -zq starting_frame float    : bitrate zone; quant
 -zw starting_frame float    : bitrate zone; weight
 -max_key_interval  integer  : maximum keyframe interval

Single Pass options:
 -cquant         float      : target quantizer (use instead of bitrate)
 -reac_delay     integer    : reaction delay factor
 -avg_period     integer    : averaging period
 -smoother       integer    : smoother

Pass 1 of 2 options:
 -full_1p                   : Full first pass

Pass 2 of 2 options:
 -size           integer    : target size of video (use instead of bitrate)
 -keyframe_boost integer    : keyframe boost (in percent)
 -close_i_red    intA intB  : I frames closer than A frames are reduced by B%
 -ccp_high       integer    : high bitrate degradation (in percent)
 -ccp_low        integer    : low bitrate improvement (in percent)
 -max_oi         integer    : max overflow improvement (in percent)
 -max_od         integer    : max overflow degradation (in percent)
 -overf_cs       integer    : overflow control strength (in percent)
 -dxn_prf        profile    : apply DXN-Profile ([HH; PPAL; PNTSC; HTPAL;
                                                  HTNTSC; HDTV])
 -vbv_size       integer    : buffer size (bits)
 -vbv_initial    integer    : initial buffer occupancy (bits)
 -vbv_maxrate    integer    : max processing bitrate (bps)
 -vbv_peakrate   integer    : max average bitrate over 3 seconds (bps)

Zone options:
 -zq starting_frame float   : bitrate zone; quant
 -zw starting_frame float   : bitrate zone; weight

Quantization options:
 -iquants        min max  : I-frame quantizer restriction ([1.31])
 -pquants        min max  : P-frame quantizer restriction ([1.31])
 -bquants        min max  : B-frame quantizer restriction ([1.31])
 -trellis                 : use trellis based R-D "optimal" quantization
 -mpeg_quant              : use MPEG quantization instead of H.263
 -mpeg_cqm       filename : use custom MPEG quantization matrix (intra & inter)
 -mpeg_intra_cqm filename : use custom MPEG quantization matrix (intra only)
 -mpeg_inter_cqm filename : use custom MPEG quantization matrix (inter only)
 -lumimasking             : use lumimasking/adaptive quantization algorithm

Motion estimation options:
 -hpel       : use half pixel ME
 -qpel       : use quarter pixel ME
 -chromap    : use P-Frame chroma for ME
 -chromab    : use B-Frame chroma for ME
 -me_advd16  : use advanced diamonds as search pattern
 -me_advd8   : use advanced diamonds for extended 8x8 search
 -me_sqr16   : use squares as search pattern
 -me_sqr8    : use squares for extended 8x8 search
 -me_hpelr16 : enable halfpel refine 16
 -me_qpelr16 : enable quarterpel refine 16
 -me_hpelr8  : enable halfpel refine 8
 -me_qpelr8  : enable quarterpel refine 8
 -me_gmer    : enable GME refine
 -me_exts16  : extend PMV by more searches
 -me_exts8   : extended 8x8 search

Rate disortion options:
 -vhq         integer  : R-D presets ([1.4])
 -rated                : enable R-D
 -rd_simple            : use simplified R-D mode decision
 -rd_bf                : enable R-D for B-frames
 -rd_hpelr16           : use R-D halfpel refine 16
 -rd_qpelr16           : use R-D quarterpel refine 16
 -rd_hpelr8            : use R-D halfpel refine 8
 -rd_qpelr8            : use R-D quarterpel refine 8
 -rd_chk_pred          : check vector equal to prediction
 -rd_exts              : perform R-D-based search using square patterns

Turbo options:
 -turbo     : turbo preset
 -t_fr16    : low-complexity 16 refinement
 -t_fr8     : low-complexity 8x8 sub-block refinement
 -t_skipds  : skip b-frame delta search
 -t_fmi     : partly skip interpolate mode
 -t_bfes    : stop b-frame search early
 -t_dsm     : detect stationary scenes

Other options
 -asm            : use assembly optmized code
 -stats          : print stats about encoded frames
 -debug          : activates xvidcore internal debugging output
 -vop_debug      : print some info directly into encoded frames
 -help           : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as many times as needed.

Please tell if I seem to have forget anything.
I´ll have a look how to add zones properly, with the flexibility to change all parameters that are allowed by the specs and are possible with xvid.
All the extended zone stuff (cartoon, greyscale, b-vop threshold...) is handled directly by the vfw and not by xvidcore and manually applied every frame.

Is there only one format for quant-matrix (I don´t know the english plural of matrix), the raw format (128 Byte)?

I don't care much about -debug , but it crashes here.

Was it the last parameter in your commandline? It seems to requires some kind of argument, but it doesn´t check if there is one.

Can this tool be patched with Haali MKV writing lib from x264, so it will write native MPEG4 MKV ?

There was C++ class for matroska writing once (based on libebml/libmatroska), i don´t know if it´s still up-to-date. It should be rather simple to add mkv support with it (as compile time option like avisynth).

eg when placing the raw m4v stream in .mp4 the vol is placed only once in the file (as it cant change anyways) and its also not placed together with the i-frames

That would of course brake changing quant-matrix.

Edit: I´m yust thinking that the option -asm should become -noasm, because most probably want to use asm and many will forget to set -asm.

[S_O]

@multicone: sure... are you volunteering your time? We've dumped too many things on poor S_O already.. I have a feeling the more it gets the more likely it'll never get done - it's kinda annoying to have a classroom full of people clamoring I want this and that

Currently I have (summer!?) holidays (also temperature is more like autumn) and so I have time.
I´ll first add all the other stuff before I start thinking about mkv.

[bond]

Originally Posted by S_O

Please tell if I seem to have forget anything.

hm the -packed option should be removed, as it only borks the stream with raw output

afaik the reduced resolution option is borked too, but i dunno the details

the dxn profiles should be grouped together with the normal profile settings, as its not possible to have both, a mpeg-4 and a dxn profile
the point of profiles is (dis)allowing specific options and setting the vbv (and setting the profile/level indication in the stream of course)
so if the tool isnt able to take this into account when the user sets a profile its better to not allow the setting of the profiles at all (and leave this up to a gui)

I´ll have a look how to add zones properly, with the flexibility to change all parameters that are allowed by the specs and are possible with xvid.

probably its a good idea to allow the same features to be set via zones as the vfw gui allows
most things (like qpel, gmc, matrix aso) are not allowed to vary in the stream anways

Is there only one format for quant-matrix (I don´t know the english plural of matrix), the raw format (128 Byte)?

yes, there is
if you search around you will find lots of cqm for xvid (i assume the vfw code handles the parsing of these)
people definitely dont use different cqm files for inter and intra

[S_O]

the dxn profiles should be grouped together with the normal profile settings, as its not possible to have both, a mpeg-4 and a dxn profile
the point of profiles is (dis)allowing specific options and setting the vbv (and setting the profile/level indication in the stream of course)
so if the tool isnt able to take this into account when the user sets a profile its better to not allow the setting of the profiles at all (and leave this up to a gui)

What I thought by separating the DXN profiles from the normal one:
The normal profiles are indicated in the VOL, DXN are not. So I can apply a DXN profile which restricts VBV buffer while at the same time the stream can be marked as AS@L3. This can be perfectly fine, if the encoding parameters are reduced to the least common denominator.
Also profile indication cannot be left to gui (except DXN), since it´s stored in the bitstream and xvidcore does not automatically set the profile, xvid_encraw (or vfw) has to tell it which profile to indicate. Currently it is hardcoded to AS@L4 (better would have been unrescricted, I haven´t done it), xvidcore doesn´t care if the stream complies to it or not. If I set profile to 0xc4, it probably won´t complain either. This is yust 1 byte information in the bitstream. It´s up to the interface if it checks the parameters to profile compliance and xvid_encraw currently does not. But I´ll add a warning if a profile is indicated thoghether with a not compliant feature.

probably its a good idea to allow the same features to be set via zones as the vfw gui allows
most things (like qpel, gmc, matrix aso) are not allowed to vary in the stream anways

It seems that many things like turbo, ME options, trellis, some rd options, force reduced resolution can be chnaged during encode, but of course the most important things are the ones included in vfw.

hm the -packed option should be removed, as it only borks the stream with raw output

Yes, you´re probably right. Also I added a warning, people might think "do not use - this must be the expert-only switch for super-high-ultra-quality, yust like -k is in lame"

if you search around you will find lots of cqm for xvid (i assume the vfw code handles the parsing of these)
people definitely dont use different cqm files for inter and intra

All I found were the normal 128 byte files, which xvid vfw seems to handle.
I also haven´t found different files for intra/inter, but maybe somebody only wants to use a custom inter matrix and leave the intra matrix untouched, or use intra and inter from different files. I´ll add it this way:
Intra/inter-matrix: only 128 byte files accepted
intra-matrix only: 64/128 byte files accepted, for 128 byte first matrix will be used.
inter-matrix only: 64/128 byte files accepted, for 128 byte second matrix will be used.
What I originally meant, there is no format containing several cqms with the names, designed goals etc. stored in the file, only raw.