xvid_encraw - Patched with AviSynth input support

[S_O]

I already made a more or less working of xvid_encraw with AviSynth input support (I simply copied the code from avs2yuv) several month ago (when 1.1.0 beta1 was released), but I somehow borked the source and I never published it.
Now I copied the changes into XviD 1.1.0 beta2, but more cleanly now and did some other changes:
-Added VBV-Buffer options (don´t ask me how to use them - I have no idea)
-Disabled the statistic display for every frame by default
-Added progress display
-Added stats at the end that displays how many I/P/B/S-VOPs with avg. quant and size.

How to use it:
For example: xvid_encraw -i script.avs -type 2 -asm -quality 6 -qpel -single -bitrate 1000 -max_bframes 2 -o rawmpeg4.m4v
For mor details use xvid_encraw -h
It only outputs raw streams, no MP4! You can use mp4box to mux the stream into mp4:
mp4box -add rawmpeg4.m4v video.mp4

Known bugs:
-Time/fps in statistics is ..hmm.. not realistic (I don´t know if I introduced the bug or if it has always been there)
-With B-Frames enabled stat says that more than 100% of the source frames were encoded.

If you like to compile it yourself using VC++, copy the xvid_encraw.cpp and the avs2yuv-dir in the examples-dir of xvidcore, open the main project, delete the xvid_encraw.c from the xvid_encraw project and add xvid_encraw.cpp.

I noticed another problem (not in this release): I had many for me unexplainable bugs in stats display, every time I fixed one another display broke. I hunted bugs over 3 hours. I couldn´t explain why a the value of variable which I set to 0 is somewhere else outputted as 875644. But when I add messagebox in between, that displays another value - it works.
I fixed the problem by changing compiler from "Optimize for speed" to "Standard" (this has only affected the frontend, not xvidcore.dll). Now it works exactly as it´s supposed to work. I don´t know if I write crappy code or Microsoft crappy compilers... (or both)

Edit: Release 3 is ready:
for changes goto:
http://forum.doom9.org/showthread.php?p=698181#post698181
http://forum.doom9.org/showthread.php?p=698555#post698555
Attachment has been approved by Koepi, please download and test, and post here how it works for you (or if you have found any bugs).

[Doom9]

thank you. Now we just need somebody to pick up the slack and add the missing options and this will be an awsome tool.

[Sirber]

Once more stable, I will add it to RealAnime :D Keep the good work!

[S_O]

thank you. Now we just need somebody to pick up the slack and add the missing options and this will be an awsome tool.What options are missing exactly? I´ll try to add them.
Once more stable, I will add it to RealAnime Keep the good work!Thanks

Edit: I yust looked into xvid.h and the vfw interface, there are lot´s of options missing. I´ll try to add them.

[Sirber]

Can mkvmerge merge M4V stream into MKV?

[Doom9]

What options are missing exactly?
interlaced: bff
quantization type
adaptive quantization
par
1 pass mode: reaction delay factor, averaging period, smoother
2 pass mode, 2nd pass: iframe boost, iframes closer than, reduced by, overflow control strength, max overflow improvement, max overflow degradation, high bitrate scenes degradation, low bitrate scenes improvement
vhq mode
vhq for b-frames
chroma motion
turbo
frame drop ratio
trellis
min/max quantizer for i/p/b frames
and zones also have: begin with keyframe, cartoon mode, grayscale, chroma optimizer and bvop sensitivity
last but not least custom quantization matrices

phew, quite a long list, isn't it?

[S_O]

Can mkvmerge merge M4V stream into MKV?No, but it shouldn´t be that difficult to add.

It seems that xvid.h offers much more options than the vfw-interface uses (some of them might be grouped toghether). I´ll try to add all.

Edit: @Doom9: The list is even longer.

[Doom9]

I'd have some more ideas but I'll keep them to myself for now, as I've already flooded you.

[Sirber]

Well, thanks a lot for your efforts! I won't bug you longer either :)

[bond]

really great stuff, thx a lot! :)

[S_O]

I yust noticed that the quality presets automatically activate features. At quality 6 for example Trellis, Inter4MV, Halfpel and Chroma Motion are always used.

Yust for curiosity: It seems that XviD allows a new Quant Matrix at every I-VOP (if I read the code correctly I can even force a new quant matrix without a new I frame).
In theory it should be possible if the VOL would be written again. But for seeking, the VOL should be repeated at every I-VOP, otherwise the decoder would need to scan the entire file back until it finds a quant matrix.
Is that allowed by the specs? Also I found informtion how inverse quantisation with the different methods work (H.263; MPEG) and how this is indicated in the bitstream, I couldn´t find one sentence saying it is allowed or not (maybe it´s more general, like "VOL is not allowed to change in a stream"). I remember XviD had an option some time ago to automatically switch between H.263 and MPEG quant, but it was removed, because its not ISO compliant. Where in the specs is that written? In theory, where a VOL can be (are they only allowed in front of a I-VOP?), there could be new quantisation.

[sysKin]

maybe it´s more general, like "VOL is not allowed to change in a stream"Yes this is exactly it. I don't remember where it is exactly, but although VOL can be repeated accross the stream, it must not change.

[edit]oh, I almost forgot: great work, I'll commit it to cvs when it's ready :)

[bond]

as syskin said its not allowed to have varying vols

eg when placing the raw m4v stream in .mp4 the vol is placed only once in the file (as it cant change anyways) and its also not placed together with the i-frames

[IgorC]

it's sumary usefull tool. Many settings were mentioned to apply. And Full quality first pass?
I don't care much about -debug , but it crashes here.

-bitrate (bps) . -bitrate 800000 . NOT -bitrate 800

[multicone]

Can this tool be patched with Haali MKV writing lib from x264, so it will write native MPEG4 MKV ?

[Doom9]

@multicone: sure... are you volunteering your time? We've dumped too many things on poor S_O already.. I have a feeling the more it gets the more likely it'll never get done - it's kinda annoying to have a classroom full of people clamoring I want this and that ;)

[S_O]

Ok, that´s all switches I want to implement, have I missed something (except zones). Note that the description is either taken from vfw or directly from xvid.h, what means I don´t what they are all good for. I also reorderd them in the help display:
Input options:
-i string : input filename (default=stdin)
-type integer: input data type (yuv=0, pgm=1, avisynth=2)
-w integer: frame width ([1.2048])
-h integer: frame height ([1.2048])
-frames integer: number of frames to encode

Output options:
-dump : save decoder output
-save : save an Elementary Stream file per frame
-o string: save an Elementary Stream for the complete sequence

Bitstream options:
-prf profile : indicated profile ([S0.S3] or [AS0.AS5])
(default=unrestricted)
-par aspect_ratio : pixel aspect ratio ([VGA11; PAL43; NTSC43; PAL169;
NTSC169]) (default=VGA11)
-custom_par X Y : custom pixel aspect ratio
-divx_ud : write DivX-userdata
-vol_ivop : repeat VOL at every I-VOP

General encoding options:
-quality integer: quality ([0..6]) presets
-hqacp : high quality ac prediction
-inter4v : use 4 motion vectors per MB
-gmc : use global motion compensation
-interlaced : use interlaced encoding (this is NOT a deinterlacer!)
-i_tff : top field first (interlaced encoding)
-i_alt : alternate scan (interlaced encoding)
-reduced_res : enable reduced resolution
-force_rrv : force all frames to be reduced resolution
-greyscale : greyscale mode, all chroma is ignored
-cartoon : use 'cartoon' mode
-chroma_opt : enable chroma-optimizer pre-filter
-frame_drop integer: frame drop ratio (in percent)

BFrames options:
-max_bframes integer : max bframes (default=0)
-bquant_ratio integer : bframe quantizer ratio (default=150)
-bquant_offset integer : bframe quantizer offset (default=100)
-packed : packed mode (DO NOT USE! WILL CORRUPT OUTPUT!)
-closed_gop : closed GOP mode

Rate control options:
-framerate float : target framerate (>0 | default=25.0)
-bitrate integer : target bitrate
-single : single pass mode
-pass1 filename : twopass mode (first pass)
-pass2 filename : twopass mode (2nd pass)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-max_key_interval integer : maximum keyframe interval

Single Pass options:
-cquant float : target quantizer (use instead of bitrate)
-reac_delay integer : reaction delay factor
-avg_period integer : averaging period
-smoother integer : smoother

Pass 1 of 2 options:
-full_1p : Full first pass

Pass 2 of 2 options:
-size integer : target size of video (use instead of bitrate)
-keyframe_boost integer : keyframe boost (in percent)
-close_i_red intA intB : I frames closer than A frames are reduced by B%
-ccp_high integer : high bitrate degradation (in percent)
-ccp_low integer : low bitrate improvement (in percent)
-max_oi integer : max overflow improvement (in percent)
-max_od integer : max overflow degradation (in percent)
-overf_cs integer : overflow control strength (in percent)
-dxn_prf profile : apply DXN-Profile ([HH; PPAL; PNTSC; HTPAL;
HTNTSC; HDTV])
-vbv_size integer : buffer size (bits)
-vbv_initial integer : initial buffer occupancy (bits)
-vbv_maxrate integer : max processing bitrate (bps)
-vbv_peakrate integer : max average bitrate over 3 seconds (bps)

Zone options:
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight

Quantization options:
-iquants min max : I-frame quantizer restriction ([1.31])
-pquants min max : P-frame quantizer restriction ([1.31])
-bquants min max : B-frame quantizer restriction ([1.31])
-trellis : use trellis based R-D "optimal" quantization
-mpeg_quant : use MPEG quantization instead of H.263
-mpeg_cqm filename : use custom MPEG quantization matrix (intra & inter)
-mpeg_intra_cqm filename : use custom MPEG quantization matrix (intra only)
-mpeg_inter_cqm filename : use custom MPEG quantization matrix (inter only)
-lumimasking : use lumimasking/adaptive quantization algorithm

Motion estimation options:
-hpel : use half pixel ME
-qpel : use quarter pixel ME
-chromap : use P-Frame chroma for ME
-chromab : use B-Frame chroma for ME
-me_advd16 : use advanced diamonds as search pattern
-me_advd8 : use advanced diamonds for extended 8x8 search
-me_sqr16 : use squares as search pattern
-me_sqr8 : use squares for extended 8x8 search
-me_hpelr16 : enable halfpel refine 16
-me_qpelr16 : enable quarterpel refine 16
-me_hpelr8 : enable halfpel refine 8
-me_qpelr8 : enable quarterpel refine 8
-me_gmer : enable GME refine
-me_exts16 : extend PMV by more searches
-me_exts8 : extended 8x8 search

Rate disortion options:
-vhq integer : R-D presets ([1.4])
-rated : enable R-D
-rd_simple : use simplified R-D mode decision
-rd_bf : enable R-D for B-frames
-rd_hpelr16 : use R-D halfpel refine 16
-rd_qpelr16 : use R-D quarterpel refine 16
-rd_hpelr8 : use R-D halfpel refine 8
-rd_qpelr8 : use R-D quarterpel refine 8
-rd_chk_pred : check vector equal to prediction
-rd_exts : perform R-D-based search using square patterns

Turbo options:
-turbo : turbo preset
-t_fr16 : low-complexity 16 refinement
-t_fr8 : low-complexity 8x8 sub-block refinement
-t_skipds : skip b-frame delta search
-t_fmi : partly skip interpolate mode
-t_bfes : stop b-frame search early
-t_dsm : detect stationary scenes

Other options
-asm : use assembly optmized code
-stats : print stats about encoded frames
-debug : activates xvidcore internal debugging output
-vop_debug : print some info directly into encoded frames
-help : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as many times as needed.Please tell if I seem to have forget anything.
I´ll have a look how to add zones properly, with the flexibility to change all parameters that are allowed by the specs and are possible with xvid.
All the extended zone stuff (cartoon, greyscale, b-vop threshold...) is handled directly by the vfw and not by xvidcore and manually applied every frame.

Is there only one format for quant-matrix (I don´t know the english plural of matrix), the raw format (128 Byte)?
I don't care much about -debug , but it crashes here.Was it the last parameter in your commandline? It seems to requires some kind of argument, but it doesn´t check if there is one.
Can this tool be patched with Haali MKV writing lib from x264, so it will write native MPEG4 MKV ?There was C++ class for matroska writing once (based on libebml/libmatroska), i don´t know if it´s still up-to-date. It should be rather simple to add mkv support with it (as compile time option like avisynth).
eg when placing the raw m4v stream in .mp4 the vol is placed only once in the file (as it cant change anyways) and its also not placed together with the i-framesThat would of course brake changing quant-matrix.

Edit: I´m yust thinking that the option -asm should become -noasm, because most probably want to use asm and many will forget to set -asm.

[S_O]

@multicone: sure... are you volunteering your time? We've dumped too many things on poor S_O already.. I have a feeling the more it gets the more likely it'll never get done - it's kinda annoying to have a classroom full of people clamoring I want this and thatCurrently I have (summer!?) holidays (also temperature is more like autumn) and so I have time.
I´ll first add all the other stuff before I start thinking about mkv.

[bond]

Please tell if I seem to have forget anything.hm the -packed option should be removed, as it only borks the stream with raw output

afaik the reduced resolution option is borked too, but i dunno the details

the dxn profiles should be grouped together with the normal profile settings, as its not possible to have both, a mpeg-4 and a dxn profile
the point of profiles is (dis)allowing specific options and setting the vbv (and setting the profile/level indication in the stream of course)
so if the tool isnt able to take this into account when the user sets a profile its better to not allow the setting of the profiles at all (and leave this up to a gui)

I´ll have a look how to add zones properly, with the flexibility to change all parameters that are allowed by the specs and are possible with xvid.probably its a good idea to allow the same features to be set via zones as the vfw gui allows
most things (like qpel, gmc, matrix aso) are not allowed to vary in the stream anways

Is there only one format for quant-matrix (I don´t know the english plural of matrix), the raw format (128 Byte)?yes, there is
if you search around you will find lots of cqm for xvid (i assume the vfw code handles the parsing of these)
people definitely dont use different cqm files for inter and intra

[S_O]

the dxn profiles should be grouped together with the normal profile settings, as its not possible to have both, a mpeg-4 and a dxn profile
the point of profiles is (dis)allowing specific options and setting the vbv (and setting the profile/level indication in the stream of course)
so if the tool isnt able to take this into account when the user sets a profile its better to not allow the setting of the profiles at all (and leave this up to a gui)
What I thought by separating the DXN profiles from the normal one:
The normal profiles are indicated in the VOL, DXN are not. So I can apply a DXN profile which restricts VBV buffer while at the same time the stream can be marked as AS@L3. This can be perfectly fine, if the encoding parameters are reduced to the least common denominator.
Also profile indication cannot be left to gui (except DXN), since it´s stored in the bitstream and xvidcore does not automatically set the profile, xvid_encraw (or vfw) has to tell it which profile to indicate. Currently it is hardcoded to AS@L4 (better would have been unrescricted, I haven´t done it), xvidcore doesn´t care if the stream complies to it or not. If I set profile to 0xc4, it probably won´t complain either. This is yust 1 byte information in the bitstream. It´s up to the interface if it checks the parameters to profile compliance and xvid_encraw currently does not. But I´ll add a warning if a profile is indicated thoghether with a not compliant feature.
probably its a good idea to allow the same features to be set via zones as the vfw gui allows
most things (like qpel, gmc, matrix aso) are not allowed to vary in the stream anwaysIt seems that many things like turbo, ME options, trellis, some rd options, force reduced resolution can be chnaged during encode, but of course the most important things are the ones included in vfw.
hm the -packed option should be removed, as it only borks the stream with raw outputYes, you´re probably right. Also I added a warning, people might think "do not use - this must be the expert-only switch for super-high-ultra-quality, yust like -k is in lame"
if you search around you will find lots of cqm for xvid (i assume the vfw code handles the parsing of these)
people definitely dont use different cqm files for inter and intraAll I found were the normal 128 byte files, which xvid vfw seems to handle.
I also haven´t found different files for intra/inter, but maybe somebody only wants to use a custom inter matrix and leave the intra matrix untouched, or use intra and inter from different files. I´ll add it this way:
Intra/inter-matrix: only 128 byte files accepted
intra-matrix only: 64/128 byte files accepted, for 128 byte first matrix will be used.
inter-matrix only: 64/128 byte files accepted, for 128 byte second matrix will be used.
What I originally meant, there is no format containing several cqms with the names, designed goals etc. stored in the file, only raw.

[bond]

What I thought by separating the DXN profiles from the normal one:
The normal profiles are indicated in the VOL, DXN are not. So I can apply a DXN profile which restricts VBV buffer while at the same time the stream can be marked as AS@L3. This can be perfectly fine, if the encoding parameters are reduced to the least common denominator.
Also profile indication cannot be left to gui (except DXN), since it´s stored in the bitstream and xvidcore does not automatically set the profile, xvid_encraw (or vfw) has to tell it which profile to indicate.well people could set a mpeg-4 profile (like simple profile) together with a dxn profile (using for example b-frames) which wouldnt fit

i think the way how its handled currently in vfw is fine: set the mpeg-4 profile being equivalent to the dxn profile
eg dxn home theather profile is equivalent to asp@l5 (it makes no sense to allow a different mpeg-4 profile to be set)

Currently it is hardcoded to AS@L4 (better would have been unrescricted, I haven´t done it), xvidcore doesn´t care if the stream complies to it or not. If I set profile to 0xc4, it probably won´t complain either. another easy possible way would be to signal simply always the highest possible profile: asp@l5

This is yust 1 byte information in the bitstream. It´s up to the interface if it checks the parameters to profile compliance and xvid_encraw currently does not. But I´ll add a warning if a profile is indicated thoghether with a not compliant feature.well i am no dev, but if encraw is already able to detect that a feature is not part of the set profile, how much work would it be to disable this feature automatically?

[yokem55]

Is there a way to build this on linux? I don't need the avs input as setting up mencoder to dump into a raw yuv fifo works fine, but the extra options would be nice to have available/play with.

[S_O]

Is there a way to build this on linux? I don't need the avs input as setting up mencoder to dump into a raw yuv fifo works fine, but the extra options would be nice to have available/play with.Yes, that will be possible, you yust need to delete/comment "#define AVISYNTH_INPUT" at the beginning.
well people could set a mpeg-4 profile (like simple profile) together with a dxn profile (using for example b-frames) which wouldnt fit

i think the way how its handled currently in vfw is fine: set the mpeg-4 profile being equivalent to the dxn profile
eg dxn home theather profile is equivalent to asp@l5 (it makes no sense to allow a different mpeg-4 profile to be set)What I mean is, that DXN restriction is partly different than the normal is (VBV buffer), and DXN profiles also only work in 2pass mode, because the vbv restriction stuff is part of the 2pass-plugin. With single pass the output probably won´t comply to the DXN specs.
That´s why I added the DXN stuff to 2pass mode in help: It only makes sense there! And the iso profiles are added to bitstream options and described "indicate profile", because it´s yust one byte in the VOL for xvidcore. I can set this byte to whatever I like (except to 0, then xvidcore sets it automatically):

if reduced_resolution -> ARTS@L4
if GMC or QPEL -> ASP@L5
everything else will get SP@L3

that´s all. In other words when I create a stream 60fps, 1440x960 with 3 bframes at 4mbps, it will automatically get SP@L3 (max: 352x288; 15fps; 384kbps; no b-frames).
The switch is yust to set the byte, not to force the profile. Of course, that means files with a wrong profile may be created, but that can happen with vfw as well, because it also doesn´t check everything.
But I will add a warning, like with " -dxn_prf XY -prf AS5 -par PAL43 -reduced_res":
Warning: Aspect Ratio for DXN-Profiles must be VGA 1:1. Stream will not comply to DXN XY Profile
Warning: Reduced resolution is not supported by Profile AS@L5. Stream will not comply to indicated profile!

Unfortunately I have no complete list of the profiles and their ids. There is a list here http://www.m4if.org/resources/profiles/ with more than 50 profiles/levels (like "Core Studio" or "Advanced Core"). Unfortunately the ids are not included.

well i am no dev, but if encraw is already able to detect that a feature is not part of the set profile, how much work would it be to disable this feature automatically?encraw is not able to detect it, I have to tell him what complies to specs and what not. Of course I can also disable stuff like Qpel, GMC or B-Frames or reduce bitrate, but fps or frame size is not easy to change (and vfw doesn´t check that either).

[bond]

What I mean is, that DXN restriction is partly different than the normal is (VBV buffer), and DXN profiles also only work in 2pass mode, because the vbv restriction stuff is part of the 2pass-plugin. With single pass the output probably won´t comply to the DXN specs.
That´s why I added the DXN stuff to 2pass mode in help: It only makes sense there!exactly the same goes for the mpeg-4 profiles...
i dunno if you know it but if you signal asp@level5 you also have to use the vbv if you want to be spec compliant...
the mpeg-4 profiles/levels influence the vbv too, not only the encoding options. basically the dxn profiles are private mpeg-4 profiles, nothing less, nothing more

check out what mpeg-4 profile/level gets signalled when a dxn profile is set! afaik the dxn home theater profile sets the asp@l5 profile/level, now what if the user additionally tries to set a different mpeg-4 profile/level
it doesnt make sense to seperate the two and its also not differentiated in the vfw gui

again, checkout how things are handled in xvid vfw! its uptodate and the people who wrote the vfw codec knew what they did (cause after all practically the xvid codec is the vfw codec)

another point showing this i brought up before too, which you seem to ignore, is that reduced resolution is not working correctly in xvid
the code might still be there but if you look at the vfw gui you will see that reduced resolution (and the arts profile) is not available, this also happens for a reason

And the iso profiles are added to bitstream options and described "indicate profile", because it´s yust one byte in the VOL for xvidcore. I can set this byte to whatever I like (except to 0, then xvidcore sets it automatically):

if reduced_resolution -> ARTS@L4
if GMC or QPEL -> ASP@L5
everything else will get SP@L3

that´s all. In other words when I create a stream 60fps, 1440x960 with 3 bframes at 4mbps, it will automatically get SP@L3 (max: 352x288; 15fps; 384kbps; no b-frames).
The switch is yust to set the byte, not to force the profile. Of course, that means files with a wrong profile may be created, but that can happen with vfw as well, because it also doesn´t check everything.no, with vfw this cant happen, as the gui doesnt let the user enable incompliant settings (of course framerate, bitrate and resolution gets ignored, but i dunno any player which cares about these three)

But I will add a warning, like with " -dxn_prf XY -prf AS5 -par PAL43 -reduced_res":
Warning: Aspect Ratio for DXN-Profiles must be VGA 1:1. Stream will not comply to DXN XY Profile
Warning: Reduced resolution is not supported by Profile AS@L5. Stream will not comply to indicated profile!

encraw is not able to detect it, I have to tell him what complies to specs and what not. Of course I can also disable stuff like Qpel, GMC or B-Frames or reduce bitrate, but fps or frame size is not easy to change (and vfw doesn´t check that either). ok but here is my point: how hard is it to make xvid automatically disable incompliant settings? i mean if you already have the goodie implemented that it tells the user that a chosen setting is incompatible

something like a message saying "b-frames disabled as not complying with chosen profile/level"

[S_O]

no, with vfw this cant happen, as the gui doesnt let the user enable incompliant settings (of course framerate, bitrate and resolution gets ignored, but i dunno any player which cares about these three)
I yust tested and confirmed result in hexeditor:
Set profile to unrescricted, enable some ASP features (except Qpel / GMC) like B-Frames and mpeg quant, now encode your file:
The stream is marked as SP@L3 also it contains B-Frames and MPEG quantization!

In my opionen the best place for automatic profile decision is xvidcore, not the frontend.
i dunno if you know it but if you signal asp@level5 you also have to use the vbv if you want to be spec compliant...
the mpeg-4 profiles/levels influence the vbv too, not only the encoding options. basically the dxn profiles are private mpeg-4 profiles, nothing less, nothing moreSeems you are right, I thought the ISO profiles were yust limiting encoding features and fps/bitrate/resolution and that´s why DXN introduced their profiles.
But since they both have this vbv stuff you are of course right and separating doesn´t makes much sense.
But I wouldn´t call them "private profiles", because they are not indicated in bitstream, if they were using some reserved values to indicate their profiles this trem would be adequate, but in my opinion it´s yust some extra rescrictions for DXN promotion that do not interfere with the ISO standard at all.
dxn home theater profile sets the asp@l5 profile/levelNo, it is set to "auto-detect" by xvidcore (see vfw source (config.c, line 149) and xvidcore-source (bitstream.c, line 1076 - 1117)!), which means, if you don´t use Qpel/GMC it´s marked as SP@L3.
another point showing this i brought up before too, which you seem to ignore, is that reduced resolution is not working correctly in xvid
the code might still be there but if you look at the vfw gui you will see that reduced resolution (and the arts profile) is not available, this also happens for a reasonSorry, I must have missed that sentence. In case it´s not working correctly it should of course be removed.
The reason why reduced resolution has been removed:
From sysKin:
* it's now a long time we planned removing support for RRV as it
adds complexity to the ME, to the decoder and this feature fits
nowhere in any MPEG4 profile we plan to support.I don´t know if it has only been removed from vfw yet, or if it´s already removed (or will be ever removed) from xvidcore and only the flags are still there.
again, checkout how things are handled in xvid vfw! its uptodate and the people who wrote the vfw codec knew what they did (cause after all practically the xvid codec is the vfw codec)That´s what I do all the time, and as far as I can tell you, profiles are not handled perfectly (see demonstration at the beginning of this post)
ok but here is my point: how hard is it to make xvid automatically disable incompliant settings? i mean if you already have the goodie implemented that it tells the user that a chosen setting is incompatible

something like a message saying "b-frames disabled as not complying with chosen profile/level"As already said, in my opionen the best way to handle this would be if xvidcore checks the features and automatically sets the profile. That ensures that every frontend sets the right profile.
When profile is set by frontend xvidcore yust confirms that the encoding features are allowed by the profile, and in case not allowed features are used it refuses to encode and exits (except and override flag is set like "XVID_VOL_IGNOREPROFILE").

The question is now: What profile has to be set when no profile matches the encoding features? The maximum resolution even for AS@L5 is 720x576.

[sysKin]

RRV has been removed from xvid 1.1. Flags will stay, so that API remains backward compatible with 1.0's API.

[bond]

I yust tested and confirmed result in hexeditor:
Set profile to unrescricted, enable some ASP features (except Qpel / GMC) like B-Frames and mpeg quant, now encode your file:
The stream is marked as SP@L3 also it contains B-Frames and MPEG quantization!hm, thats not good :/

As already said, in my opionen the best way to handle this would be if xvidcore checks the features and automatically sets the profile. That ensures that every frontend sets the right profile.indeed, but i assume its hard to code taking everything into account
and than there is the issue of enforcing the vbv too (but i assume the profile code can do that already, at least in the case of the dxn profiles its a must, or is this done by the vfw gui?)

When profile is set by frontend xvidcore yust confirms that the encoding features are allowed by the profile, and in case not allowed features are used it refuses to encode and exits (except and override flag is set like "XVID_VOL_IGNOREPROFILE").cool

The question is now: What profile has to be set when no profile matches the encoding features? The maximum resolution even for AS@L5 is 720x576. well i would simply ignore the framerate and resolution issue, cause i really dunno any player who needs this to be correct

if you ignore the two than the profile to be set should be imho simply asp@l5, cause it covers everything (except reduced resolution, which is borked anyways)

[S_O]

RRV has been removed from xvid 1.1. Flags will stay, so that API remains backward compatible with 1.0's API.Unfortunately this is not written in xvid.h. The flags are there yust like all others, without any notice that they are only for compatibility and rrv doesn´t work. Also there is still code inside xvidcore to deal with rrv, the function BitstreamWriteVolHeader checks for rrv being enabled and sets the flags in bitstream.
and than there is the issue of enforcing the vbv too (but i assume the profile code can do that already, at least in the case of the dxn profiles its a must, or is this done by the vfw gui?)Yes, everything is done by the frontend, xvidcore doesn´t know at all a DXN profile is applied (it doesn´t even know that they exist).
When a profile is selected, vfw enables/disables the features in the dialog and sets the vbv buffer stuff (and for DXN profile also maximum peak-bitrate). In case of ISO profiles it also sets the profile id (the 5th byte in a raw stream, directly after the video object sequence start code).
Without 2pass mode the vbv buffer is not set, since this feature is part of 2pass plug-in.
well i would simply ignore the framerate and resolution issue, cause i really dunno any player who needs this to be correct

if you ignore the two than the profile to be set should be imho simply asp@l5, cause it covers everything (except reduced resolution, which is borked anyways)OK, I yust looked in the specs and this page: http://www.m4if.org/resources/profiles/ again. fps is nowhere mentioned, and resolution is called "typical resolution" not "maximum resolution". In other words, we don´t need to care about that. Unfortunately I can´t find ISO 14496-2 AMD2 containing advanced simple profile defiition.
If I get a full definition of the profiles I´ll see how difficult it is to implement working auto-profile setting in xvidcore.

Oh, in case you haven´t notice: I uploaded a new version (contains only changed files, if you do not have XviD 1.1 installed you need xvidcore.dll from zip of first post).
Chnages:
Lot´s of (nearly all) options added. See -h for details. You may notice that some options are not -xy but +xy: "+" means enable, "-" disable. So you can use presets (-quality, -vhq, -turbo), but you can easily disable some options again like "-quality 6 -trellis", means use -quality 6 (which enables trellis), but disable trellis. Disable switches always override enable switches.
I also added this switches, because I have an idea how to handle advanced zones and there I also need them.

Note 1: cartoon is not exactly the same here and in vfw! When you check cartoon in vfw, it enables cartoon mode and "detect static motion". So "+cartoon +t_dsm" equals vfw cartoon mode (I may change it to avoid confusion).

Note 2: I coded several hours and did one test. I mean one 30 sec clip (but that was fine). So the chances for bugs are very high. Consider this as very alpha. And please report all bugs here. The frame display is not yet fixed. And I said the encoding time is screwed up: It´s not: It´s _only_ the encoding time. If encoding takes 3 minutes but encraw says 1 minute, it means that the other 2 minutes was avisynth processing!

Still missing: Profile settings, fix quant mode, full 1st pass and advanced zones.

Edit: Outdated, see first post for latest

[buzzqw]

would be possible to implement an automatic 2 pass mode to hit final size (mb)

thanks

BHH

[Doom9]

would be possible to implement an automatic 2 pass mode to hit final size (mb)That's a feature for a GUI. MeGUI and RealAnime will do that for you when the time is right.

[S_O]

would be possible to implement an automatic 2 pass mode to hit final size (mb)I not quite sure what you mean, but you can use -size instead of -bitrate for 2pass mode, allowing you to set the final size of the video (note: Frame overhead is 0, but in vfw it´s set to 24 bytes (AVI), so you have calculate the container overhead yourself)

[Sirber]

That's a feature for a GUI. MeGUI and RealAnime will do that for you when the time is right.
So far, RealAnime don't produce a fixed size and work by bitrate or quality only. It include at least a bitrate calculator :).

[Doom9]

note: Frame overhead is 0, but in vfw it´s set to 24 bytes (AVI), so you have calculate the container overhead yourselfdoes that have any influence in bitrate mode? in the VfW and mencoder, the bitrate contains 24 byte container overhead.

[S_O]

does that have any influence in bitrate mode? in the VfW and mencoder, the bitrate contains 24 byte container overhead.No, it doesn´t seem so. I looked into xvid source and it seems that this parameter is only used with filesize parameter in 2pass mode, very simple, like this:
newsize = oldsize - frames*overhead.
(see plugin_2pass2.c; line 404).
Because xvid_encraw outputs raw stream I set this to 0 (with vfw the avi-file have the filesize enterd, with encraw the raw-file will have the filesize). So this parameter isn´t special, you can calculate it yourself.

Has nobody noticed that I uploaded a new version? No feedback until now... I don´t think that´s why because it works perfect and does not contain bugs.

[CiNcH]

No, it is because download doesn't work ;) ... still not approved.

[S_O]

http://forum.doom9.org/attachment.php?attachmentid=4477
Works for me, also after log-out.

[CiNcH]

Not for me.

Invalid attachment specified. If you followed a valid link, please notify the webmaster

[S_O]

You seem to be right. It seems to work because of cookies, when I try another browser it doesn´t work.
I´ll see if I can upload it somewhere else.

[Doom9]

It might just be that you can see your own attachments even while they're still pending.. I know that happens with mine but then again as admin I get to see everything. I have just approved the attachment. It might not be a bad idea to have just one attachment in the first post and edit the thread with changelog updates rather than to have multiple attachments all over the place.

[S_O]

It might not be a bad idea to have just one attachment in the first post and edit the thread with changelog updates rather than to have multiple attachments all over the place.Yes, I´ll do so next time and delete the others then.

[S_O]

OK, next release is ready:
Added advanced zones:
Now you have -z "zone_type start_frame quant/weight [advanced options]"
To see which options are allowed as advanced zones options see -h.
Note: Like in VirtualDub first frame is 0, not 1!

Fixed bugs:
Frame display finally works, you encode 100% of your movie.

Known bugs:
-Advanced zones will not start exactly at the specified frame in case B-Frames are enabled
-N-VOPs are not displayed correctly in stats (always 0 N-VOPs).

Missing features:
-Single pass fixed quant encoding
-Full 1st pass
-Profiles

Download is in first post!

[IgorC]

VHQ for b-frames? is it -rd_bf?

[S_O]

VHQ for b-frames? is it -rd_bf?Yes.
-rd_bf : enable R-D for B-framesR-D is Rate Disortion, "VHQ" is just another name for it (in fact the "VHQ"-modes are some presets for the R-D switches).
Edit:
How I can disable a setting which is enabled by default?Nothing is enabled by default. But if you have a preset activated like -quality 6, you can disable parts of it again, with, for example "-trellis" (disable trellis). Only the switches marked with "+" in the help screen can be disabled.
Tomorrow (or today, when I wake up again), I´ll post the normal switches that eqauls to.

[IgorC]

chroma optimization is bugy with this coomand line

xvid_encraw -i mxskal.avs -o raw.m4v -type 2 -asm -quality 6 -bitrate
900 -max_bframes 2 -croma_opt

[S_O]

xvid_encraw -i mxskal.avs -o raw.m4v -type 2 -asm -quality 6 -bitrate
900 -max_bframes 2 -croma_optYou disabled chroma optimizer (it is already disabled by default).
chroma optimizer is enable/disable switch, to enable it you have to use +chroma_opt.

Edit: it is +chroma_opt, not +croma_opt

[IgorC]

ok , I get it.

chroma was my fault. Thanks to point

[S_O]

+trellis
Yes. Trellis is enabled with this switch
+lumimasking
No. Lumimasking cannot be enabled/disabled. It´s yust -lumimasking
+chroma opt
Yes, it´s +chroma_opt to enable it and, if you like to disable it, for example inside a zone: use -chroma_opt as advanced parameter in the zone.

[IgorC]

I think a good GUI will be nice :)

[S_O]

I think a good GUI will be nice Yes.
I have taken this idea with "+" and "-" switches from a Real/Helix tool, I think it was dt_drive, it has also this kind of options.

[Doom9]

I think a good GUI will be niceMeGUI will eventually support it.
Speaking of which, two things that will be requested at some point will be MP4 and MKV output. Considering that such output is currently broken at 23.976fps in x264, and that that kind of output is most likely going to be integrated in the very same way, I'd hold off until these bugs are fixed. On the other hand, AVI output might be something where those problems won't ocurr, and with AVI output, encraw can serve as a full mencoder replacement in MeGUI.

[bond]

MeGUI will eventually support it.
Speaking of which, two things that will be requested at some point will be MP4 and MKV output. Considering that such output is currently broken at 23.976fps in x264, and that that kind of output is most likely going to be integrated in the very same way, I'd hold off until these bugs are fixed. On the other hand, AVI output might be something where those problems won't ocurr, and with AVI output, encraw can serve as a full mencoder replacement in MeGUI.if .avi output is wanted, why not use mencoder right away? or even the vfw codec

i dont see the point in wasting time adding avi output to this tool, if xvidvfw's avi output is stable for years already. sounds like reinventing the wheel

[Doom9]

if .avi output is wanted, why not use mencoder right away?Because mencoder is bloated beyond reason, doesn't fully support xvid and cannot support any option that contains double points (so xvid custom quantizer matrices as an example) because double points are used to separate encoder options in mencoder.
Considering that you can put any VfW codec into MKV using MKVToolnix, by your analogy mkv output in another other tool would also be reinventing the wheel ;)

[bond]

my few cents:

1) this +/- sheme is confusing. on some options there is only the "-" valid on others both? on some you enable the feature with "-" on others you disable it?

imho this is not really good. a better way would be imho to simply disable a feature when its not explicitely set in the cmdl (meaning by default everything is disabled).
or do it like x264 and enable some features by default which are known to be useful, and than offer an option to disable them, like "-no-qpel"

2) about .mp4 output

the mpeg4ip project offers an own commandline encoder using xvid and outputting .mp4 directly via the mpeg4ip mp4 lib. maybe its easy to take the .mp4 writing code from that and add it to "xvidcli" too?

the mpeg4ip lib should work perfectly fine

3) about .mkv output
Considering that you can put any VfW codec into MKV using MKVToolnix, by your analogy mkv output in another other tool would also be reinventing the wheel ;)nope, cause afaik no mkv tool can atm create native mpeg-4 in mkv fully supporting all features (b-frames)

so if native mpeg-4 mkv output is added to this tool this would be something new and it would also make sense to use this cli tool than ;)

[Doom9]

the mpeg4ip project offers an own commandline encoder using xvid and outputting .mp4 directly via the mpeg4ip mp4 lib. maybe its easy to take the .mp4 writing code from that and add it to "xvidcli" too?You told me once that GPAC creates less overhead... I don't recall the technical details why though.
nope, cause afaik no mkv tool can atm create native mpeg-4 in mkv fully supporting all features (b-frames)What's wrong with VfW mode? While not officially supported, it works just fine.. muxing avc-in-avi into mkv via mkvmerge has yet to cause any problems.

and then there's the old mp4box can handle raw input argument.. since you'd hardly find yourself not adding any audio, raw will work just fine for mp4, but where's the avi packer for raw streams? Just not liking AVI doesn't count ;)

[stephanV]

What's wrong with VfW mode? While not officially supported, it works just fine.. muxing avc-in-avi into mkv via mkvmerge has yet to cause any problems.
You get decoding delays. 1 frame in case of b-frames +1 extra frame if b-pyramid is used.

there is a tool called AVC2AVI provided with the x264 source code,

[bond]

You told me once that GPAC creates less overhead... I don't recall the technical details why though.it does, but with avc, not asp

What's wrong with VfW mode? While not officially supported, it works just fine.. muxing avc-in-avi into mkv via mkvmerge has yet to cause any problems. well stephanv answered + the mkv specs define how to place mpeg-4 (avc and asp) in .mkv (aka native mode) and thats not followed when going via vfw, which might cause interoperability problems aso

and then there's the old mp4box can handle raw input argument..hm i dont understand what you mean with that. mp4creator can handle raw streams fine too

since you'd hardly find yourself not adding any audio, raw will work just fine for mp4, but where's the avi packer for raw streams? Just not liking AVI doesn't count ;) what do container muxer tools have to do with thinking that adding avi output to this xvid encoder should be really low priority, as you dont get anything new if you add it?

i am pretty sure that its possible to mux raw streams into avi, eg via ffmpeg

[Doom9]

hm i dont understand what you mean with that. mp4creator can handle raw streams fine toowhy should encraw support direct mp4 output if you're going to mux audio (and subs, chapters, whatnot) afterwards anyways?
You get decoding delays. 1 frame in case of b-frames +1 extra frame if b-pyramid is used.alright, that's a very good point. By the way, does that apply to both ASP and AVC? Right now I use AVI as intermediate format for MKV output for Snow, lavc MPEG-4 and XviD.
there is a tool called AVC2AVI provided with the x264 source codeand its relevance for asp would be? Can it mux raw ASP content as well?
I'd like to have AVI output to be able to support all 3 containers directly in MeGUI. I don't have any vested agenda in any container, unlike some other participants here. I'll rest my case now.

[stephanV]

alright, that's a very good point. By the way, does that apply to both ASP and AVC? Right now I use AVI as intermediate format for MKV output for Snow, lavc MPEG-4 and XviD.
It applies to both ASP and H264 yes. It has no relevance to snow as it doesn't have b-frames AFAIK (VFW mode is fine there) and you could work around the delay with XviD if you use packed bistream... but uhm... yeah. :)

There is a native MPEG4 ASP MKV mode in MKVMerge which *should* also accept AVI input, but I forgot the command switch. Anyways, using the VFW mode for XviD is still the "normal" way for MKVMerge so I don't see why you should worry about that either then. I also believe Mosu has stated the same before.

A 1 frame decoding delay is not that big of a deal in any case.


and its relevance for asp would be? Can it mux raw ASP content as well?
Not any. Admittedly I got a bit confused in all the XviD and x264 crosstalk and I thought to understand you were looking for a way to put raw H264 in AVI. Pardon me.

[bond]

why should encraw support direct mp4 output if you're going to mux audio (and subs, chapters, whatnot) afterwards anyways?because with .mp4 output you can edit the output in various tools and dont need mp4box or mp4creator :p

[Doom9]

but I forgot the command switch.I'm off to the mkvmerge manpage then.

[multicone]

I'm off to the mkvmerge manpage then.

I guess you won't find it there, because it hasn't been officially released yet. But it works quite stable already, more testing welcome of course.

--engage -nativeMPEG4

This was difficult to add IIRC, as Mosu needed code to read the MPEG4ES headers to find out the exact frame type ( P or B ), when parsing the AVI as it only know P frames.

[S_O]

1) this +/- sheme is confusing. on some options there is only the "-" valid on others both? on some you enable the feature with "-" on others you disable it?Yes, it´s bit confusing, but it´s needed for zones, but it´s needed for zones, if you want to enable something for zones use +xyz, if you like to disable it, use -xyz. I thought this is simpler than -xyz and -no-xyz.

You can remember it like this: Options that don´t need an argument are +/- , all others are yust - (I know, that´s not true for all, like gmc or qpel, but I´ll change it).

My thoughts about the container stuff:
When mp4, matroska, avi, ogm (?)... support is added this encoder becomes more or less a second ffmpeg, I don´t think that is what it is supposed to be. Also it´s nonsense to support these formats while there is no support for directly muxing audio (which means you have to remux it anyway). It´s better to add m4v support to the muxing apps, because in probably 98% of all encoding situation you want add audio. If you go for mp4, it doesn´t matter if mp4box / mp4creator reads the video from m4v or mp4.
But I´ll also understand that it´s annoying first to create a m4v raw stream, then mux it to mp4 and then to make mkv or what ever of it (or edit it as Bond said).
So my idea: stdin/stdout: I already added stdout support to xvid_encraw and I was able to successfully watch the video with a for stdin input patched tmp4 (Skal´s MPEG-4 Encoder/Decoder) while encoding.
I also tried to mux, but neiter mp4box, mp4creator nor ffmpeg seem to support m4v via stdin.
I tried to add stdin support to mp4box, but without any success. Is anybody here in contact with the author? Maybe someone could ask him to implement stdin support (it shouldn´t be to difficult for him, since he knows his code).

This was difficult to add IIRC, as Mosu needed code to read the MPEG4ES headers to find out the exact frame type ( P or B ), when parsing the AVI as it only know P frames.If he has that already and is quite familar with basic m4v, it shouldn´t be too difficult to add a complete m4v parser.

BTW: @Doom9: Why don´t you add xvid directly to MeGUI? If it is coded in C/C++ it should be quite simple.

[Doom9]

If it is coded in C/C++ it should be quite simple.It isn't, and simply being able to plug in a new exe without having to worry about recompiling is a nice thing to have.

[Mosu]

I guess you won't find it there, because it hasn't been officially released yet. But it works quite stable already, more testing welcome of course.

--engage -nativeMPEG4

Not quite, it's "--engage native_mpeg4". It should be working well by now. The only reason I don't make this the default is that there would be problems on playback because most apps don't support V_ISO/MPEG4/* (apart from AVC) yet, I guess. It would be deliberately breaking playback for a LOT of people, something that I have done in the past and don't really want to repeat.

[bond]

My thoughts about the container stuff:
When mp4, matroska, avi, ogm (?)... support is added this encoder becomes more or less a second ffmpeg, I don´t think that is what it is supposed to be. Also it´s nonsense to support these formats while there is no support for directly muxing audio (which means you have to remux it anyway). It´s better to add m4v support to the muxing apps, because in probably 98% of all encoding situation you want add audio. If you go for mp4, it doesn´t matter if mp4box / mp4creator reads the video from m4v or mp4.
But I´ll also understand that it´s annoying first to create a m4v raw stream, then mux it to mp4 and then to make mkv or what ever of it (or edit it as Bond said).
So my idea: stdin/stdout: I already added stdout support to xvid_encraw and I was able to successfully watch the video with a for stdin input patched tmp4 (Skal´s MPEG-4 Encoder/Decoder) while encoding.
I also tried to mux, but neiter mp4box, mp4creator nor ffmpeg seem to support m4v via stdin.
I tried to add stdin support to mp4box, but without any success. Is anybody here in contact with the author? Maybe someone could ask him to implement stdin support (it shouldn´t be to difficult for him, since he knows his code).well i wouldnt say that necessarily every output will get muxed with audio, some might use the output for editing or whatever
its like saying that virtualdub outputting to .avi even without encoding audio is senseless, i think its not
an encoder should be able to output something useable right away imho, and raw .m4v isnt imho, cause you need a second tool for muxing the stream and there arent lots of tools being able to handle raw input
(btw ffmpeg's mp4 writing is not working correctly, only mp4creator from mpeg4ip and mp4box from gpac do that correctly)

if what you say is true, that raw output should be the way to go, than i wonder why every normal encoder tool on earth outputs into a container ;)

but well you are the dev, i cant force you to anything...
all i can say it might be very easy to add mp4 output when looking at the mpeg4ip sources i pointed you to
or you use the lib from gpac (as used in mp4box)

Not quite, it's "--engage native_mpeg4". It should be working well by now. The only reason I don't make this the default is that there would be problems on playback because most apps don't support V_ISO/MPEG4/* (apart from AVC) yet, I guess. It would be deliberately breaking playback for a LOT of people, something that I have done in the past and don't really want to repeat.this also works correctly with packed bitstream, packed bitstream + real n-vops and vfw delay frames?

[S_O]

well i wouldnt say that necessarily every output will get muxed with audio, some might use the output for editing or whatever
its like saying that virtualdub outputting to .avi even without encoding audio is senseless, i think its notI think in most cases audio is used. Also there is difference between xvid_encraw and VirtualDub: VirtualDub is an audio/video processing application, xvid_encraw is an XviD MPEG-4 video encoder.
an encoder should be able to output something useable right away imho, and raw .m4v isnt imho, cause you need a second tool for muxing the stream and there arent lots of tools being able to handle raw inputYes, that´s indeed a good point: m4v is not supported very well (only ffmpeg, MP4Box and mp4creator are able to read it (and tmp4)).
if what you say is true, that raw output should be the way to go, than i wonder why every normal encoder tool on earth outputs into a containerAll MPEG Audio Layer 1/2/3 encoder I know output raw streams (In fact, most MP3 players cannot even handle MP3 muxed into normal MPEG-1, also it´s the standard container for it).
but well you are the dev, i cant force you to anything...
all i can say it might be very easy to add mp4 output when looking at the mpeg4ip sources i pointed you to
or you use the lib from gpac (as used in mp4box)OK, mp4 support might be good, because raw m4v is nearly unuseable, I´ll have at look at it.
It isn't, and simply being able to plug in a new exe without having to worry about recompiling is a nice thing to have.Plug in a new xvidcore.dll... without having to worry about changed command line syntax... But it´s your tool, your decision.

Has anybody tested the tool so far? Please post your experience with it.

[bond]

any news about the project?

[S_O]

I´m currently waiting for bug reports. I don´t think it´s fine to always add new features without fixing bugs, that makes the tool unuseable.
So please, report something like
"I used it with commandline ... and noticed that ...." or yust "I used it with commandline ... and it seemed to work fine"
But without any feedback I can´t fix anything (or I doesn´t even know that there is nothing to fix).

[Doom9]

is there going to be any change in the commandline interface (akin what bond suggested) or is it going to remain as it is?
And at the risk of flogging a dead horse, so far ffmpeg is the only app that can put a raw m4v into an AVI. For the forseeable future, XviD is most likely to be stored in AVI (existing tools that are available, standalone compatiblity). This might change over time but for now it would be really nice to not having to use ffmpeg. raw streams are much more workable when you add them to mp4 because that's where we have the tools that can handle those streams. avi would also help for mkv (at least until the native storage becomes the standard for asp content).

[buzzqw]

maybe :stupid:

but i not realized how to use -size parameter (i always get the same size)

could you write 2 line for example ?

Thanks a lot

BHH

[bond]

i think one of the points of using this tool is to make it easier to not have to go via .avi
i doubt many people will use it for .avi output, cause vfw is simply better known and accepted

better add .mp4 support :D

[Doom9]

cause vfw is simply better known and acceptedTrue.. but think if we can get people to move away from VDub.. it's not like most users insist on VDub, it's that the apps they use (GKnot, AutoGK, etc) require these tools. If they used a generic purpose encoder (like MeGUI does), it will become much easier to switch out the container once they are ready to.. MP4 output instead of AVI is just a few clicks away in MeGUI. But that move will not happen over night.. it takes time and it's best to give people a soft update path (first the tool, with the output staying the same), then at some later point the container can change as well.

[bond]

thats indeed a point

i just fear the glue of virtualdub. simply look at how many people use x264vfw, altough the cli blows the vfw codec away clearly :scared:

[Sirber]

it takes time and it's best to give people a soft update path (first the tool, with the output staying the same), then at some later point the container can change as well.Like when I removed RMVB output from RealAnime? ;)

[Doom9]

Like when I removed RMVB output from RealAnime?You can of course always force people to their luck :)
i just fear the glue of virtualdubVirtualdub is sticky mostly because of the editing features. I think encoding could easily be switched out (except of course for the codecs that are only available as VfW version, filtering has long since moved mostly to AviSynth, audio encoding is done via other tools because there's no VBR in VDub, but there's no tool that allows you to open an MP4 or MKV and to navigate to frames, set cut points, split and merge. True, we can split and merge but those are non graphical tools, and as long there's no preview, the "why doesn't VirtualDub open MP4 files" question will stick around. But perhaps a DShow based preview tool that serves as a cutting / merging and muxing / demuxing frontent for the existing mp4/mkv tools might scratch that itch just enough.

[S_O]

is there going to be any change in the commandline interface (akin what bond suggested) or is it going to remain as it is?
I´m currently thinking about it. Do you (or anybody else) have a good idea how to change it in a way, that it is still possible to aktivate greyscale in general, but deactivate it again in a zone etc.
XviD options can be very complex, controlling it all via commandline is difficult.
And at the risk of flogging a dead horse, so far ffmpeg is the only app that can put a raw m4v into an AVI. For the forseeable future, XviD is most likely to be stored in AVI (existing tools that are available, standalone compatiblity). This might change over time but for now it would be really nice to not having to use ffmpeg. raw streams are much more workable when you add them to mp4 because that's where we have the tools that can handle those streams. avi would also help for mkv (at least until the native storage becomes the standard for asp content).
Yesterday 23:34Is there no application for avi muxing that accepts stdin input? It should be quite simple for every GUI to directly mux to avi. I´m not familar with avi, but it should be possible.
but i not realized how to use -size parameter (i always get the same size)

could you write 2 line for example ?It could be like this:
xvid_encraw -i script.avs -type 2 -o output.m4v -quality 6 -pass1 statsfile
for first pass and for second pass:
xvid_encraw -i script.avs -type 2 -o output.m4v -quality 6 -pass2 statsfile -size 102400
This will (should) create a 100MB m4v.
i think one of the points of using this tool is to make it easier to not have to go via .avi
i doubt many people will use it for .avi output, cause vfw is simply better known and accepted

better add .mp4 supportThat´s also my opinion
True.. but think if we can get people to move away from VDub.. it's not like most users insist on VDub, it's that the apps they use (GKnot, AutoGK, etc) require these tools. If they used a generic purpose encoder (like MeGUI does), it will become much easier to switch out the container once they are ready to.. MP4 output instead of AVI is just a few clicks away in MeGUI. But that move will not happen over night.. it takes time and it's best to give people a soft update path (first the tool, with the output staying the same), then at some later point the container can change as well.You´re right, but I still think avi support is not directly needed inside, what about an external avi muxer, with stdout/stdin
Virtualdub is sticky mostly because of the editing features. I think encoding could easily be switched out (except of course for the codecs that are only available as VfW version, filtering has long since moved mostly to AviSynth, audio encoding is done via other tools because there's no VBR in VDub, but there's no tool that allows you to open an MP4 or MKV and to navigate to frames, set cut points, split and merge. True, we can split and merge but those are non graphical tools, and as long there's no preview, the "why doesn't VirtualDub open MP4 files" question will stick around.A new VirtualDub is really needed, not based on the limitations of vfw/acm, and platform-indipendant, using it´s own plug-in system, also for muxing/demuxing. ChristianHJW claimed to start a tool like this months/years ago (I think it was called "The Core Media Editor"). Never seen anything of it.

[Sirber]

Could you add MKV support too? I'm not very friendly with MP4... :D Thanks!!

[buzzqw]

Quote:
but i not realized how to use -size parameter (i always get the same size)

could you write 2 line for example ?
It could be like this:
xvid_encraw -i script.avs -type 2 -o output.m4v -quality 6 -pass1 statsfile
for first pass and for second pass:
xvid_encraw -i script.avs -type 2 -o output.m4v -quality 6 -pass2 statsfile -size 102400
This will (should) create a 100MB m4v.

Thanks !

BHH

P.S. : i like too mp4/mkv support

[dimzon]

i like too mp4/mkv support
And don't forget about CQM ;)
:thanks:

[S_O]

And don't forget about CQMIs already supported:
-mpeg_quant -mpeg_cqm filename
CQM File is the same format used by XviD vfw or ffdshow

With containers, I´ll see.

[Brother John]

I´m currently thinking about it. Do you (or anybody else) have a good idea how to change it in a way, that it is still possible to aktivate greyscale in general, but deactivate it again in a zone etc.
Two ways of doing this come to my mind:

1.
Exclusively use + to turn an option on and - to turn it off. For the cases, where this doesn't make sense (e.g. input file) use --. That way we would get a nice and clear
xvid_encraw --i script.avs --type 2 --o output.m4v --pass1 stats.file +greyscale --z w 1000 0.5 -greyscale

2.
Always use - to start an option and add a trailing + or - where appropriate. That way -gmc+ would turn GMC on and -gmc- would turn GMC off.

[bond]

is using "+" also possible on for example linux based systems?

[S_O]

is using "+" also possible on for example linux based systems?I can use everything. There is no rule that says commandline switches have to start with a special char.
Exclusively use + to turn an option on and - to turn it off. For the cases, where this doesn't make sense (e.g. input file) use --. That way we would get a nice and clearIndeed, that´s pretty nice.
I think it´s better than a trailing + or -

[bond]

I can use everything. There is no rule that says commandline switches have to start with a special char.hm there are things which can conflict, eg you cant use ";" in a cmdl for linux prompt compatibility

Indeed, that´s pretty nice.yep, i also think that it should be the same for all options: + for enabling, - for disabling

i dont think the -- is necessary, it maybe just makes things complicated, i would simply use + here too

I think it´s better than a trailing + or - hm i would prefer the trailing +/- cause using + at the front somehow looks very very strange :D

[The Link]

i dont think the -- is necessary, it maybe just makes things complicated, i would simply use + here too

Wouldn't "--" make sense if one just want to use a quality preset (and this way doesn't have to care about individually switching features on or off)? Or did I misunderstand sth. in this regard?

[dimzon]

yep, i also think that it should be the same for all options: + for enabling, - for disabling
i prefer 1 for enabling and 0 for disabling :)

[bond]

i would say keep it simple and use the same logic everywhere

[Doom9]

for booleans, wouldn't it be better to either have the commandline contain something, or not contain anything which would mean the opposite? e.g. why -gmc if gmc is not turned on by default? why not -gmc to turn it on, and if there's no option called gmc, it's not turned on at all? a boolean, after all, can only have two values: true or false... there's no undetermined state so to me it makes no sense to introduce such a third state via commandline. After all, what do you do if somebody leaves out -/+gmc? Aborting in this state would meant that every possible parameter has to be crammed into the commandline, which may just cause problems simply for the fact that commandlines can't have an arbitrary length (so long zone strings are potential cause for cutoffs), and then with such long commandlines there's a much higher probability of a typing error.

And for zones, why not use a similar format to what other cli encoders. I envisaged the following format if I ever added full xvid zones to mencoder (I wrote the patch to add basic zones): -zones 1,100,w,0.5,KGOC3/200,300,q,20,K/400,500,q,20/600,700,q,20

where KGOC are exactly the options used in the VfW: K = start with a keyframe, G = grayscale mode, O = use chroma optimizer, C = cartoon mode and if there's any number in that string, it's the bvop sensitivity. The rest of the options is as follows:
start frame, end frame, weight/quantizer, modifier. If there's a 4th comma, whatever comes after it and before the / (or a space which signifies the end of the zones string) are modifiers for the current zone.

And I'm not aware of a commandline that muxes raw streams from the stdin. Plus there's always the issue about these tools not always be willing to cooperate when used from a programming environment. E.g. whihle avs2yuv and mencoder can be piped just fine on the commandline, or running a batch file, no matter how I tried, I couldn't pipe the avs2yuv stdout to mencoder's stdin.. mencoder would always exit for no good reason, leaving avs2yuv pumping out data that had to be discarded.

[bond]

for booleans, wouldn't it be better to either have the commandline contain something, or not contain anything which would mean the opposite? e.g. why -gmc if gmc is not turned on by default? why not -gmc to turn it on, and if there's no option called gmc, it's not turned on at all? +/- allows easier handling of zones

if you only have one option for enabling you either
1) cant disable anything in a zone which is set in the main settings if the main settings are valid for the whole enocde
2) or you have to set all options in every zone seperately if there are no main settings valid for the whole encode, which is a real pita (imagine someone setting a keyframe for every chapterpoint (very likely situation imho) you would need to repeat the whole settings for every chapterpoint zone...)

[Doom9]

1) cant disable anything in a zone which is set in the main settings if the main settings are valid for the whole enocdeuhh, the way I read the source code of the VfW encoder, before a frame is encoded, the zones are consulted. If a zone is found which includes the frame in question, the properties requested from the encoder are updated to match what the zone asks for. Thus, it's perfectly possible to have the bvop sensitivity on a zone override the one set as global parameter. Likewise for chroma, and the rest of the options are mutually exclusive (there's no option in the zones and in the general config), at least in the VfW.

I've just been browsing the code again and looking at the platform SDK in the background: A frame is compressed via the ICM_COMPRESS (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/multimed/htm/_win32_icm_compress.asp) and this is mapped to the compress function in codec.c (the mapping is in driverproc.c). And if you look at that function (which is called for every frame), it contains a call to apply_zone_modifiers and that function does exactly what I described above. Hence your point is invalid, it's perfectly possible to apply these 5 modifiers for each zone completely independent of what's configured for the entire movie. Furthermore, my suggestion also takes care of the following problem: a user does not realize that a frame extends from it's start frame to the start of the subsequent zone.. it's perfectly possible to add a zone end frame (not so straightforward to code though because the zone struct still makes it to the encoder, so you can't just change it and the alternative is having a separate list of zone ends, or make a patch to the core which adds that property - then somebody could expose ithis in the VfW frontend as well.


or you have to set all options in every zone seperately if there are no main settings valid for the whole encode, which is a real pitafor the abovementioned, this isn't the case at all. Stuff like the number of b-frames, quantizer matrix, qpel, gmc, etc, it's all global, the only thing that applies to zones are the 5 parameters I mentioned.

Plus you know very well that zones to match chapters are completely superfluous. Any proper DS decoder can jump to any given frame, and even standalone players have no problem jumping to an arbitrary position via the go-to function.. at least the 3 I have can do that without problem (but not a single one supports chapters ;)) An "apply these zone modifiers to all zones" would be a GUI function imho.. not even the VfW offers this and I've not come across any complaints there, but if there's a demand for such a function, I could always add it to MeGUI.

[Doom9]

upon looking at the full list of supported options in the latest release... phew, that's a lot to take in. I expected to only find what's in the VfW. How is the separation between "applies to a zone" and applies overall made? Using an approach where you have one zone string would take care of that.. anything outside that string is a general option, anything inside a zone string only applies to that zone.

Here's an example: -zones 0,q,20,+trellis+hpel/101,w,1.0,+greyscale
(other options with some separator between the modifiers are also possible of course, like a comma)

Using that approach, it's even possible to switch out +option with -option. Looking at the options that begin with a +, none uses a minus anywhere in the string, so you could use the standard -option to activate, and nothing means not activated. That way, there is no dual sign approach that might be confusing. I think that when zones are used, they would be grouped together in the commandline anyway for a better overview, so why not join them into a single string.

[bond]

Using that approach, it's even possible to switch out +option with -option. Looking at the options that begin with a +, none uses a minus anywhere in the string, so you could use the standard -option to activate, and nothing means not activated. That way, there is no dual sign approach that might be confusing. I think that when zones are used, they would be grouped together in the commandline anyway for a better overview, so why not join them into a single string.hm now if you want to be sure that 50 frames become keyframes how would the cmdl look like?

[Doom9]

-zones A,w,1.0,-if/B,w,1.0,-if/C,w,1.0,-if/etc

If I'm reading the manual right, right now it would look like this:

-z "A w 1.0 -if" -z "B w 1.0 -if" -z "C w 1.0 -if" etc

there's no "make these 50 frames to a keyframe" function anyway, you have to define every zone start frame, zone type, modifier (1.0 since you only want to force a keyframe), and any other options you like (in your case -if to force the I-frame at the start). Thus either way, there's a lot of typing to be done. But I could imagine a file open dialog in a GUI app that would read a list of frame numbers from a file, and create such zones automatically for you to simplify things. Thinking about megui, that wouldn't be so hard to add with the zone functionality I already have. Unlike xvid's core, I can extend the zone struct to accomodate additional flags (and they already support a start and end frame, the end frame is just never used in case of xvid.. unlike this is about to change in xvid's core) and -if is definitely a flag I would add.

[Sagittaire]

@ S_O

this project will be really interessing I think ...

Ready for global ISO 14496 open source project ... ???

x264 is perhabs at this time the best video codec (only beta for Nero HP). XviD or Libavcodec are some excellent MPEG4 ASP codec. FAAC a good AAC encoder. MP4Box an excellent muxer, splitter and more for mp4 ...

Why not a global fusion of all these project:
- ISO 14496-14 container MP4: MP4Box
- ISO 14496-17 subtitles TTXT: MP4Box
- ISO 14496-02 video MPEG4 ASP: XviD or Libavcodec
- ISO 14496-10 video MPEG4 AVC: x264
- ISO 14496-03 audio MPEG4 AAC: FAAC

ISO14496.exe CLI like producer.exe CLI for RV10 ... ???

[Doom9]

@saggitaire: isn't that just a tad bit ot here?

[Doom9]

@S_O: any thoughts on the proposed changes?

[Isochroma]

I cannot find a way to set the framerate - the -framerate option specified anywhere in the command line doesn't affect the output - which is always 25.00 FPS!

xvid_encraw -i "G:\Shinjuku\AVISynth Scripts\Ghost in the Shell\GITS-RS.Tex.AVS" -framerate 23.976 -type 2 -w 720 -h 464 -custom_par 32 27 -quality 6 +hqacp +inter4v -max_bframes 4 -bitrate 10000000 -single -qpel +me_advd16 +me_advd8 +me_hpelr16 +me_hpelr8 +me_qpelr16 +me_qpelr8 +me_gmer +me_exts16 +me_exts8 +rated +rd_bf +rd_hpelr16 +rd_hpelr8 +rd_qpelr16 +rd_qpelr8 +rd_exts -iquants 1 1 -pquants 1 1 -o "d:\test.m4v"

[Doom9]

I've found a means to mux raw m4v into AVI so I'm all game for integrating encraw into MeGUI, but I'd like to have the commandline thing settled..

[S_O]

I've found a means to mux raw m4v into AVI so I'm all game for integrating encraw into MeGUI, but I'd like to have the commandline thing settled..Expect a new release in a few days, the commandline structure will change like proposed here.
I cannot find a way to set the framerate - the -framerate option specified anywhere in the command line doesn't affect the output - which is always 25.00 FPS!I can confirm the bug, the fps-switch is ignored when reading avs, the avs framerate is always used (you can work around by setting another framerate in your script). Expect it to be fixed in next release.

[Isochroma]

The framerate in my AVS was 23.976

[S_O]

The framerate in my AVS was 23.976Are you sure? What has been displayed?
Oh, I think I have an possible explanation of what happend:
xvid_encraw has used 23,976 fps, no problems. But framerate is not stored in the bitstream, so when you mux it with mp4box, mp4box uses a default framerate of 25fps. You need to add "-fps 23.976" in the mp4box commandline.

[Doom9]

doesn't mp4box only pick up the framerate if you feed it with an mp4 containing video (or perhaps avi as well). iirc you always need to specify the framerate for raw content, regardless of whether the bitstream contains the framerate or not.

[S_O]

In avi or mp4 the bitrate is stored, so mp4box reads it when it opens such a file. But in raw m4v a bitrate is never stored. So mp4box uses default value of 25fps. In case you mux a raw-movie now with a framerate other than 25fps, you need specify this in the commandline using the -fps switch, otherwise the movie will have a wrong framerate of 25fps, and I think that´s exactly the problem that Isochroma has noticed.

[JoeBG]

Expect a new release in a few days, the commandline structure will change like proposed here.


Any new releases? Great projekt :thanks:

[Doom9]

since it seems to take longer than anticipated (hey, it happens), could you tell us which of the proposed schemes you're going to adopt so that I can start my preparations?

[JoeBG]

Hi guys,

since 4 hours I´m testing my tesclip (885 frames) xvid_encraw. I get horrible pictures with the following skript and can´t improve it with ever I try:

echo Pass 1 Xvid
echo ************
echo.
xvid_encraw -i Film.avs -type 2 -asm -par PAL169 -quality 6 -pass1 Xvid.log -bitrate 1400 -max_bframes 2 -qpel -vhq 4 -turbo -z "q 800 40" -o Film.m4v
echo.
echo Pass 2 Xvid
echo ************
echo.
xvid_encraw -i Film.avs -type 2 -asm -par PAL169 -quality 6 -pass2 Xvid.log -bitrate 1400 -max_bframes 2 -qpel -vhq 4 -z "q 800 40" -o Film.m4v

[JoeBG]

I found the mistake myself. If I rename "Xvid.log" to "statsfile" in both passes, everything works fine. I really don´t know why.

Edit:
Just one word to the tool: It´s great. Mencoder Xvid is slower and always creates too small outputs. But my first tests with xvid_encraw are all very good.

[JoeBG]

Sorry guys, I´m testing the tool and there are two other questions coming up :)

-vhq, default is [1.4]. Does this mean a range between 1 and 4 and can I choose 4 like in vfw codec?

-quality 6: Which features does this include? All features with a "+" ? So that I can deaktivate them when I want? I´m interested in a complete list please. :)

Sorry for so many questions. But it´s agreat tool and I will use it correct :)

[Yuri Khan]

I've found a means to mux raw m4v into AVI so I'm all game for integrating encraw into MeGUI, but I'd like to have the commandline thing settled..
Could you please elaborate? Is it an external commandline muxer that accepts m4v input, or just an idea how to implement one?

To put it short, my typical use case is a series of 26 24-minute episodes, each of which has to be encoded in a separate file and wants an individual bitrate (sometimes up to 100% difference), but with all other settings common. So the only part of the encoder settings that changes is the stats filename and the target size. And I’m looking for a command line driven encoder that I could invoke from a generated batch file (or maybe even from a makefile).

For now, I see three candidates: (1) VirtualDubMod (but then I have to understand how it encodes the binary codec settings and to watch for settings changes when upgrading the codec); (2) mencoder (but it undersizes each episode by ~20MB); and (3) xvid_encraw which seems almost perfect but I need to find out what to do with raw m4v output.

[JoeBG]

...but I need to find out what to do with raw m4v output.

mux it into mp4 with mp4box or mux it into mkv with mkvtoolnix

[Yuri Khan]

-quality 6: Which features does this include? All features with a "+" ? So that I can deaktivate them when I want? I´m interested in a complete list please. :)
Judging by the source:
-quality 1 implies +me_advd16
-quality 2 adds +me_hpelr16 and +hpel
-quality 3 adds +me_advd8, +me_hpelr8, +inter4v, and if you’re using -qpel, then also +me_qpelr8
-quality 4 adds +chromap and +chromab
-quality 5 adds +trellis
-quality 6 adds +me_exts16, +me_exts8 and +hqacp

mux it into mp4 with mp4box or mux it into mkv with mkvtoolnix
And then remux into avi… I think I’d rather take a plunge into mencoder sources :)

[Doom9]

mencoder can mux raw ASP streams into an avi.. the commandline is the same as what megui uses for the AVI muxer, just with an .m4v instead of an .avi input. Once encraw is ready to be incorporated, I'll unlock the filetype selector in the avi muxer to make it possible to directly mux in MeGUI. even raw AVC muxing should be possible, but due to yet another bug in mencoder, only if you specify the FPS of the source.

[Yuri Khan]

Heh, figured it out too…
mencoder -ovc copy -ffourcc XVID test.m4v -o test.aviSomehow, without -ffourcc, it puts FMP4 in the headers.

Not that I’m happy with using a sophisticated tool like mencoder for mundane muxing tasks, but hey, if it works, why not :)

[Beave]

to mix the raw file into avi with mencoder I tried the cli from megui:
"mencoder.exe" "test.m4v" -ovc copy -oac copy -audiofile "test.ac3" -mc 0 -noskip -o "test.avi"
It just stops without ending. I guess I need to add framerate and Aspect information to the commandline, right?

[Yuri Khan]

@Beave:
You need to specify the audio parameters such as bitrate, sample rate and channel count. Interestingly, you can get the required information with mplayer:
mplayer.exe -frames 0 -identify test.ac3and watch for lines starting with:
ID_AUDIO_BITRATE
ID_AUDIO_RATE
ID_AUDIO_NCHYou then issue a command like this:
mencoder.exe -ovc copy -oac copy -audiofile test.ac3 -audio-demuxer 20 -demuxer lavc -rawaudio format=0x2000 -rawaudio bitrate=384000 -rawaudio channels=6 -rawaudio rate=48000 test.m4v -ffourcc XVID -o test.avisubstituting your own values for 384000, 6 and 48000.

@Doom9:
I have found that, with most raw XviD .m4v files, mencoder selects libavformat demuxer and the resulting avi is all right, but some files are identified as MPEG4-ES. In the latter case, it produces an avi file whose headers contain zeros in the video width and height fields. It is not exactly b0rked, as mplayer still can play them; but also not completely compatible as, for example, Media Player Classic refuses to render such 0×0 files. Thus, I suggest that the switch -demuxer lavf be used explicitly to select libavformat demuxer, which passes the correct width and height to the muxer.

[Doom9]

@S_O: any news?

[naysayer]

I have a feature request: output to stdout.

I used to pipe avs2yuv to ffmpeg via stdin/stdout, with no temp file, to save to avi like this:
avs2yuv 1.avs - | ffmpeg -f yuv4mpegpipe -i - 1.avi

I've been using xvid_encraw now, and saving with "-o out.m4v", and then using mencoder or ffmpeg to mux into an avi.
What would be great though, is if I could avoid the temp m4v file and pass the video to ffmpeg like I used to with avs2yuv, like:
xvid_encraw -i 1.avs -o - | ffmpeg -f rawvideo -i - 1.avi

I tried variations of "xvid_encraw -i 1.avs -o -" already, and it only writes to disk with a file named " - "...

[708145]

I have a feature request: output to stdout.

I used to pipe avs2yuv to ffmpeg via stdin/stdout, with no temp file, to save to avi like this:
avs2yuv 1.avs - | ffmpeg -f yuv4mpegpipe -i - 1.avi

I've been using xvid_encraw now, and saving with "-o out.m4v", and then using mencoder or ffmpeg to mux into an avi.
What would be great though, is if I could avoid the temp m4v file and pass the video to ffmpeg like I used to with avs2yuv, like:
xvid_encraw -i 1.avs -o - | ffmpeg -f rawvideo -i - 1.avi

I tried variations of "xvid_encraw -i 1.avs -o -" already, and it only writes to disk with a file named " - "...

what about named pipes? they work under cygwin at least.

bis besser,
T0B1A5

[708145]

Is -frames ignored in avs mode?
And a related question: could -start be added as well as in x264.exe?

That would help me a lot with ELDER :)

P.S.: What's the status on avi output? If it's not planned then I have to ship ffmpeg with ELDER... just want to avoid depending on too many tools.

bis besser,
T0B1A5

[JoeBG]

@S_O: any news?

Any News?

[Doom9]

I know S_O is still around but not posting anymore. Sad :(

[JoeBG]

I know S_O is still around but not posting anymore. Sad :(

He is preparing something special I hope :yes:

[mirthandir]

When I compile and run the xvid_encraw.cpp posted it throws an exception

LN 1004: "res = env->Invoke("Import", AVSValue(&arg, 1));"

Anybody have any suggestions? Is there something moronic that I'm missing?

[JoeBG]

Any news?

[Zero1]

It's great to be able to finally use encraw :)
I was going to make some anamorphic ASP in MP4, for some testing, and I came across this weird error. To try and eliminate the cause I re-encoded it without the AR tag.

I was wondering if there are any knowns errors, or perhaps someone could tell what the problem is by looking at it and suggest disabling a feature manually?

This is the command I am using:
xvid_encraw.exe -i "Video1.avs" -framerate 23.976 -type 2 -w 704 -h 480 -quality 6 -vhq 4 -chroma_opt -max_bframes 2 -bitrate 1500000 -pass2 stats.log -max_key_interval 240 -iquants 2 31 -pquants 2 31 -bquants 2 31 -o "test2.m4v"

And here is the "artifact" (not so much of an artifact, just looks like b0rked Macroblocks?).

http://img420.imageshack.us/img420/6896/encrawblocks6wj.th.jpg (http://img420.imageshack.us/my.php?image=encrawblocks6wj.jpg)

(That was made using directshowsource)

Many thanks

[JoeBG]

Still hoping for an update :)

[Doom9]

Me too - I'd love to use this in megui to offer the full featureset of xvid, but it looks like somebody else has to start coding.

[Kopernikus]

Is there anything else than zones for enc_raw that I didnt see when looking through the thread?

I think zones could easy be implemented as a plugin, but I don't know if I have time to try it.

[Doom9]

zones are included.. it's just the syntax that is a bit weird. S_O said he was going to change it - then nothing ever happened...

[Kopernikus]

is the syntax proposed by BrotherJohn the one that should be implemented?

[Doom9]

I still insist the one I proposed (which corresponds to syntax other codecs use) is better. Are you volunteering to finishing encraw? I would except I have no time (there will be a new codec comparison, and when I have time I should continue working on MeGUI) and don't like C (and I don't suppose you could do this project in Visual Studio).

[JoeBG]

For me there is still the open question with the stats file. You have to give the the stats file of the second pass another name than the one of the first pass. This looks like as if the stats file of the first pass is not used in the second pass. I tested this and if you only make a second pass ( without any first pass) the quality is the same as if you make a first pass and a second pass. => The first pass with xvid_encraw seems to be totally useless. Can someone confirm?

[Zero1]

Syntax like the ones used in x264.exe CLI would be absolutely ideal in my opinion.

Any comments on my messed up screenshot yet guys?

Cheers :)

[JoeBG]

Syntax like the ones used in x264.exe CLI would be absolutely ideal in my opinion.

Any comments on my messed up screenshot yet guys?

Cheers :)

Where is your total commandline? No first pass is used?

[Zero1]

This wasn't the command I used at the time (since then I've edited my batch and lost the settings) but these settings reproduce the problem.

xvid_encraw.exe -i "Video1.avs" -framerate 23.976 -type 2 -w 704 -h 480 -quality 6 -vhq 4 -chroma_opt -max_bframes 2 -bitrate 1500000 -pass1 stats.log -max_key_interval 240 -iquants 2 31 -pquants 2 31 -bquants 2 31 -o "test1.m4v"
xvid_encraw.exe -i "Video1.avs" -framerate 23.976 -type 2 -w 704 -h 480 -quality 6 -vhq 4 -chroma_opt -max_bframes 2 -bitrate 1500000 -pass2 stats.log -max_key_interval 240 -iquants 2 31 -pquants 2 31 -bquants 2 31 -o "test2.m4v"

Yes, the first file was 2 pass, the line I posted was from the second pass. IIRC the first pass was identical except for --pass and the output was to a different file. It still used the same log file as pass 2 though.

Thanks :)

[JoeBG]

This wasn't the command I used at the time (since then I've edited my batch and lost the settings) but these settings reproduce the problem.

xvid_encraw.exe -i "Video1.avs" -framerate 23.976 -type 2 -w 704 -h 480 -quality 6 -vhq 4 -chroma_opt -max_bframes 2 -bitrate 1500000 -pass1 stats.log -max_key_interval 240 -iquants 2 31 -pquants 2 31 -bquants 2 31 -o "test1.m4v"
xvid_encraw.exe -i "Video1.avs" -framerate 23.976 -type 2 -w 704 -h 480 -quality 6 -vhq 4 -chroma_opt -max_bframes 2 -bitrate 1500000 -pass2 stats.log -max_key_interval 240 -iquants 2 31 -pquants 2 31 -bquants 2 31 -o "test2.m4v"

Yes, the first file was 2 pass, the line I posted was from the second pass. IIRC the first pass was identical except for --pass and the output was to a different file. It still used the same log file as pass 2 though.

Thanks :)


So you have the same problem like me. Give the statsfile of the second pass another name and it will look better. But you (and me) will never know if the first pass was totally useless.

[Doom9]

It seems encraw in the CSV has been updated as well and it should support all options available in the VfW except the profiles, and AviSynth input. So fire up your compilers ladies and gentlemen.

[Kopernikus]

I dont see any changes in the CVS. The last update of encraw dates 8 weeks ago. (at least here :))

[JoeBG]

Sorry for pointing this again:

Xvid Encraw ist useless because it does not use the statsfile of the first. Can someone confirm this?

[Doom9]

well, Isibaar told me what I posted here.. I'm sure I'll get a copy shortly to check this for myself and since I'm using MP4 for the next codec comparison, I plan to use the commandline encoder rather than the VfW encoder if somehow possible.

[JoeBG]

well, Isibaar told me what I posted here.. I'm sure I'll get a copy shortly to check this for myself and since I'm using MP4 for the next codec comparison, I plan to use the commandline encoder rather than the VfW encoder if somehow possible.

Sorry for bagging you so soon. Any news?

[Doom9]

when the time comes you'll know... it is not up to me to release unreleased software.

[JoeBG]

when the time comes you'll know... it is not up to me to release unreleased software.

But it´s hot news. So please keep us informed - thank you :)

[sysKin]

Hi, what's the current status of xvid_encraw? I volunteer to add anything I can add - this weekend if needed. What's left/broken/wrong? Any patches waiting to be checked in?

[Doom9]

@syskin: you might wanna talk to Isibaar about that.

@JoeBG: I run a news site so you can bet you'll know when I have something to share.

[Sagittaire]

Hi, what's the current status of xvid_encraw? I volunteer to add anything I can add - this weekend if needed. What's left/broken/wrong? Any patches waiting to be checked in?

cool ... :cool:

1) psnr calculation
2) it's xvid_encoraw but mp4 output could be great
3) all possible advanced ME search
4) multipasse encoding : we can use pseudo multipass with XviD vfw and zone option. For example if my final average quant encoding is ~q4 I use q4 for first pass in zone option. First pass -> write 1pass.log, second pass -> read 1pass.log and write 2pass.log ... etc

[bond]

basically enabling all xvid options in the cli would be great (you can use S_Os code as a startpoint)
.mp4 output, well ;) you can use mpeg4ip's or gpac's lib which are save (mpeg4ip has a xvid cli outputting mp4 in their cvs, which might be a modified encraw?)
.avs input

[Doom9]

guys, please. I have it on good authority that encraw is done (except for the profiles you can select in the GUI), it would make little sense for syskin to re-invent the wheel.

[bond]

guys, please. I have it on good authority that encraw is done (except for the profiles you can select in the GUI), it would make little sense for syskin to re-invent the wheel.with .mp4 output and .avs input?

[Doom9]

I doubt there's mp4 output. But if anybody were going to add that, it's better to wait until the CSV code (I don't know why you can't see it.. is there a shadow CSV? In the past year I've not seen much of activity anywhere so I'm wondering if development has moved to channels I'm not aware). Anyway, the submission deadline for codecs is the 16th so by that time I should have that encraw and if it's ready for release it will show up here as well.

[bond]

I doubt there's mp4 output. But if anybody were going to add that, it's better to wait until the CSV code (I don't know why you can't see it.. is there a shadow CSV? In the past year I've not seen much of activity anywhere so I'm wondering if development has moved to channels I'm not aware). Anyway, the submission deadline for codecs is the 16th so by that time I should have that encraw and if it's ready for release it will show up here as well.so .avs input is supported?

[Doom9]

yes, as I said, there is no point continuing the source from this thread. Now, patience is a virtue..

[Sagittaire]

$Id : xvid_encraw.c, v1.11.2.282003/06/2523:23:21 edgomez Exp $

xvid_encraw is edgomez's code ... ???

[JoeBG]

Hi, what's the current status of xvid_encraw? I volunteer to add anything I can add - this weekend if needed. What's left/broken/wrong?

I have posted the following problem very often but never got a feedback:
For me it seems to be that the first pass in xvid_encraw is useless because I have the suspicion, that xvid_encraw does not use the informations of the statsfile. The second pass is totally independent from the first pass. I have just the request, that someone with more skills than me can check this before the next release - thank you very much :)

[sysKin]

For me it seems to be that the first pass in xvid_encraw is useless because I have the suspicion, that xvid_encraw does not use the informations of the statsfile. The second pass is totally independent from the first pass. I have just the request, that someone with more skills than me can check this before the next release - thank you very much :)

I just looked around, apparently you need to specify stats filename with
-pass1 xvid.stats

and then

-pass2 xvid.stats

If you don't, everything goes weird. I fully admit encoder should be more rebust than this (and should not allow -pass1 and -pass2 in the same command, just in case).

[JoeBG]

I just looked around, apparently you need to specify stats filename with
-pass1 xvid.stats

and then

-pass2 xvid.stats

If you don't, everything goes weird. I fully admit encoder should be more rebust than this (and should not allow -pass1 and -pass2 in the same command, just in case).

I don´t think so. Please look at the problem if the commandlinde works:

@echo off
title Jempeg4-1400
echo.
echo Pass 1 Xvid
echo ************
echo.
%Xvid% -i %Videofolder%\Film.avs -type 2 -pass1 %Videofolder%\xvid.stats -asm -quality 6 +chroma_opt -max_bframes 2 -bquant_ratio 150 -bquant_offset 100 -bf_thres 0 -bitrate %bitrate% -max_key_interval 250 -pquants 1 31 -bquants 1 31 -iquants 1 31 +trellis -mpeg_quant -lumimasking +hpel -qpel -vhq 4 +rd_bf -turbo -o %Videofolder%\Film.m4v
cls
echo.
echo Pass 2 Xvid
echo ************
echo.
%Xvid% -i %Videofolder%\Film.avs -type 2 -pass2 %Videofolder%\xvid.stats -asm -quality 6 +chroma_opt -max_bframes 2 -bquant_ratio 150 -bquant_offset 100 -bf_thres 0 -bitrate %bitrate% -max_key_interval 250 -pquants 1 31 -bquants 1 31 -iquants 1 31 +trellis -mpeg_quant -lumimasking +hpel -qpel -vhq 4 +rd_bf -keyframe_boost 100 -close_i_red 1 20 -ccp_high 0 -ccp_low 5 -max_oi 5 -max_od 6 -overf_cs 10 -o %Videofolder%\Film.m4v
cls



To say it directly: This does not work: statsfile will not get read in the second pass and it would be very nice, if someone could check this or if someone could tell me my the mistake.

I can only point out, that xvid_encraw does not work and I hope that I´m making a mistake. Please confirm :)

[Sagittaire]

little test with xvid_encraw.exe

1) 2 pass encoding work fine
2) it's possible to make NPass encoding ... lol

here CLI for Multipasse with XviD

xvid_encraw.exe -i Encodage.avs -type 2 -o first.m4v -pass1 statsfile1 -asm +hqacp +inter4v +hpel +chromap +chromab +me_advd16 +me_advd8 +me_sqr16 +me_sqr8 +me_hpelr16 +me_hpelr8 +me_gmer +me_exts16 +me_exts8 +rated +rd_bf +rd_hpelr16 +rd_hpelr8 +rd_chk_pred +rd_exts +trellis -lumimasking -max_bframes 2 -bquant_ratio 150 -bquant_offset 0 -stats
xvid_encraw.exe -i Encodage.avs -type 2 -o second.m4v -pass1 statsfile2 -pass2 statsfile1 -size 13500 -asm +hqacp +inter4v +hpel +chromap +chromab +me_advd16 +me_advd8 +me_sqr16 +me_sqr8 +me_hpelr16 +me_hpelr8 +me_gmer +me_exts16 +me_exts8 +rated +rd_bf +rd_hpelr16 +rd_hpelr8 +rd_chk_pred +rd_exts +trellis -lumimasking -max_bframes 2 -bquant_ratio 150 -bquant_offset 0 -stats
xvid_encraw.exe -i Encodage.avs -type 2 -o third.m4v -pass2 statsfile2 -size 13500 -asm +hqacp +inter4v +hpel +chromap +chromab +me_advd16 +me_advd8 +me_sqr16 +me_sqr8 +me_hpelr16 +me_hpelr8 +me_gmer +me_exts16 +me_exts8 +rated +rd_bf +rd_hpelr16 +rd_hpelr8 +rd_chk_pred +rd_exts +trellis -lumimasking -max_bframes 2 -bquant_ratio 150 -bquant_offset 0 -stats

MP4Box.exe -nodrop -add third.m4v third.mp4


Multipass example:

I want 13500 Ko final size for my encoding

first pass q2 done 34741 Ko
second pass with first stat file done 13312 Ko
third pass with second stat file done 13500 Ko

and quality for 3 pass is better than 2 pass quality with better target bitrate and certainely better vbv reliability (if vbv are used) ... lol

[IgorC]

Sagi
thank you for information.

[JoeBG]

little test with xvid_encraw.exe

1) 2 pass encoding work fine
2) it's possible to make NPass encoding ... lol

here CLI for Multipasse with XviD

xvid_encraw.exe -i Encodage.avs -type 2 -o first.m4v -pass1 statsfile1 -asm +hqacp +inter4v +hpel +chromap +chromab +me_advd16 +me_advd8 +me_sqr16 +me_sqr8 +me_hpelr16 +me_hpelr8 +me_gmer +me_exts16 +me_exts8 +rated +rd_bf +rd_hpelr16 +rd_hpelr8 +rd_chk_pred +rd_exts +trellis -lumimasking -max_bframes 2 -bquant_ratio 150 -bquant_offset 0 -stats
xvid_encraw.exe -i Encodage.avs -type 2 -o second.m4v -pass1 statsfile2 -pass2 statsfile1 -size 13500 -asm +hqacp +inter4v +hpel +chromap +chromab +me_advd16 +me_advd8 +me_sqr16 +me_sqr8 +me_hpelr16 +me_hpelr8 +me_gmer +me_exts16 +me_exts8 +rated +rd_bf +rd_hpelr16 +rd_hpelr8 +rd_chk_pred +rd_exts +trellis -lumimasking -max_bframes 2 -bquant_ratio 150 -bquant_offset 0 -stats
xvid_encraw.exe -i Encodage.avs -type 2 -o third.m4v -pass2 statsfile2 -size 13500 -asm +hqacp +inter4v +hpel +chromap +chromab +me_advd16 +me_advd8 +me_sqr16 +me_sqr8 +me_hpelr16 +me_hpelr8 +me_gmer +me_exts16 +me_exts8 +rated +rd_bf +rd_hpelr16 +rd_hpelr8 +rd_chk_pred +rd_exts +trellis -lumimasking -max_bframes 2 -bquant_ratio 150 -bquant_offset 0 -stats

MP4Box.exe -nodrop -add third.m4v third.mp4


Multipass example:

I want 13500 Ko final size for my encoding

first pass q2 done 34741 Ko
second pass with first stat file done 13312 Ko
third pass with second stat file done 13500 Ko

and quality for 3 pass is better than 2 pass quality with better target bitrate and certainely better vbv reliability (if vbv are used) ... lol

You are not right with this. 2 pass Encoding is broken. If you try the following you will see, that you only get ggod quality if you rename statsfile of second pass to statsfile 2 and that you get the same quality without any first pass.

But try yourself:
xvid_encraw.exe -i Encodage.avs -type 2 -o first.m4v -pass1 statsfile1 -asm +hqacp +inter4v +hpel +chromap +chromab +me_advd16 +me_advd8 +me_sqr16 +me_sqr8 +me_hpelr16 +me_hpelr8 +me_gmer +me_exts16 +me_exts8 +rated +rd_bf +rd_hpelr16 +rd_hpelr8 +rd_chk_pred +rd_exts +trellis -lumimasking -max_bframes 2 -bquant_ratio 150 -bquant_offset 0 -stats
xvid_encraw.exe -i Encodage.avs -type 2 -o second.m4v -pass2 statsfile1 -size 13500 -asm +hqacp +inter4v +hpel +chromap +chromab +me_advd16 +me_advd8 +me_sqr16 +me_sqr8 +me_hpelr16 +me_hpelr8 +me_gmer +me_exts16 +me_exts8 +rated +rd_bf +rd_hpelr16 +rd_hpelr8 +rd_chk_pred +rd_exts +trellis -lumimasking -max_bframes 2 -bquant_ratio 150 -bquant_offset 0 -stats

[JoeBG]

No Reaction?

Sorry, but this problem is bagging me since months and Sagittaire did not use a 2 pass commandline for his test. I´m really wondering, why noone is interested in this? Did noone test this? Xvid_Encraw is such an interesting project. Noone interested in this? Noone is testing a 2 pass commandline since months and noone is corresponding to this problem.

@ All

If noone will answer on this problem again, the xvid commandline for xvid_encraw will be dead - and I really wonder why noone is interested.

@ doom9

Don´t use xvid_encraw before this important problem is fixed. The first pass is useless.

But more important is, why noone is intersted in this problem? MeGUI with mencoder-Xvid is very bad because it never matches the size. Xvid Encraw is a solution but noone is really interested. But why? I dont understand all this. Has someone answers?

[dimzon]

Xvid Encraw is a solution but noone is really interested. But why? I dont understand all this. Has someone answers?
I can tell you only about myself. I'm switched from XviD to x264 encoding 6 month back...

[708145]

broken 2pass explains a lot. I use xvid_encraw in ELDER's xvid mode and had strange size matching problems.

A bugfix would help me, too.

bis besser,
T0B1A5

[JoeBG]

broken 2pass explains a lot. I use xvid_encraw in ELDER's xvid mode and had strange size matching problems.

A bugfix would help me, too.

bis besser,
T0B1A5

But noone is interested in this - there will be no help for you :(

I can tell you only about myself. I'm switched from XviD to x264 encoding 6 month back...

Do you think we shoul close this request? I´m really looking every day at this threat but noone is answering and noone is interested.

@ doom9

I´m so disappointed with this project. Mencoder is really a bad solution for meGUI, xvid_encraw is really a much worser solution than mencoder for meGui. So we should forget our targets for a Xvid commandline solution? Have you ever seen a video with a 2 pass xvid_encraw commandline?

[bond]

didnt doom9 say that there is already someone working on the cli? simply be patient ;)

[JoeBG]

didnt doom9 say that there is already someone working on the cli? simply be patient ;)

OK. But after such a long time with this problem and no solution for it I really don´t think that doom9 has the time to test it - especially the quality of the resulting videos. Do you really have hope for this project?

[Kurtnoise]

@echo off
title Jempeg4-1400
echo.
echo Pass 1 Xvid
echo ************
echo.
%Xvid% -i %Videofolder%\Film.avs -type 2 -pass1 %Videofolder%\xvid.stats -asm -quality 6 +chroma_opt -max_bframes 2 -bquant_ratio 150 -bquant_offset 100 -bf_thres 0 -bitrate %bitrate% -max_key_interval 250 -pquants 1 31 -bquants 1 31 -iquants 1 31 +trellis -mpeg_quant -lumimasking +hpel -qpel -vhq 4 +rd_bf -turbo -o %Videofolder%\Film.m4v
cls
echo.
echo Pass 2 Xvid
echo ************
echo.
%Xvid% -i %Videofolder%\Film.avs -type 2 -pass2 %Videofolder%\xvid.stats -asm -quality 6 +chroma_opt -max_bframes 2 -bquant_ratio 150 -bquant_offset 100 -bf_thres 0 -bitrate %bitrate% -max_key_interval 250 -pquants 1 31 -bquants 1 31 -iquants 1 31 +trellis -mpeg_quant -lumimasking +hpel -qpel -vhq 4 +rd_bf -keyframe_boost 100 -close_i_red 1 20 -ccp_high 0 -ccp_low 5 -max_oi 5 -max_od 6 -overf_cs 10 -o %Videofolder%\Film.m4v
cls



To say it directly: This does not work: statsfile will not get read in the second pass and it would be very nice, if someone could check this or if someone could tell me my the mistake.

I can only point out, that xvid_encraw does not work and I hope that I´m making a mistake. Please confirm :)

I just tested with xvid_encraw from the 1.1.0 sources and it seems to work fine without specifying an output file for the 1st pass...

xvid_encraw.exe -i Birth_test.avs -type 2 -pass1 Birth_test.stats -quality 6 -bitrate 50000 -max_bframes 2 -turbo
xvid_encraw.exe -i Birth_test.avs -type 2 -pass2 Birth_test.stats -o Birth_out.m4v -quality 6 -bitrate 50000 -max_bframes 2 -vhqmode 4 -bvhq

[snherbst]

Hi

Iam not good in this commandline tool either. Thats proberly because its not possible to just chose a profile. But any way which command line would be equal to the "Advanced Simple @ L5" Profile.

[Sagittaire]

I just tested with xvid_encraw from the 1.1.0 sources and it seems to work fine without specifying an output file for the 1st pass...

you can post this version here ... ???

[Kurtnoise]

sure...http://kurtnoise.free.fr/xvid_encraw-1.1.0.zip

[JoeBG]

sure...http://kurtnoise.free.fr/xvid_encraw-1.1.0.zip

You make my day :) By the way: -bitrate 50000 <- seems to be a littly high ;)

Edit:

Same Problem here. You have to give the statsfile of the second pass another name than the statsfile of the first pass if you want quality. => Xvid_Encraw does not read from statsfile of the first pass -> the statsfile of the first pass and the first pass itself is totally useless. If you only make a second pass without the first pass you have the same quality.

=> Sorry Kurt, your commandline does not work.

[Kurtnoise]

By the way: -bitrate 50000 <- seems to be a littly high ;)
It was just a crash test... :)


Same Problem here. You have to give the statsfile of the second pass another name than the statsfile of the first pass if you want quality.
huh ? for what for ?

=> Xvid_Encraw does not read from statsfile of the first pass
mmh...to be sure : make only a 1st pass first. Then, rename the stats file or put it in other folder. Second, make an encode with 2 pass (by using the same filename concerning stats file) and compare the different stats files...(diff is good for that). I'm pretty sure that the files are the same. This means that xvid_encraw read the stats from the 1st pass.

[buzzqw]

very dumb question

how i can mux/play with the resultin .raw file ??? :stupid:

BHH

[Yong]

very dumb question

how i can mux/play with the resultin .raw file ??? :stupid:

BHH
First you have to rename the raw xvid file to *.xvid extension,
then mux the raw xvid file with mp4box or mp4creator.
You can playback the raw xvid file with mplayer. ;)

[buzzqw]

@Yong
Thanks !

i successfully muxed raw xvid with mp4creator but latest versione of mp4bix (on celticdruid site) failed with an "unknow input file type"

thanks !

BHH

[Doom9]

mp4box can mux it just fine.. but it expects the .m4v extension. That also works for mp4creator by the way.

[buzzqw]

Thanks too Doom9 !

renaming movie.raw to movie.m4v resolved the unkown input error with mp4box !

BHH

[JoeBG]

Same Problem here. You have to give the statsfile of the second pass another name than the statsfile of the first pass if you want quality. => Xvid_Encraw does not read from statsfile of the first pass -> the statsfile of the first pass and the first pass itself is totally useless. If you only make a second pass without the first pass you have the same quality.

=> Sorry Kurt, your commandline does not work.

I really think, that noone ever tested xvid_encraw like I did.

Which tool makes snapshots from raw files or mp4 files? I will post my results in pictures to show you the problem.

@ Kurt

I have made a first pass, then copied the statsfile into a new folder. Then I made the second pass. How can I compare the two statsfiles? Diff is not a commandline function isn´t it?

[Kurtnoise]

this is a command line tool...;) Check this site (http://gnuwin32.sourceforge.net/packages.html). (diffutils)

And to be more clear...I said.

1/ Make an encode in one pass and save the statsfile.
2/ Make a full encode with 2 passes like you do by changing the filename.
3/ Then compare the stats files...

[Yong]

I think this cli apps indeed have problem,
like JoeBG said.
I tried encode video with 2nd pass without 1st pass stats file, it doesnt show any error when encoding start :rolleyes:

But xvid_encraw does use 1stpass stats file, without it xvid_encraw will create a big files ;)

[JoeBG]

And to be more clear...I said.

1/ Make an encode in one pass and save the statsfile.
2/ Make a full encode with 2 passes like you do by changing the filename.
3/ Then compare the stats files...

I will never understand this testing procedure or the difference to what I already did.


1/ Make an encode in one pass and save the statsfile.
Do you mean a first pass of an 2 Pass Encode? That´s what I already did. I wrote a skript with a stop after the first pass.



2/ Make a full encode with 2 passes like you do by changing the filename.
Why not starting the second pass? Why new encoding in 2 passes. Why changing the filename. I already stored a copy of the statsfile in a different folder.
=> I think I´m not intelligent enough to understand the procedure :(


3/ Then compare the stats files..

Which ones?

[bond]

guys, please. I have it on good authority that encraw is done (except for the profiles you can select in the GUI), it would make little sense for syskin to re-invent the wheel.ok now 1 month is gone. any news on this "done" tool? isibaar? :D

[Doom9]

well.. what was done was posted here. I compiled a list of missing options from that release and sent it to syskin.

[bond]

well.. what was done was posted here. I compiled a list of missing options from that release and sent it to syskin.hm, you mean the 1.1 compile from kurtnoise includes all the changes from isibaar? or did i oversee some other thingie being posted?

i wonder because when i look at the cvs (http://www.xvid.org/cvs/chora/cvs.php/xvidcore/examples?login=2&sbt=1)i only see changes being 3 months old, made by suxen_drol, but not by isibaar!?

[Doom9]

hm, you mean the 1.1 compile from kurtnoise includes all the changes from isibaar?I don't know who wrote it, but it's the current state of encraw. And considering there's a whole new codec not in the CVS I could start a whole series of rumours about a parallel secret CVS or whatnot. Bottom line.. you can get what's posted in this thread and that's it. MeGUI already supports that featureset (not publicly released yet).. if syskin makes any update to the CSV they will be supported.

[bond]

well time for testing then :D

[bond]

if thats the final version it definitely cant keep up with S_O's version. altough this +/- sheme is messy it still offers as good as all options, whereas this 1.1 build definitely does not

[Doom9]

Well, there are in fact multiple major issues I found when giving it more than a 1000 frames to test. I'm in the process of sending a long bugreport to Michael. Some of the issues JoeBG reported are definitely there, but it does get a lot worse :(

Encoding past frame 9999 is impossible.

CBR bitrate control is broken (encodes everything at Q2).
2 pass encoding with a non existing statsfile is possible, and it results in a Q2 stream
2 pass encoding with an existing statsfile results in the majority of all frames being encoded at Q31.. naturally that means the size is off.
The statsfile generated during the first pass is correct, and it can be used to make a second pass via VfW that comes out just as it should.
This behavior seems to happen irrespective of the settings (I tried a bunch of settings but obviously not every permutation).

@JoeBG: that's the kind of analysis I think is reasonable to make.. it gives you a lot more pointers than your "it doesn't work, try it" ;)

[squid_80]

Encoding past frame 9999 is impossible.That's a simple #define in the source code, there's a comment that it's only for testing short sequences.2 pass encoding with a non existing statsfile is possible, and it results in a Q2 streamSame thing happens with the vfw interface if the statsfile is bad.

If people want the features of S_O's build why not just merge the sources with the current CVS code?

[Doom9]

If people want the features of S_O's build why not just merge the sources with the current CVS code?somebody needs to authorize that.. and S_O's build is quite messy, so I'd probably not looking forward to that if I'd be in a position to chose.

P.S. The bitrate has to be given in bits/s but all the rate control issues are still there.

[Doom9]

I have an update: CBR RC is not broken.. but encraw needs the bitrate in bits/s (I suspected but because the results varied all over the place I got confused), and it does work.. it's not as accurate as 2 pass though (2 pass in the VfW of course). I did another 2 pass test with bits/s bitrate for both passes.. it was only a trailer, and it wasn't spot on either, but 200KB off.. doing a full movie now to check.

[squid_80]

If I'm reading the code right, all the extra parameters for 2nd pass like overflow treatment and curve compression are left as 0. Same goes for the CBR settings (Reaction delay, averaging period and smoother). Fixing them to defaults is pretty easy, if anyone's still interested I can post a modified build.

[dimzon]

because with .mp4 output you can edit the output in various tools and dont need mp4box or mp4creator :p
And Yet Another Reason - you can preview right after encoding (without any additional tools)

Seem's like I can spend a little time to add AVI output to this utility (reuse some part of avs2avi code)...

[squid_80]

Seem's like I can spend a little time to add AVI output to this utility (reuse some part of avs2avi code)...
Done it already (didn't mention that in my previous post :)).

[bond]

if you want .avi output why not simply use vfw in virtualdub?

better add support for some real containers, like mkv or mp4 ;)

[squid_80]

if you want .avi output why not simply use vfw in virtualdub?

better add support for some real containers, like mkv or mp4 ;)
..And here in person is the reason why I didn't say anything. ;)
.avi output is a start. Once I figure out what the calls are for mkv (maybe mp4, but look at the trouble gpac causes with x264) I'll probably add that too.

[bond]

..And here in person is the reason why I didn't say anything. ;) you guessed right ;)

edit:
1) shame on you added avi support!
2) if you add mkv support make sure the cli writes the stream in "native mpeg-4" and not in the vfw mode as virtualdubmod, as i really dont see the sense in recreating with the cli whats already possible easily in other ways...

(maybe mp4, but look at the trouble gpac causes with x264) I'll probably add that too. gpac doesnt really cause trouble to x264... the only issue caused by gpac itself i know is that sometimes plain cvs checkouts bork, but then again its no wonder that cvs isnt always stable. after all its also not needed to use latest cvs in most cases
the rest was caused by issues in x264's calling of gpac (one time not setting the track enabled flag, the other time having issues with the 64bit times)
thats basically it

than again you can also use the mp4 lib from mpeg4ip if you dont like gpac...

[Doom9]

@squid_80:
Here's the list of missing features I compiled:

Additional zone options: (start with i-frame, grayscale, cartoon, chroma optimizer, bvop sensitivity),
minimum and maximum quantizer, trellis,
top field first mode for interlaced, PAR/DAR settings, VBV options and chroma motion.

Fixing them to defaults is pretty easy, if anyone's still interested I can post a modified build.It would be nice to be able to control those since they're currently missing (goes for both 2 pass and 1 pass RC settings)
all the RC related options aren't there.


And when it comes to XviD 1.2, a new option like -nbthreads would be nice to control how many threads are being used.

Done it alreadyIs it already in the CVS?

[Zero1]

I guess if you are going to add MKV and MP4 support that's great, so long as we can still output raw (outputting raw and muxing later has saved me a few times when people have been complaining about x264's borked mp4 out).

[squid_80]

@squid_80:
Here's the list of missing features I compiled:

Additional zone options: (start with i-frame, grayscale, cartoon, chroma optimizer, bvop sensitivity),
minimum and maximum quantizer, trellis,
top field first mode for interlaced, PAR/DAR settings, VBV options and chroma motion.
It would be nice to be able to control those since they're currently missing (goes for both 2 pass and 1 pass RC settings)
all the RC related options aren't there. Here's what I've changed the default settings to (most of this is just to match vfw):
- bframes to 2, adjustable using -max_bframes
- Key frame interval to 300, adjustable using -max_key_interval
- Default quality = 6, should match vfw's motion search precision settings
- Unrestricted profile (currently not adjustable)
- Chroma motion on, can be disabled with -nochromamotion
- Trellis on, can be disabled with -notrellis
- Default VHQ mode = 1, should match vfw's settings
- Packed mode and closed gop are on, can be disabled with -nopacked and -noclosed_gop
- PAR can be set using -par <integer> where 1 = square, 2 = 4:3 PAL, 3 = 4:3 NTSC etc... currently working on custom support.
- AVI output possible using -avi <filename.avi> (for verification purpose only, the avi files are produced using the AVIFile API so they're a bit rough, most people would probably remux them with audio anyway.) Note that it's possible to output a raw file and avi at the same time.
- Fixed single, 1pass and 2pass parameters to default vfw values. I'll make cli parameters for these as soon as I get a chance. Also look for the stats file in 2pass and don't proceed if it's not there.
- Force min/max quants to 2/31. Again cli parameters are coming (I'm thinking probably -imin, -imax, -pmin etc.)
And when it comes to XviD 1.2, a new option like -nbthreads would be nice to control how many threads are being used. There's actually a way for xvidcore.dll to provide the number of cpu cores to the calling program, but at the moment it doesn't work (returns 0). So I'll make it use this value by default then when the functionality is added to xvidcore it should select the correct number of threads automatically. I'll add an override switch anyway, since encraw is all about diagnostics.
Is it already in the CVS?
Nope, I don't have access to xvid's CVS. I'm just working on my own here (but I see Isibaar did recently check in a fix for that 9999 frame limit).

[Doom9]

@squid: sounds like it's finally moving forward :) Doesn't VDub use AVIFile as well to write AVIs? Naturally in the end it would be great if you could submit a patch to xvid-devel so that the whole thing can be merged.
Isibaars patch finally fixed encoding for me.. I'm currently encoding the second movie and then check for size discrepancies if there's any.

[Doom9]

Alright, I did some more testing with a build that no longer has the frame number limit and no bugs with regards to that. I encoded two movies in two pass, the filesize matched what I wanted by a few 100 KB.. so the usual XviD accuracy. I'm now redoing one second pass with a non existing stats file to see what that results in. But I think it's safe to say that encraw does what it's supposed two in two pass mode.
With 10% done, it looks like without a stats file everything is being encoded at quantizer 2.. the projected filesize is way too big.

[squid_80]

It'll still meet the filesize but with all the 2nd pass RC options set to 0 doesn't it look a bit...off? Especially if there's sudden jumps from high to low motion or vice-versa.

[sam1974]

Is it possible that it breaks encodig after frame no. 9999. After this -1 follows and it breaks. I've tested it with different sources.

[Doom9]

@sam1974: I think reading up a bit would hurt: http://forum.doom9.org/showpost.php?p=767689&postcount=187

[squid_80]

Is it possible that it breaks encodig after frame no. 9999. After this -1 follows and it breaks. I've tested it with different sources.
I'm guessing you mean this type of output: 9997: key=0, time= 0, len= 6 | type=P, quant= 4, len= 5883
9998: key=0, time= 47, len= 10416 | type=B, quant= 7, len= 2884
9999: key=0, time= 0, len= 6 | type=P, quant= 4, len= 7538
-1: key=0, time= 47, len= 11666 | type=B, quant= 7, len= 3460
-1: key=0, time= 0, len= 6 | type=P, quant= 4, len= 8212
-1: key=0, time= 0, len= -5
Tot: enctime(ms) =9495.00, length(bytes) = 1842441
Avg: enctime(ms) = 31.44, fps = 31.81, length(bytes) = 6100The -1 reference just means it's reached the end of the input frames and is flushing any remaining output frames. It happens at frame 9999 because that's a hardcoded limit that used to exist. It has since been removed.

[handtruck]

I looked up and down the thread.. What happend to full quality first pass.. It's listed on the first page as part of the switches (-full_1p), but doesn't work in usage, and doesn't show up in -help.

[shon3i]

What i get will full quality first pass. Did i get better quality.

[Kopernikus]

I looked up and down the thread.. What happend to full quality first pass.. It's listed on the first page as part of the switches (-full_1p), but doesn't work in usage, and doesn't show up in -help.

Whose build do you use?

[handtruck]

I used the attachment on the very first post of this thread (which I believe has been updated), and I also tried the most recent link on this thread as well.

[woah!]

where do you get your patched versions squid_80?

the one in the first post isnt your newer one is it, or arent you releasing them just yet?

[squid_80]

where do you get your patched versions squid_80?

the one in the first post isnt your newer one is it, or arent you releasing them just yet?
I haven't released any yet, as soon as I implement all the options I mentioned in my post to Doom9 I'll make it available.

[woah!]

ah thats cool i will stop searching for it then heh..

[squid_80]

Ok, here's an unrefined build: ftp://squid80.no-ip.com/xvid_encraw.zip

The usage text should list the new options, unless I've left them off. Note the bitrate is in kbps, not bps. The reason I'm posting it unfinished is because I've just spent two hours figuring out why I was losing/gaining frames with the .avi output - it should be fixed now, but if you end up with the wrong number of frames please let me know and try and copy the final output screen. Also please try out the zone options, the functionality hasn't changed but the code behind them has.
Todo: implement the remaining options (rate control, zone options, custom par/dar, vbv... anything else?)
Stricter argument checking.
Support for other output formats (hoping for DaveEL to dig up his avs2matroska sources).
Sleep..

Bugs: -debug is broken, don't know why (it was like that when I got here!).
It's probably not a good idea to run more than one instance simultaneously.
Parameters can be repeated with different values on the command line if you're feeling silly - don't think I'll bother fixing this.

[buzzqw]

@squid_80
just to reserve you some bandwith i made a link on my site

www.64k.it/andres/xvid_encraw-build-20-01-2006.zip

if is OK i will host it until a new build come up

thanks !

BHH

[buzzqw]

xvid_encraw.exe -i movie.avs -type 2 -pass1 xvid.stats -bitrate 1000 -full1pass -max_bframes 2 -bquant_ratio 150 -bquant_offset 100 -framerate 25.000000 -turbo -quality 6 -vhqmode 1 -max_key_interval 250 -imin 2 -bmin 2 -pmin 2 -par 1 -qtype 0 -avi movie.avi

is not fuctional, the bug(?) is in full1pass. If i remove it the string is ok

BHH

[buzzqw]

even -nopacked is broken. It will encode all file, then will continue ... (ultil a ctrl+c ;) )

BHH

[squid_80]

Can you try again with the new download please?

[buzzqw]

OK !!!

both

xvid_encraw.exe -i movie.avs -type 2 -pass1 xvid.stats -bitrate 515 -full1pass -max_bframes 2 -bquant_ratio 150 -bquant_offset 100 -framerate 25.000000 -quality 6 -vhqmode 1 -noclosed_gop -max_key_interval 250 -imin 2 -bmin 2 -pmin 2 -par 1 -qtype 0 -avi movie.avi

and nopacked is OK!

i have already uploaded the new build at www.64k.it/andres/xvid_encraw-build-20-01-2006.zip

:thanks: squid_80 !

BHH

[squid_80]

BTW, the only thing -full1pass seems to do (going from the vfw code) is encode the whole movie at quant 2. So -bitrate, -imin -pmin etc. have no effect.

[buzzqw]

xvid_encraw.exe -i "movie.avs" -type 2 -pass1 xvid.stats -bitrate 515 -full1pass -max_bframes 2 -bqu
ant_ratio 150 -bquant_offset 100 -framerate 25.000000 -turbo -quality 6 -vhqmode 1 -noclosed_gop -ma
x_key_interval 250 -imin 2 -bmin 2 -pmin 2 -par 1 -qtype 0 -avi movie.avi"
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

Trying to retrieve width and height from input header
0: key=2, time= 0, len= 2145 | type=I, quant= 2, len= 2145
1: key=0, time= 0, len= 0
2: key=0, time= 0, len= 0
3: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
4: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
5: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
6: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
7: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
8: key=0, time= 31, len= 6 | type=P, quant= 2, len= 6
9: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
10: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
11: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
12: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
13: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
14: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
15: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
16: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
17: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
18: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
19: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
20: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
21: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
22: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
23: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
24: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
25: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
26: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
27: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
28: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
29: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
30: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
31: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
32: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
33: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
34: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
35: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
36: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
37: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
38: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
39: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
40: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
41: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
42: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
43: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
44: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
45: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
46: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
47: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
48: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
49: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
50: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
51: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
52: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
53: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
54: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
55: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
56: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
57: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
58: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
59: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
60: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
61: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
62: key=0, time= 32, len= 3463 | type=B, quant= 4, len= 764
63: key=0, time= 15, len= 764 | type=B, quant= 4, len= 764
64: key=0, time= 0, len= 6 | type=P, quant= 2, len= 2705
65: key=2, time= 15, len= 5031 | type=B, quant= 4, len= 1507
66: key=0, time= 0, len= 6 | type=I, quant= 2, len= 3530
67: key=2, time= 0, len= 4877 | type=I, quant= 2, len= 4877
68: key=2, time= 0, len= 4615 | type=I, quant= 2, len= 4615
69: key=2, time= 0, len= 4221 | type=I, quant= 2, len= 4221
70: key=2, time= 0, len= 4866 | type=I, quant= 2, len= 4866
71: key=0, time= 94, len= 5021 | type=P, quant= 2, len= 5021
72: key=0, time= 109, len= 6668 | type=B, quant= 4, len= 1825
73: key=0, time= 0, len= 6 | type=P, quant= 2, len= 4849
74: key=0, time= 109, len= 8748 | type=B, quant= 4, len= 1988
75: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6766
76: key=0, time= 62, len= 10099 | type=B, quant= 4, len= 1913
77: key=0, time= 0, len= 6 | type=P, quant= 2, len= 8192
78: key=0, time= 47, len= 9879 | type=B, quant= 4, len= 1688
79: key=0, time= 16, len= 1531 | type=B, quant= 4, len= 1531
80: key=0, time= 0, len= 6 | type=P, quant= 2, len= 8197
81: key=0, time= 63, len= 9110 | type=B, quant= 4, len= 1515
82: key=0, time= 16, len= 1411 | type=B, quant= 4, len= 1411
83: key=0, time= 0, len= 6 | type=P, quant= 2, len= 7601
84: key=0, time= 62, len= 10160 | type=B, quant= 4, len= 1515
85: key=0, time= 16, len= 1495 | type=B, quant= 4, len= 1495
86: key=0, time= 0, len= 6 | type=P, quant= 2, len= 8651
87: key=0, time= 62, len= 9014 | type=B, quant= 4, len= 1552
88: key=0, time= 0, len= 6 | type=P, quant= 2, len= 7468
89: key=0, time= 63, len= 10907 | type=B, quant= 4, len= 2165
90: key=0, time= 0, len= 6 | type=P, quant= 2, len= 8748
91: key=0, time= 79, len= 8409 | type=P, quant= 2, len= 8409
92: key=2, time= 16, len= 10202 | type=I, quant= 2, len= 10202
93: key=0, time= 63, len= 7777 | type=P, quant= 2, len= 7777
94: key=0, time= 46, len= 8951 | type=P, quant= 2, len= 8951
95: key=0, time= 47, len= 10556 | type=P, quant= 2, len= 10556
96: key=0, time= 47, len= 10345 | type=P, quant= 2, len= 10345
97: key=0, time= 63, len= 14491 | type=B, quant= 4, len= 2349
98: key=0, time= 0, len= 6 | type=P, quant= 2, len= 12148
99: key=0, time= 62, len= 11077 | type=B, quant= 4, len= 2044
100: key=0, time= 16, len= 6 | type=P, quant= 2, len= 9039
101: key=0, time= 47, len= 9762 | type=B, quant= 4, len= 1462
102: key=0, time= 0, len= 1556 | type=B, quant= 4, len= 1556
103: key=0, time= 0, len= 6 | type=P, quant= 2, len= 8306
104: key=0, time= 32, len= 9558 | type=B, quant= 4, len= 1496
105: key=0, time= 16, len= 1327 | type=B, quant= 4, len= 1327
106: key=0, time= 0, len= 6 | type=P, quant= 2, len= 8068
107: key=0, time= 31, len= 8950 | type=B, quant= 4, len= 1248
108: key=0, time= 0, len= 1321 | type=B, quant= 4, len= 1321
109: key=0, time= 0, len= 6 | type=P, quant= 2, len= 7708
110: key=0, time= 47, len= 9213 | type=B, quant= 4, len= 1294
111: key=0, time= 15, len= 1344 | type=B, quant= 4, len= 1344
112: key=0, time= 0, len= 6 | type=P, quant= 2, len= 7925
113: key=0, time= 31, len= 8991 | type=B, quant= 4, len= 1269
114: key=0, time= 16, len= 1398 | type=B, quant= 4, len= 1398
115: key=0, time= 0, len= 6 | type=P, quant= 2, len= 7728
116: key=0, time= 47, len= 9809 | type=B, quant= 4, len= 1318
117: key=0, time= 15, len= 1817 | type=B, quant= 4, len= 1817
118: key=0, time= 0, len= 6 | type=P, quant= 2, len= 8497
119: key=0, time= 62, len= 13535 | type=B, quant= 4, len= 2567
120: key=0, time= 0, len= 6 | type=P, quant= 2, len= 10974
121: key=0, time= 47, len= 9184 | type=P, quant= 2, len= 9184
122: key=0, time= 31, len= 8674 | type=P, quant= 2, len= 8674
123: key=0, time= 32, len= 8736 | type=P, quant= 2, len= 8736
124: key=0, time= 32, len= 8660 | type=P, quant= 2, len= 8660
125: key=0, time= 31, len= 7806 | type=P, quant= 2, len= 7806
126: key=0, time= 32, len= 7735 | type=P, quant= 2, len= 7735
127: key=0, time= 32, len= 8509 | type=P, quant= 2, len= 8509
128: key=0, time= 47, len= 9179 | type=P, quant= 2, len= 9179
129: key=0, time= 31, len= 10274 | type=P, quant= 2, len= 10274
130: key=0, time= 31, len= 10651 | type=P, quant= 2, len= 10651
131: key=0, time= 32, len= 11225 | type=P, quant= 2, len= 11225
132: key=0, time= 47, len= 11357 | type=P, quant= 2, len= 11357
133: key=0, time= 31, len= 9925 | type=P, quant= 2, len= 9925
134: key=2, time= 31, len= 15167 | type=B, quant= 4, len= 2965
135: key=0, time= 0, len= 6 | type=I, quant= 2, len= 12208
136: key=0, time= 78, len= 10860 | type=B, quant= 4, len= 2203
137: key=0, time= 0, len= 6 | type=P, quant= 2, len= 8663
138: key=0, time= 47, len= 9996 | type=P, quant= 2, len= 9996
139: key=0, time= 47, len= 10427 | type=P, quant= 2, len= 10427
140: key=0, time= 47, len= 10991 | type=P, quant= 2, len= 10991
141: key=0, time= 62, len= 10725 | type=P, quant= 2, len= 10725
142: key=0, time= 31, len= 9995 | type=P, quant= 2, len= 9995
143: key=0, time= 47, len= 13505 | type=B, quant= 4, len= 2435
144: key=0, time= 0, len= 6 | type=P, quant= 2, len= 11076
145: key=0, time= 46, len= 10644 | type=B, quant= 4, len= 1762
146: key=0, time= 0, len= 6 | type=P, quant= 2, len= 8888
147: key=0, time= 47, len= 11643 | type=B, quant= 4, len= 1792
148: key=0, time= 16, len= 1619 | type=B, quant= 4, len= 1619
149: key=0, time= 0, len= 6 | type=P, quant= 2, len= 9857


the first frames (black almost) is encoded at quant 2 but other at 2 or 4 (for b frames)

BHH

[squid_80]

the first frames (black almost) is encoded at quant 2 but other at 2 or 4 (for b frames)

BHH
....yeah, that's what I said. Bitrate and quantizer min/max don't do anything with -full1pass active. Quantizer for B-frames:
= avg(prev p quant + next p quant)*bquant_ratio + bquant_offset
= avg(2 + 2) *1.5 + 1
= 2*1.5+1
= 4.

This however worries me: 64: key=0, time= 0, len= 6 | type=P, quant= 2, len= 2705
65: key=2, time= 15, len= 5031 | type=B, quant= 4, len= 1507
66: key=0, time= 0, len= 6 | type=I, quant= 2, len= 3530
67: key=2, time= 0, len= 4877 | type=I, quant= 2, len= 4877
68: key=2, time= 0, len= 4615 | type=I, quant= 2, len= 4615A B-frame reported as a key frame and an I-frame not being reported as a key frame. Looks kinda weird but I think that's what you get with packed bitstream. The .avi file might be a bit b0rked, open it with vdub and skip through using the key buttons and see if any keyframes look messed up.

[buzzqw]

Ok, this time without packed frames

this is the command line

xvid_encraw.exe -i "C:\Programmi\PureBasic\Prove\movie.avs" -type 2 -pass1 xvid.stats -bitrate 515 -max_bframes 2 -bquant_ratio 150 -bquant_offset 100 -framerate 25.000000 -quality 6 -vhqmode 1 -nopacked -noclosed_gop -max_key_interval 250 -imin 2 -bmin 2 -pmin 2 -par 4 -qtype 0 -avi "C:\Programmi\PureBasic\Prove\movie.avi"

and this is the first encoded frames


C:\Programmi\PureBasic\Prove>C:\Programmi\PureBasic\Prove\exe\encoder\xvid_encraw.exe -i "C:\Progra
mmi\PureBasic\Prove\movie.avs" -type 2 -pass1 xvid.stats -bitrate 515 -max_bframes 2 -bquant_ratio 1
50 -bquant_offset 100 -framerate 25.000000 -quality 6 -vhqmode 1 -nopacked -noclosed_gop -max_key_in
terval 250 -imin 2 -bmin 2 -pmin 2 -par 4 -qtype 0 -avi "C:\Programmi\PureBasic\Prove\movie.avi"
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

Trying to retrieve width and height from input header
0: key=2, time= 0, len= 1677
1: key=0, time= 0, len= 0
2: key=0, time= 0, len= 0
3: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
4: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
5: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
6: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
7: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
8: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
9: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
10: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
11: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
12: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
13: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
14: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
15: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
16: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
17: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
18: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
19: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
20: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
21: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
22: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
23: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
24: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
25: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
26: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
27: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
28: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
29: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
30: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
31: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
32: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
33: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
34: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
35: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
36: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
37: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
38: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
39: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
40: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
41: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
42: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
43: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
44: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
45: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
46: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
47: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
48: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
49: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
50: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
51: key=0, time= 15, len= 6 | type=P, quant= 2, len= 6
52: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
53: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
54: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
55: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
56: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
57: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
58: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
59: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
60: key=0, time= 16, len= 6 | type=P, quant= 2, len= 6
61: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
62: key=0, time= 0, len= 2115 | type=P, quant= 2, len= 6
63: key=0, time= 0, len= 600 | type=B, quant= 4, len= 600
64: key=0, time= 0, len= 600 | type=B, quant= 4, len= 600
65: key=2, time= 0, len= 2999 | type=P, quant= 2, len= 2115
66: key=0, time= 0, len= 1184 | type=B, quant= 4, len= 1184
67: key=2, time= 16, len= 4177 | type=I, quant= 2, len= 2999
68: key=2, time= 16, len= 3557 | type=I, quant= 2, len= 4177
69: key=0, time= 15, len= 3950 | type=I, quant= 2, len= 3557
70: key=0, time= 16, len= 4714 | type=P, quant= 2, len= 3950
71: key=0, time= 16, len= 5261 | type=P, quant= 2, len= 4714
72: key=0, time= 0, len= 1893 | type=B, quant= 4, len= 1893
73: key=0, time= 32, len= 6198 | type=P, quant= 2, len= 5261
74: key=0, time= 15, len= 1482 | type=B, quant= 4, len= 1482
75: key=0, time= 15, len= 8606 | type=P, quant= 2, len= 6198
76: key=0, time= 16, len= 1953 | type=B, quant= 4, len= 1953
77: key=0, time= 16, len= 6563 | type=P, quant= 2, len= 8606
78: key=0, time= 16, len= 1307 | type=B, quant= 4, len= 1307
79: key=0, time= 0, len= 7858 | type=P, quant= 2, len= 6563
80: key=0, time= 0, len= 1414 | type=B, quant= 4, len= 1414
81: key=0, time= 0, len= 1272 | type=B, quant= 4, len= 1272
82: key=0, time= 16, len= 7627 | type=P, quant= 2, len= 7858
83: key=0, time= 0, len= 1216 | type=B, quant= 4, len= 1216
84: key=0, time= 16, len= 1168 | type=B, quant= 4, len= 1168
85: key=0, time= 15, len= 8266 | type=P, quant= 2, len= 7627
86: key=0, time= 15, len= 1301 | type=B, quant= 4, len= 1301
87: key=0, time= 0, len= 1275 | type=B, quant= 4, len= 1275
88: key=0, time= 0, len= 7126 | type=P, quant= 2, len= 8266
89: key=0, time= 16, len= 1272 | type=B, quant= 4, len= 1272
90: key=2, time= 16, len= 11673 | type=P, quant= 2, len= 7126
91: key=0, time= 15, len= 2345 | type=B, quant= 4, len= 2345
92: key=2, time= 0, len= 8483 | type=I, quant= 2, len= 11673
93: key=0, time= 0, len= 6748 | type=I, quant= 2, len= 8483
94: key=0, time= 0, len= 8067 | type=P, quant= 2, len= 6748
95: key=0, time= 15, len= 9605 | type=P, quant= 2, len= 8067
96: key=0, time= 16, len= 11307 | type=P, quant= 2, len= 9605
97: key=0, time= 31, len= 2107 | type=B, quant= 4, len= 2107
98: key=0, time= 16, len= 10798 | type=P, quant= 2, len= 11307
99: key=0, time= 16, len= 1950 | type=B, quant= 4, len= 1950
100: key=0, time= 46, len= 8874 | type=P, quant= 2, len= 10798
101: key=0, time= 16, len= 1219 | type=B, quant= 4, len= 1219
102: key=0, time= 0, len= 1157 | type=B, quant= 4, len= 1157
103: key=0, time= 47, len= 8092 | type=P, quant= 2, len= 8874
104: key=0, time= 0, len= 1167 | type=B, quant= 4, len= 1167
105: key=0, time= 0, len= 1348 | type=B, quant= 4, len= 1348
106: key=0, time= 16, len= 7644 | type=P, quant= 2, len= 8092
107: key=0, time= 15, len= 1140 | type=B, quant= 4, len= 1140
108: key=0, time= 0, len= 1141 | type=B, quant= 4, len= 1141
109: key=0, time= 15, len= 7585 | type=P, quant= 2, len= 7644
110: key=0, time= 0, len= 1152 | type=B, quant= 4, len= 1152
111: key=0, time= 16, len= 1008 | type=B, quant= 4, len= 1008
112: key=0, time= 0, len= 7306 | type=P, quant= 2, len= 7585
113: key=0, time= 16, len= 900 | type=B, quant= 4, len= 900
114: key=0, time= 16, len= 1017 | type=B, quant= 4, len= 1017
115: key=0, time= 15, len= 7645 | type=P, quant= 2, len= 7306
116: key=0, time= 0, len= 988 | type=B, quant= 4, len= 988
117: key=0, time= 16, len= 1008 | type=B, quant= 4, len= 1008
118: key=0, time= 31, len= 8684 | type=P, quant= 2, len= 7645
119: key=0, time= 16, len= 1797 | type=B, quant= 4, len= 1797
120: key=0, time= 15, len= 9793 | type=P, quant= 2, len= 8684
121: key=0, time= 0, len= 2150 | type=B, quant= 4, len= 2150
122: key=0, time= 16, len= 9496 | type=P, quant= 2, len= 9793
123: key=0, time= 15, len= 2105 | type=B, quant= 4, len= 2105
124: key=0, time= 16, len= 8932 | type=P, quant= 2, len= 9496
125: key=0, time= 16, len= 1927 | type=B, quant= 4, len= 1927
126: key=0, time= 15, len= 9062 | type=P, quant= 2, len= 8932
127: key=0, time= 0, len= 1830 | type=B, quant= 4, len= 1830
128: key=0, time= 15, len= 11486 | type=P, quant= 2, len= 9062
129: key=0, time= 16, len= 2487 | type=B, quant= 4, len= 2487
130: key=0, time= 15, len= 13209 | type=P, quant= 2, len= 11486
131: key=0, time= 16, len= 3100 | type=B, quant= 4, len= 3100
132: key=0, time= 0, len= 10700 | type=P, quant= 2, len= 13209
133: key=0, time= 16, len= 10458 | type=P, quant= 2, len= 10700
134: key=0, time= 15, len= 2690 | type=B, quant= 4, len= 2690
135: key=0, time= 16, len= 8613 | type=P, quant= 2, len= 10458
136: key=0, time= 15, len= 1913 | type=B, quant= 4, len= 1913
137: key=2, time= 0, len= 11732 | type=P, quant= 2, len= 8613
138: key=0, time= 32, len= 1976 | type=B, quant= 4, len= 1976
139: key=0, time= 15, len= 9670 | type=I, quant= 2, len= 11732
140: key=0, time= 0, len= 10419 | type=P, quant= 2, len= 9670
141: key=0, time= 32, len= 13053 | type=P, quant= 2, len= 10419
142: key=0, time= 15, len= 2471 | type=B, quant= 4, len= 2471
143: key=0, time= 0, len= 10982 | type=P, quant= 2, len= 13053
144: key=0, time= 0, len= 1906 | type=B, quant= 4, len= 1906
145: key=0, time= 15, len= 10414 | type=P, quant= 2, len= 10982
146: key=0, time= 16, len= 1450 | type=B, quant= 4, len= 1450
147: key=0, time= 15, len= 1623 | type=B, quant= 4, len= 1623
148: key=0, time= 16, len= 9577 | type=P, quant= 2, len= 10414
149: key=0, time= 0, len= 1126 | type=B, quant= 4, len= 1126
150: key=0, time= 0, len= 1229 | type=B, quant= 4, len= 1229


I big note: without packed bitstream the file size is correct, but with is not so correct (aiming at 800kb)

second pass with -nopacked -> size 988KB
-1: key=0, time= 0, len= -5 | type=P, quant= 3, len= 4234
Tot: enctime(ms) =5481.00, length(bytes) = 1002142
Avg: enctime(ms) = 17.29, fps = 57.84, length(bytes) = 3161

second pass without -> size KB 793
-1: key=0, time= 16, len= 3801 | type=P, quant= 3, len= 3801
-1: key=0, time= 0, len= -5
Tot: enctime(ms) =5655.00, length(bytes) = 802655
Avg: enctime(ms) = 17.84, fps = 56.06, length(bytes) = 2532


i know the sample is short ... i would like to know why :stupid:

thanks

BHH

[squid_80]

Ok, this time without packed frames

this is the command line

xvid_encraw.exe -i "C:\Programmi\PureBasic\Prove\movie.avs" -type 2 -pass1 xvid.stats -bitrate 515 -max_bframes 2 -bquant_ratio 150 -bquant_offset 100 -framerate 25.000000 -quality 6 -vhqmode 1 -nopacked -noclosed_gop -max_key_interval 250 -imin 2 -bmin 2 -pmin 2 -par 4 -qtype 0 -avi "C:\Programmi\PureBasic\Prove\movie.avi"Aargh.. "-type 2 -max_bframes 2 -bquant_ratio 150 -bquant_offset 100 -framerate 25.000000 -quality 6 -vhqmode 1 -qtype 0" are all defaults. No need to clog up the commandline with them. Also bitrate does nothing on 1st pass, it is only useful for -single or -pass2.
and this is the first encoded frames


61: key=0, time= 0, len= 6 | type=P, quant= 2, len= 6
62: key=0, time= 0, len= 2115 | type=P, quant= 2, len= 6
63: key=0, time= 0, len= 600 | type=B, quant= 4, len= 600
64: key=0, time= 0, len= 600 | type=B, quant= 4, len= 600
65: key=2, time= 0, len= 2999 | type=P, quant= 2, len= 2115
66: key=0, time= 0, len= 1184 | type=B, quant= 4, len= 1184
67: key=2, time= 16, len= 4177 | type=I, quant= 2, len= 2999
68: key=2, time= 16, len= 3557 | type=I, quant= 2, len= 4177
69: key=0, time= 15, len= 3950 | type=I, quant= 2, len= 3557
70: key=0, time= 16, len= 4714 | type=P, quant= 2, len= 3950
71: key=0, time= 16, len= 5261 | type=P, quant= 2, len= 4714
72: key=0, time= 0, len= 1893 | type=B, quant= 4, len= 1893
That makes it a bit easier to see. The stats on the left are the true order, the stats on the right are from the stored order. You can tell by matching up the sizes - b-frames are stored after the frames they reference.
I big note: without packed bitstream the file size is correct, but with is not so correct (aiming at 800kb)

second pass with -nopacked -> size 988KB
-1: key=0, time= 0, len= -5 | type=P, quant= 3, len= 4234
Tot: enctime(ms) =5481.00, length(bytes) = 1002142
Avg: enctime(ms) = 17.29, fps = 57.84, length(bytes) = 3161

second pass without -> size KB 793
-1: key=0, time= 16, len= 3801 | type=P, quant= 3, len= 3801
-1: key=0, time= 0, len= -5
Tot: enctime(ms) =5655.00, length(bytes) = 802655
Avg: enctime(ms) = 17.84, fps = 56.06, length(bytes) = 2532


i know the sample is short ... i would like to know why :stupid:

thanks

BHHPacked bitstream is not a parameter that can be changed without redoing the first pass.

[JoeBG]

Works great, no problems anymore.

Next I try a whole film and not only my testclips.

Is there any update from the old xvidcore.dll ?

[Doom9]

Is there any update from the old xvidcore.dll ?Since the API has not been changed afaik, it should be possible to replace xvidcore.dll with the latest one.. even one compiled for 2 cores.

[buzzqw]

two note:

i had redo both first and secon pass with nopacked, and another two round without (but not deleting stats file :o )

i am using latest xvidcore dll for 2 core

a very big :thanks: squid_80 ! for your good work !!

BHH

[JoeBG]

In encodet the same video which I tried before with mencoder xvid. Mencoder xvid never fitted the size for me, but with xvid_encraw it works. Thanks to all

[woah!]

i have a weird problem

my encodes come out at 29.000 frames instead of 29.970 ?

same avs encded with mencoder gives the correct 29.970fps so i dont think its a script error

crop(8,8,-8,-8)
LeakKernelDeint(order=1, sharp=true)
ColorMatrix("Rec.709->Rec.601",mmx=true,hints=false)
LanczosResize(704,400)
trim(20000, 20900)

heres the xvid info:

-type 2 -cq 3 -avi

http://images.dr3vil.com/files/default/xvid_encraw.jpg

heres the mencoder info:

-ovc xvid -xvidencopts fixed_quant=3:max_key_interval=300:packed:vhq=1:qpel:chroma_me:quant_type=mpeg:trellis:lumi_mask:bvhq=1

http://images.dr3vil.com/files/default/mencoder.jpg

also heres the info from MPC:

Video: XVID 704x400 29.00fps 968Kbps [Video 0]


cant seem to get away from 29.000fps even.

[squid_80]

Yeah, I realized today that I hadn't done any tests with anything other than 25fps material so I apologize to all the NTSC people out there. On the upside it should be easy enough to change the framerate if you mux audio in later.

[woah!]

true but i thought i should at least let you know there is an issue :)

otherwise it works great :)

[squid_80]

Well that's what testing is for...

MKV output is very close. It's working when I run it from inside the debugger, but crashes when I try it normally. :mad:
(bond: you'll be happy to hear it's native V_MPEG4/ISO/ASP in mkv.)

[bond]

MKV output is very close. It's working when I run it from inside the debugger, but crashes when I try it normally. :mad:
(bond: you'll be happy to hear it's native V_MPEG4/ISO/ASP in mkv.)great stuff!!!
edit: how are you going to handle packed bitstream with mkv? simply disable it?

tough when will we see real native mpeg-4 (meaning in .mp4 ;) )?

[squid_80]

I could borrow Mosu's bitstream parser from mkvmerge to split the packed frames. Or I could just disable simultaneous .avi and .mkv output, which is probably more sensible.

[bond]

how are you going to remove all the other avi specific fud when outputting native mkv: like placing the vol on every keyframe?

[bond]

heres the xvid info:

-type 2 -cq 3 -avi

http://images.dr3vil.com/files/default/xvid_encraw.jpgjust realised this: where does the "coremp4" come from?

[squid_80]

Whatever codec is grabbing the FourCC code from the avi?

[bond]

sure, but i am wondering cause i dont find this "coremp4" decoder anywhere?

[bond]

squid_80, one error: packed bitstream is enabled by default. this leads to the situation that if someone doesnt disable pb and outputs raw .m4v he will get an incorrect stream full of the dummy placeholder n-vops

a possible solution would be to allow pb only with avi output (and not with raw, mkv and a hopefully to come mp4 output), as after all pb is only a hack for avi, and/or disable pb by default (which imho would be a good thing)

[dimzon]

Well that's what testing is for...

MKV output is very close. It's working when I run it from inside the debugger, but crashes when I try it normally. :mad:
(bond: you'll be happy to hear it's native V_MPEG4/ISO/ASP in mkv.)
Seems like you have invalid field align for structures. Just disable this option in optimizer or use pragma pack directive...

[squid_80]

Actually it seems I was forgetting to set the global timecode for the cues :o Works properly now, just fine tuning.

[squid_80]

how are you going to remove all the other avi specific fud when outputting native mkv: like placing the vol on every keyframe?
squid_80, one error: packed bitstream is enabled by default. this leads to the situation that if someone doesnt disable pb and outputs raw .m4v he will get an incorrect stream full of the dummy placeholder n-vops

a possible solution would be to allow pb only with avi output (and not with raw, mkv and a hopefully to come mp4 output), as after all pb is only a hack for avi, and/or disable pb by default (which imho would be a good thing)
OK, before I finalize my solution I'm going to ask for advice because I'm not an expert.
I've made a quick and dirty function which searches the buffer returned by the encoder for the sequence 00 00 01 b6 (vop start code).
If the frame is the first keyframe of the sequence, store everything before this location (the vol headers) in the codec private element of the mkv and store everything after as a key frame. If it's not the first keyframe just throw the headers away and store the frame.
If it's not a keyframe check for 2 vop start codes to indicate a packed frame. If so write the first vop as a p frame and the second as a b frame.
If the total length of a vop is less than 10 it's probably a n-vop. Check if the size of the buffer returned by the encoder matches the length of the encoded frame and if not throw it away. Otherwise write it as a frame with a reference to the previous one.

I'll try and make it a bit clearer:xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
0: key=2, time= 16, len= 33779 | type=I, quant= 4, len= 33779
1: key=0, time= 0, len= 0
2: key=0, time= 79, len= 15268 | type=B, quant= 7, len= 4985
3: key=0, time= 0, len= 6 | type=P, quant= 4, len= 10289
4: key=0, time= 62, len= 15242 | type=P, quant= 4, len= 15242
5: key=2, time= 16, len= 44086 | type=I, quant= 4, len= 44086
6: key=0, time= 62, len= 15406 | type=P, quant= 4, len= 15406
7: key=2, time= 15, len= 39235 | type=I, quant= 4, len= 39235
8: key=0, time= 63, len= 11424 | type=P, quant= 4, len= 11424
9: key=0, time= 62, len= 13839 | type=P, quant= 4, len= 13839
-1: key=0, time= 63, len= 15438 | type=P, quant= 4, len= 15438
-1: key=0, time= 0, len= -5
Tot: enctime(ms) = 438.00, length(bytes) = 203723
Avg: enctime(ms) = 39.82, fps = 25.11, length(bytes) = 18520
Frame 0 has the headers before the vop start stripped off and they are stored in the codec private element. The vop is stored as a frame with no references.
Frame 1 doesn't return anything from the encoder.
Frame 2 is scanned for vop start codes and 2 are found. The first vop is stored as a frame with a reference to the last i frame, the second vop is stored as a frame with references to the last i frame and p frame.
Frame 3 has a buffer with a length smaller than 10 and different size to the corresponding encoded frame ( 6 != 10289). It's a fake n-vop so throw it away.
Frame 4 has only one start code and is a p frame. Store it with a reference to the previous I frame.
Frame 5 is an I frame. Find the vop start, throw away everything before it and store the frame with no references.

You get the idea (I hope). If anyone can spot any flaws please speak up now before I waste too much time implementing/testing.

[squid_80]

So either I've utterly confused everyone, or I'm gonna go with this :)

[dimzon]

@squid_80
please, take look @ this:
http://forum.doom9.org/showthread.php?p=774869#post774869

[Doom9]

if your goal is to just avoid packed bitstream except for AVI output, why not override whatever the user set? It's the solution I should've adopted in MeGUI long ago (add that to the long list of "should haves").

[squid_80]

The headers still need to be stripped from every I frame (there's global flags to turn this off but they're not implemented in xvidcore).

[squid_80]

@squid_80
please, take look @ this:
http://forum.doom9.org/showthread.php?p=774869#post774869
really, it encoder reques avs with yv12 support it means encoder are working with avs via avisynth.dll (there are no other way to get yv12 from it, AviFile32 API performs RGB24 conversion).This isn't right at all. If it were true how do x264 and xvid_encraw currently get yv12 frames using avifile? Besides, xvid supports lots of different input colorspaces but support for others just hasn't been implemented in encraw yet. It's yet another thing to be done.

Edit: No-one took a scrap of notice when I said I'd enhanced x264.exe to print the error message if opening an .avs script fails (which is pretty useful, imho).

[bond]

simply drop avi output and with it all the avi-centric crap ;)

[dimzon]

This isn't right at all. If it were true how do x264 and xvid_encraw currently get yv12 frames using avifile?
can you post code (i want take look @ it)

[Doom9]

simply drop avi output and with it all the avi-centric crapNot gonna happen any time soon with a millions of XviD capable standalones out there. I'm afraid XviD in AVI still makes a lot of sense.

[squid_80]

can you post code (i want take look @ it)
For the avisynth error handling? Well this is how I changed x264.c:

extern "C" const GUID IID_IAvisynthClipInfo // {E6D6B708-124D-11D4-86F3-DB80AFD98778}
= {0xe6d6b708, 0x124d, 0x11d4, {0x86, 0xf3, 0xdb, 0x80, 0xaf, 0xd9, 0x87, 0x78}};


struct IAvisynthClipInfo : IUnknown {
virtual int __stdcall GetError(const char** ppszMessage) = 0;
virtual bool __stdcall GetParity(int n) = 0;
virtual bool __stdcall IsFieldBased() = 0;
};

static int open_file_avis( char *psz_filename, hnd_t *p_handle, x264_param_t *p_param )
{
AVISTREAMINFO info;
PAVISTREAM p_avi = NULL;
PAVIFILE m_AVIFile;
int i;
IAvisynthClipInfo *pAvisynthClipInfo;

*p_handle = NULL;

AVIFileInit();
if( AVIFileOpen(&m_AVIFile, psz_filename, OF_READ, NULL))
{
AVIFileExit();
return -1;
}

// Check if we're using avisynth and display any script errors (taken from virtualdub)
if (FAILED(m_AVIFile->QueryInterface(IID_IAvisynthClipInfo, (void **)&pAvisynthClipInfo)))
pAvisynthClipInfo = NULL;
else {
const char *s;

if (pAvisynthClipInfo->GetError(&s)) {
printf("Avisynth open failure:\n%s\n", s);
pAvisynthClipInfo->Release();
m_AVIFile->Release();
AVIFileExit();
return -1;
}
}

if (pAvisynthClipInfo!=NULL)
pAvisynthClipInfo->Release();

if( AVIFileGetStream(m_AVIFile, &p_avi, streamtypeVIDEO, 0) )
{
AVIFileRelease(m_AVIFile);
AVIFileExit();
return -1;
}
// Close m_AVIFile now that we have the stream
AVIFileRelease(m_AVIFile);

if( AVIStreamInfo(p_avi, &info, sizeof(AVISTREAMINFO)) )
......

[JoeBG]

Just finished the third complete video. Size fitted, quality is best. Congratulations to the tool - much more satisfying than mencoder xvid.

[Doom9]

So either I've utterly confused everyone, or I'm gonna go with this I think the people who could give proper advice have yet to read this.. perhaps it would be best to just run with it and see where it takes you.

[squid_80]

It was a busted idea anyway. Since mkv requires a timecode to be stored with every frame, I can't store a reference frame without knowing how many b frames are before it; I can't store the b frames as they come because the reference frame needs to be written first. Catch 22.
So now I buffer everything in memory from one reference frame to the next, which seems to work pretty well.
The other thing I was doing wrong was assuming a packed frame would always be PB. If closed gop is turned off it's possible to get IB instead.

[squid_80]

There's a new build available with very experimental matroska output.

[bond]

ftp://squid80.no-ip.com/xvid_encraw.zip

squid_80: i tried playing a file produced by this with the haali splitter and ffdshow and i saw lots of artefacts, which seemed to me like maybe wrongly ordered files? wrong references? i used two b-frames

at the end it caused my pc to restart :D

so i think there might be something not really working correctly ;)
you might want to release the code so people can look at whats going wrong

[squid_80]

Argh! What the hell happened, it was fine a minute ago. Does look pretty funky.

Now I can't seem to recreate it. Still got the command line handy?

[bond]

xvid_encraw.exe -i input.avs -type 2 -bitrate 700 -single -max_key_interval 250 -max_bframes 2 -qpel -gmc -stats -mkv xvid.mkv

[squid_80]

S-VOPS! Why didn't anyone remind me about them!

Edit: Can you try without -gmc, pretty please?

[Doom9]

at the end it caused my pc to restartWell.. squid sure showed you what you should do with your WinMEgacrap ;)

[bond]

works indeed without gmc :)
why did it bork things? arent s-vops, plain p-vops?

btw next to I/P/B and S-vops, there are also N-vops
(you still have packed bitstream on by default creating borked raw streams! disable it! :p )

edit:
Well.. squid sure showed you what you should do with your WinMEgacrap ;)WinME rocks! ;)
btw i am using win2000 for some months now, so that excuse doesnt count :p

[squid_80]

works indeed without gmc :)
why did it bork things? arent s-vops, plain p-vops?
Kinda, but they have a different vop type. 0 = IVOP, 1 = PVOP, 2 = BVOP, 3 = SVOP. I forgot to check for SVOPs.

Edit: If I'd left one line as else instead of else if, it would have worked :)

btw next to I/P/B and S-vops, there are also N-vops
(you still have packed bitstream on by default creating borked raw streams! disable it! :p )
Hmmm I remembered to drop the fake n-vops, but forgot to keep the real ones. Whoops.
If you want unborked raw streams, just use -nopacked. Packed bitstream isn't that hard to handle, any worthwhile muxer should be able to process it correctly.

[bond]

Hmmm I remembered to drop the fake n-vops, but forgot to keep the real ones. Whoops.
If you want unborked raw streams, just use -nopacked. Packed bitstream isn't that hard to handle, any worthwhile muxer should be able to process it correctly. the thing is if someone wants a correct raw .m4v stream he shouldnt need to know about packed bitstream.

packed bitstream is a hack for avi and nothing else.
there is no tool in the world expecting fake n-vops in a raw stream, cause they simply dont belong there

[squid_80]

Alrighty. I won't disable packed by default, but I'll make sure fake n-vops are left out of raw output.

[bond]

Alrighty. I won't disable packed by default, but I'll make sure fake n-vops are left out of raw output.better make sure that packed bitstream is disabled for every output mode except avi. will be easier to add mp4 output that way ;)

[bond]

btw i have checked the timestamps of frames with the following samples:
- 0 b-frames, no n-vops
- 2 b-frames, no n-vops
- 0 b-frames, n-vops (drop 100), everything except the keyframes are n-vops
- 2 b-frames, n-vops (drop 100), ~ 3 n-vops used in the stream

and it seems they are correct! great job :)

[squid_80]

btw i have checked the timestamps of frames with the following samples:
- 0 b-frames, no n-vops
- 2 b-frames, no n-vops
- 0 b-frames, n-vops (drop 100), everything except the keyframes are n-vops
- 2 b-frames, n-vops (drop 100), ~ 3 n-vops used in the stream

and it seems they are correct! great job :)
Have you tried each option with/without -nopacked?
Hmmm I remembered to drop the fake n-vops, but forgot to keep the real ones. Whoops.
No you didn't, you dummy. N-VOPS have the same identifier as P-VOPS.:o

[bond]

Have you tried each option with/without -nopacked?the .mkvs were all produced with packed enabled
the raw streams have been produced with -nopacked

is it necessary to also seperately test mkv without packed set? shouldnt that work exactly the same way as with packed?

No you didn't, you dummy. N-VOPS have the same identifier as P-VOPS.:oand? are you sure that the n-vops have different identifiers?

btw make sure to use latest libmatroska:
http://www.matroska.org/downloads/windows.html

[bond]

here another reason why packed bitstream should be disabled by default:
as packed bitstream is enabled by default the cli writes always a fake divx userdata before the xvid userdata, also with raw and mkv output. this divx userdata signals that the stream is packed (which it obviously isnt with raw and mkv and a hopefully to come mp4 output). some decoders (like the one from divx and some standalones propably) take this flag into account.
obviously signalling this with a non packed stream is incorrect and will potentially bork things, therefore this flag needs to be disabled too with every outputmode, except avi output

people should simply realise that avi is a complete mess and the earlier people stop using it the better :D


btw i compared a 2 b-frame, no drop, mkv with and without packed bitstream: same timestamps

[squid_80]

the .mkvs were all produced with packed enabled
the raw streams have been produced with -nopacked

is it necessary to also seperately test mkv without packed set? shouldnt that work exactly the same way as with packed?I just wanted to make sure I'm not throwing away any real n-vops. I rely on the stats returned from the encoder for this, and they can be a bit screwy with -nopacked:
C:\xvid\build\win32\bin>xvid_encraw -i g:\vid\new.avs -frames 10 -start 2000 -no
packed -drop 100
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
0: key=2, time= 15, len= 19838
1: key=0, time= 0, len= 0
2: key=0, time= 0, len= 0
3: key=0, time= 31, len= 9295 | type=I, quant= 4, len= 19838
4: key=0, time= 15, len= 624 | type=B, quant= 7, len= 624
5: key=0, time= 16, len= 624 | type=B, quant= 7, len= 624
6: key=0, time= 31, len= 6 | type=P, quant= 4, len= 6
7: key=0, time= 31, len= 9203 | type=P, quant= 4, len= 6
8: key=0, time= 16, len= 565 | type=B, quant= 7, len= 565
9: key=0, time= 31, len= 1271 | type=P, quant= 4, len= 9203
-1: key=0, time= 16, len= 302 | type=B, quant= 7, len= 302
-1: key=0, time= 0, len= 277 | type=B, quant= 7, len= 277
-1: key=0, time= 0, len= -5 | type=P, quant= 4, len= 1271
Tot: enctime(ms) = 202.00, length(bytes) = 42005
Avg: enctime(ms) = 16.83, fps = 59.41, length(bytes) = 3500If you look at the stats, it looks like there were two n-vops (dropped frames). But in reality the first of those P frames had a length of 9295 bytes. I think this is a bug, but it's a quick way to spot real n-vops vs fake n-vops. I'm still not completely convinced of it's reliability though.
and? are you sure that the n-vops have different identifiers?
No they don't, they're the same as P frames. That's why it works even though I forgot about handling them.
btw make sure to use latest libmatroska:
http://www.matroska.org/downloads/windows.htmlI am. I just copied the line that sets the lib version from test6.cpp and forgot to change it.
here another reason why packed bitstream should be disabled by default:
as packed bitstream is enabled by default the cli writes always a fake divx userdata before the xvid userdata, also with raw and mkv output. this divx userdata signals that the stream is packed (which it obviously isnt with raw and mkv and a hopefully to come mp4 output). some decoders (like the one from divx and some standalones propably) take this flag into account.
obviously signalling this with a non packed stream is incorrect and will potentially bork things, therefore this flag needs to be disabled too with every outputmode, except avi outputDo any of these players actually support anything other than avi? Anyway, I'll just drop the p off the divx userdata for mkv and mp4 (if it happens). I'll do the same for ES stream output (-o) and apply the checks for fake n-vops. That way mkv and raw will give the same output whether or not packed is used. But the -save option (one file for each frame) stays as it is - it's too invaluable as a diagnostic tool to mess around with.

(hehe: I just searched xvidcore's code for divx and found a rude word.)

[squid_80]

btw i compared a 2 b-frame, no drop, mkv with and without packed bitstream: same timestamps
It's possible timestamps could screw up on long encodes/exotic framerates. The whole idea of having a timestamp for every frame, while being handy for VFR material, is a bit of a headache (why is there a default duration if timestamps are required anyway?) and I'm not sure an overflow can't occur with the formula I'm using to calculate them.
Is there some hidden documentation for libmatroska that clearly explains when and which timecode scale applies? This is the main reason cues aren't written - I couldn't get the damn timecodes to work for them, even though they work fine for the blocks/clusters.

[bond]

I just wanted to make sure I'm not throwing away any real n-vops. I rely on the stats returned from the encoder for thisin what way do you rely on the stats? just by checking the "len" values?

i am repeating myself over and over again: why not simply disable packed bitstream by default for every mode except avi? that way you know that for raw and mkv there simply are no fake n-vops and therefore there also isnt anything to drop. is it so hard to add a check in the cmdl parsing which only takes packed into account if avi output is used
in the case of avi with packed bitstream there is no problem with the real n-vops as the fake n-vops dont need to be dropped anyways
in x264cli there are lots of checks which only allow options to be set if another option is set too aso...

why are you trying to make things in a difficult and potentially borking way if its so easy to solve this?


If you look at the stats, it looks like there were two n-vops (dropped frames). But in reality the first of those P frames had a length of 9295 bytes. I think this is a bug, but it's a quick way to spot real n-vops vs fake n-vops. I'm still not completely convinced of it's reliability though.whats the meaning of the two length values?

It's possible timestamps could screw up on long encodes/exotic framerates. The whole idea of having a timestamp for every frame, while being handy for VFR material, is a bit of a headache (why is there a default duration if timestamps are required anyway?) and I'm not sure an overflow can't occur with the formula I'm using to calculate them.
Is there some hidden documentation for libmatroska that clearly explains when and which timecode scale applies? This is the main reason cues aren't written - I couldn't get the damn timecodes to work for them, even though they work fine for the blocks/clusters. i dunno where in the mkv specs its written, but how you have set the timecodes till now is correct

you combine the displaytimecode (or cts in mp4 language) with each frame (the frames themselves are stored in decoding order)

about the timescale: as you atm dont seem to set an own timescale you are using the default one (which is 1000 or multiple of that dunno exactly)
of course you can also use an own timescale (which might be better with not nice framerates like 25fps), but i dunno how to do this with libmatroska

[Doom9]

@bond: I already suggested the commandline override.. but according to squid, it's not that simple:
The headers still need to be stripped from every I frame (there's global flags to turn this off but they're not implemented in xvidcore).

[squid_80]

Re-arranging your words slightly to explain things better...
in what way do you rely on the stats? just by checking the "len" values?
whats the meaning of the two length values?
Here's the code that checks for fake n-vops:
if (m4v_size > 10 || (m4v_size == stats_length))
MKVStreamWrite(myMKVStream, output_num, 1, mp4_buffer, m4v_size, key ? AVIIF_KEYFRAME : 0, NULL, NULL);
In xvid_encraw's output text, m4v_size is the len value on the left. stats_length is the len value on the right. m4v_size is the size of the data returned from the encoder. stats_length is supposedly the encoded length of the input frame. But as demonstrated the stats don't necessarily match the output from the encoder - look at the example I posted, even the stats for the I frame are 3 input frames late. But in the case of real n-vops the stats do always seem to match up, which is how I identify them. I said it was a quick way, maybe I should have said quick n dirty. So far I haven't seen it break.
i am repeating myself over and over again: why not simply disable packed bitstream by default for every mode except avi? that way you know that for raw and mkv there simply are no fake n-vops and therefore there also isnt anything to drop. is it so hard to add a check in the cmdl parsing which only takes packed into account if avi output is used
in the case of avi with packed bitstream there is no problem with the real n-vops as the fake n-vops dont need to be dropped anyways
in x264cli there are lots of checks which only allow options to be set if another option is set too aso...
why are you trying to make things in a difficult and potentially borking way if its so easy to solve this?
If an avi is made without packed bitstream by accident, can it be fixed?
I want to have the least amount of restrictions possible when it comes to parameters. -avi should not imply packed or -nopacked. Neither should -mkv or -o. As I showed above the code to check for real vs. fake n-vops is one line and while that holds up I think it's best to leave packed as default. The idea of allowing multiple simultaneous output files was so they could be compared and make sure everything is behaving correctly - if the mkv is borked check the failsafe avi. So it's better to have a working avi in all cases, getting screwed because I forgot to use the -packed option (ever seen a report of xvid crashing with divide by 0 error? Bet it was unpacked and open gop...) is not something I want to see. -nopacked is always available as an option. If you can break the real vs. fake n-vop check (the divx userdata doesn't count as I haven't fixed it yet :p ) then I'll make -nopacked the default choice.
i dunno where in the mkv specs its written, but how you have set the timecodes till now is correct

you combine the displaytimecode (or cts in mp4 language) with each frame (the frames themselves are stored in decoding order)Ahh, but if displaytimecode has been rounded we get cumulative rounding errors... Leading to audio desync. At the moment I use timecode(ms) = 1000*framenum*framescale/framerate.
about the timescale: as you atm dont seem to set an own timescale you are using the default one (which is 1000 or multiple of that dunno exactly)
of course you can also use an own timescale (which might be better with not nice framerates like 25fps), but i dunno how to do this with libmatroskaThe problem is every section seems to have it's own timescale - segment vs tracks vs clusters. Some seem to multiply the given timecode (segment) while others seem to divide (tracks). Then again some timecodes aren't scaled at all.

P.S. Doom9 rubs it in... I'd forgotten about that. No matter what I do, the encoder still spits out avi sympathetic data :sly: And that extra data is dealt with in a very similar way to packed frames.

[bond]

Here's the code that checks for fake n-vops:
if (m4v_size > 10 || (m4v_size == stats_length))
MKVStreamWrite(myMKVStream, output_num, 1, mp4_buffer, m4v_size, key ? AVIIF_KEYFRAME : 0, NULL, NULL);fake n-vops will hopefully also be dropped for raw stream in the future? or not?

In xvid_encraw's output text, m4v_size is the len value on the left. stats_length is the len value on the right.
m4v_size is the size of the data returned from the encoder. stats_length is supposedly the encoded length of the input frame. hm i still dont really get the difference? :confused:

If an avi is made without packed bitstream by accident, can it be fixed? i think there is an important confusion on your side.
avi files do NOT need to be packed when using b-frames!!! let me say it again: there is no need for packed bitstream when using avi with b-frames
(for how b-frames work without packed in avi read this (http://forum.doom9.org/showthread.php?s=&threadid=80430))

xvid produced for years avi files by default which are not packed with b-frames (and as you see in the xvidvfw gui there is also the option to not use packed bitstream). thats said nopacked works great with xvid and always has been that way (i have NOT been using packed bitstream for years and didnt have a problem)
in fact as good as all of the standalones handling xvid handle no packed bitstreams, as that was simply the way xvid did it for years

apart from that the libavcodec asp encoder also doesnt output packed bitstream in any case (eg if used via mencoder with avi output)

I want to have the least amount of restrictions possible when it comes to parameters. -avi should not imply packed or -nopacked. Neither should -mkv or -o. As I showed above the code to check for real vs. fake n-vops is one line and while that holds up I think it's best to leave packed as default. mkv and raw output should not imply -nopacked??? of course they should imply that, as it simply doesnt make sense to use packed with them
i dont get why simply automatically enabling -nopacked with mkv and raw output is such a tragedy?

-nopacked is always available as an option. If you can break the real vs. fake n-vop check (the divx userdata doesn't count as I haven't fixed it yet :p ) then I'll make -nopacked the default choice.so the user first needs to waste a day of encoding to find out he should have enabled -nopacked with mkv and raw output altough packed bitstream makes no sense with mkv and raw anyways and therefore nopacked should be enabled right away with mkv and raw output always???

The idea of allowing multiple simultaneous output files was so they could be compared and make sure everything is behaving correctly - if the mkv is borked check the failsafe avi.you are serious about the multiple simultaneous outputs? you do know that noone will use this ;)
i would propose to simply drop the idea of simultenous different outputs as noone will use this anyways
i also dont really see the sense in using the different output modes together at the same time as its absolutely no problem to remux from one format to the other afterwards (and thats more likely to be done by users than to output the same file multiple times via the encoder while encoding...)
also i have never seen any encoder offering outputting the same stream in different formats _at the same time_...

/rant

anyways to make it short, i have said my opinion about this and i do know it counts nothing as you are the dev ;)
basically all i want is to have a correctly working cli with correct raw and mkv output (and hopefull mp4 output too) without the user needing to know anything about all the avi shit floating around (userdata, vol, packed bitstream, fake n-vops and whatever else)
i dont care how you get the cli to manage to do this as long as it works correctly :) and as long as its not total hackery ;)

Ahh, but if displaytimecode has been rounded we get cumulative rounding errors... Leading to audio desync. At the moment I use timecode(ms) = 1000*framenum*framescale/framerate.
The problem is every section seems to have it's own timescale - segment vs tracks vs clusters. Some seem to multiply the given timecode (segment) while others seem to divide (tracks). Then again some timecodes aren't scaled at all.sorry i dont know much about the internals, you will need to have to ask a mkv guy about this
all i can say is that mkvinfo tells me that there is no explicit timescale set in your files which means that the default timescale is used
regarding the timecodes i checked them on a ~7000 frames mkv clip with 2 b-frames and drop 100 and they have been correct. that was with a 25fps clip

P.S. Doom9 rubs it in... I'd forgotten about that. No matter what I do, the encoder still spits out avi sympathetic data :sly: And that extra data is dealt with in a very similar way to packed frames.yeah, this avi shit is a total mess and noone should have ever come up with it...
thank god people tend to not use avi for avc anymore

[squid_80]

fake n-vops will hopefully also be dropped for raw stream in the future? or not?Yes. For the -o option, but not -save.
hm i still dont really get the difference? :confused:
Watch the output. The numbers on the left of | refer to the encoder output, they are in the encoded order. The numbers on the right refer to input frames, they are in frame order. By chance or bug or dare I say it by design, when there's a real n-vop the lengths match up. When it's a fake, they don't.
i think there is an important confusion on your side.
avi files do NOT need to be packed when using b-frames!!! let me say it again: there is no need for packed bitstream when using avi with b-frames
(for how b-frames work without packed in avi read this (http://forum.doom9.org/showthread.php?s=&threadid=80430))
encraw doesn't output 0x7f for delays; it doesn't output anything
To be properly seekable, packed is needed. If I'm at frame 50 then that's what I want, not frame 49.xvid produced for years avi files by default which are not packed with b-frames (and as you see in the xvidvfw gui there is also the option to not use packed bitstream). thats said nopacked works great with xvid and always has been that way (i have NOT been using packed bitstream for years and didnt have a problem)With avi output?in fact as good as all of the standalones handling xvid handle no packed bitstreams, as that was simply the way xvid did it for yearsSo why oh why is it enabled by default in the vfw interface?
apart from that the libavcodec asp encoder also doesnt output packed bitstream in any case (eg if used via mencoder with avi output)This is xvid_encraw, not mencoder.
mkv and raw output should not imply -nopacked??? of course they should imply that, as it simply doesnt make sense to use packed with them
i dont get why simply automatically enabling -nopacked with mkv and raw output is such a tragedy?

so the user first needs to waste a day of encoding to find out he should have enabled -nopacked with mkv and raw output altough packed bitstream makes no sense with mkv and raw anyways and therefore nopacked should be enabled right away with mkv and raw output always???
When the next build is done, mkv and raw output will be the same whether packed bitstream is enabled or not. Avi output will be packed by default unless -nopacked is specified. Same goes for the files produced by -save, since it's mainly a diagnostic output and can come in very handy when combined with the output log from encraw.
you are serious about the multiple simultaneous outputs? you do know that noone will use this ;)
i would propose to simply drop the idea of simultenous different outputs as noone will use this anyways
i also dont really see the sense in using the different output modes together at the same time as its absolutely no problem to remux from one format to the other afterwards (and thats more likely to be done by users than to output the same file multiple times via the encoder while encoding...)
also i have never seen any encoder offering outputting the same stream in different formats _at the same time_...
You don't see this as a good option? Having to remux into a different container later on is better than being able to output both at the same time?
Further back in this very thread:an encoder should be able to output something useable right away imho, and raw .m4v isnt imho, cause you need a second tool for muxing the stream and there arent lots of tools being able to handle raw input
So if we're outputting raw to mux into mp4 later on, we can check the avi/mkv output straight away to make sure it's ok. Silly idea or worthwhile endeavour?

/rant

anyways to make it short, i have said my opinion about this and i do know it counts nothing as you are the dev ;)
basically all i want is to have a correctly working cli with correct raw and mkv output (and hopefull mp4 output too) without the user needing to know anything about all the avi shit floating around (userdata, vol, packed bitstream, fake n-vops and whatever else)
i dont care how you get the cli to manage to do this as long as it works correctly :)
It will be done. and as long as its not total hackery ;) Like I said, if you manage to break that fake/real n-vop trick, then I'll change things.

[bond]

Yes. For the -o option, but not -save.good :)

To be properly seekable, packed is needed. If I'm at frame 50 then that's what I want, not frame 49.well you can also seek on not packed avi files with b-frames. try it
in fact the divx encoder outputs in his multiple b-frames mode also NOT packed bitstream.
why? because xvid did it that way long before divx was able to output multiple b-frames and therefore standalone players were written to handle it

With avi output?of course

So why oh why is it enabled by default in the vfw interface?because without packed you have the delay and the delay message shown which newbies always complained about. so packed was enabled by default to shut newbies up
there is no problem or bug with not packed output

It will be done. Like I said, if you manage to break that fake/real n-vop trick, then I'll change things. ok so i couldnt convince you that "outputting the stream packed and unpacking it at the same time" doesnt make any sense, well its your choice :p

edit: dont get me wrong, its great you are working on this. its just that i fear things getting borky (anyways as you said i am working hard at trying to bork your algo ;) )

[squid_80]

well you can also seek on not packed avi files with b-frames. try it
in fact the divx encoder outputs in his multiple b-frames mode also NOT packed bitstream.
why? because xvid did it that way long before divx was able to output multiple b-frames and therefore standalone players were written to handle it
I have tried. If I skip using vdub's keyframe keys, I get the previous keyframe instead of the one I want. The frames shown are nearly always the previously requested one instead of the currently requested. Except if I'm on an I frame and I go backward, I get the next P frame. It's wacky, compared to packed which gives me exactly what I ask for.

because without packed you have the delay and the delay message shown which newbies always complained about. so packed was enabled by default to shut newbies up
there is no problem or bug with not packed output
I thought the lag message was disabled to shut the newbies up and packed was enabled for compatability.
ok so i couldnt convince you that "outputting the stream packed and unpacking it at the same time" doesnt make any sense, well its your choice :p
Not so much that it doesn't make sense, just that it's less work to undo. Especially since I-VOPS always have to have the headers stripped off anyway.

[bond]

Not so much that it doesn't make sense, just that it's less work to undo. Especially since I-VOPS always have to have the headers stripped off anyway.ok if its less work that way its ok, i just thought that automatically enabling the -unpacked flag together with -o and -mkv would be as good as no work at all

about the the header stripping: isnt this an independant issue of the packed bitstream thing as its also necessary with -unpacked?

[bond]

as i couldnt convince you to simply enforce -nopacked with mkv and raw output i thought about your method of differentiating real n-vops from fake n-vops:

as i see it you say:
1) if a frame has a size of smaller than 10 on the coding order side (left side) its a n-vop.

i would say this is a not necessarily always correct workaround as at small bitrates and resolutions it would be imagineable to also have frames with a small size
better would be to check if the not coded flag is set for identifying n-vops, cause than you know it for sure


2) if this p-vop has a different size in coding order than in display order you throw it away, as its a fake n-vop
two things coming to my mind:

a) thinking it through this is (might be ;) ) true for packed bitstream streams, but as your own example about the -nopacked stream also shows this is not true for not packed streams
still its not a problem because of course the whole searching for and dropping fake n-vops isnt necessary with not packed streams, as in this case there are only real n-vops
right?

b) your method is wrong when a real frame (be it a real n-vop or a normal p-vop), that has the same size as the fake n-vop, gets packed together with a b-vop
in this case things would look like this:

len= 1309 | type=B, quant= 11, len= 1295
len= 1217 | type=B, quant= 11, len= 1217
len= 6 | type=P, quant= 7, len= 6the P gets packed together with the first B, but your algo doesnt find the fake n-vop cause it has the same size as the real vop



btw some potential bugs i found in the stats of xvid:
- i also think that there is a bug in this value 6 for the p-vop (it should be 9295) you showed above. i have reproduced this on other streams too, it seems to be a common problem
- also a common bug could be len values in the case of packed frames. shouldnt the size of the packed frames be the same as the sum of the not packed ones?
- the following borked coding order sizes created with -nopacked are also often the case but dont make sense:
6133: key=0, time= 300, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.95, psnr u = 39.68, psnr v = 38.35
6134: key=0, time= 291, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6135: key=2, time= 40, len= 13205 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6136: key=0, time= 230, len= 6 | type=P, quant= 2, len= 6, psnr y = 26.52, psnr u = 44.55, psnr v = 44.81
6137: key=0, time= 271, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.22, psnr u = 37.26, psnr v = 36.73
6138: key=0, time= 290, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6139: key=2, time= 40, len= 10332 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6140: key=0, time= 251, len= 6 | type=P, quant= 2, len= 6, psnr y = 27.14, psnr u = 46.42, psnr v = 45.23
6141: key=0, time= 270, len= 6 | type=P, quant= 2, len= 6, psnr y = 24.35, psnr u = 44.41, psnr v = 43.27maybe you can find the reason for these bugs (?) and fix them.

[squid_80]

1) if a frame has a size of smaller than 10 on the coding order side (left side) its a n-vop.

i would say this is a not necessarily always correct workaround as at small bitrates and resolutions it would be imagineable to also have frames with a small size
better would be to check if the not coded flag is set for identifying n-vops, cause than you know it for sure


2) if this p-vop has a different size in coding order than in display order you throw it away, as its a fake n-vop
two things coming to my mind:

a) thinking it through this is (might be ;) ) true for packed bitstream streams, but as your own example about the -nopacked stream also shows this is not true for not packed streams
still its not a problem because of course the whole searching for and dropping fake n-vops isnt necessary with not packed streams, as in this case there are only real n-vops
right?

b) your method is wrong when a real frame (be it a real n-vop or a normal p-vop), that has the same size as the fake n-vop, gets packed together with a b-vop
in this case things would look like this:

len= 1309 | type=B, quant= 11, len= 1295
len= 1217 | type=B, quant= 11, len= 1217
len= 6 | type=P, quant= 7, len= 6the P gets packed together with the first B, but your algo doesnt find the fake n-vop cause it has the same size as the real vopYou're pretty much right. But are these circumstances actually possible? What is the minimum size of a p-vop without the not coded flag?

btw some potential bugs i found in the stats of xvid:
- i also think that there is a bug in this value 6 for the p-vop (it should be 9295) you showed above. i have reproduced this on other streams too, it seems to be a common problem
- also a common bug could be len values in the case of packed frames. shouldnt the size of the packed frames be the same as the sum of the not packed ones?
When packed bitstream is on, the reported size of the P frames has the bytes for the fake-nvop added in. I think that's what you're seeing.
- the following borked coding order sizes created with -nopacked are also often the case but dont make sense:
6133: key=0, time= 300, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.95, psnr u = 39.68, psnr v = 38.35
6134: key=0, time= 291, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6135: key=2, time= 40, len= 13205 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6136: key=0, time= 230, len= 6 | type=P, quant= 2, len= 6, psnr y = 26.52, psnr u = 44.55, psnr v = 44.81
6137: key=0, time= 271, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.22, psnr u = 37.26, psnr v = 36.73
6138: key=0, time= 290, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6139: key=2, time= 40, len= 10332 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6140: key=0, time= 251, len= 6 | type=P, quant= 2, len= 6, psnr y = 27.14, psnr u = 46.42, psnr v = 45.23
6141: key=0, time= 270, len= 6 | type=P, quant= 2, len= 6, psnr y = 24.35, psnr u = 44.41, psnr v = 43.27maybe you can find the reason for these bugs (?) and fix them.
It's the same bug. I think in reality it should be something like this:
6133: key=0, time= 300, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.95, psnr u = 39.68, psnr v = 38.35
6134: key=0, time= 291, len= 6 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6135: key=2, time= 40, len= 13205 | type=P, quant= 4, len= 6, psnr y = 20.72, psnr u = 39.74, psnr v = 38.09
6136: key=0, time= 230, len= 6 | type=I, quant= 2, len= 13205, psnr y = 26.52, psnr u = 44.55, psnr v = 44.81
6137: key=0, time= 271, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.22, psnr u = 37.26, psnr v = 36.73
6138: key=0, time= 290, len= 6 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6139: key=2, time= 40, len= 10332 | type=P, quant= 2, len= 6, psnr y = 21.66, psnr u = 41.96, psnr v = 40.32
6140: key=0, time= 251, len= 6 | type=I, quant= 2, len= 10332, psnr y = 27.14, psnr u = 46.42, psnr v = 45.23
6141: key=0, time= 270, len= 6 | type=P, quant= 2, len= 6, psnr y = 24.35, psnr u = 44.41, psnr v = 43.27Whether it's a bug or by design I don't know. But if the bad lengths are being used for first pass stats (I have a sneaking suspicion they are) the filesize is going to be way off. Can you do that same stream again with -pass1, then examine the stats file for any weirdness?

[bond]

You're pretty much right. But are these circumstances actually possible? What is the minimum size of a p-vop without the not coded flag?i have been able to create smaller than 10 real p-vops with a very small res and very small bitrate. i doubt anyone will ever do something like that, its just that it should be known as an potential issue

about the not coded flag: will you identify n-vops by the flag or by the size?
doing it by the flag will make sure no small p-vop will get dropped and will always be correct

and what are you answer to my question whether you are doing the fake n-vop check also with not packed streams?

When packed bitstream is on, the reported size of the P frames has the bytes for the fake-nvop added in. I think that's what you're seeing.well yeah, still i dont think the size of the packed frame(s) (shown on the left) is correct.
imho it would be correct if it would be the sum of the B and P sizes (shown on right), but it isnt

It's the same bug. I think in reality it should be something like this:what you are saying is more than just a wrong size.
what you are saying is that the frametype is shown wrongly (I vs. P) which would be a more serious bug than just an off size imho

Whether it's a bug or by design I don't know. But if the bad lengths are being used for first pass stats (I have a sneaking suspicion they are) the filesize is going to be way off. Can you do that same stream again with -pass1, then examine the stats file for any weirdness? actually now that you are saying i have got indeed wrong filesizes with no packed output together with n-vops enabled (with one pass encoding)

[squid_80]

i have been able to create smaller than 10 real p-vops with a very small res and very small bitrate. i doubt anyone will ever do something like that, its just that it should be known as an potential issueSmaller than 7 bytes? That's the largest n-vop I've seen. 10 was chosen to allow a few extra bytes, just in case.
about the not coded flag: will you identify n-vops by the flag or by the size?
doing it by the flag will make sure no small p-vop will get dropped and will always be correctSpeed is the issue. The check has to be done for every frame outputted by the encoder, so the simpler it is, the better.
and what are you answer to my question whether you are doing the fake n-vop check also with not packed streams?At the moment yes, because it means one less if statement in the code.
well yeah, still i dont think the size of the packed frame(s) (shown on the left) is correct.
imho it would be correct if it would be the sum of the B and P sizes (shown on right), but it isntTake the size of the P frame from the right. Subtract the size of the encoded n-vop. Add the size of the B frame from the right. The result should match the size of the encoded packed frame. If it doesn't, give an example.
what you are saying is more than just a wrong size.
what you are saying is that the frametype is shown wrongly (I vs. P) which would be a more serious bug than just an off size imhoWell the encoder's saying the encoded frame is a key frame, which means it's an I frame. And the PSNR calculations would seem to back that up. I think that with -nopacked, the stats (right side) are one frame behind i.e. they are actually for the encoded frame from the line above. Except if there's a n-vop, then the stats will match the encoded frame from the same line. This has the side effect of making some stats never appear in the output at all.
actually now that you are saying i have got indeed wrong filesizes with no packed output together with n-vops enabled (with one pass encoding)
Like I said, do a first pass and check the stats file. There's definitely weirdness in there. They even crash Koepi's statsreader.

Edit: hlength in the statsfile - my understanding is that it's the length of the headers in each frame e.g. it won't change no matter what quantizer is used. So a negative value is probably really bad, right?

[squid_80]

Ok, I put on my bug hunting hat and found a few things. Firstly there's this in encoder.c (function FrameCodeP)
/* frame drop code */
if (current->sStat.kblks + current->sStat.mblks <=
(pParam->frame_drop_ratio * mb_width * mb_height) / 100 &&
( (pEnc->bframenum_head >= pEnc->bframenum_tail) || !(pEnc->mbParam.global_flags & XVID_GLOBAL_CLOSED_GOP)) )
{
current->sStat.kblks = current->sStat.mblks = 0;
// Squid's fix
// current->sStat.iTextBits = 0;
current->sStat.ublks = mb_width * mb_height;
I think current->sStat.iTextBits should be set to 0 since an n-vop contains no texture information. I tested and it fixed the bitrate issues for packed bitstreams with frame drops.
For unpacked bitstream there's another problem. These 2 sections of code (again from encoder.c):
/* for unpacked bframes, output the stats for the last encoded frame */
if (!(pEnc->mbParam.global_flags & XVID_GLOBAL_PACKED) && pEnc->mbParam.max_bframes > 0)
{
if (pEnc->current->stamp > 0) {
call_plugins(pEnc, pEnc->reference, &pEnc->sOriginal, XVID_PLG_AFTER, NULL, NULL, stats);
}
else
stats->type = XVID_TYPE_NOTHING;
}
if ( FrameCodeP(pEnc, &bs) == 0 ) {
/* N-VOP, we mustn't code b-frames yet */
// Squid's fix
// if ((pEnc->mbParam.global_flags & XVID_GLOBAL_PACKED) || pEnc->mbParam.max_bframes == 0)
call_plugins(pEnc, pEnc->current, &pEnc->sOriginal, XVID_PLG_AFTER, NULL, NULL, stats);
goto done;
}
The upshot of this is when packed bitstream isn't used and a n-vop is encoded, the plugins are called twice. This explains how I was seeing double entries in the stats file for n-vops. The fix I've come up with seems to solve the problem, but breaks encraw like I thought it would. :(

[squid_80]

I'm confused; need help from a matroska expert.
Haali says ( http://haali.cs.msu.ru/mkv/codecs.pdf ) that the mpeg4 headers up to the first VOP go in the CodecPrivate element and the stream itself should only contain VOPs and no headers, but this borks things if the VOL header changes which seems to happen sometimes (use a constant quant zone at some point after the start). Maybe it's just a first pass quirk, I haven't got that far yet. Keeping the VOL header with the I frames in the mkv seems to work, but is apparently prohibited. Any idea which is correct?

[bond]

mpeg-4 defines that the vol can never change throughout the stream
if a mpeg-4 encoder outputs varrying vols its incompliant to the mpeg-4 standard

[squid_80]

That's what I thought. I'll ignore it and move on to 2nd pass...

[bond]

so will you output varrying vols or will you do it as haali writes?

[squid_80]

Like Haali says. Unless people want busted streams, then I might make an option for it.

[bond]

actually if xvid outputs varrying vols its a serious bug which needs to be fixed :>

[squid_80]

I've only seen it doing first pass with full1pass disabled, which the vfw says: "If you don't discard first pass but keep full quality disabled, the resulting 1st pass stream might not be mpeg-4 compliant."

[bond]

yeah, the results from the fastfirstpass are not useable for anything

[JoeBG]

@ squid_80

Do you plan a new release? Last release works perfect for me with xvidcore 1.2 beta single threat version -> So it´s just a question. :)

[squid_80]

Of course, I'm working on nothing else at the moment (except my real-life job to pay the bills).

@bond: Have you heard anything back about your post to the xvid-devel mailing list? Because I think I pretty much solved the bitrate problems but unfortunately the bug is in xvidcore, I can implement a fix in encraw but it would mean duplicating a lot of code.

[bond]

@bond: Have you heard anything back about your post to the xvid-devel mailing list? Because I think I pretty much solved the bitrate problems but unfortunately the bug is in xvidcore, I can implement a fix in encraw but it would mean duplicating a lot of code.nope i didnt. if you know how to fix it it would be best to do in the core itself. i am sure the xvid devs will apply a working patch from your side to the cvs

[Doom9]

@squid_80: are you making any progress on the remaining commandline options?

[squid_80]

Working on it as we speak, should have something in about 1.5 hours.

[squid_80]

Damnit, not going to get zones finished off properly tonight but there's a new build with rate control options and custom PAR settings. Display AR should now be set for mkv, I/P/B min quants are 2 by default, the trailing p is dropped from the divx userdata for mkv and m4v ES output(not sure if this is done correctly), cartoon mode is no longer accidentally activated, GMC works with mkv, fake vs. real n-vop detection no longer relies on a bug in xvidcore.

BTW I strongly recommend not using frame drops and b-frames together; the encoder is smart enough to not drop the P frame immediately after a B frame series, but not smart enough to know not to use B frames after encoding an n-vop.
159: key=0, time= 0, len= 6 | type=P, quant= 7, len= 12462
160: key=0, time= 125, len= 20950 | type=B, quant= 11, len= 5124
161: key=0, time= 0, len= 6 | type=P, quant= 7, len= 15832
162: key=0, time= 94, len= 6 | type=P, quant= 7, len= 6
163: key=0, time= 125, len= 12282 | type=B, quant= 13, len= 1443
164: key=0, time= 0, len= 6 | type=P, quant= 9, len= 10845
165: key=0, time= 94, len= 5873 | type=B, quant= 15, len= 1705
The highlighted P frame is an n-vop; the following B frame shows artifacts since it references a frame that isn't coded.
I've also seen artifacts when using -noclosed_gop, but haven't had time to look into it.

[squid_80]

Re:zones, I was going to be clever and allocate them dynamically (instead of having a limit of 64 max) but realized this was silly since it would probably be impossible to specify that many via the command line anyway. But now I'm thinking of adding an option to pass a filename (e.g. -zonefile xxx) which would be a text file containing the zones in a specified format. Good idea or bad idea?

[buzzqw]

But now I'm thinking of adding an option to pass a filename (e.g. -zonefile xxx)

Good Idea ! but don't drop support for old command line zones

BHH

[bond]

how are you detecting fake n-vops now?
btw its not possible to deactivate real n-vops placement in xvid

about zones: why not simply do it like other cmdl tools (mp4box, mencoder...):
you specify a zone via -zone and than add specific commands you want right after it with a ":"

like -zone startframe=3000:weight=0.5:cartoonmode:greyscale

that would be straight forward and easy to set and understand

[squid_80]

Good Idea ! but don't drop support for old command line zonesI won't. I'm trying hard not to bork any of the original options for backwards compatability.
how are you detecting fake n-vops now?
btw its not possible to deactivate real n-vops placement in xvid
fake n-vop=a n-vop that follows a b-vop when using packed bitstream.
The real ones that show up when frame-drop=0 are identical to the frame before, so the artifacts aren't visible.
about zones: why not simply do it like other cmdl tools (mp4box, mencoder...):
you specify a zone via -zone and than add specific commands you want right after
it with a ":"

like -zone startframe=3000:weight=0.5:cartoonmode:greyscale

that would be straight forward and easy to set and understand
There will be something like that; I was going to go with doom9's suggestion since it's nice and compact ( http://forum.doom9.org/showthread.php?p=702759#post702759 ) but drop the end zone parameter.

Something I forgot: -pass1 and -pass2 default to using xvid.stats as the statsfile if one isn't specified. Also packed bitstream is disabled if avi output isn't used - this may cause slight oversize in the second pass, we'll see what happens.

[bond]

There will be something like that; I was going to go with doom9's suggestion since it's nice and compact ( http://forum.doom9.org/showthread.php?p=702759#post702759 ) but drop the end zone parameter. nice, tough i prefer a = instead of a , for signalling what belongs together, but thats a matter of taste

Also packed bitstream is disabled if avi output isn't used - this may cause slight oversize in the second pass, we'll see what happens. great!
tough disabling packed bitstream or not should not make any difference on the stream (except the n-vops of course) as packed bitstream is only a "packing mode" on how frames are stored. if with or without packed doesnt give the same results there is a bug (maybe its the same bug i mentioned on the maillist)

[squid_80]

tough disabling packed bitstream or not should not make any difference on the stream (except the n-vops of course) as packed bitstream is only a "packing mode" on how frames are stored. if with or without packed doesnt give the same results there is a bug (maybe its the same bug i mentioned on the maillist)

I wouldn't really label it a bug; just that the statsfile is different depending if packed or nopacked is used, due to each fake n-vops taking up a few bytes. I *think* rate-control will compensate for it, if it does the stream will be encoded differently (when comparing packed/unpacked) and if not there might be a slight oversize. I'll have to do more tests.

[bond]

as i understand it the stats file should be exactly the same no matter if packed bitstream is enabled or not

[squid_80]

BUG: when using raw input and not specifying a maximum number of frames, EVERY frame is encoded as a P frame. Whoops!

[squid_80]

New build with -zones parameter, should work as described except there's no end frame parameter; a zone lasts until the start of the next one just like vfw. There will probably be a bug or two, please report any problems.

Edit: sorry, as described here: http://forum.doom9.org/showthread.php?p=702759#post702759

[JoeBG]

New build with -zones parameter, should work as described except there's no end frame parameter; a zone lasts until the start of the next one just like vfw. There will probably be a bug or two, please report any problems.

Edit: sorry, as described here: http://forum.doom9.org/showthread.php?p=702759#post702759


Where is the download? :) :thanks:

[squid_80]

Same as before; ftp://squid80.no-ip.com/xvid_encraw.zip or http://okejl.dk/dunstan/xvid_encraw.zip

[woah!]

hi again.

so i see that the 29fps bug id faxed now as i get 29.97 fps :)

now to the next question, is pass2 borked? i cant get a readable output file from any encode i do pass2 :(

same setting in either quant or bitrate mode is fine, but 2pass crashes mpc.

pity as its nearly there isnt it ..


heres my cmd line:

xvid_encraw.exe -i file.avs -type 2 -pass1 file.stats -framerate 29.970000 -qtype 1 -lumimasking -bitrate 1830

xvid_encraw.exe -i file.avs -type 2 -pass2 file.stats -o file.avi -framerate 29.970000 -qtype 1 -lumimasking -bitrate 1830

no avi output file for 1st pass only for the second

nothing to fancy as you see.

[squid_80]

<edited out rubbish>

-o is for saving an elementary stream. -avi is for saving an avi file. Use -avi instead.

[woah!]

trim(30000, 30230)

file is small as i am trying to get this to work before i go with a full encode.

i am encoding about 44mins of 1920x1080i HD source to 704x400 and i have no issues doing the same avs script using mencoder.

[woah!]

noooooo say it aint so, how did i miss that heh... couldnt see the forest for the trees.. thx for this it is now working great :)

[Doom9]

@squid: are the new zone options actually in and just the cli help is not up-to-date? Also, now that it seems the missing options are supported, any chance of a -threads parameter?

[squid_80]

Yeah I haven't got around to updating the help screen. But they should be working. (except the -zonefile idea I had, haven't done that yet.)
-threads I can put in, but it won't actually do anything until we get a fixed SMP version of xvidcore which doesn't have the number of threads hardcoded.

[JoeBG]

Can I have a list of all options? I just thought zones and mkv are new.

[squid_80]

c:\>xvid_encraw -help
xvid_encraw built at 22:29:08 on Feb 9 2006
Usage : xvid_encraw [OPTIONS]

Input options:
-i string : input filename (stdin)
-type integer: input data type (yuv=0, pgm=1, avi/avs=2)
-w integer: frame width ([1.2048])
-h integer: frame height ([1.2048])
-frames integer: number of frames to encode

Output options:
-dump : save decoder output
-save : save an Elementary Stream file per frame
-o string : save an Elementary Stream for the complete sequence
-avi string: save an AVI file for the complete sequence
-mkv string: save a MKV file for the complete sequence

BFrames options:
-max_bframes integer: max bframes (2)
-bquant_ratio integer: bframe quantizer ratio (150)
-bquant_offset integer: bframe quantizer offset (100)

Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-max_key_interval integer : maximum keyframe interval (300)

Single Pass options:
-reaction integer : reaction delay factor (16)
-averaging integer : averaging period (100)
-smoother integer : smoothing buffer (100)

Second Pass options:
-kboost integer : I frame boost (10)
-kthresh integer : I frame reduction threshold (1)
-kreduction integer : I frame reduction amount (20)
-ostrength integer : overflow control strength (5)
-oimprove integer : max overflow improvement (5)
-odegrade integer : max overflow degradation (5)
-chigh integer : high bitrate scenes degradation (0)
-clow integer : low bitrate scenes improvement (0)
-overhead integer : container frame overhead (24)
-vbvsize integer : use vbv buffer size
-vbvmax integer : vbv max bitrate
-vbvpeak integer : vbv peak bitrate over 1 second

Other options
-noasm : do not use assembly optmized code
-turbo : use turbo presets for higher encoding speed
-quality integer : quality ([0..6]) (6)
-vhqmode integer : level of R-D optimizations ([0..4]) (1)
-bvhq : use R-D optimizations for B-frames
-qpel : use quarter pixel ME
-gmc : use global motion compensation
-qtype integer : quantization type (H263:0, MPEG4:1) (0)
-qmatrix filename : use custom MPEG4 quantization matrix
-interlaced : use interlaced encoding
-nopacked : Disable packed mode
-noclosed_gop : Disable closed GOP mode
-lumimasking : use lumimasking algorithm
-stats : print stats about encoded frames
-debug : activates xvidcore internal debugging output
-vop_debug : print some info directly into encoded frames
-nochromame : Disable chroma motion estimation
-notrellis : Disable trellis quantization
-imin integer : Minimum I Quantizer (1..31) (2)
-imax integer : Maximum I quantizer (1..31) (31)
-bmin integer : Minimum B Quantizer (1..31) (2)
-bmax integer : Maximum B quantizer (1..31) (31)
-pmin integer : Minimum P Quantizer (1..31) (2)
-pmax integer : Maximum P quantizer (1..31) (31)
-drop integer : Frame Drop Ratio (0..100) (0)
-start integer : Starting frame number
-par integer[:integer] : Set Pixel Aspect Ratio.
1 = 1:1
2 = 12:11 (4:3 PAL)
3 = 10:11 (4:3 NTSC)
4 = 16:11 (16:9 PAL)
5 = 40:33 (16:9 NTSC)
other = custom (width:height)
-help : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as needed.
-zones integer,q/w,float[,flags]
integer = start frame of zone
q/w = q for quantizer zone or w for weight zone
float = quant or weight (floating point value)
optional flags = K for start with keyframe, C for cartoon mode, O for chroma optimizer, G for greyscale or an integer for BVOP sensitivity. Order isn't important.
No spaces anywhere in the zones options.
Separate multiple zones with / . Or I guess you could use multiple -zones but I haven't tested it since it wasn't designed this way.
Also there's a -packed option to force packed bitstream on since it's off by default if there's no .avi output. Kinda weird I know but as we've discussed the bitstream setting affects what is written to the statsfile and that in turn has a substantial effect on the final encoded stream.

[JoeBG]

c:\>xvid_encraw -help
xvid_encraw built at 22:29:08 on Feb 9 2006
Usage : xvid_encraw [OPTIONS]

Input options:
-i string : input filename (stdin)
-type integer: input data type (yuv=0, pgm=1, avi/avs=2)
-w integer: frame width ([1.2048])
-h integer: frame height ([1.2048])
-frames integer: number of frames to encode

Output options:
-dump : save decoder output
-save : save an Elementary Stream file per frame
-o string : save an Elementary Stream for the complete sequence
-avi string: save an AVI file for the complete sequence
-mkv string: save a MKV file for the complete sequence

BFrames options:
-max_bframes integer: max bframes (2)
-bquant_ratio integer: bframe quantizer ratio (150)
-bquant_offset integer: bframe quantizer offset (100)

Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-max_key_interval integer : maximum keyframe interval (300)

Single Pass options:
-reaction integer : reaction delay factor (16)
-averaging integer : averaging period (100)
-smoother integer : smoothing buffer (100)

Second Pass options:
-kboost integer : I frame boost (10)
-kthresh integer : I frame reduction threshold (1)
-kreduction integer : I frame reduction amount (20)
-ostrength integer : overflow control strength (5)
-oimprove integer : max overflow improvement (5)
-odegrade integer : max overflow degradation (5)
-chigh integer : high bitrate scenes degradation (0)
-clow integer : low bitrate scenes improvement (0)
-overhead integer : container frame overhead (24)
-vbvsize integer : use vbv buffer size
-vbvmax integer : vbv max bitrate
-vbvpeak integer : vbv peak bitrate over 1 second

Other options
-noasm : do not use assembly optmized code
-turbo : use turbo presets for higher encoding speed
-quality integer : quality ([0..6]) (6)
-vhqmode integer : level of R-D optimizations ([0..4]) (1)
-bvhq : use R-D optimizations for B-frames
-qpel : use quarter pixel ME
-gmc : use global motion compensation
-qtype integer : quantization type (H263:0, MPEG4:1) (0)
-qmatrix filename : use custom MPEG4 quantization matrix
-interlaced : use interlaced encoding
-nopacked : Disable packed mode
-noclosed_gop : Disable closed GOP mode
-lumimasking : use lumimasking algorithm
-stats : print stats about encoded frames
-debug : activates xvidcore internal debugging output
-vop_debug : print some info directly into encoded frames
-nochromame : Disable chroma motion estimation
-notrellis : Disable trellis quantization
-imin integer : Minimum I Quantizer (1..31) (2)
-imax integer : Maximum I quantizer (1..31) (31)
-bmin integer : Minimum B Quantizer (1..31) (2)
-bmax integer : Maximum B quantizer (1..31) (31)
-pmin integer : Minimum P Quantizer (1..31) (2)
-pmax integer : Maximum P quantizer (1..31) (31)
-drop integer : Frame Drop Ratio (0..100) (0)
-start integer : Starting frame number
-par integer[:integer] : Set Pixel Aspect Ratio.
1 = 1:1
2 = 12:11 (4:3 PAL)
3 = 10:11 (4:3 NTSC)
4 = 16:11 (16:9 PAL)
5 = 40:33 (16:9 NTSC)
other = custom (width:height)
-help : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as needed.
-zones integer,q/w,float[,flags]
integer = start frame of zone
q/w = q for quantizer zone or w for weight zone
float = quant or weight (floating point value)
optional flags = K for start with keyframe, C for cartoon mode, O for chroma optimizer, G for greyscale or an integer for BVOP sensitivity. Order isn't important.
No spaces anywhere in the zones options.
Separate multiple zones with / . Or I guess you could use multiple -zones but I haven't tested it since it wasn't designed this way.
Also there's a -packed option to force packed bitstream on since it's off by default if there's no .avi output. Kinda weird I know but as we've discussed the bitstream setting affects what is written to the statsfile and that in turn has a substantial effect on the final encoded stream.


Thanks :)
:stupid:
weight is in percent? Are the following examples ok?

-z 58000, q, 8 -> will start a zone beginning with frame 58000 with q=8
-z 58000, w, 20 -> will start a zone beginning with frame 58000 with 20 % of quality of the main video.

[squid_80]

weight is in percent? Are the following examples ok?

-z 58000, q, 8 -> will start a zone beginning with frame 58000 with q=8
-z 58000, w, 20 -> will start a zone beginning with frame 58000 with 20 % of quality of the main video.
Weight is a percentage but it's in decimal form. 1 = 100%, 0.1 = 10%, 0.001 = 1% etc.

-zones 58000,q,8/58001,w,0.2
A zone starting at frame 58000 with a quantizer of 8 and another zone starting at 58001 with a weight of 20%. Very important that there's no spaces anywhere.

[JoeBG]

Thanks. I updated this info in the german boards. We are all testing :) :)

[buzzqw]

@squid_80

for your bandwith safe here the link at actual (13/02/2006) build www.64k.it/andres/xvid_encraw-13022006.zip

BHH

[dimzon]

@squid_80
Great Job! Can you add cqmintra && cqminter command-line switches to be capable use CQM without cqm file
sample:
-cqmintra 16,16,...,127 -cqminter 32,32,...,32

[bond]

squid_80, someone has posted a patch on the xvid maillist which seems to fix my issues with high framedrop values, maybe you are interested in it (maybe it was you who wrote the patch? :D )

[squid_80]

Yeah it was.

[squid_80]

@squid_80
Great Job! Can you add cqmintra && cqminter command-line switches to be capable use CQM without cqm file
sample:
-cqmintra 16,16,...,127 -cqminter 32,32,...,32
Should be easy enough. I assume the order of the values is just reading the matrix left to right, one row at a time e.g. for a matrix like this:
8, 11, 13
12, 11, 12
12, 12, 13
You would put 8,11,13,12,11,12,12,12,13?
(I know q-matrices are 16x16, I'm just using 3x3 for a small example)
Just wanted to be sure in case someone expects the values to be in the other order that sometimes gets used (which would be 8,12,11,13,11,12,12,13).

[Sagittaire]

Possible to make that ... ???

- pass1 : write always stat files
- pass2 : read stat files

but with pass1 function always active even if pass2 command is enabled:

xvid_encraw.exe -i file.avs -type 2 -pass1 file1.stats ....
xvid_encraw.exe -i file.avs -type 2 -pass2 file1.stats -pass1 file2.stats ...
xvid_encraw.exe -i file.avs -type 2 -pass2 file2.stats -pass1 file3.stats ...

or perhaps other function like -writestats [filename] with exactly the same file structure than 1pass files. With this function it will be possible to make multipass encoding (better target bitrate, better predictibility for frame size, better quality ...)

[squid_80]

BTW, avi output is broken for files > 2gb in size (thanks to avifile). They might be recoverable using vdub with the rederive keyframe flags option, I haven't let it run long enough to find out. mkv and raw streams seem to be ok.
Possible workarounds: a) Get an avi writer capable of producing odml files and use that instead (probably from virtualdub) or b) output multiple files, splitting at 2gb boundaries.
Of course there's option c) ignore it and hope no-one notices.

[squid_80]

but with pass1 function always active even if pass2 command is enabled:

xvid_encraw.exe -i file.avs -type 2 -pass1 file1.stats ....
xvid_encraw.exe -i file.avs -type 2 -pass2 file1.stats -pass1 file2.stats ...
xvid_encraw.exe -i file.avs -type 2 -pass2 file2.stats -pass1 file3.stats ...

At the moment -pass1 will disable -pass2 and vice versa. Whichever option comes last in the commandline is the one that will be active.
I'll run some tests and see if it's worth implementing multi-pass. I'm not sure if they will actually work together to give better results.

[bond]

i think there is a bug in labelling s-vops as the following doesnt make sense for me (framedropratio of 100):
len= 6 | type=P
len= 6 | type=P
len= 6 | type=S
len= 6 | type=S
len= 6 | type=S
len= 6 | type=Sobviously the frames with size 6 are n-vops, still they are labelled as s-vops

[hellfred]

Of course there's option c) ignore it and hope no-one notices.
I saw it, I noticed it, TOO LATE NOW :D

[squid_80]

i think there is a bug in labelling s-vops as the following doesnt make sense for me (framedropratio of 100):
len= 6 | type=P
len= 6 | type=P
len= 6 | type=S
len= 6 | type=S
len= 6 | type=S
len= 6 | type=Sobviously the frames with size 6 are n-vops, still they are labelled as s-vops
Are not-coded S-VOPs against specs? I know why it happens; the decision to drop a frame is made after encoding, based on the number of not-coded macroblocks. If the percentage of not-coded mbs is greater than or equal to the frame drop ratio the whole frame is not coded.
I can definitely see how this could be bad with s-vops. But it's a problem in xvidcore rather than encraw.

[Wilbert]

@squid_80,

I know why it happens; the decision to drop a frame is made after encoding, based on the number of not-coded macroblocks. If the percentage of not-coded mbs is greater than or equal to the frame drop ratio the whole frame is not coded.
Is it possible to make that decision before the encoding?

[squid_80]

It wouldn't really save much since some sort of motion estimation would still need to be done. If you have a source with lots of close to identical frames and expect lots of N-VOPs it would be better IMO to run a filter that removes dupes and dumps a timecode file to be used later. (Assuming there's a filter to do this and it actually works with MPEG4 ASP in mkv; I haven't got around to finding that out, but I plan to.)

[bond]

It wouldn't really save much since some sort of motion estimation would still need to be done. If you have a source with lots of close to identical frames and expect lots of N-VOPs it would be better IMO to run a filter that removes dupes and dumps a timecode file to be used later. (Assuming there's a filter to do this and it actually works with MPEG4 ASP in mkv; I haven't got around to finding that out, but I plan to.)hehe i didnt want to ask for vfr with mkv ;)

[bond]

squid_80, someone has posted a patch on the xvid maillist which seems to fix my issues with high framedrop values, maybe you are interested in it (maybe it was you who wrote the patch? :D )was applied to cvs

squid: is xvidcli supposed to work with the 1.2 line of xvid?

[dimzon]

Should be easy enough. I assume the order of the values is just reading the matrix left to right, one row at a time e.g. for a matrix like this:
8, 11, 13
12, 11, 12
12, 12, 13
You would put 8,11,13,12,11,12,12,12,13?
(I know q-matrices are 16x16, I'm just using 3x3 for a small example)
Just wanted to be sure in case someone expects the values to be in the other order that sometimes gets used (which would be 8,12,11,13,11,12,12,13).
YES EXATLY

[squid_80]

squid: is xvidcli supposed to work with the 1.2 line of xvid?
It should, that's what I'm using. It uses the current version of xvid.h so it should support all versions of xvidcore that are backwards compatible with that.

YES EXATLY
I'll try it but I don't know if specifying 256 values for each matrix will fit in the command line.

What I've been twiddling over the last few days is a -mkvvfr <filename> option, which produces a vfr mkv file by dropping n-vops completely and setting the duration and timecode of each frame appropriately.

[dimzon]

I'll try it but I don't know if specifying 256 values for each matrix will fit in the command line.
I'm not shure but seem's like there are no restrictions in modern OS (Win2K/WinXP)

[bond]

whats the advantage of specifying a cqm via cmdl and not using an external file?

edit:
What I've been twiddling over the last few days is a -mkvvfr <filename> option, which produces a vfr mkv file by dropping n-vops completely and setting the duration and timecode of each frame appropriately.is it necessary to make this vfr option so mkv centric? i mean regarding an hopefully coming mp4 output which also would be able to handle vfr its maybe better to keep as much vfr things as container agnostic as possible

[dimzon]

whats the advantage of specifying a cqm via cmdl and not using an external file?
to avoid unnececary dependencies when creating profiles in MeGUI ;)

[squid_80]

is it necessary to make this vfr option so mkv centric? i mean regarding an hopefully coming mp4 output which also would be able to handle vfr its maybe better to keep as much vfr things as container agnostic as possible
The main reason I'm using -mkvvfr is so -mkv can still be used at the same time in case of errors. If (when) mp4 output is added I can add a -vfr option which will make vfr mkv+mp4 output streams.

[bond]

why not use -vfr right away?

[ImmortAlex]

Hi! I have some questions:

1. In GUI there is option to specify target file size for 2nd pass. In xvid_encraw I see only target bitrate. Is this equivalent? For example, if bitrate calculator tells me that size is 617820 kbytes and bitrate is 821 kbps, is specifing "-bitrate 821" in encraw equals to specifing 617820 in GUI?

2. Are units for VBV options equals to GUI ? Are "-vbvsize 3145728 -vbvmax 4854 -vbvpeak 8000000" right options, equals to "Cinema Plus" profile?

[squid_80]

Hi! I have some questions:

1. In GUI there is option to specify target file size for 2nd pass. In xvid_encraw I see only target bitrate. Is this equivalent? For example, if bitrate calculator tells me that size is 617820 kbytes and bitrate is 821 kbps, is specifing "-bitrate 821" in encraw equals to specifing 617820 in GUI?Yes. I plan to add a -size parameter later.
2. Are units for VBV options equals to GUI ? Are "-vbvsize 3145728 -vbvmax 4854 -vbvpeak 8000000" right options, equals to "Cinema Plus" profile?
Yes they match the VFW options.

[ImmortAlex]

Thanks, but at least -vbvmax is not in kbps.
"-vbvmax 4854" leads to quant=31 for all frames. Maybe I need "-vbvmax 4854000" ? BTW, it shows like "4854.000" in GUI.

Also I don't understand how to set zones. For example, "-frames 2000 -zones 0,w,1.0/1000,w,0.01" doesn't produce a clip with worse quality in its 2nd half. What is my mistake?

[squid_80]

Thanks, but at least -vbvmax is not in kbps.
"-vbvmax 4854" leads to quant=31 for all frames. Maybe I need "-vbvmax 4854000" ? BTW, it shows like "4854.000" in GUI.
I can't check the source code ATM but you're probably right. The value internally is bps, the vfw gui just chooses to show it as kbps and I missed the decimal point when I checked before.
Also I don't understand how to set zones. For example, "-frames 2000 -zones 0,w,1.0/1000,w,0.01" doesn't produce a clip with worse quality in its 2nd half. What is my mistake?
Probably my fault - I made a mistake by not sorting the zones before passing them to the encoder. Try switching the order of the zones e.g. -zones 1000,w,0.01/0,w,1.0 and see if that helps.

[squid_80]

why not use -vfr right away?
Actually I now think it's a dumb idea anyway, because:
- possibly breaks specs; no frame found matching expected timecode (not sure about this, need a mpeg4asp expert)
- Only saves about 6/7/8 bytes per dropped n-vop, not really worth the effort.

[bond]

Actually I now think it's a dumb idea anyway, because:
- possibly breaks specs; no frame found matching expected timecode (not sure about this, need a mpeg4asp expert)sounds more like a container-centric problem?

- Only saves about 6/7/8 bytes per dropped n-vop, not really worth the effort. yeah, but vfr support is a funky feature ;)

[squid_80]

sounds more like a container-centric problem?That's why I need an expert. Each VOP does have a timecode of sorts based on dwrate (I dunno the real terminology, that's the avi term for it, where framerate = dwrate/dwscale; xvid source code uses fbase/fincr), how it is used I'm not sure. It looks like it can be used for determining if a n-vop is real or fake - at least the way xvid does it. Might just be something to do with error correction.
It would be the same as replacing n-vops in avi (real ones) with actual dropped frames - more or less pointless.

[bond]

i would assume that a vfr mkv file simply doesnt have a framerate (dwrate) anymore?

maybe you want to send Mosu a message on how its handled

[foxyshadis]

It has a default duration; frames that deviate from it have their durations embedded in the frame header. I believe mp4 works the same way.

[squid_80]

I'm not talking about the container, I'm talking about what's in the raw m4v stream - specifically the first 3 or 4 bytes after the vop start code. For example here's a frame I picked at random:
00 00 01 B6 51 60 54 1D....
00 00 01 B6 = vop start code. Now we break the stream down into bits. 51 60 54 =
01 0 1 00010 1 10000001010100
01 = vop type (P)
0 = seconds. Not really seconds, kinda got something to do with fbase. The value is calulated by counting the number of preceding 1s before the 0.
1 = marker bit.
00010 = ticks, measured in fbase units. So for each successive frame, ticks increases by one. It rolls back to 0 when it reaches fbase, and the seconds field is incremented. But then the seconds field goes back to 0 after a few frames... the exact point at which it does seems to be max_bframes dependant.
1 = vop coded, the rest of the bits are to do with the encoded frame. (If it's not coded there's just padding up to the next byte.)

The width of the ticks field varies, it is equal to the base 2 log of fbase-1 (my maths is rusty, can't remember how to write that correctly as text). That's why the size of n-vops can vary depending on the framerate (the seconds field also varies).
What I would like to know is what a decoder is allowed to do with these values and what effect (if any) there would be if there was a sudden jump forwards/backwards in ticks.

[Zero1]

I tried encraw again since the last time I had problems (when I posted the screenshot). It works fine now, no visual errors at all ;) The command line and switches are pretty straight forward too.

Great work and thanks to everyone involved, it's great to be able to totally bypass VfW and produce native, spec compliant encodes.

Just a couple of questions, are there options within the CLI that are, or could be accessible, which would help improve the quality (over the VfW counterpart). I might be wrong, but aren't there some other search modes or anything in there that aren't accessible by the VfW?
Also, is it much hassle to update it's internals to something like 1.1.0 final, or maybe even the current 1.2? Even if the multithreading does not work, maybe the trellis improvments would be handy ;)

Cheers

[squid_80]

Just a couple of questions, are there options within the CLI that are, or could be accessible, which would help improve the quality (over the VfW counterpart). I might be wrong, but aren't there some other search modes or anything in there that aren't accessible by the VfW?
The way the quality and vhqmode options work is the best way to do it IMO; they group together search modes of the same approximate quality. I don't think it would be an improvement to allow individual modes to be (de)selected. If anyone used S_O's build and thought otherwise, feel free to mention it.
Also, is it much hassle to update it's internals to something like 1.1.0 final, or maybe even the current 1.2? Even if the multithreading does not work, maybe the trellis improvments would be handy ;)
It uses whatever version of xvidcore.dll you have installed. If that's the current version then you're already using the improvements.

(note to self: add output message to show the version of xvidcore.dll being used, if possible.)

[squid_80]

New build up. Actually there was a newer build up over 24 hours ago, I just updated it again. Nothing real groundbreaking:
-size option (help text is formatted wrong :( )
-pass1 and -pass2 can be used at the same time even though I couldn't get any improvement from nth pass
-interlaced option has support for TFF
-threads option is implemented and should work when support is added to xvidcore
- information about detected xvidcore version is displayed
-progress option, which I added in about 2 minutes because I was sick of watching the scrolling text output. It's a bit buggy.
-vfr option (not in help text) which should drop n-vops from mkv output streams and make the stream vfr. Haven't played with it much but seems to work.

Edit: Forgot that I also fixed zone handling to sort zones properly.

[shon3i]

Where i can download this lastest version.

[buzzqw]

link is same as usual ftp://squid80.no-ip.com/xvid_encraw.zip

or my (tiny) mirror www.64k.it/andres/xvid_encraw-23022006.zip

BHH

[shon3i]

link is same as usual ftp://squid80.no-ip.com/xvid_encraw.zip

or my (tiny) mirror www.64k.it/andres/xvid_encraw-23022006.zip

BHH
Thanks

[bond]

pretty much a small revolution having a xvid vfr encoder

thx squid!

[bond]

btw for stressing the vfr option via the drop option you should make sure to use a recent 1.2 xvid compile, cause the older ones (eg official 1.1) have a bug with n-vops

[unplugged]

Does this recent xvidenc_raw is compatible with current MeGUI?

[squid_80]

Not as far as I know since I broke the -bitrate parameter (whoops). Might be possible to make a fix - I could assume any value > 10000 is in bps and any value <= 10000 is in kbps. Does anyone regularly use bitrates less than 10kbps or greater than 10 mbps?

[squid_80]

btw for stressing the vfr option via the drop option you should make sure to use a recent 1.2 xvid compile, cause the older ones (eg official 1.1) have a bug with n-vops
Is that the n-vop with unpacked bitstream bug that I fixed? That shouldn't affect the vfr stuff, I though it just mucks up the stats file and bitrate distribution.

[Doom9]

Does this recent xvidenc_raw is compatible with current MeGUI?Not now, but the next stable megui will be.. it has no release date and will be available when it's done.

@squid: when I saw -overhead integer : container frame overhead (24) I wanted to run away screaming.. did you just reintroduce all the mencoder problems or can I make it so that encraw doesn't give a flying hoot about container knowledge? If I give it a bitrate of 700, I want my output video stream to have a bitrate of exactly 700 kbit/s.. and the file having a larger size than length * 700 kbit/s because of the container overhead. So my raw stream would have exactly the desired size, my avi and mkv would be bigger. It would be a shame if different calculation mechanisms would have to be reintroduced everywhere because xvid is so tied to avi. can I force -overhead 0 and never have to worry about container overhead ever again? (well, megui does.. but I have generic calculations for that.. I don't want codec specific ones)

[squid_80]

If the passed bitrate is what you want the raw stream to be, then use -overhead 0. I only made the default=24 because it's better to get an undersized file rather than oversized.

I actually abuse it and use -overhead 665 -size 143360 to get an output avi that I can mux with matching 128kbps mp3 audio and end up with 140megs :) If I had an automated reliable way of cutting out commercials, backing up DVB captures would almost be too easy...

[bond]

Is that the n-vop with unpacked bitstream bug that I fixed? That shouldn't affect the vfr stuff, I though it just mucks up the stats file and bitrate distribution.that bug lead to wrong target bitrates and different, wrong frametype decisions too

a fixed 1.2 compile is available here (http://rapidshare.de/files/14009418/xvidcore.dll.html)

[Doom9]

fyi, on the way from the office I added support for the latest encraw to megui.. I just have to solve a few crucial issues first before making another release, and then refactor some more.

[Beave]

If the passed bitrate is what you want the raw stream to be, then use -overhead 0. I only made the default=24 because it's better to get an undersized file rather than oversized.

I actually abuse it and use -overhead 665 -size 143360 to get an output avi that I can mux with matching 128kbps mp3 audio and end up with 140megs :) If I had an automated reliable way of cutting out commercials, backing up DVB captures would almost be too easy...

This sounds great. How can I calculate that? Say I want to mux a 384 kbit/s ac3 file?
Thanks for all the great work on xvid_encraw, btw!

[squid_80]

This sounds great. How can I calculate that? Say I want to mux a 384 kbit/s ac3 file?

Just to be clear, there's much better ways of doing this. I'm just lazy.

Divide the audio bitrate by 8 to get bytes/sec (384000/8 = 48000).
Divide that number by the framerate to get bytes/frame (48000/25 = 1920, or 48000/23.976 = 2002, or 48000/29.97 = 1602).
Add avi overhead (24) plus 1 (1920 + 24 + 1 = 1945).
The resulting output file should be pretty close to -size once the audio is muxed in.
It might not fit all audio formats but should work for CBR mp3 and ac3.

[Beave]

Just to be clear, there's much better ways of doing this. I'm just lazy.

Divide the audio bitrate by 8 to get bytes/sec (384000/8 = 48000).
Divide that number by the framerate to get bytes/frame (48000/25 = 1920, or 48000/23.976 = 2002, or 48000/29.97 = 1602).
Add avi overhead (24) plus 1 (1920 + 24 + 1 = 1945).
The resulting output file should be pretty close to -size once the audio is muxed in.
It might not fit all audio formats but should work for CBR mp3 and ac3.

Well, I find your way of specifying the overhead great. This way I don't need to calculate the length of the movie nor getting/calculating the filesize of the ac3. All I need to know is the framerate and the audio bitrate.

Thanks for your great input.

One question:
Why do I need the parameter -framerate? Can't xvid_encraw get that framerate info from the avs/d2v automaticly? If I don't use that -framerate switch, will my 23.976 video be converted to 25 fps?

I did a full encode, but I got a totally undersized file. I can't find the problem though:

Starting 1st Pass...

"xvid_encraw.exe" -i "test.avs" -avi "test.avi" -pass1 "test.stats" -size 1362413 -turbo -max_key_interval 240 -qmatrix "HVS Best Picture.qmatrix" -bvhq -framerate 23.976 -zq 0 22 -zw 151 0.6 -zw 2334 1 -zq 80422 20

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
xvidcore build version: xvid-1.1.0
Bitstream version: 1.1.0
Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
Detected 0 cpus, defaulting to 1 thread
83001 frames(100%) encoded, 8.28 fps, Average Bitrate = 3030kbpss
Tot: enctime(ms) =10022143.00, length(bytes) = 1311303606
Avg: enctime(ms) = 120.75, fps = 8.28, length(bytes) = 15798
I frames: 1178 frames, size = 151918/178959925, quants = 1 / 1.29 / 22
P frames: 32472 frames, size = 1515/49198004, quants = 1 / 1.61 / 22
B frames: 49350 frames, size = 21948/1083145677, quants = 2 / 3.02 / 31

Starting 2nd Pass...

"xvid_encraw.exe" -i "test.avs" -avi "test.avi" -pass2 "test.stats" -size 1362413 -max_key_interval 240 -qmatrix "HVS Best Picture.qmatrix" -bvhq -framerate 23.976 -zq 0 22 -zw 151 0.6 -zw 2334 1 -zq 80422 20

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
xvidcore build version: xvid-1.1.0
Bitstream version: 1.1.0
Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
Detected 0 cpus, defaulting to 1 thread
83001 frames(100%) encoded, 10.91 fps, Average Bitrate = 1160kbps
Tot: enctime(ms) =7608925.00, length(bytes) = 501897660
Avg: enctime(ms) = 91.67, fps = 10.91, length(bytes) = 6046
I frames: 1178 frames, size = 22479/26480984, quants = 2 / 7.17 / 22
P frames: 32472 frames, size = 10904/354106754, quants = 2 / 5.70 / 24
B frames: 49350 frames, size = 2458/121309922, quants = 4 / 8.28 / 31

Can someone detect the problem? The filesize should be about 1330 MB, but I only got 480 MB unfortunatly.

[Doom9]

All I need to know is the framerate and the audio bitrate.
Actually.. you need to know a lot more.. video overhead for the given container, audio overhead for the given container, subtitle overhead for the given container ;)

[dimzon]

@Beave
Seems like your source is clear enought to fit it into 480MB ;)
In this case you will obtain perfect 1CD rip instead of 2-cd

[bond]

squid_80, when will your code be ready for cvs commit? ^^

[Beave]

@Bond
No way, sorry. This is a 720p encode and plenty noise in it. It's an episode of a series which I encoded before with VirtualDubMod. If I encode it in Quant=2 I get around 1800MB.

Now I did the second pass again changing -size 1362413 to -bitrate 3221. This time I got the right size. So there must be something wrong with the -size setting.

[squid_80]

One question:
Why do I need the parameter -framerate? Can't xvid_encraw get that framerate info from the avs/d2v automaticly? If I don't use that -framerate switch, will my 23.976 video be converted to 25 fps?
Who says you always need to specify -framerate? It's only needed when using raw input or if you want to change it.

I did a full encode, but I got a totally undersized file. I can't find the problem though:

Starting 1st Pass...

"xvid_encraw.exe" -i "test.avs" -avi "test.avi" -pass1 "test.stats" -size 1362413 -turbo -max_key_interval 240 -qmatrix "HVS Best Picture.qmatrix" -bvhq -framerate 23.976 -zq 0 22 -zw 151 0.6 -zw 2334 1 -zq 80422 20

83001 frames(100%) encoded, 8.28 fps, Average Bitrate = 3030kbpss
Tot: enctime(ms) =10022143.00, length(bytes) = 1311303606
Avg: enctime(ms) = 120.75, fps = 8.28, length(bytes) = 15798
I frames: 1178 frames, size = 151918/178959925, quants = 1 / 1.29 / 22
P frames: 32472 frames, size = 1515/49198004, quants = 1 / 1.61 / 22
B frames: 49350 frames, size = 21948/1083145677, quants = 2 / 3.02 / 31
That command line doesn't seem to quite match, there's a progress report but no -progress option, and you've got quants < 2 but no -imin/pmin/bmin...

Even with an average quantiser close to 1, it's undersized.
Try in order:
Use -qtype 1 so the custom matrix is actually used (someone found a bug; yay!).
Turn off turbo.
Drop the zones.
Use 1 max bframe or even perhaps 0.
If it's still undersized set the minimum quants to 1. Also maybe update to 1.2.0...
Now I did the second pass again changing -size 1362413 to -bitrate 3221. This time I got the right size. So there must be something wrong with the -size setting.Redo both first and second pass with -qtype 1. Otherwise you're just wasting space.
Looks like you're right about -size though - I think any value > 262144 will bork. Easily fixed.
squid_80, when will your code be ready for cvs commit? ^^
When it's not buggy anymore... Remember when I said it was a bad idea to run 2 copies at once? Last week I ignored my own advice and ended up with a movie that changed PAR about one third of the way through. Still can't recreate the situation either.

[woah!]

@Bond
No way, sorry. This is a 720p encode and plenty noise in it. It's an episode of a series which I encoded before with VirtualDubMod. If I encode it in Quant=2 I get around 1800MB.

Now I did the second pass again changing -size 1362413 to -bitrate 3221. This time I got the right size. So there must be something wrong with the -size setting.


did you try -size 1330

[Beave]

That command line doesn't seem to quite match, there's a progress report but no -progress option, and you've got quants < 2 but no -imin/pmin/bmin...

Yes, you are right about one part. I added the -progress 10 on the cmd window, but did not update it in my batch file. I definately did not use any imin/pmin/bmin though. I'm 100% sure about it, I was wondering myself about the quants=1. Since the filesize was off and fps didn't quite match my normal fps I didn't pay too much attention to this.


Try in order:
Use -qtype 1 so the custom matrix is actually used (someone found a bug; yay!).
Turn off turbo.
Drop the zones.
Use 1 max bframe or even perhaps 0.
If it's still undersized set the minimum quants to 1. Also maybe update to 1.2.0...

So in the future releases -qtype 1 will not be nessesary to use a -qmatrix, right? So for the next try I will put them right after each other:

-qtype 1 -qmatrix "file.qmatrix"

in hopes it will use the CQ and not the MPEG Matrix.
Using -bitrate 3221 gives me a perfect sized result.
Your other comments where only valid with the wrong size in mind, right?

Redo both first and second pass with -qtype 1. Otherwise you're just wasting space.

I don't get this. Why am I wasting space?

Looks like you're right about -size though - I think any value > 262144 will bork. Easily fixed.

That's good to hear it's easily fixed.

did you try -size 1330
Why should I use an even smaller number if the file is already undersized?

[squid_80]

So in the future releases -qtype 1 will not be nessesary to use a -qmatrix, right? So for the next try I will put them right after each other:

-qtype 1 -qmatrix "file.qmatrix"

in hopes it will use the CQ and not the MPEG Matrix.
Using -bitrate 3221 gives me a perfect sized result. Yes, specifying a custom matrix should turn on MPEG quantization but it didn't. Fixed in next build.
Your other comments where only valid with the wrong size in mind, right?They were things to try if it still comes out undersized after the custom matrix is used.
I don't get this. Why am I wasting space?Because it used h263 quantization, using the same parameters with -bitrate 3221 is just going to lower the (already quite low) quantizers for no visible gain. You're telling it to use a bitrate higher than what your 1st pass ended up with.

[squid_80]

Yes, you are right about one part. I added the -progress 10 on the cmd window, but did not update it in my batch file. I definately did not use any imin/pmin/bmin though. I'm 100% sure about it, I was wondering myself about the quants=1. Since the filesize was off and fps didn't quite match my normal fps I didn't pay too much attention to this.
Do you still have the stats file from the 1st pass? I'd like to know where the quant1 frames are, it's probably a zones screw up since any frame not in a quant zone should get quant=2 during first pass.

[Kopernikus]

Hi,

is it possible to get the a copy of the sources?

Have you thought about sending a patch to the mailinglist? BTW, is your 64bit stuff already in CVS?

[squid_80]

Actually.. you need to know a lot more.. video overhead for the given container, audio overhead for the given container, subtitle overhead for the given container ;)
You also need to know it doesn't work for vfr encodes. Wish I'd realized that about 10 hours ago. :angry:

[bond]

any news? ^^

[squid_80]

any news? ^^
Yeah. Instead of doing what I should be (reorganizing the code so it's nice and neat, fixing the >2gb avi output problem, adding mp4 support etc.) I've been playing around with dedup making vfr encodes... Made a new parameter -timecode <file> which takes a mkv timecode file and adds dropped frames to the output avi to simulate vfr. You'd think it would be simple to add it to the mkv output module (since timecodes are pretty simple in mkv), but it's not since the timecodes are in display order and the frames are written in encoded order. Just got to figure out the most sensible way to buffer the times until they're written.

[JoeBG]

Any news for the mp4 output? :)

[Zero1]

Why MP4 output? MP4box handles it fine.

[bond]

Why MP4 output? MP4box handles it fine.because direct .mp4 output (maybe even with vfr) without needing to remux would rock much more :D

[squid_80]

It looks like code from mpeg4ip will work out pretty well. I'm just trying to keep it all neat and tidy and trimmed down by cutting out parts of code that aren't needed.

[dimzon]

It looks like code from mpeg4ip will work out pretty well. I'm just trying to keep it all neat and tidy and trimmed down by cutting out parts of code that aren't needed.
Wow, fine news.

[fight2win]

i tried xvid_encraw in megui, it said that the options i choose in encoding not supported!

[buzzqw]

you can try my application

BHH

[JoeBG]

i tried xvid_encraw in megui, it said that the options i choose in encoding not supported!

Because MeGui does not support the commands from xvid_encraw. MeGui uses mencoder commandlines for xvid_encraw. The support for xvid_encraw is planned, but not finished.

Mkv Magic does the job very good :)

[Doom9]

@JoeBG: not true. Public megui releases support the CSV builds of encraw.. but not squid's custom builds. A yet unreleased version will support squid's build.

[JoeBG]

@ doom9

OK, sorry, misunderstanding

[bond]

It looks like code from mpeg4ip will work out pretty well. I'm just trying to keep it all neat and tidy and trimmed down by cutting out parts of code that aren't needed.great stuff! will vfr also work?

[snherbst]

@bond

because direct .mp4 output (maybe even with vfr) without needing to remux would rock much more :D

I thought that this prog only could encode video content and not audio content, or am I very wrong?

As I see it the benefit in getting a MP4 output file vould be that the frame rate is part of a container content.

[Beave]

@squid_80

I am wondering if you fixed some of the bugs in xvid_encraw yet and could maybe provide some sort of beta version? I'm especially interested in using the -size parameter with bigger file sizes.

[squid_80]

Just posted this in another thread: http://okejl.dk/dunstan/xvid_encraw_megui_test.zip

I *think* I fixed that bug by reordering the size->bitrate equation to prevent overflow. I might be mistaken since I can't remember what it used to look like. Try it and see.
This build may contain unfinished code too, since it's my work-in-progress build. In particular, make sure the input is raw YV12 (you might get away with a compressed avi that decompresses to YV12 by default) because it won't throw an error if it's not - you'll just end up with a) garbage b) a cropped picture or c) a crash :)
For the adventurous, you can try making a packed .avi output file using -timecode <file> where file is a matroska v2 timecode file (like the one dedup spits out, for example). The .avi file should have dropped frames inserted to make up the time differences between frames.

[Elic]

squid_80
> Just posted this in another thread: http://okejl.dk/dunstan/xvid_encraw_megui_test.zip
Oops... Server seems to be offline (ping has time out). Are there another copies, please?

[squid_80]

Not unless anyone is able to mirror it. I had to take my ftp server down about a month ago (ftp://squid80.no-ip.com), the okejl.dk webspace is from a 1024/512 ADSL connection in Denmark, provided by someone who offered a home to my 64-bit stuff (thanks Asbjørn!). He did say it was only temporary so if it's down, it's down - He's been very generous and I'm not going to bug him about it.

[woah!]

i have some gigs to shares if you need a host.

i just need the file first

[Elic]

I had to take my ftp server down about a month agoOK, I have noticed that http://okejl.dk/ is online again. And I dowloaded your file and uploaded it into http://elicit.vn.ua/files/hosted/xvid_encraw_megui_test.zip, just to be on the safe side, OK? :)

[kurt]

I got a zone error with:
-zones 1,w,0.5,25,O/359,w,1,O/161519,w,0.04,25,O,G

Error in zone 161519,w,0.04,25,O,G option ,
Error in zone 1,w,0.5,25,O option ,
without flags there are no errors:
-zones 1,w,0.5,25/359,w,1/161519,w,0.04,25

Any ideas, what's going wrong?

btw: thx for your great work, squid_80!

[Adub]

Squid, in case you missed it in the other topic, I can confirm that your knew test build works in MeGUI. It works quite well infact, I was getting about 46fps on the first pass and about 38 fps on the second. Excellent job man! It didn't crash or anything. Of course I wasn't using every setting so I can't tell for sure, but it work for simple encoding purposes. Again, excellent job.

[squid_80]

I got a zone error with:
-zones 1,w,0.5,25,O/359,w,1,O/161519,w,0.04,25,O,G

Don't put commas between the optional flags (including bvop sensitivity), e.g:
-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG

Squid, in case you missed it in the other topic, I can confirm that your knew test build works in MeGUI.Excellent. Now I don't have to be so hard on myself for breaking it in the first place.

[Adub]

Yeah man don't worry about it. It worked wonderfully. Keep going with everything that you are doing to it to. It is faster than the build off of berrinam's page, and it no longer follows the 9999 rule. It's fast and the quality is perfect. Freakin' sweet job.

[kurt]

Don't put commas between the optional flags (including bvop sensitivity), e.g:
-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG

thx, no errors in first pass. :)
(I have to test it also for second pass)

question about -nopacked:

if I choose matroska output, this writes a native stream. So I didn't have to put -nopacked in the line, right? (I don't want to use packet bitstream)

[squid_80]

question about -nopacked:

if I choose matroska output, this writes a native stream. So I didn't have to put -nopacked in the line, right? (I don't want to use packet bitstream)
Right. -packed/-nopacked change the way the encoder passes back data, but mkv output will (at least it should) be the same either way. Packed bitstream is off by default anyway unless you are outputting to an .avi file or force it on using -packed.

[kurt]

Right. -packed/-nopacked change the way the encoder passes back data, but mkv output will (at least it should) be the same either way. Packed bitstream is off by default anyway unless you are outputting to an .avi file or force it on using -packed.
thx for the info. Maybe you should update the -help parameter
(there is only -nopacked) :)

[JoeBG]

Hi,

I have seen a logfile from MeGUI with the following parameters:

-closed_gop : I thought encraw does not have this command. It only has noclosed_gop
-bitrate 4169000. I thought squid changed this. 4169 should be correct

@ squid

I´m a little bit confused now :) Is it possible to update the helpfunction in encraw. I use your tool nearly daily, would be nice to have the correct commands

[squid_80]

The options are there for compatability. You're not missing anything by not knowing them.

-closed_gop more or less does nothing, since closed gop is on by default. (I have experimented with open gop and strongly suggest it shouldn't be used at all.)
-bitrate can be in bits/sec or kbits/sec - if the number given is less than or equal to 10000 it is assumed to be in kbits/sec, otherwise it is assumed to be bits/sec. This makes it possible to replace the original xvid_encraw with this version and not have to change other programs like MeGUI.

encraw is long overdue for an update, but I've been busy with other things. There is a x64 build available now though (as well as an updated avisynth64 and lame64).

[JoeBG]

Thanks for your answer.


encraw is long overdue for an update, but I've been busy with other things. There is a x64 build available now though (as well as an updated avisynth64 and lame64).

Very good.

[squid_80]

BTW, I had a quick look through the german forum (I don't speak german) and if the log you're referring to is this one, (http://forum.gleitz.info/showpost.php?p=261774&postcount=3) I think the problem might be that -o "G:\HD.avi" will not create a valid avi file; it will create a raw m4v stream in a file called hd.avi. If that commandline is being created by MeGUI, it's broken (although I doubt that's the case).

[bond]

i think people dont really get that -o is not useable for every output filetype

maybe -o should be renamed to -raw, or -o can be made useable with all filetypes (detected by the extension)

[JoeBG]

BTW, I had a quick look through the german forum (I don't speak german) and if the log you're referring to is this one, (http://forum.gleitz.info/showpost.php?p=261774&postcount=3) I think the problem might be that -o "G:\HD.avi" will not create a valid avi file; it will create a raw m4v stream in a file called hd.avi. If that commandline is being created by MeGUI, it's broken (although I doubt that's the case).


Yes, and you solved the problem. I will post your answer there in german.

[stax76]

Would be great having CLI frontend and VFW frontend all in one package meaning the CLI frontend is incl. in Koepi's (and others) builds.

Another thing that would be great is building the CLI on top of COM+ automation, interfaces that also allow to utilize the native XviD GUI. It would be trivial then to use XviD without VFW from *any* programming or scripting language, lot's of great tools could be build on top of that and lot's of already existing application could use it.

[shon3i]

where i can find lastest xvid_encraw. Thanks

[Brother John]

Elic provided a link here (http://forum.doom9.org/showpost.php?p=801447&postcount=400).

As I have quite a bit of unused traffic allowance right now I've put the archive on my site too: http://brother-john.net. Let's see how much traffic that'll produce.

[henryho_hk]

Has the "-cartoon" switch gone?

[squid_80]

What build had a -cartoon switch? It is available via -zones.

[bond]

squid_80, when can we see a new build? and when can we see your sources (just in case you abandon encraw so your changes are not lost...)? :)

[squid_80]

Hmm.... I'll go out on a limb and say tomorrow night (roughly 24 hours). If anyone has any feature requests (apart from mp4 output, that's too much testing to do in such a short time) say them now and if they're easy enough to do you might get lucky.

[bond]

moo? ;)

[IgorC]

Maybe I lost something here but question :
Does 2 pass or multipass work? It seems to be broked since some time.

[squid_80]

bond: The source code is there, but my ftp server's not :(

IgorC: Can you be a bit more specific on why it doesn't work?

[IgorC]

Seems 2 pass didn't work because of RC limits 9999 frames.

[kurt]

Did you use squid's latest build? http://forum.doom9.org/showthread.php?p=800747#post800747

no problems here with 2pass and more than 9999 frames :)

[IgorC]

Kurt
Thank you. I will try it.
And 3 pass mode? How it works? How to set it?

[kurt]

Sorry, I didn't try 3pass yet.... (and can't try it for the next few days either)...

How to set it?
I guess for setting up a 3pass encode use the same commandline as for 2pass. Just change the numbers ...

[bond]

bond: The source code is there, but my ftp server's not :(i am sure someone would be willing to host it :)

[Adub]

yeah man, some one would be sure to post it if you emailed it to them or put it in a zip file on the forum.
I can't wait to play with it! Ha!

[ChronoCross]

I can host it. pm me if you want.

[squid_80]

I don't like having stuff hosted by other people if I don't have direct access to it i.e. username and password to be able to upload new versions.

Anyway, turns out my ftp server wasn't down at all, it was just full up with connections left open. I've since reduced the connection time-out to 5 minutes. Also after 2.5 weeks I've *nearly* finished compressing 124gigs of DVB recordings, getting back some much needed space - so now I can get back to work.

[ChronoCross]

well I can create a user account for you if need be if you need off site mirroring and whatnot. just like I did for soulhunter. He has ftp access to his directory and thus can do what he pleases. Whatever you decide.

[Adub]

Squid,
your server is working again. Which Xvid encraw is the latest? is just the one labled "Xvid Encraw.zip". If so, is it compatible with MeGUI, as the specific MeGUI version is?

[squid_80]

I think the megui_test build is the most recent (apart from the x64 build). If you run xvid_encraw -h it shows the build date and time at the top. The older build won't work with MeGUI because it expects bitrate in kbps.

[DarkZell666]

Heh, for a wierd reason, xvid_encraw_test_megui produces invalid files ?

I used the megui_test build, and the resulting file (which was the correct filesize) wasn't playable. MPC just said : couldn't render the file.

I tried muxing the file into an mp4 using YAMB with no success, same goes for demuxing ...
What can I do to get the file to work ? Is it an actual .avi or a (borked) raw stream ?

Note : Input was an avs script (obviously :p, with ConvertToYV12() at the end), and MeGUI generated the following cmdlines :

Pass 1 :

Starting job job1-1 at 22:55:55
Job is a video job. encoder commandline:
-i "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avs" -o NUL: -pass1 "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.stats" -bitrate 800000 -turbo -max_key_interval 300 -vhqmode 4 -closed_gop -max_bframes 4 -bvhq -bquant_ratio 160 -bquant_offset 200
successfully started encoding
Processing ended at 23:43:40


Pass 2 :

job job1-1 has been processed. This job is linked to the next job: job1-2
Starting job job1-2 at 23:43:40
Job is a video job. encoder commandline:
-i "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avs" -pass2 "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.stats" -bitrate 800000 -max_key_interval 300 -vhqmode 4 -closed_gop -max_bframes 4 -bvhq -bquant_ratio 160 -bquant_offset 200 -o "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avi"
successfully started encoding
Processing ended at 00:44:59


(and don't laugh about the pokemon bit, this DVD has such a shaky picture it's perfect for my encoding tests :p)

[squid_80]

You can't use MeGUI to generate avi files using xvid_encraw. The -o parameter is to output a raw stream - that's what "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avi" will be.

(Why MeGUI uses -o NUL: for the first pass is beyond me; encraw has never required an output file to be explicitly set and it just wastes time dumping data down a sink.)

I was thinking of writing a (mpeg4asp) raw to avi muxer, but bond would probably kill me. :devil:

[DarkZell666]

I tried muxing the file into an mp4 using YAMB with no success,
The -o parameter is to output a raw stream - that's what "D:\POKEMON_5_HEROES\VIDEO_TS\pokemon.avi" will be.

Ok, after making my brain work a couple of seconds I renamed the .avi to .m4v and threw it at YAMB's face ... it worked ;)
I also tried .raw but it failed miserably ^^

It's wierd that YAMB (or mp4box, no idea) doesn't analyse the file but relies on the extension to determine if it is muxable or not.

Anyway thx for your help :)

[Kurtnoise]

Keep in mind that raw asp streams are stored in m4v files. That's all. So, it's completely useless to name or rename files with .raw extension.

and btw, analyse a file is not tricky because you must seek trough it. Thus, very time consuming in order to retrieve some infos, especially big files.

[squid_80]

Erm, did I just accidently break something or have the output .avi files ALWAYS had dwWidth and dwHeight swapped in their headers?

[bond]

heya squid_80. any news on the source (or a new build?)

[squid_80]

source: ftp://squid80.no-ip.com/xvid_encraw_src.zip
new build that should handle compressed (and raw colorspaces other than yv12) avi input files: ftp://squid80.no-ip.com/xvid_encraw.zip

(Also fixes the bug I mentioned above which no-one took any notice of.)

[Adub]

I was wondering about that bug.
Anyways, thanks for the new build! I can't wait to test this baby!
Laters,
Merlin Archangel

[stax76]

I can't download it: The server at squid80.no-ip.com is taking too long to respond

[buzzqw]

@stax

same problem here... waiting for better news :scared:

BHH

[squid_80]

@#$#cking firewall config has reset itself. See mirror at http://okejl.dk/dunstan

Edit: Main server should be back now.

[buzzqw]

quick direct link www.64k.it/andres/xvid_encraw-02-05-2006.zip

BHH

[stax76]

Are there cmdl examples for the most common uses? I find common use examples very helpful to learn a new CLI.

[Elic]

buzzqw > quick direct link www.64k.it/andres/xvid_encraw-02-05-2006.zip
Also http://elicit.vn.ua/files/hosted/xvid_encraw_src.zip, http://elicit.vn.ua/files/hosted/xvid_encraw-02-05-2006.zip

[squid_80]

Thank you for the mirrors. There's been about 120+ downloads in the past 24 hours, I thought there would only be a handful of people using it.

Command line examples
I decided this was a pretty good idea, since there's a couple of undocumented switches.
Here's my standard first pass command:
c:\>xvid_encraw -i final.avs -pass1 -packed -par 4 -bvhq -vhqmode 4 -progress -uses final.avs as input
-pass1 specifies first pass and since no log name follows, the default of xvid.stats is used.
-packed specifies packed b-frames mode, since my final target will be avi.
-par 4 means pal 16:9 pixel aspect ratio (not strictly necessary for first pass).
-bvhq means use VHQ for b-frames.
-vhqmode 4 means VHQ mode 4 - Wide Search.
-progress replaces the standard encoding status output with a minimal display which updates every 10 frames.

For 2nd pass, it's a bit more complicated:
c:\>xvid_encraw -i final.avs -pass2 -par 4 -bvhq -vhqmode 4 -progress -avi "vid final.avi" -size 179200 -overhead 665 -timecode mkvtimes.txt
-pass2 specifies 2nd pass using the default stats file name of xvid.stats
-avi "vid final.avi" means use vid final.avi as an output file.
-size 179200 specifies a target size of 179200 Kb (170Mb).
-overhead 665 = 665 bytes of overhead per frame. Default for avi is roughly 24, 128kbps audio + 25fps video = 640 bytes of audio per frame, 1 extra byte to be sure = 665 total. Better explanation is elsewhere in this thread.
-timecode mkvtimes.txt specifies mkvtimes.txt as a matroska v2 timecode file. If the output is an avi file, dropped frames will be inserted to sync the encoded frames to their timecodes. If the output is a mkv file, the timestamps will be copied thru (should be re-arranged correctly to suit b-frames, but I haven't tested much).

[stax76]

Thanks, when I've got a couple of hours time I'll make a TreeView based GUI for StaxRip.

[squid_80]

If it helps you any, -o will soon become the only option needed for output i.e. -o file.avi will create an avi file and -o file.mkv will create a mkv file. -o any_other_extension.xyz will still create a raw m4v stream.
-avi and -mkv will still be there, this is just to help out MeGUI users who keep getting stuck with raw m4v files which look like something else thanks to incorrect file extensions.

[Brother John]

Thx for the new version, squid_80. I've one question, though.

In your example above you mention -packed to enable packed bframes. Encraw's online help only mentions -nopacked to disable packed streams. No I'm wondering what is the proper way to en/disabled packed bitstream?

[SeeMoreDigital]

Thanks, when I've got a couple of hours time I'll make a TreeView based GUI for StaxRip.If it becomes your intension to incorporate this into StaxRip, by all means please let me know by PM and I'll be happy to test it for you before going live :)


Cheers

[squid_80]

In your example above you mention -packed to enable packed bframes. Encraw's online help only mentions -nopacked to disable packed streams. No I'm wondering what is the proper way to en/disabled packed bitstream?
Deja vu... (http://forum.doom9.org/showthread.php?p=803819#post803819) Packed bitstream is off by default unless outputting to an .avi file, which is not the case in my first pass. But as my second pass target is a packed .avi file, for best results the stats file figures SHOULD include the fake n-vops produced by packed bitstream. It's not absolutely necessary, filesize and quality will probably be more or less the same either way but I'm pedantic.

Also -packed was an option of the original xvid_encraw, hence it remains for compatability.

[SeeMoreDigital]

If say, I wanted to generate an encode with b-frames straight to MP4 (ie: packed bitstream free). How will the b-frames be ordered, in relation to the I and P frames?


Cheers

[squid_80]

Umm, however mp4creator does it? That's where the code is coming from.

[foxyshadis]

mp4 always has to be in jumbled mpeg order anyway, splitters would probably crash if you somehow shoehorned packed into mp4.


IPBBPBBPBIPPBPP
031264587902134

[SeeMoreDigital]

Umm, however mp4creator does it? That's where the code is coming from.
mp4 always has to be in jumbled mpeg order anyway, splitters would probably crash if you somehow shoehorned packed into mp4.


IPBBPBBPBIPPBPP
031264587902134
So to confirm... the B frame order for MP4 will not be the same as it is when they are hacked into AVI?


Cheers

[squid_80]

No, that is how they're stored in avi. Except in packed mode, the first B frame in a run is stored packed with the preceeding reference frame and an extra P (n-vop) is stored at the tail of the run.

Packed mode only exists for avi, for mkv and mp4 there is only one way to do things.

[SeeMoreDigital]

Thanks Squid.... :D

[Brother John]

Thanks Squid.... :D
Thx from me as well. Your answer saved me some boring testing. ;)

Btw: I've updated the enraw mirror on my site (http://brother-john.net) to the current version.

[bond]

Umm, however mp4creator does it? That's where the code is coming from.is xvid_encraw now supposed to support .mp4 output? if yes how?

[squid_80]

is xvid_encraw now supposed to support .mp4 output? if yes how?
Not yet. "That's where the code is coming from," means it's on its way but not quite here yet. But it would be just like me to release a new version and not say anything. :)

[SeeMoreDigital]

Not yet. "That's where the code is coming from," means it's on its way but not quite here yet. But it would be just like me to release a new version and not say anything. :)Great... I'm looking forward to an .MP4 output support :)

[bond]

:thanks:

[MatMaul]

I think I found a very minimal bug :
C:\Documents and Settings\MatMaul\Bureau\xvid_encraw>xvid_encraw -i "D:\lotr1\vi
deo.avs" -type 2 -pass2 -bitrate 2863 -max_bframes 2 -bquant_ratio 162 -bquant_o
ffset 0 -zones 0,w,1,O -quality 6 -vhqmode 4 -bvhq -qpel -qmatrix "SixOfNine.cqm
" -nopacked -imin 2 -imax 5 -pmin 2 -pmax 5 -bmin 2 -bmax 5 -chigh 10 -clow 3 -m
ax_key_interval 250 -progress -mkv "D:\lotr1\video\lotr1.mkv"
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
Input colorspace is YV12
xvidcore build version: xvid-1.2.0-dev
Bitstream version: 1.2.-127
Detected CPU flags: ASM MMX MMXEXT SSE SSE2 TSC
Detected 1 cpus, using 1 threads.
288651 frames(100%) encoded, 6.68 fps, Average Bitrate = -113kbps
Tot: enctime(ms) =43188075.00, length(bytes) = -163231213
Avg: enctime(ms) = 149.62, fps = 6.68, length(bytes) = -565
I frames: 3977 frames, size = 53748/213756687, quants = 2 / 2.33 / 3
P frames: 119693 frames, size = -12651/-1514325311, quants = 2 / 2.66 / 4
B frames: 164825 frames, size = 6900/1137336171, quants = 3 / 3.64 / 5
N frames: 155 frames, size = 8/ 1240

The avg bitrate reported by the log is negative.
The real bitrate of this encoded video is good, so it's not very important.

[squid_80]

Yes, the variables that hold the sizes are signed int32s so when the output grows larger than 2gb they wrap around to negative values. This throws out all the average calculations. The sensible thing to do would probably be for me to change them to uint64s.
The bitrate should be pretty spot on though, I'm guessing the final size was around 3940M? And the source has a framerate of 25fps?

[MatMaul]

yes the real final bitrate is good.

[berrinam]

Hi there squid_80,

Would it be possible to make the messaging system in xvid_encraw more informative? I have something in mind like how x264.exe does it:
x264 [error/warning/info]: message
where error means a fatal error, warning means a non-fatal error, and info means info. Then it would mean that a parser (like MeGUI) could tell whether the encode error'd out or not.

It could also be a good way of choosing the verbosity of messages: the user can choose which level of message they want in the commandline, and xvid_encraw.exe then only prints messages of that level or of more importance.

[squid_80]

Certainly. I've been meaning to do that for a while, the only reason I haven't changed any of the output messages was in case some programs used the original encraw and modifying the messages would break compatability. I'll go poke around in x264 and figure out what constitutes a fatal/non-fatal/info message so things match up nicely.

[acrespo]

I don't know if you saw the MEGUI bug report thread but I will post here again as I think that is a xvid_encraw problem. I receive this error when try to encode a avisynth script:

comand line:
-i "C:\encodes\ygh1.AVS" -pass1 "C:\encodes\ygh1.stats" -bitrate 606 -kboost 100 -overhead 0 -turbo -vhqmode 1 -closed_gop -imax 31 -max_bframes 2 -bmax 31 -threads 1

Result:
Trying to retrieve width and height from input header
Wrong Framerate 0.000000
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

I don't know what's wrong, because I can open this script in virtualdubmod and media player classic.

[squid_80]

I saw it, but I really need to know exactly what is in the avisynth script.

[acrespo]

Here is the script:


SetMemoryMax(64)
LoadPlugin("c:\encodes\filtros\EEDI2.dll")
LoadPlugin("c:\encodes\filtros\tdeint.dll")
LoadPlugin("c:\encodes\filtros\dgdecode.dll")
LoadPlugin("c:\encodes\filtros\tivtc.dll")
LoadPlugin("c:\encodes\filtros\dup.dll")
LoadPlugin("c:\encodes\filtros\RemoveDirt10SSE2.dll")
LoadPlugin("c:\encodes\filtros\RemoveGrain10SSE2.dll")
LoadPlugin("c:\encodes\filtros\RepairSSE2.dll")
LoadPlugin("c:\encodes\filtros\masktools-v2.0a28.dll")
LoadPlugin("c:\encodes\filtros\gradfun2db.dll")
LoadPlugin("c:\encodes\filtros\frfun7.dll")

import("c:\encodes\filtros\RemoveDust.avs")
import("c:\encodes\filtros\FastLineDarken_1.3.avs")

MPEG2Source("c:\encodes\2006-04-01\ygh2.d2v",cpu=6)

AssumeTFF()
Interp = SeparateFields().EEDI2(field=-2)
Deinted = TDeint(mode=2,edeint=Interp)
TFM(PP=7,Clip2=Deinted)
TDecimate(mode=1)

ColorYUV(levels="PC->TV")
Crop(8,8,-8,-8,align=true)
Spline16Resize(320,240)
RemoveDirt()
frfun7()

FastLineDarken2(thinning=0)
dup(threshold=1,blend=true,maxcopies=8)
gradfun2db()

str = "WAS Hub - dchub://wasbr.com - " + Time("%d") + "/" + Time("%m") + "/" + Time("%Y")
Subtitle (str,15,10,0,240,"Courier New", 14)
Subtitle ("WAS",258,202,0,300000,"Courier New",20)

[berrinam]

squid: Basically, the fatal errors in x264 are the ones in which there is no way to continue, like the video is stuffed or the log file is missing, etc. The warnings tend to be settings that are wrong for some reason, but can be fixed by x264, so the warning just tells you that it isn't encoding quite with the settings you asked for.

[squid_80]

acrespo: I can't find anything wrong. Can you do me a favour, add info() to the bottom of that script, open it in virtualdub(mod) and tell me what it says on the Frames per second line? Specifically what the numbers in the brackets are.

[acrespo]

Here is the exact information extracted from info() filter in avisynth:

Frame: 0
ColorSpace: YV12
Width: 320 pixels, Height: 240 pixels.
Frames per second: 23.9760
FieldBased (Separated) Video: NO
Parity: Assumed Top Field First
Video Pitch: 640 bytes
Has Audio: NO
Audio Channels: 0
Sample Type: NONE
Samples Per Second: 0
CPU detected: x87 MMX ISSE SSE SSE2 SSE3

[squid_80]

The only thing I can suggest is updating avisynth and dgindex/dgdecode; I think this might be related to the way avisynth used to give some framerates large numerator/denominator values.

[bond]

The only thing I can suggest is updating avisynth and dgindex/dgdecode; I think this might be related to the way avisynth used to give some framerates large numerator/denominator values.avisynth 2.5.7alpha doesnt do this anymore

[squid_80]

True. It also displays dwrate and dwscale via info(), which is what I really need because I can't reproduce this error. Closest I can get is a crash in xvidcore when dwscale is >2^31.

[sysKin]

I want to put it in CVS.
However, I was crazy enough to try to compile it first >_< and, well, it doesn't because it's trying to use c++ syntax in c file.

What do I do about IID_IAvisynthClipInfo to make it C? :)

[squid_80]

I just compile it as c++ :D
I did try to convert that code to C, but couldn't get it to work. However I'll try again and see how it goes.

[squid_80]

Looks like this works, but I dunno if it's completely correct. First define the interface inside the #ifdef XVID_AVI_INPUT block:

#ifdef XVID_AVI_INPUT
/* Avisynth related */
const GUID IID_IAvisynthClipInfo // {E6D6B708-124D-11D4-86F3-DB80AFD98778}
= {0xe6d6b708, 0x124d, 0x11d4, {0x86, 0xf3, 0xdb, 0x80, 0xaf, 0xd9, 0x87, 0x78}};

#undef INTERFACE
#define INTERFACE IAvisynthClipInfo
DECLARE_INTERFACE_ (INTERFACE, IUnknown)
{
STDMETHOD (QueryInterface) (THIS_ REFIID, void **) PURE;
STDMETHOD_ (ULONG, AddRef) (THIS) PURE;
STDMETHOD_ (ULONG, Release) (THIS) PURE;
STDMETHOD_ (INT, GetError) (THIS_ char**) PURE;
STDMETHOD_ (LONG, GetParity) (THIS_ INT) PURE;
STDMETHOD_ (LONG, IsFieldBased) (THIS) PURE;
};
#endif

(Bool wouldn't work, so I used LONG for the return type of GetParity and IsFieldBased. Shouldn't be relevant, they're not used here.)
Then change the avisynth checking stuff to match the new interface:

if (FAILED(avi_file->lpVtbl->QueryInterface(avi_file, &IID_IAvisynthClipInfo, (void **)&pAvisynthClipInfo)))
pAvisynthClipInfo = NULL;
else {
const char *s;

if (pAvisynthClipInfo->lpVtbl->GetError(pAvisynthClipInfo, &s)) {
fprintf(stderr, "xvid [error]: Avisynth open failure:\n%s\n", s);
pAvisynthClipInfo->lpVtbl->Release(pAvisynthClipInfo);
AVIFileRelease(avi_file);
AVIFileExit();
return (-1);
}
}


if (pAvisynthClipInfo != NULL)
pAvisynthClipInfo->lpVtbl->Release(pAvisynthClipInfo);

Then there were a few spots where I've declared and assigned variables in the middle of code which plain C doesn't like. Easy enough, just move the declarations up.

[sysKin]

Looks like this works, but I dunno if it's completely correct.
Well, it didn't work :/

I removed the avisynth error catching and commited the file.

I officially declare this is the ugliest piece of code in XviD's CVS. It's technically incorrect (static variables), hackish to the bone, and probably not cross-platform at all.

It needs an urgent rewrite. If doing it properly requires xvid interface changes (wrt packed bitstream and userdata) then so be it. Better that than hackish removal of userdata from encoded frames.

Volunteers?....

[shpitz]

i'm using latest build and for some reason it always says detected 0 cpus and using 0 threads.

if i force -threads 2 or 4 it will still be using only 1 cpu as my cpu usage is stuck on 25%.

i have a dual xeon 3ghz with HT enabled.

Detected 0 cpus, using 4 threads.
701 frames( 31%) encoded, 7.19 fps, Average Bitrate = 14231kbps

that's 1st pass, 2nd pass goes down to 3fps.

i'm using the batch file made by henry_hk in the settings thread by teegedeck.

any ideas about what i'm doing wrong?

[SeeMoreDigital]

any ideas about what i'm doing wrong?It would be handy if you could confirm the resolutions you are encoding to!!!

[shpitz]

oh, sorry, i was encoding a 720p clip to a 720p output resolution.

what was suspicious to me was that encraw reported 'xvidcore build version: xvid-1.1.0'

so i figured it must be using an old version of xvid that is not multithreaded.

i installed a newer build and now it is working great, getting 2x the speed, and # of cpus is detected properly now.

so that's all it was, using an older version of xvidcore.dll .

i will do another test with 2 threads, right now i'm doing with 4 threads.

[mod]

Hi all.
I'm not able to limit the number of frames encoded. Arguments:

xvid_encraw -i pippo.avs -type 2 -frames 200 -custom_par 80 81 +chroma_opt -quality 6 -max_bframes 1 -framerate 25 -bitrate 2000 -pass1 pippo.log -iquants 2 31 -pquants 2 31 -bquants 2 31 +trellis -lumimasking -vhq 4

it simply doesn't stop at frame 200. Am I making some error?
Thanks!

[buzzqw]

Hi mod !

first of all: you are using the old sintax...
second: latest build of xvid_encraw doesn't have -frame parameter... just -start for specifying the START frames...

i can suggest to use zones... to lower quant

BHH

[mod]

Really thanks for the info.
I'm using the version linked a 1st page, and the syntax is the one I got with

xvid_encraw -help ..

EDIT: I found this (http://forum.doom9.org/showthread.php?p=800912#post800912) one and supports -frames. Is this the latest?

I'm really sorry if I missed some link.

[kurt]

latest is this (AFAIK): http://forum.doom9.org/showthread.php?p=822253#post822253

[buzzqw]

please mod ... don't complain !

i don't know older built fupporting this features... but... may i suggest to use the TRIM feature in your avs script ?

BHH

[SeeMoreDigital]

Excuse for my ignorance but I have to ask....

Is one of the goals of XviD EncRAW to create an non VfW, ie: CLI version of XviD?


Cheers

[imcold]

yes /I believe :D

[buzzqw]

yes ! for sure... but what's the matter ? :confused:

BHH

[john27]

small bug:
#include <ctype.h> is missing //toupper(c)

[JoeBG]

Excuse for my ignorance but I have to ask....

Is one of the goals of XviD EncRAW to create an non VfW, ie: CLI version of XviD?


Cheers

Yes...and it works great. I use it for all my xvid encodes. You can create a proper raw or *.mkv stream without packet bitstreams.

[shpitz]

Yes...and it works great. I use it for all my xvid encodes. You can create a proper raw or *.mkv stream without packet bitstreams.

Joe, and your encodes have no issues with fast forwarding (seeking)? if so what do you use to mux the encoded video with the audio?

[SeeMoreDigital]

Yes...and it works great. I use it for all my xvid encodes. You can create a proper raw or *.mkv stream without packet bitstreams.That's encouraging news :)

I wonder if Stax can code it into StaxRip before he abandons it altogether?


Cheers

[foxyshadis]

Is it possible to split this thread from page 10 or so, or better have a mod edit the first post to include some information on getting current versions of encraw?

[mod]

I made a little GUI for xvid_encraw, maybe someone finds it useful.

here (http://forum.doom9.org/showthread.php?t=112808)

I use it for SAP, so not really a lot of options (ie only avi output), but if somene wants I can add what needed, no problem :D

In the next days I'll try to write something in english, but it's really easy, doesn't need any real explanation..

[JoeBG]

@ shpitz

Just tested it only for you :) (I never use fast forwarding) and it works great. I always use ZoomPlayer with HaaliSplitter and ffdshow for xvid and aac. I mux the video with mp4box.

[shpitz]

@ shpitz

Just tested it only for you :) (I never use fast forwarding) and it works great. I always use ZoomPlayer with HaaliSplitter and ffdshow for xvid and aac. I mux the video with mp4box.

haha, thanks m8, i guess i'll give it a shot. i'm using MPC myself.

[MatMaul]

I have a problem with the mkv output of xvid_encraw (the version of megui).
I can play the mkv in mpc but it's horrible in vlc (macroblock).

If I use the raw output and I remux it in mp4 with mp4box I have an error : the file is playable but not seekable.

If I use the xvid_encraw of celtic druid, no problem with raw output (good mux in mp4), but it doesn't support mkv output.

I tried to compile my own version of xvid_encraw with mkv output but I can't find the needed file "matroska.cpp" in the source of matroska or mkvtoolnix : I just find a file "r_matroska.cpp" in mkvtoolnix source. Where I can find the file please?

EDIT : all the squid80's modifications are committed to the cvs of xvid project or not ?
EDIT2 : I have found matroska.cpp, I tried to compile my own build and I report the results

[bond]

If I use the raw output and I remux it in mp4 with mp4box I have an error : the file is playable but not seekable.what player? try another player

[MatMaul]

sorry you are right, it's my version of mp4box the problem.

but the mkv problem exists and I don't think it's related to the player (vlc), because raw->mp4->mkv works good.

I can't compile a version with mkv support, I don't now how to do that.

[holzi]

I don't know if the problem was already posted coudn't find anything.
I get a strange result when encoding in 2 pass mode.
It's done by megui:
Here the log:


encoder commandline:
-i "O:\DVD-out\Temp\fight.avs" -pass1 "O:\DVD-out\Temp\fight.stats" -bitrate 1076 -kboost 100 -overhead 0 -max_key_interval 250 -vhqmode 4 -closed_gop -threads 1
successfully started encoding
Processing ended at 01:33:44
----------------------------------------------------------------------------------------------------------

Log for job job1

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Tot: enctime(ms) =2545641.00, length(bytes) = 821595521
Avg: enctime(ms) = 12.72, fps = 78.62, length(bytes) = 4104
I frames: 2508 frames, size = 17424/43701787, quants = 2 / 2.00 / 2
P frames: 73684 frames, size = 8340/614550880, quants = 2 / 2.00 / 2
B frames: 123908 frames, size = 1318/163342488, quants = 4 / 4.00 / 4
N frames: 46 frames, size = 7/ 366
Trying to retrieve width and height from input header
Input colorspace is YV12
xvidcore build version: xvid-1.2.0-dev
Bitstream version: 1.2.-127
Detected CPU flags: ASM MMX MMXEXT SSE 3DNOW 3DNOWEXT TSC
Detected 1 cpus, using 1 threads.

----------------------------------------------------------------------------------------------------------
Job completed successfully and deletion of intermediate files is activated
Starting job job2 at 01:33:44
encoder commandline:
-i "O:\DVD-out\Temp\fight.avs" -pass2 "O:\DVD-out\Temp\fight.stats" -bitrate 1076 -kboost 100 -overhead 0 -max_key_interval 250 -vhqmode 4 -closed_gop -threads 1 -avi "O:\DVD-out\Temp\fight.avi"
successfully started encoding
Processing ended at 04:42:15
----------------------------------------------------------------------------------------------------------

Log for job job2

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

Trying to retrieve width and height from input header
Input colorspace is YV12
xvidcore build version: xvid-1.2.0-dev
Bitstream version: 1.2.-127
Detected CPU flags: ASM MMX MMXEXT SSE 3DNOW 3DNOWEXT TSC
Detected 1 cpus, using 1 threads.

Tot: enctime(ms) =5988780.00, length(bytes) = 770489159
Avg: enctime(ms) = 29.92, fps = 33.42, length(bytes) = 3849
I frames: 2508 frames, size = 17244/43249140, quants = 2 / 2.00 / 2
P frames: 73684 frames, size = 7516/553839680, quants = 2 / 2.00 / 2
B frames: 123953 frames, size = 1398/173400333, quants = 4 / 4.00 / 4
N frames: 1 frames, size = 6/ 6
desired video bitrate of this job: 1076 kbit/s - obtained video bitrate (approximate): 774 kbit/s


Why doesn't encraw use the hole bitrate of 1076?

My avs script I used:

DGDecode_mpeg2source("O:\DVD-out\Temp\VTS_01_1.d2v",info=3)
ColorMatrix(hints=true)
#Not doing anything because the source is progressive
crop( 0, 74, 0, -76)

LanczosResize(576,240) # Lanczos (Sharp)
Undot() # Minimal Noise

[foxyshadis]

I frames: 2508 frames, size = 17244/43249140, quants = 2 / 2.00 / 2
P frames: 73684 frames, size = 7516/553839680, quants = 2 / 2.00 / 2
B frames: 123953 frames, size = 1398/173400333, quants = 4 / 4.00 / 4


You're already maxing out the codec quality, anything more is just throwing bits away. Assuming you'd rather increase quality than simply add a hundred megs of padding, at the very least use MPEG matrix, or check out the xvid presets thread and load the presets into megui. Use one of the higher ones.

[MatMaul]

lol I ever have problem with the output of xvid_encraw.

If I use the direct mkv output, problem to play the file in vlc (big macroblock on the video).

If I do m4v->mp4->mkv, no problem with vlc BUT problem with the ordered chapters function in mkv,an example : I want to display the first 30s of a video, then I want to see the last 30s of the video (the video is longer than 1mn) and I do that with ordered chapters (=>the timeline have a length of 1mn and the real video have a length > 1mn). No problem with the first 30s, but when the player seek to the last 30s of the video, big freeze of the video.
I have test with some couple of mp4box/mkvmerge, same problem.
And I don't have this problem with mkv direct output.

any idea to have a correct display in vlc and ordered chapters function ?

Do you think anyone can modify mkv output to have a good picture in vlc, or it's a vlc problem ?

Thanks !

[Adub]

Yo, Squidy, whats with the new version that you updated today?

bug fixes, enhancements, what?

[squid_80]

Neater error reporting and dynamic linking with xvidcore.dll.

[MatMaul]

@squid : any news for my mkv problem in the new version ?

[squid_80]

Is your problem mainly that the mkv output from xvid_encraw is unplayable in vlc? What if you use mkvmerge to remux it to a new file?

[MatMaul]

the mkv output is playable in vlc but with big and horrible macroblocks. Same problem if I remux the mkv in a new mkv with mkvmerge.

If I do m4v->mp4->mkv, I don't have this problem but an other problem, but I think this new problem is related to mkvmerge (I have report the problem in the mkvtoolnix thread).

[squid_80]

Are you using GMC on your clips?

[MatMaul]

no.
I use the preset 58hq :
pass1 : xvid_encraw.exe -i video.avs -type 2 -pass1 xvid.stats -max_key_interval 250 -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -vhqmode 1 -qtype 1 -qmatrix Didees-SixOfNine.cqm -nopacked -quality 5 -nochromame -turbo -zones 0,q,3,O
pass2 : xvid_encraw.exe -i video.avs -type 2 -mkv video.mkv -pass2 xvid.stats -bitrate 2500 -max_key_interval 250 -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -vhqmode 4 -bvhq -qpel -qtype 1 -qmatrix Didees-SixOfNine.cqm -nopacked -imin 2 -imax 4 -bmin 2 -bmax 5 -pmin 2 -pmax 5 -quality 6 -chigh 10 -clow 3 -ostrength 0 -zones 0,w,1,O

[squid_80]

VLC seems to be buggy in the way it handles native mpeg4 in mkv. Any advanced options such as qpel, gmc, mpeg quantization etc. seem to produce artifacts. I've checked by converting other avi files (not created with xvid_encraw) to mkv with mkvmerge using --engage native_mpeg4 and they show the same errors when played in VLC.

[MatMaul]

But do you know why it works good if I do m4v->mp4->mkv ?
I have report this problem in the videolan forum and the videolan dev list with a sample but no answers.

[squid_80]

When you convert from mp4 to mkv do you use --engage native_mpeg4?

[MatMaul]

When you convert from mp4 to mkv do you use --engage native_mpeg4?

No I don't use --engage native_mpeg4 but mp4 uses raw mpeg4 data, without any hacks, so I think when I remux a mp4 to mkv, it uses native mpeg4 data.
I test --engage native_mpeg4 tomorrow to be sure.

[squid_80]

When I tested mp4->mkv using mkvmerge, it gave me a vfw mkv file (with the frames in the wrong order around I frames - I believe bond reported this long ago). When I used --engage native_mpeg4, it gave me a native mkv file but there was something strange; the VOL information was appended to each P frame preceeding an I frame. In a native mpeg4 matroska file that information should only occur once in the CodecPrivate section. If I had to guess, I would say VLC isn't reading this at all and is relying on the information being present in the stream which is the wrong way to do it.

[MatMaul]

OK after test you are right, mkvmerge use vfw mode if I do m4v->mp4->mkv, so the problem is vlc with native mpeg4 stream.

Can you post this technical details in this thread (http://forum.videolan.org/viewtopic.php?t=21189) on videolan forum please ?

EDIT : an other question, how do you compile xvid_encraw with mkv support ?

I have all the dependencies but I don't know how compile a c file with c++ file included in it.

[kypec]

...and dynamic linking with xvidcore.dll.

Sorry, but what exactly does it mean to me as a plain user of xvid_encraw?
Do I still need to install some XviD build (like 1.2) separately?
@squid_80
BTW, can you confirm that this download link of dunstan's (http://okejl.dk/dunstan/xvid_encraw.zip) contains the most up-to-date release of your builds?
I'm asking this because your FTP server is not working again :mad:

P.S. thanks for your great effort put into this CLI of XviD
I just like to write my scripts/batches once and use them million times:cool:

[squid_80]

EDIT : an other question, how do you compile xvid_encraw with mkv support ?

I have all the dependencies but I don't know how compile a c file with c++ file included in it.
Compile it as a c++ file. :) You'll also need to link it with libmatroska and libebml.

Sorry, but what exactly does it mean to me as a plain user of xvid_encraw?
Do I still need to install some XviD build (like 1.2) separately?Yes, it just means it should work properly with any version of xvidcore.dll. You still need it installed from somewhere.
BTW, can you confirm that this download link of dunstan's contains the most up-to-date release of your builds?
Yes.

[MatMaul]

thanks !

[bond]

MatMaul: native asp support in videolan is buggy

vlc is not able to play correct native asp files created by xvidencraw or haalis muxer (yeah, xvidencraw creates correct native asp files!!!).
it also doesnt play native asp files created by mkvmerge (via .mp4 source (avi/raw source doesnt work))

in all cases vlc shows artefacts. its a known problem i have told robux4 a long time ago already, but which was never fixed

[mod]

I'm trying to make a change but I can't find (or maybe I'm blind ok..) the source files needed by matroska.h, could anyone please give me a link to an updated page with all source files?
Thanks!

[squid_80]

Get libmatroska and libebml from matroska.org.

[mod]

Thanks A LOT :)

[squid_80]

MatMaul: native asp support in videolan is buggy

vlc is not able to play correct native asp files created by xvidencraw or haalis muxer (yeah, xvidencraw creates correct native asp files!!!).
it also doesnt play native asp files created by mkvmerge (via .mp4 source (avi/raw source doesnt work))

in all cases vlc shows artefacts. its a known problem i have told robux4 a long time ago already, but which was never fixed
Despite certain people in the videolan forums declaring asp-in-mkv "proprietary" and "non-standard", VLC has been patched and when I downloaded the latest nightly snapshot it played my test files without problems. So much for having to petition all the major players and codec vendors blah blah blah...

[bond]

great :)

[Brother Darrell]

How about a quick post on proper commandline syntax using Xvid_encraw? So far, the only thing I have found are snippets I have copied from Bond or Squid or anyone else who seemed to be getting ANY result different from mine.
I spent a couple of hrs trying to determine why my command line would only do a 1 pass 4quant encode with b-frames. (No b frames in the stats file). As it turns out, it's because I was using "-type 2" before the avs file instead of after it. So I think a small guide on the proper syntax for the command line would be great.

Edit: Duh!!...If I had only read past 2 more threads, I would have found it...You would have thought I could have narrowed it down a great deal using the search funtion for the forums but nooo, As much as I love this site, I think the search function for the forums Is WAY too limited..(Speaking of syntax).

[MatMaul]

Despite certain people in the videolan forums declaring asp-in-mkv "proprietary" and "non-standard", VLC has been patched and when I downloaded the latest nightly snapshot it played my test files without problems. So much for having to petition all the major players and codec vendors blah blah blah...

yes the problem of blocky images is solved, but I actually can't seek in the file, and the length of the video isn't recognized.

[squid_80]

As it turns out, it's because I was using "-type 2" before the avs file instead of after it.
Can you elaborate a bit? Like what was the failing command line and what you ended up using to make it work? There is a *very minor* bug with -type but I don't know how it would cause a lack of b frames.
yes the problem of blocky images is solved, but I actually can't seek in the file, and the length of the video isn't recognized.
Remux the file. I never fixed encraw to write cues correctly which probably explains the lack of seeking, I don't know why the duration is missing as well.

[MatMaul]

ok thanks, no problem after remux.

[bond]

are cues mandatory for seeking?

[squid_80]

I don't think so, they just help.

[bond]

I don't think so, they just help.so vlc is not yet perfect?

[Brother Darrell]

Can you elaborate a bit? Like what was the failing command line and what you ended up using to make it work? There is a *very minor* bug with -type but I don't know how it would cause a lack of b frames.

Sorry. I've tried to reproduce the problem. Now I can't. Damn thing won't break now that I want it to. I can tell you I think it was an earlier version of encraw I was using.(From February I think). Once I got the syntax correct & installed a newer version, I was good to go. Sorry man.. I will continue to try & break it though. :D

[Brother Darrell]

Uh oh. My file came out great but with absolutely no header info. I figured out how to use mencoder for the fourcc and bitrate issue (actually, mencoder took care of the bitrate issue). But it says my resolution is 0x0. How do I tell it W/H of the video? I thought encraw did it but obviously not. this is my command line:

mencoder.exe "movie1.m4v" -ovc copy -mc 0 -noskip -ofps 23.976 -ffourcc XVID -o "test.avi"

I have tried aspect=16/9 I have tried -ow 720 -oh 432. I have also tried verbal abuse & bribery. What's the key?

[squid_80]

Are you trying to use mencoder to mux an elementary stream into avi? Why not just output .avi directly from encraw?

[Brother Darrell]

Tried that as well. I have played with it enough , so I think I have down the syntax. MediaInfo sees everything I have added (info-wise) except the width & hieght. Avi is seen as 0x0 as well. I can't imagine this being a bug. No one else is complaining. So it has got to be my command-line. Recoding with the following commandline using encraw:

"xvid_encraw.exe " -i "D:\Blah.avs" -quality 6 -vhqmode 1 -imin 2 -imax 31 -pmin 2 -pmax 5 -overhead 650 -framerate 23.976 -pass2 "D:\blah.pass" -qmatrix "D:\hvc matrix.txt" -w 720 -h 432 -par 5 -o "D:\movie1.m4v" -size 2073344 -progress 5

[squid_80]

So if you use -avi "d:\movie1.avi" instead of -o "d:\movie1.m4v", the avi file has dimensions of 0x0?

[mod]

I think the line must be:

"xvid_encraw.exe " -i "D:\Blah.avs" -type 2 -quality 6 -vhqmode 1 -imin 2 -imax 31 -pmin 2 -pmax 5 -overhead 650 -framerate 23.976 -pass2 "D:\blah.pass" -qmatrix "D:\hvc matrix.txt" -w 720 -h 432 -par 5 -o "D:\movie1.m4v" -size 2073344 -progress 5[/QUOTE]

[squid_80]

-type 2 is implied if the input file has an .avs or .avi extension. Width and height should be retrieved automatically, but I think if you manually set them smaller than the actual values the output will be cropped (don't set them bigger, I can't remember what will happen but it certainly won't enlarge the image to fit).

[mod]

I prefer to set -type 2 ("I sleep better" ^^).
Hmm.. In my experience it has never failed getting the correct dimensions of the video, so I don't see any reason to use it, unless, as you just wrote, to perform a crop (if this is what happens btw). Maybe it's a bit slower (dunno) but I wouldn't mind of that.

[squid_80]

Mainly -w and -h are for setting width and height when the input is raw yv12 (or stdin).

[mod]

Ok, didn't know (I never used it with that input). Thanks for the info.

[Brother Darrell]

AS far as an AVI file, that is correct, at least on my (very short) test clip. I am trying again (m4v) with the command line above. It may very well be I'm using the wrong switches. So I'm looking as always for proper commandline syntax. Is there anything else that would build the header for me, other than mencoder?
And isn't the output I'm producing a raw file? I thought it was as my very 1st attempt gave me NOTHING for header information(mediainfo).

Edit: Well, I got a file which so far seems to be correct in all aspects. MediaInfo sees the dimensions & par correctly. I haven't run it thru mencoder yet but I suspect it will see the info as well. I sometimes tend to get impatient & stop the encode simply to view the results. This may be my issue all along. Mencoder & MediaInfo won't see it for a very good reason. I broke the file.
(Plausible..yes?)

[squid_80]

If you forcibly close encraw before it is finished then the avi file will be broken.

[henryho_hk]

The xvid_encraw compile at http://celticdruid.no-ip.com/xvid/ seems to be the most recent (Jul 06) but the result are always changed to 25fps.

[squid_80]

You're right; in my builds I made the default framerate 0 so it would report an error instead of assuming 25fps, since I found this to be undesirable behaviour. The code in xvid's CVS has reverted to a default of 25fps.

[henryho_hk]

squid_80, shouldn't it detect the framerate from the source?

[squid_80]

It should, but making 25fps the default breaks it. The intended mode of operation (as I coded it) is like this:

Set framerate=0 (default)
Check command line for -framerate parameter. Adjust framerate to given value if found.
If framerate==0, set it to the framerate of the input source if possible (not possible if input is stdin or pgm).
If framerate==0, show an error and abort.

[henryho_hk]

Yeah, that version crashes if I put -framerate 0

[Brother Darrell]

So far I have managed to create several -avi files with no issue. Still can't get the W/H to be seen in mencoder if original file is a raw stream. BUT!, moving on...Avi is wonderful. I LOVE the fact I can do multiple 1st passes WHILE doing a second pass. :D ..
I am an n-pass junkie, sad but true.
A question though. I don't see an option for B frame sensitivity. Is there such an option & I am just missing it?

[kurt]

it's part of the zones, eg
-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG
--> first zone got a bframe sens. of 25, 2nd 0, 3rd 25 again...

[Brother Darrell]

-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG

I don't know this switch.
so...-zones
1=the zone or starting frame
w(weight) ,(ok to put a delimiter here?).05,250 (I noticed you used an 'oh', instead of 'zero' but it is zero, correct?)
And to use it globally, I simply change the weight to 1 and leave it as is. -zq I guess will work as well as -zw.
Thanks! I'll give it a try.
Oh. 1 more thing...Why is -stats suppressed (if being written to a log file) if the -progress switch is used?
Like this: -stats>> "stats.log" -progress 10

Same result regardless of where the -progress switch is located. ie: before the -stats switch or after the -stats switch..doesn't matter. It will still not output. Take the -progress switch out though & my log file is produced.

[squid_80]

The -zones parameter works like described here: http://forum.doom9.org/showthread.php?p=702759#post702759
except there's no parameter for the end frame; a zone runs until the next zone or the end of the stream. In kurt's example it is in fact the letter "O", for chroma optimizer.
As for the stats/progress thing... -progress turns off all frame-by-frame output, otherwise there's not much point in having it (unless stdout is redirected, like you're doing). If you really want the progress update and stats to happen at the same time I guess I could add an optional filename parameter for -stats, to specify the file to output to.

[Brother Darrell]

The only reason I was wanting a stats output was because of the PSNR values. It's just something for me to play with, not something essential.
If it's not alot of trouble to do so then YES! I would LOVE to have that feature incorporated! I would buy you many cold beers for the effort! That is, if you are open to such blatant bribery.

:thanks:
To you & Kurt for the info about zones. The only switches I know are the ones from -help.
Are there any other switches I can't get from the -help option?

[Brother Darrell]

I found them listed in one place.
For anyone else is having an issue finding the parameters not listed with -help..
look here->http://forum.doom9.org/showthread.php?p=784465#post784465
It includes syntax and options.

[Brother Darrell]

Is there an alternate way to set chapter points on the 1st or 2nd pass without using the -zones parameter to force an I-frame? Or editing the pass file directly?

Lets say I have 40 chapter points. That's ALOT of typing!
So I was hoping there was an easier way, such as feeding (porting?) a text file with the frames needed, into the command line, forcing an I-frame where I think I need it.
I realize MkvToolnix can mux an xml or ogg file into the finished product..but what if the chapter point falls on a b-frame? Heck...what if I just want to split the stream at the chapter point(s)?
For now, I'll either do it with the -zones or directly edit my pass file.
I can't help but think there must be (or should be) an easier/proper way. Too much to hope for?

[Sharktooth]

The chapter start is always set at the beginning of a new scene and the codec will almost likely detect the scene change and place an I-Frame.
MeGUI supports chapters creation and muxing, so it should be ok as soon as this version of xvid will be supported (not until the patches go official...)

[Brother Darrell]

Does anyone recognise this specifically?
.0 0. ax + + 1 - bn

I was running a batch file using Encraw with the following command line:

xvid_encraw -i "Versus.avs" -pass2 "5th.pass" -max_key_interval 0 -qmatrix matrix.txt -framerate 23.976 -max_bframes 120 -bquant_offset 000 -bquant_ratio 162 -imin 2 -imax 3 -pmin 2 -pmax 3 -bmin 3 -bmax 5 -nopacked -quality 6 -vhqmode 4 -avi "VsBframes2.avi" -size 1706540 -pass1 "6th.pass" -stats>>"6thpass.stats" -progress 10.

I started running it in a command window when I left home this morning. When I got home, the file was finished, but the above er..characters appeared immediately afterwards, as though it was part of my original command line, with the exception of course of XP not recognising anything as a batch process or command...The last part however, -bn, did seem to process, at least XP didn't say "not recognized as an internal or external command,operable program or batch file".
Any clues? Did I break something?

Edit: sorry, the matrix file I used was HvsBest.txt under the name Matrix.txt

[squid_80]

If the command window has focus while processing and you type something, it will appear at the c:\> prompt when processing finishes. Is that what happened?

[Brother Darrell]

That's entirely possible. I didn't think about that. So something I was running at the time may have simply dumped garbage. Maybe something that happens all the time but would normally not be seen. That sounds reasonable. I was concerned because it looked a little like code. As far as me typing something while the command window has focus, It certainly would not have been that.

[henryho_hk]

was that something in your batch file?

[Brother Darrell]

Not explicitly. I don't even recognize the -bn as a parameter I use for anything. This all appeared after the batch completed.
I would have blown it off as a catwalk over the keyboard, except my keyboard was on top of my monitor. My cat is FAR too old to jump 5 feet onto the top of my monitor, land on the keyboard, type something for the ax register and jump off without knocking down the keyboard.
(Ok, maybe she can...but there is still the fact she never passed her computer basics class...waste of money, that).
Hell.. I don't know what it was. But no harm seems to have come to my file or computer. So I ain't gonna sweat it. I just wanted to make sure it didn't have to do with the batch file doing something Xvid_encraw didn't like.

[henryho_hk]

Is there an alternate way to set chapter points on the 1st or 2nd pass without using the -zones parameter to force an I-frame?

I have made a batch file implementing Teegedeck's quality presets as well as zone input thru a plain text file. Please refer to http://forum.doom9.org/showthread.php?p=874332#post874332
BTW, there is basically no documentation :p :p :p .

[henryho_hk]

squid_80, I found that zone quant. settings seem to be reseting the b-frame max-consec., ratio and offset settings. I need to put the zone parameters before the b-frame parameters so as to use something like 1/1.62/0. Is it intended?

[Champs]

Hi, first of all thank you squid_80 for taking over and improving this cli program, I really like it a lot.

I've spent a few days going through every single page of this thread and found several posts by squid_80 and others about encoded outputs set to 25fps without setting the fps explicitly regardless of the what the input avisynth script is serving. I read them but I can't really grasp what is really implied.

My question is, is it simply not possible to change the source code so that the default output framerate is the same as what the avs is serving, or will such change be implemented in the future? I guess I have no problem with preparing a set of batch scripts, each with different fps set explicitly and also sorting avs scripts into different folders by framerates, but I just feel xvid_encraw.exe should be able to do what x264.exe can do.
(I know next to nothing about programming so please forgive me if I'm saying something stupid.)

Apart from this I have nothing to say and want to express my deepest gratitude!

[squid_80]

squid_80, I found that zone quant. settings seem to be reseting the b-frame max-consec., ratio and offset settings. I need to put the zone parameters before the b-frame parameters so as to use something like 1/1.62/0. Is it intended?
Not sure if it's encraw related or xvidcore, can you give me a complete example command line and brief description of what output you expect?

I've spent a few days going through every single page of this thread and found several posts by squid_80 and others about encoded outputs set to 25fps without setting the fps explicitly regardless of the what the input avisynth script is serving.
My builds of xvid_encraw set the output framerate based on the input. xvid_encraw builds made from xvid's cvs code are forced to 25fps. My latest available compiles should be available from http://members.optusnet.com.au/squid_80/

I may not be available in this forum for a while, so please have patience if reporting problems.

[Champs]

squid_80, thank you so much for a quick reply! I was using the 12th of July version that I got off this thread and having the afformentioned trouble. Just downloaded your 13th of Sept version now and tried it. Yes it does indeed take the correct fps from the input avs! I'm sincerely sorry for not having been aware of this more recent version, and thanks again!!

[squid_80]

I have made a new build available for testing: http://members.optusnet.com.au/squid_80/xvid_encraw.zip

Changes:

-o can be used for avi and mkv files, it decides what format to write based on file extension.
-stats takes an optional filename parameter, to specify a file to write the stats to. I guess if you want no output at all (no progress display or stats) you can use -stats nul.
The avi writing code is all new (borrowed from virtualdub), this should fix the > 2GB problems people were having. It may contain large bugs, particularly if you run out of space.
MKV writer interface was changed a bit, should be transparent (but may have introduced bugs).
If ctrl-c is used to terminate the program it should exit gracefully (output files should be flushed, statistics shown).


The new avi writing code is a big change, if there are any problems with the avi files it creates please me know ASAP.

Keep aviwriter.dll with xvid_encraw.exe, so if I change it (likely) it can be overwritten without too much hunting around on your hard drive.

If you get errors about missing msvc dlls, get the vs2005 redistributable pack from here:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=32BC1BEE-A3F9-4C13-9C99-220B62A191EE
I know it's big (2.6mb) but most people will probably already have it and if you don't you'll need it sooner or later, unless you never plan on installing any new software.

@henryho_hk: Any news on your problem? I can't see anything wrong with it here. The numbers for -bquant_ratio and -bquant_offset have to be multiplied by 100 i.e. 162 for 1.62, that's about the only thing I can think the problem would be.

[henryho_hk]

@henryho_hk: Any news on your problem? I can't see anything wrong with it here.

I cannot reproduce the problem again. I think I had done something wrong in the parameters.

[henryho_hk]

squid_80, AVIMux is complaining on the output avi files when the framerate is not an integer.

xvid_encraws 20060621 gives:

Version: 1.17.7, Aug 8 2006
-----------------------------------------------
AVI-Type: standard
Writing-App : n/a
Title : n/a
resolution : 640x480
FourCC : xvid
biCompression value : XVID
framerate (strh): 29.97 fps
framerate (avih): 29.9697 fps
number of frames : 501 (0:00:16.717)
thereof keyframes : 8
thereof deltaframes : 493
thereof dropped frames : 0
strh[0].dwLength : 501

MainAVIHeader.dwTotalFrames : 501
real value : 501
suggested buffer size : 7809

Flags set: : 0x00000810
AVIF_HASINDEX
AVIF_TRUSTCKTYPE
size of video stream : 208,705 bytes
video data rate : 12 kByte/s


xvid_encraws 20061005 gives:

Version: 1.17.7, Aug 8 2006
-----------------------------------------------
AVI-Type: standard
Writing-App : n/a
Title : n/a
resolution : 640x480
FourCC : xvid
biCompression value : XVID
framerate (strh): 29.97 fps
framerate (avih): 29.9706 fps
The framerate value has appearently been written by a broken program!
number of frames : 501 (0:00:16.717)
thereof keyframes : 8
thereof deltaframes : 493
thereof dropped frames : 0
strh[0].dwLength : 501

MainAVIHeader.dwTotalFrames : 501
real value : 501
suggested buffer size : 0

Flags set: : 0x00000110
AVIF_HASINDEX
AVIF_ISINTERLEAVED
size of video stream : 208,705 bytes
video data rate : 12 kByte/s

[squid_80]

If you use virtualdub to do the processing, it'll say the same thing about the output file. Basically it's being picky; the framerate in the stream header(strh) is specified as a rational number (rate/scale) while the framerate in the avi file header is given in Microseconds per frame. 29.97fps is normally represented as 30000/1001 in the stream header, which gives 0.0333666.... microseconds per frame. Virtualdub seems to truncate this value rather than rounding it and avimux_gui complains because it's not as accurate as possible. But applications shouldn't be using the avih framerate value anyway since the strh framerate is more accurate.

[Blue_MiSfit]

Hey everyone...

So, I haven't ever trid XviD through MeGUI before, but I tried giving it a whirl last night and came up with some problems.


Isolated it to the CQM. I remember some talk of xvid_encraw not supporting CQMs, is that the case?


I got the latest xvid_encraw from squid80's optusnet site, pointed MeGUI at it, and built a profile that matches my usual high bitrate virtualdub settings:


C:\Program Files\megui\tools\xvid_encraw\xvid_encraw.exe -i "F:\Movies Work\City of God\God.avs" -pass2 ".stats" -bitrate 700 -kboost 100 -overhead 0 -max_key_interval 240 -nopacked -vhqmode 4 -qpel -qmatrix "D:\XviD Matricies\Sharktooth's Matricies\eqm_v3hr (gp hb).xcm" -closed_gop -lumimasking -max_bframes 3 -bvhq -threads 1 -avi "F:\Movies Work\City of God\God.avi"


script:

LoadPlugin("D:\Archives\Video Software\AviSynth Plugins\Common\RemoveGrainSSE3.dll")

DGDecode_mpeg2source("F:\Movies Work\City of God\VTS_01_1.d2v",info=3)
ColorMatrix(hints=true)

RemoveGrain(mode=5)


It's automated 2 pass, so the bitrate should get overridden..

Anyway, when I try to encode my AviSynth script that works perfectly in virtualdub or x264.exe, it crashes... I get the unexpectedly closed box, and it references xvidcore.dll.

Maybe my xvidcore.dll is out of date? It's 856,064b and in the same folder as xvid_encraw... Am I just really n00bing it and totally missing some crucual step here?

Sorry if I'm asking about a totally common thing, I did some search and didn't find anything..

Also tried a 1 pass CQ encode and had the same problem.

Suggestions?
~MiSfit

[henryho_hk]

Why is "-overhead 0"? And I think XviD wants us to do a pass1 and then 1 pass2 manually.

[squid_80]

Isolated it to the CQM. I remember some talk of xvid_encraw not supporting CQMs, is that the case?


I use Sharktooth's cqms a lot with no problems. Does xvid_encraw give any text output at all before it crashes?

[Blue_MiSfit]

Well the megui log says:

Generating jobs. Desired size: 2306867200 bytes
No audio encoding. Calculating desired video bitrate directly.
Encoded audio file is present: F:\Movies Work\City of God\VTS_01_1 T01 3_2ch 448Kbps DELAY 0ms.mp4 It has a size of 228491910 bytes.
Setting video bitrate for the video jobs to 2131 kbit/s
Setting desired size of video to 2074358784 bytes
Starting job job1-1 at 9:03:29 PM
encoder commandline:
-i "F:\Movies Work\City of God\God.avs" -pass1 "F:\Movies Work\City of God\God.stats" -bitrate 2131 -kboost 100 -overhead 0 -max_key_interval 240 -nopacked -vhqmode 4 -qpel -qmatrix "D:\XviD Matricies\Sharktooth's Matricies\eqm_v3hr (gp hb).xcm" -closed_gop -lumimasking -max_bframes 3 -bvhq -threads 1
successfully started encoding


and then I get the generic "xvid_encraw.exe has encountered a problem and needs to close..." error box.

~MiSfit

[Brother John]

You isolated it to the CQM? Then with a standard matrix everything works fine?

Corrupted XCM file? Can you use it with XviD VfW?
Just to rule out path name problems: Try a path with only letters and no spaces like "D:\eqm_v3hr.xcm". Spaces shouldn't be a problem, but for some weird reason the ' might.
Which version of xvid_encraw do you use? Run "xvid_encraw.exe -help" to find out.

[squid_80]

Probably easiest to just post the matrix file, or a link to it. encraw checks to make sure it's 128 bytes long but that's about all it looks for. There's no string manipulation on the filename so spaces and slashes shouldn't be an issue. (Yes it freads in binary mode, in case anyone wondered.)

[Blue_MiSfit]

My CQM definately works in VFW XviD.

I'm using the October 5 2006 build (from 'xvid_encraw.exe -help')

http://rapidshare.de/files/36538364/eqm_v3hr__gp_hb_.xcm.html

Pretty sure it's not the matrix itself, as it works in vfw just fine..

Tried again, with a no space name (called it eqm_v3hr.xcm and put it in the root of my D: drive...

Very strange indeed..

[squid_80]

There's something wrong with that matrix file. I started xvid vfw's config and went to the custom matrix section, when I loaded your matrix it showed a 121 and 0 at the end of the second row of the inter matrix. When OK is selected the VFW config corrects the 0 into a 1, xvid_encraw isn't that smart so it crashes.

[Blue_MiSfit]

How goofy.

I will re-download Sharktooth's CQM pack. Thanks for the help squid_80...


Did that and everything works perfectly. Nice bit of code gentlemen!! No more vdub encoding for me :)


~MiSfit

[MetalPhreak]

I either found a bug or I'm doing something wrong:
Using the latest encraw by squid_80 I cannot specify a negative b-frame sensititvity in -zones. Eg. -zones 1000,q,2,-20 the "-" gets ignored and encraw uses the integer entered as a positive value.
Any help would be greatly appreciated.

--EDIT--
Maybe I should also add this is with XviD v1.1.2.
Also I'm going away tomorrow so I won't be able to reply for about a week (if more info should be needed)

[henryho_hk]

Try inserting e.g., "K" (start with keyframe), O (chroma optimizer), etc. and see if it is parsed correctly.


-zones 0,q,3,KO-3/1,q,3,KO-10

[squid_80]

Using the latest encraw by squid_80 I cannot specify a negative b-frame sensititvity in -zones. Eg. -zones 1000,q,2,-20 the "-" gets ignored and encraw uses the integer entered as a positive value.

I can't reproduce your results (i.e. it works fine here), can you post your full command line in case it's something else?
G:\vid1>xvid_encraw -i sources.avs -frames 200 -progress
I frames: 3 frames, size = 43890/ 131671, quants = 4 / 4.00 / 4
P frames: 76 frames, size = 24035/1826677, quants = 4 / 4.00 / 4
B frames: 121 frames, size = 5087/ 615613, quants = 7 / 7.00 / 7

G:\vid1>xvid_encraw -i sources.avs -frames 200 -progress -zones 0,q,4,-20
I frames: 3 frames, size = 43890/ 131671, quants = 4 / 4.00 / 4
P frames: 124 frames, size = 20997/2603670, quants = 4 / 4.00 / 4
B frames: 73 frames, size = 3940/ 287642, quants = 7 / 7.00 / 7

[MetalPhreak]

Try inserting e.g., "K" (start with keyframe), O (chroma optimizer), etc. and see if it is parsed correctly.

Already tried that, eg. forcing keyframes work but same result with b-frame bias.

I can't reproduce your results (i.e. it works fine here), can you post your full command line in case it's something else?

xvid_encraw -single -cq 2 -zones 50,q,2,K-30 -quality 6 -vhqmode 4 -bvhq -qpel -gmc -qtype 0 -max_bframes 2 -bquant_ratio 100 -bquant_offset 100 -i "C:\GARDEN_STATE\VIDEO_TS\gs.avs" -mkv "F:\xvid_encraw\test.mkv" -max_key_interval 250 -par 64:45 -progress 10

--EDIT--
Nevermind, just tried again and now it works perfectly. I was probably half asleep the last time I tried and messed something up. Thanks henryho_hk and squid_80 for your help anyway.

[Adub]

Yo, Squidy!
Any new developments in the works? Some crazy optimizations or insane quality modes that blow x264 out of the water? ;p

[squid_80]

encraw's only a frontend for xvid. If you want some new options to play with, check this thread: http://forum.doom9.org/showthread.php?t=114811

[henryho_hk]

squid_80, it seems both the stat file output and the AVI output routines are quite stable. Will you consider sending the source patch to xvid-devel?

[SeeMoreDigital]

Hi guys....

Can somebody please confirm where we got to concerning XvidEncRAW and Packed Bit-stream.

Can XvidEncRAW be configured to generate MPEG-4 part-2 streams with PBS. And if so, is it really neccessary to include such an implementation?


Cheers

[squid_80]

Default setting with .avi output is to use packed bitstream, if there is no .avi output no packed bitstream.
It is also possible to force it on/off with -packed/-unpacked. This does not affect matroska output.

[SeeMoreDigital]

Default setting with .avi output is to use packed bitstream, if there is no .avi output no packed bitstream.
It is also possible to force it on/off with -packed/-unpacked. This does not affect matroska output.I see... So when generating and outputting RAW "MPEG-4_ASP.M4V" streams, there is no packed bit-stream... Excellent :)

Pardon my ignorance... But may I ask why it was felt necessary to develop XvidEncRAW to include PBS at all, even when outputting to AVI?

[squid_80]

AVIs without packed bitstream are hard to edit due to the lag between the frame being displayed and the frame being decoded. For example with virtualdub the image that is displayed on screen does not necessarily come from the frame indicated by the slider. Also don't some SAPs actually require packed bitstream?

It's good to have it as an example of how to use the xvid API as well, since that's where the functionality really comes from. xvid_encraw simply passes a flag to the encoder to tell it to deliver a packed bitstream or not.

[708145]

AVIs without packed bitstream are hard to edit due to the lag between the frame being displayed and the frame being decoded.

Why use avi at all then? If it is not even useful for editing?

[SeeMoreDigital]

It's good to have it as an example of how to use the xvid API as well, since that's where the functionality really comes from. xvid_encraw simply passes a flag to the encoder to tell it to deliver a packed bitstream or not.I see....

And as a back-up, for all those people who find their SAP's need PBS, but forgot to include it. There is always Moitah's excellent MPEG4 Modifier application - which was recently upgraded to add PBS to MPEG-4 (in AVI) streams too ;)

[squid_80]

Why use avi at all then? If it is not even useful for editing?
It's fine with packed bitstream. That's all I'm going to say since I don't want this to turn into another avi/vfw war thread.

[708145]

It's fine with packed bitstream. That's all I'm going to say since I don't want this to turn into another avi/vfw war thread.

OK then.

[weaver4]

I see that xvid-encraw has picked up a lot of momentum. So, I want to learn a little more about it. Are there any guides or manuals that I could look at.

[squid_80]

xvid_encraw -help and this thread from about page 11 onwards.

[elguaxo]

I am using the xvid_encraw version that comes with MeGUI. I don't remember when it was last updated, but the Greyscale (-grey) option is not available:

D:\megui\tools\xvid_encraw>xvid_encraw -help
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


xvid_encraw built at 17:33:35 on Apr 30 2006
Usage : xvid_encraw [OPTIONS]

Input options:
-i string : input filename (stdin)
-type integer: input data type (yuv=0, pgm=1, avi/avs=2)
-w integer: frame width ([1.2048])
-h integer: frame height ([1.2048])
-frames integer: number of frames to encode

Output options:
-dump : save decoder output
-save : save an Elementary Stream file per frame
-o string : save an Elementary Stream for the complete sequence
-avi string: save an AVI file for the complete sequence
-mkv string: save a MKV file for the complete sequence

BFrames options:
-max_bframes integer: max bframes (2)
-bquant_ratio integer: bframe quantizer ratio (150)
-bquant_offset integer: bframe quantizer offset (100)

Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-size integer : target size in kilobytes
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-max_key_interval integer : maximum keyframe interval (300)

Single Pass options:
-reaction integer : reaction delay factor (16)
-averaging integer : averaging period (100)
-smoother integer : smoothing buffer (100)

Second Pass options:
-kboost integer : I frame boost (10)
-kthresh integer : I frame reduction threshold (1)
-kreduction integer : I frame reduction amount (20)
-ostrength integer : overflow control strength (5)
-oimprove integer : max overflow improvement (5)
-odegrade integer : max overflow degradation (5)
-chigh integer : high bitrate scenes degradation (0)
-clow integer : low bitrate scenes improvement (0)
-overhead integer : container frame overhead (24)
-vbvsize integer : use vbv buffer size
-vbvmax integer : vbv max bitrate
-vbvpeak integer : vbv peak bitrate over 1 second

Other options
-noasm : do not use assembly optmized code
-turbo : use turbo presets for higher encoding speed
-quality integer : quality ([0..6]) (6)
-vhqmode integer : level of R-D optimizations ([0..4]) (1)
-bvhq : use R-D optimizations for B-frames
-qpel : use quarter pixel ME
-gmc : use global motion compensation
-qtype integer : quantization type (H263:0, MPEG4:1) (0)
-qmatrix filename : use custom MPEG4 quantization matrix
-interlaced [integer] : interlaced encoding (BFF:1, TFF:2) (1)
-nopacked : Disable packed mode
-noclosed_gop : Disable closed GOP mode
-lumimasking : use lumimasking algorithm
-stats : print stats about encoded frames
-debug : activates xvidcore internal debugging output
-vop_debug : print some info directly into encoded frames
-nochromame : Disable chroma motion estimation
-notrellis : Disable trellis quantization
-imin integer : Minimum I Quantizer (1..31) (2)
-imax integer : Maximum I quantizer (1..31) (31)
-bmin integer : Minimum B Quantizer (1..31) (2)
-bmax integer : Maximum B quantizer (1..31) (31)
-pmin integer : Minimum P Quantizer (1..31) (2)
-pmax integer : Maximum P quantizer (1..31) (31)
-drop integer : Frame Drop Ratio (0..100) (0)
-start integer : Starting frame number
-threads integer : Number of threads
-progress [integer] : Show progress updates every n frames (10)
-par integer[:integer] : Set Pixel Aspect Ratio.
1 = 1:1
2 = 12:11 (4:3 PAL)
3 = 10:11 (4:3 NTSC)
4 = 16:11 (16:9 PAL)
5 = 40:33 (16:9 NTSC)
other = custom (width:height)
-help : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as needed.

I downloaded the latest version available here (http://members.optusnet.com.au/squid_80/), but it is not there either:

c:\temp>xvid_encraw -help
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


xvid_encraw built at 02:09:09 on Oct 5 2006
Usage : xvid_encraw [OPTIONS]

Input options:
-i string : input filename (stdin)
-type integer: input data type (yuv=0, pgm=1, avi/avs=2)
-w integer: frame width ([1.2048])
-h integer: frame height ([1.2048])
-frames integer: number of frames to encode

Output options:
-dump : save decoder output
-save : save an Elementary Stream file per frame
-o string : save an Elementary Stream for the complete sequence
-avi string: save an AVI file for the complete sequence
-mkv string: save a MKV file for the complete sequence

BFrames options:
-max_bframes integer: max bframes (2)
-bquant_ratio integer: bframe quantizer ratio (150)
-bquant_offset integer: bframe quantizer offset (100)

Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-size integer : target size in kilobytes
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-max_key_interval integer : maximum keyframe interval (300)

Single Pass options:
-reaction integer : reaction delay factor (16)
-averaging integer : averaging period (100)
-smoother integer : smoothing buffer (100)

Second Pass options:
-kboost integer : I frame boost (10)
-kthresh integer : I frame reduction threshold (1)
-kreduction integer : I frame reduction amount (20)
-ostrength integer : overflow control strength (5)
-oimprove integer : max overflow improvement (5)
-odegrade integer : max overflow degradation (5)
-chigh integer : high bitrate scenes degradation (0)
-clow integer : low bitrate scenes improvement (0)
-overhead integer : container frame overhead (24)
-vbvsize integer : use vbv buffer size
-vbvmax integer : vbv max bitrate
-vbvpeak integer : vbv peak bitrate over 1 second

Other options
-noasm : do not use assembly optmized code
-turbo : use turbo presets for higher encoding speed
-quality integer : quality ([0..6]) (6)
-vhqmode integer : level of R-D optimizations ([0..4]) (1)
-bvhq : use R-D optimizations for B-frames
-qpel : use quarter pixel ME
-gmc : use global motion compensation
-qtype integer : quantization type (H263:0, MPEG4:1) (0)
-qmatrix filename : use custom MPEG4 quantization matrix
-interlaced [integer] : interlaced encoding (BFF:1, TFF:2) (1)
-nopacked : Disable packed mode
-noclosed_gop : Disable closed GOP mode
-lumimasking : use lumimasking algorithm
-stats : print stats about encoded frames
-debug : activates xvidcore internal debugging output
-vop_debug : print some info directly into encoded frames
-nochromame : Disable chroma motion estimation
-notrellis : Disable trellis quantization
-imin integer : Minimum I Quantizer (1..31) (2)
-imax integer : Maximum I quantizer (1..31) (31)
-bmin integer : Minimum B Quantizer (1..31) (2)
-bmax integer : Maximum B quantizer (1..31) (31)
-pmin integer : Minimum P Quantizer (1..31) (2)
-pmax integer : Maximum P quantizer (1..31) (31)
-drop integer : Frame Drop Ratio (0..100) (0)
-start integer : Starting frame number
-threads integer : Number of threads
-progress [integer] : Show progress updates every n frames (10)
-par integer[:integer] : Set Pixel Aspect Ratio.
1 = 1:1
2 = 12:11 (4:3 PAL)
3 = 10:11 (4:3 NTSC)
4 = 16:11 (16:9 PAL)
5 = 40:33 (16:9 NTSC)
other = custom (width:height)
-help : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as needed.

Where can I find a version that includes the -grey option?

[buzzqw]

while i perfectly agree... i never seen too -grey... but i will prefer the greyscale from avisynth

BHH

[kurt]

greyscale is part of the zones and it's missing in MeGUI. What you can do is to use a custom commandline:

eg:
-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG
--> last zone starts with frame 161519, has chroma optimizer and greyscale enabled, bframe sensivity of 25 and a weight of 0.04...

here are the complete zones parameters:


Rate control:
Weight 2.01 => ,w,2.01
Quantizer 5.01 => ,q,5.01

Begin with keyframe => ,K
Greyscale encoding => ,G
Chroma optimizer enabled => ,O
Cartoon Mode => ,C
BVOP sensivity 45=> ,45

[elguaxo]

I am pretty sure it was there sometime. It is even an option in MeGUI's XviD Configurations.

Edit: Great! Thanks kurt!!!

[squid_80]

New build available (for the second time this week): http://members.optusnet.com.au/squid_80/xvid_encraw.zip

Changes include plugh's alternative 2 pass algorithm (thread (http://forum.doom9.org/showthread.php?t=118419)) and Kopernikus's HVS mods (thread (http://forum.doom9.org/showthread.php?t=114811)). Also some SSIM options from xvid's CVS.

[henryho_hk]

I am using Celtic Druid's 2006.12.08 CVS compile of XviDcore.dll on a Core 2 Duo E6300 1.86GHz. The encoding FPS reported by xvid_encraw seems incorrect. A single pass encode of a 2hr50m 29.9fps was reported to be encoding at 34fps; however, it took 4hr40m totally, way slower than the reported speed. Or have I misunderstood the encoding fps?

[MatMaul]

I am using Celtic Druid's 2006.12.08 CVS compile of XviDcore.dll on a Core 2 Duo E6300 1.86GHz. The encoding FPS reported by xvid_encraw seems incorrect. A single pass encode of a 2hr50m 29.9fps was reported to be encoding at 34fps; however, it took 4hr40m totally, way slower than the reported speed. Or have I misunderstood the encoding fps?
Do you have fft3dGPU in your avs script ?
because when I use it, wrong fps is reported by xvid_encraw and x264.

[henryho_hk]

Do you have fft3dGPU in your avs script?

I am just using mpeg2source(), colormatrix(), tdeint(), degrainmedian() and crop().

[squid_80]

Or have I misunderstood the encoding fps?
It's encoding fps; it only measures the time taken to encode frames. It doesn't take into account the time taken to get the input frames i.e. avisynth processing.

[henryho_hk]

I don't get it. Playing back the AVS in media player classic takes about 25% CPU load only. How can it contribute for 16fps lost in encoding speed?

[squid_80]

So it's a simple enough script that can playback in realtime without lagging? I'd assumed it was more complicated. If you really want to benchmark the avs script open it with virtualdub, set video processing mode to direct stream copy, leave video compression set to none and run a video analysis pass.
Another thing is to make sure the script is outputting yv12, if it's not then some vfw codec might be used for the conversion.

[henryho_hk]

It's a rather simple script. I can play it realtime at 25% CPU load and a direct stream copy of VirtualDubMod runs at about 50fps. VirtualDubMod says "YV12" and "ATI YVU12 4:2:0 Planar".


vf="test.d2v"
mpeg2source(vf)
colormatrix(d2v=vf)
tdeint(order=1,mtnmode=0)
crop(8,0,704,480)
removegrain(mode=5)


I am now testing "-threads 3" (#core + 1) and see if it helps.

[mixanobios]

i have experienced the same problem with henryho_hk. if you want i can search for my megui logs

[squid_80]

I'm in the middle of building a dual core machine (literally putting it together), when it's properly setup I will add threaded code for the input stage and fix the encoding timers (and hopefully remember once and for all to also fix the size variables so they don't wrap to negatives with large files).

[G_M_C]

Do you have fft3dGPU in your avs script ?
because when I use it, wrong fps is reported by xvid_encraw and x264.

Could you please report that in the FFT3DGPU thread please ?
http://forum.doom9.org/showthread.php?t=89941&page=28

You may have to add a script-example + a clip example to make ... reproduce the error.

I ask this because i use fft3dgpu very often, and making it bug-free is offcourse imporatnt enough for all of us that do too.

[MatMaul]

I don't think it's a bug of fft3dGPU but just the way fps is calculated in xvid_encraw.

[Brother John]

Hi, squid_80.

Everytime I look at "xvid_encraw -help" I stumble over the outdated zones options. Isn't it time to document the zones feature properly? To do this you should remove the obsolete -zq and -zw lines from "Rate control options" and replace the note at the very end with a more detailed explanation. I put together the following text. What do you think about it?


Zones options:
You can define up to 64 zones using the -zones option as described below.

-zones start,mode,value[,options][/start,mode,value[,options]]...

Parameters of a zones use the comma (,) as delimiter. Multiple zones are
separated by a slash (/). The end of each zone is defined by either the start
of the following zone or the last frame of the input file.

start : Start frame of the zone.
mode : weight zone = w, quantizer zone = q
value : Depending on mode either the zone's weight or quantizer.
options : Enables certain encoder features for the zone. Each feature is
represented by a single letter. An integer number stands for
b-frame sensitivity. To enable multiple features at the same time
combine the appropriate symbols without any delimiting characters.
K = Begin with keyframe
O = Enable chroma optimizer
G = Greyscale encoding
C = Cartoon mode
integer = B-Frame sensitivity

Example:
To create a first zone starting at frame 0 with weight 1.0, all options
enabled and b-frame sensitivity -5, and a second zone starting at frame 1000
with constant quant 4 and no options enabled you would use the -zones option
like this:

-zones 0,w,1.0,-5KOGC/1000,q,4

[squid_80]

The old options (-zq and -zw) are still useful so I'll leave them in, but I'll add a small description to be displayed with -help and your longer description can be displayed with "-help zones" or if an error is detected in the -zones parameters.

With regards to the fps bugs, I am seeing very irregular things happening with my dual core machine and I have a hunch that it's due to the main thread jumping between cores. I remember reading that separate cores don't always have their clocks synchronised exactly, hence if the start time is taken from one core and the finish time is taken from the other the result isn't valid. This may be BS but I'm not sure how else to explain the fact that I'm seeing frames pop out with negative encoding times.

[foxyshadis]

Maybe it's one of those things that only works right if you start windows with /usepmtimer. If you already have it, then... dunno. What do you use? GetSystemTickCount?

[Brother John]

I’ve been wondering for some time now about a specific function in encraw, but search didn’t turn up anything useful.

In Xvid VfW you can set 1-pass target quant in the main dialog and 1-pass constant quant via the zones window. However in Xvid_Encraw I only find -cq for constant quant. No target quant at all. Am I blind? Is target quant not supported in encraw? Or is the VfW GUI misleading?

Also (if it does exist), would it be fair to say that Xvid’s target quant is a similar function to x264’s constant rate factor? Very roughly speaking, maybe. Just to get the general idea.

[Kopernikus]

There is no equivalent to x264s crf in xvid, IIRC target quant is equivalent to constant quant. If the target is not a integer, the quants are distributed to achieve a average quant near the target.

[henryho_hk]

Is target quant not supported in encraw?

Both are supported: -cq and -zones

Xvid’s target quant is a similar function to x264’s constant rate factor?

I am afraid not. But you can take a look at the thread " Isn't it time for a new compression mode in XviD?".

[elfurbe]

So, it seems that no matter how many threads I specify, xvid_encraw never spawns more than one, at least according to task manager. I've got an A64 4400+ dual-core, so I used -threads 3 (cause I've found one more threads than cores is ideal for most tasks) but Task Manager shows it at 1 thread which is disturbing slow (5.6fps) on pass2 and pretty damn slow on pass1 (~20fps). The AVS framesize is 960x544, down-resing from 720p HDTV content.

What am I doing wrong? Here's my avs file and encode lines:

AVS:
LoadPlugin("c:\plugins\dgdecode.dll")
LoadPlugin("c:\plugins\TIVTC.dll")
mpeg2source("C:\file.d2v")
SelectEven()
Tdecimate()
LanczosResize(960,544,0,0,1280,720)
UnDot()

Pass 1:
xvid_encraw -bitrate 2300 -pass1 -threads 3 -progress -turbo -i file.avs -mkv file.mkv

Pass 2:
xvid_encraw -bitrate 2300 -pass2 -threads 3 -progress -qpel -gmc -i file.avs -mkv file.mkv

[foxyshadis]

How many fps do you get if you load the script into virtualdub and hit "run video analysis pass"? That's the maximum upper bound. Using the multithreaded avisynth and appropriate syntax can help. I suspect most of the speed hit comes from TDecimate, but there's not much you can do about that. And of course gmc - didn't I read somewhere that gmc wasn't multithreaded?

If speed is of the utmost importance, you can always do a lossless pre-render.

[squid_80]

xvid_encraw reports the xvidcore version and the number of threads being used. Can you post this information?

[elfurbe]

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


xvid [warn]: Can't find xvid_plugin_ssim, ssim calculations will be disabled
Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [info]: Input is 960 x 544, 23.976fps (24000/1001), starting from frame 0
xvid [info]: Number of frames to encode: 62979, Bitrate = 2235kbps
xvid [info]: xvidcore build version: xvid-1.1.0
xvid [info]: Bitstream version: 1.1.0
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
xvid [info]: Detected 0 cpus, using 3 threads.

That's what she outputs on start, but in Task Manager it shows 1 thread.

I don't use virtual dub for anything in my encoding chain, so I wouldn't even know how to go about doing what you're talking about.

I ran another encode this morning, and it looks like I underestimated my -turbo pass, it's getting about 32fps which isn't terrible, but pass two is still super slow.

[squid_80]

xvid [info]: xvidcore build version: xvid-1.1.0
xvid [info]: Detected 0 cpus, using 3 threads.
You need xvid 1.2.0 for multithreading.

(Also, note to self: Try changing "using x threads" to "requesting x threads" and append "x threads being used", if possible. Also remove ssim warning unless ssim calculations are specifically requested.)

[elfurbe]

Well, that could certainly do it, I suppose. Is this:
http://www.koepi.org/XviD-1.2.-127-25022006.exe
what I need to install?

I guess I thought the xvid encoder was rolled into the xvid_encraw binary, a-la x264.exe. Is that not the case?

[squid_80]

Well, that could certainly do it, I suppose. Is this:
http://www.koepi.org/XviD-1.2.-127-25022006.exe
what I need to install?Yep, that should work.
I guess I thought the xvid encoder was rolled into the xvid_encraw binary, a-la x264.exe. Is that not the case?No, it uses whatever xvidcore.dll it can find. It helps keep the filesize down a bit and makes it possible to update the core encoder without updating xvid_encraw.exe.

[elfurbe]

So if I put xvidcore.dll in the same folder as xvid_encraw, would that be the default one it uses or does it check system first?

[squid_80]

Yes, if xvidcore.dll is in the same directory as xvid_encraw.exe it will be used. At least I'm pretty sure that's how windows dll loading rules work.

[elfurbe]

Alright, threw the xvidcore.dll from xvid 1.2 in there, and rolled it out, got this on start:
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


xvid [warn]: Can't find xvid_plugin_ssim, ssim calculations will be disabled
Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [info]: Input is 960 x 544, 23.976fps (24000/1001), starting from frame 0
xvid [info]: Number of frames to encode: 62994, Bitrate = 2234kbps
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 TSC
xvid [info]: Detected 2 cpus, using 3 threads.

This is on my computer at work, Core Duo so no 3DNOW, but I've found encoding speeds are similar to my 4400+. Anyway, I'm seeing better first pass (40fps at the moment), but the thread count still stays at 1. I do occasionally see it jump to three, but then right back to one instantly. Is that normal behavior?

[squid_80]

(I never noticed task manager had a thread count column available till now, duh.)

Yes that's normal behaviour. x264 does the same sort of thing.

[elfurbe]

Neat. My second pass encode speed is up to 12fps or so, which is obviously much better. I suppose my expectations are a little off, as I usually deal with anamorphic DVD conversions to x264, but man this seems slow. Does this seem like a reasonable speed for a 1.8GHz Core Duo on pass 2 for this frame size?

[squid_80]

New build is available (http://members.optusnet.com.au/squid_80/xvid_encraw.zip).

Cosmetic changes only:
- fps figures should be more accurate (the time taken to fetch the input frame is included)
- ssim warning should only be displayed if ssim options are used and they are unavailable (in which case it is treated as an error)
- zones help, available via "-help zones". Also displayed if an error is detected in the -zones string
- frame/file sizes are 64-bit integers so there should be no more negative numbers when the output filesize goes over 2gb
- changed the help text for -o to reflect that it can be used for .avi and .mkv
- changed the xvid[info] threads text, still needs a little tweak to distinguish between how many threads are requested vs. how many are actually being used (hint: if detected cpus is 0, multi-threading is probably not available)

[henryho_hk]

Excellent. ^_^ I recall that I once did a 20G encode. The numbers went positive, negative, positive, negative .....

Is your next move putting the input and output in separate threads?

[squid_80]

It's one of the ideas I have. I don't know if there will be much improvement since cpu usage normally already hits 100% if you're using MT avisynth. Elfurbe's posts raise the point that the extra threads for multithreading are created/destroyed for each frame; I'm wondering if that's causing an overhead penalty that can be avoided.

[elfurbe]

Didn't know there was a multithreaded AviSynth, downloaded and installed for the win. Interesting thing though. I added a multithread on the TDecimate filter, since I expected that it was the heaviest non-resizing filter in my chain, and it spawns more threads, now my thread count sits around 3 the whole time, but my cpu usage only barely climbs above 50% indicating that it's not really getting into the second core at all. It basically doesn't matter what I do with thread counts in both places, I can't get more than 53% CPU usage. Wondering what you make of that?

Also, I threw the new version of xvid_encraw in and my framerate went to the toilet. I went from doing 30-40 for this file (with/without MT avis) to doing 12. Switching back to the last version puts the framerates right back up. Is it just a display bug?

[squid_80]

I went from doing 30-40 for this file (with/without MT avis) to doing 12. Switching back to the last version puts the framerates right back up. Is it just a display bug?
The difference between the old/new versions is that the old version only measured the time taken by xvid to encode the frame, whereas the new version measures the time taken to retrieve the input frame and encode it. So (given your script) the time reported by the new version is encoding time + mpeg2 decoding + selecteven (negligible?) + tdecimate + resizing + undot. What you could do is try short encodes (e.g. try adding -frames 2000) with both versions and manually time them (don't rely on the times reported by xvid_encraw). The actual time taken should be the same, and the fps reported by the new xvid_encraw should prove more accurate if you work it out by hand.

[SealTooGreat]

Can someone suggest me up-to-date GUI for xvid_encraw?!

[Adub]

MeGUI.

[squid_80]

Virtualdub and xvid's vfw interface. The idea of encraw was to make all the vfw options available via command line, making a gui for it seems a bit backwards.

[elguaxo]

I want to run encraw with the -ssim_file option, but I'm getting this error: Can't find xvid_plugin_ssim

Do I need a newer xvidcore.dll? Where can I get one that supports this? Thanks!

[squid_80]

Get an updated build from celtic_druid: http://ffdshow.faireal.net/mirror/XviD/
Go for the build with the most recent date. :)

[elguaxo]

Perfect. Thanks!

[smok3]

squid_80: what is the correct latest file?

[henryho_hk]

squid_80, I have recently “upgraded” my Core2Duo (1.86GHz) + 865PE + 1GB DDR-I-400 to Core2Duo (@3GHz) + P965 + 2GB DDR-II-800@860. You know what; the encoding speed does not get increased! Same script (mpeg2source + colormatrix + tdeint + bicubicresize + degrainmedian => 25fps ) I must have messed something up….

[squid_80]

That's weird. Try using -threads 1 on both machines and see if there's a difference.

[henryho_hk]

No, I can't... It's the same CPU. :devil:

I have installed Celtic Druid's 2007/03/10 XviD compile in the new setup. I would try the last version and see if it's better.

[squid_80]

No, I can't... It's the same CPU. :devil:
You mean it's the same chip overclocked to 3ghz? It might be getting speed throttled due to overheating. Try checking the temps when it's encoding.

[henryho_hk]

It might be getting speed throttled due to overheating.

It's staying at 58~60 C at full load. Even if it's throttling, it's at 6x @ 2.5G, still way faster than 1.86G. That's why i am puzzled.

[squid_80]

Try running avsutil on the script (the newest version that reports the total time taken to play the script) at the new clock speed then drop to a lower clock speed and see if the time taken increases. If the time measurements are the same it's an avisynth problem; if the avsutil times are proportional to cpu clock speed(as they should be) then there's a problem with xvid.

[henryho_hk]

avsutil's clock time seems to be consistent with the bus speed (hence cpu & ram). It looks like a xvid related. I will try to test the encoding speed in Virtualdub tonight.
---------------
I tested avs2avi instead. At the same settings (TG's comp test, #threads=1), avs2avi is about 50% faster than xvid_encraw. That's far more than I would ever expect.
---------------
Seems the "threads=2" parameter of Colormatrix 2.1 is causing slowness in xvid_encraw. After I change it to "threads=1", the xvid_encraw is just a few fps slower than avs2avi.
---------------
An sincere apology to squid_80. After testing long encodes (>1hr movie, Colormatrix threads=1), the encoding speed difference further diminishes to 2~3fps. Yet the Colormatrix speed issue (threads=2) still remains.

[Brother John]

Here’s a strange thing a have been noticing for some time.

Situation: A video encoded with newest xvid_encraw and Matroska output (-mkv parameter). Xvid settings and 1pass/2pass don’t affect the problem.
When I try to play such a video, the player accesses the hard drive for quite a while. It looks like it’s buffering the video. After a while (about 20 secs for a 2 GB movie) it plays and seeks ok. This happens with all DirectShow players, all the common splitters and decoders as well as with VLC.
When I remux the file with MKVMerge (no special settings, simple blocks disabled because I assume encraw doesn’t use them) it plays immediately without the »buffering« effect.

It’s not a huge issue because remuxing is necessary anyway to add audio, but I’m still wondering: could this be a bug in encraw’s MKV handling?

[708145]

It’s not a huge issue because remuxing is necessary anyway to add audio, but I’m still wondering: could this be a bug in encraw’s MKV handling?

I had the same effect with an old version of mkvmerge.
So it might be a setting with the mkv dictionary.

[foxyshadis]

I think the large read comes from an insufficient cue contents. (Similar to the avi index.) mkvmerge has three options, none, key-frames, or all frames, and currently defaults to keyframes; most other muxers will write no cues, because they aren't known ahead of time. Without cues, the splitter will read the whole file to extract keyframes, or seeking would be impossible. (It could read on the first seek, but they don't....)

(That's not entirely true. In 2pass mode encraw has all i-frames known, and the locations can be filled with dummy values to be filled in at the end of the encode. mkvmerge probably works that way. Up to squid or someone else to consider implementing that though.)

Simpleblock can still help for xvid, btw. It's the audio tracks that really get the gain from using it, especially AC3 and DTS.

[squid_80]

It’s not a huge issue because remuxing is necessary anyway to add audio, but I’m still wondering: could this be a bug in encraw’s MKV handling?
It's not really a bug, xvid_encraw just doesn't put cues (used for quick seeking) in files that it outputs. They're still valid mkv files and I never put in the extra effort since as you pointed out remuxing is normally necessary.

[Brother John]

Thx for the explanation. As you said, it's not too important. Maybe some time, if you get really bored... ;)

[henryho_hk]

squid_80, have you tried the 01/04 (hope it's not a joke) GCC Core2 compile for XviD (http://ffdshow.faireal.net/mirror/XviD/gcc/xvidcore.Core2.7z)?

The encoding time and the fps are shown as -1:

25701 frames( 99%) encoded, -1.#J fps, Average Bitrate = 1808kbps
Tot: enctime(ms) = -1.#J, length(bytes) = 193906924
Avg: enctime(ms) = -1.#J, fps = -1.#J, length(bytes) = 7532

[kurt]

Had the same issue and posted on celtic_druids forum (http://celticdruid.no-ip.com/phpBB2/viewtopic.php?p=785#785).

for me the P4 & Prescott builds work fine...

[chipzoller]

I noticed that encoding with xvid_encraw sets the fourcc in the output file as MP4V (I think, anyway). Since I really haven't messed much with xvid_encraw I was just wondering why this isn't setting the fourcc to XVID or something. I wanted to try some experiments on decoding this output file (640x480, 60fps) using ffdshow and divx to see how each affected the playback on this system. I used the >30% comp. check (hq) in megui to encode the file.

[henryho_hk]

Do you mean xvid_encraw's MKV output?

[chipzoller]

Do you mean xvid_encraw's MKV output?
Yes. I just noticed I can force a fourcc when I remux in mkvmerge, so maybe that's just the simplest answer. I was just curious as to why xvid_encraw's MKV output set that fourcc.
EDIT: Evidently if using MKV as the output in encraw, remuxing in mkvmerge won't correct/replace the fourcc. Evidently this has to be done only with avi output.

[Brother John]

Xvid_Encraw creates MKVs with native MPEG-4 video streams. The ID system for those doesn't identify encoders but video formats (MPEG-4 ASP, AVC, MPEG-2, WMV etc). The FourCC is only used for VfW compatibility mode streams, like those created by the Xvid VfW encoder (e.g. through VDub).

Xvid, DivX, 3ivx and some others are all MPEG-4 ASP encoders and are identified accordingly.

[chipzoller]

Then is the only way to test decoders ffdshow and DivX by having xvid_encraw output to AVI then change fourCC when muxing to MKV (or leave alone if already set to XVID)? Because it seems DivX will not decode content if the fourCC is set to MP4V. Disabling the decoding options for both xvid and MPEG-4 generic in ffdshow renders the stream undecodable even with DivX is installed and generic Mpeg-4 playback set in the decoder options.

[henryho_hk]

Is there any way to extract these native MPEG-4 video streams from the MKV file and mux it in an avi?

[squid_80]

Because it seems DivX will not decode content if the fourCC is set to MP4V. Sounds like a DivX issue to me. Go bug them to support generic mpeg-4 asp in their decoder. Or just output to avi instead of mkv and change the fourcc.
Is there any way to extract these native MPEG-4 video streams from the MKV file and mux it in an avi?I think you have to do mkv->native stream->mp4->avi. Not too sure since I haven't done it myself but I think Bond has posted how to do it.
squid_80, have you tried the 01/04 (hope it's not a joke) GCC Core2 compile for XviD ?

The encoding time and the fps are shown as -1:That used to happen with ANY xvidcore build, when I first fixed the negative bitrate/time issues; I had to re-arrange the code to get it to work correctly. Seems like a missing emms instruction or possibly a non-volatile register not being saved, I just need time to look into it.
I thought there was also a post here about disabling N-VOPs thru xvid_encraw, I'm not sure if that's possible but will look into it.

[vanger]

I have strange problem with avs2qxvid...
Preset to use = TGHQ-58

start "TGHQ-58 - Pass 1/2" /b /wait /belownormal "D:\avs2qxvid\bin\xvid_encraw_2
0061213.exe" -zones 0,q,3,KO -threads 2 -progress 100 -max_key_interval 250 -pac
ked -quality 5 -vhqmode 1 -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -qty
pe 1 -qmatrix "D:\avs2qxvid\matrix\Didees-SixOfNine.cqm" -nochromame -turbo -pas
s1 "D:\avs2qxvid\tmp\zxxxz2.pass" -type 2 -i "D:\avs2qxvid\zxxxz2.avs"

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [info]: Input is 512 x 336, 25.000fps (25/1), starting from frame 0
xvid [info]: Number of frames to encode: 73744
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
xvid [info]: Detected 2 cpus, using 2 threads.
73701 frames( 99%) encoded, 112.14 fps, Average Bitrate = 1367kbps
Tot: enctime(ms) =657434.00, length(bytes) = 503621549
Avg: enctime(ms) = 8.91, fps = 112.17, length(bytes) = 6829
I frames: 556 frames, size = 28837/16033540, quants = 3 / 3.00 / 3
P frames: 26237 frames, size = 11261/295456472, quants = 3 / 3.00 / 3
B frames: 46951 frames, size = 4092/192131537, quants = 4 / 4.00 / 4

start "TGHQ-58 - Pass 2/2" /b /wait /belownormal "D:\avs2qxvid\bin\xvid_encraw_2
0061213.exe" -zones 0,w,1.0,KO -threads 2 -progress 100 -max_key_interval 250 -p
acked -quality 6 -vhqmode 4 -qpel -max_bframes 2 -bquant_ratio 162 -bquant_offse
t 0 -qtype 1 -qmatrix "D:\avs2qxvid\matrix\Didees-SixOfNine.cqm" -imin 3 -imax 4
-pmin 3 -pmax 5 -bmin 3 -bmax 5 -chigh 10 -clow 3 -bvhq -bitrate 1000 -pass2 "D
:\avs2qxvid\tmp\zxxxz2.pass" -stats "D:\avs2qxvid\tmp\zxxxz2.stats" -type 2 -i "
D:\avs2qxvid\zxxxz2.avs" -avi "D:\avs2qxvid\zxxxz2_TGHQ-58_br_1000.avi"

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


xvid [info]: outputting statistics to D:\avs2qxvid\tmp\zxxxz2.stats
Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [info]: Input is 512 x 336, 25.000fps (25/1), starting from frame 0
xvid [info]: Number of frames to encode: 73744, Bitrate = 1000kbps
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
xvid [info]: Detected 2 cpus, using 2 threads.
73701 frames( 99%) encoded, 35.35 fps, Average Bitrate = 1000kbps
Tot: enctime(ms) =2085700.00, length(bytes) = 368580465
Avg: enctime(ms) = 28.28, fps = 35.36, length(bytes) = 4997, psnr y = 42.92
, psnr u = 49.28, psnr v = 50.30
I frames: 556 frames, size = 25511/14184197, quants = 3 / 3.64 / 4
P frames: 26237 frames, size = 8172/214430966, quants = 3 / 3.81 / 5
B frames: 46951 frames, size = 2981/139965302, quants = 4 / 4.79 / 7

All completed..............

16.04.2007 0:46:20,03 - Checking Source
16.04.2007 0:46:24,01 - Loading Preset
16.04.2007 0:46:24,12 - Encoding Video
16.04.2007 0:46:24,12 - Encoding Pass 1
start "TGHQ-58 - Pass 1/2" /b /wait /belownormal "D:\avs2qxvid\bin\xvid_encraw_20061213.exe" -zones 0,q,3,KO -threads 2 -progress 100 -max_key_interval 250 -packed -quality 5 -vhqmode 1 -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -qtype 1 -qmatrix "D:\avs2qxvid\matrix\Didees-SixOfNine.cqm" -nochromame -turbo -pass1 "D:\avs2qxvid\tmp\zxxxz2.pass" -type 2 -i "D:\avs2qxvid\zxxxz2.avs"
16.04.2007 2:37:16,64 - Encoding Pass 2
start "TGHQ-58 - Pass 2/2" /b /wait /belownormal "D:\avs2qxvid\bin\xvid_encraw_20061213.exe" -zones 0,w,1.0,KO -threads 2 -progress 100 -max_key_interval 250 -packed -quality 6 -vhqmode 4 -qpel -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -qtype 1 -qmatrix "D:\avs2qxvid\matrix\Didees-SixOfNine.cqm" -imin 3 -imax 4 -pmin 3 -pmax 5 -bmin 3 -bmax 5 -chigh 10 -clow 3 -bvhq -bitrate 1000 -pass2 "D:\avs2qxvid\tmp\zxxxz2.pass" -stats "D:\avs2qxvid\tmp\zxxxz2.stats" -type 2 -i "D:\avs2qxvid\zxxxz2.avs" -avi "D:\avs2qxvid\zxxxz2_TGHQ-58_br_1000.avi"
16.04.2007 4:51:54,85 - All Completed
The real fps for 1st pass is ~11fps, for 2nd - ~9fps.

the avs script is:
DGDecode_mpeg2source("D:\capture\video\wsa.d2v",info=3).trim(0,73743)
ColorMatrix(hints=true)

crop(14,36,702,504)

FFT3DFilter(sigma=2,plane=4)

LanczosResize(512,336)

I think on my config (AMD X2 3800+ (2500Mhz)) it is too slow...
I have the same fps with the same avs-script on 2nd pass with x264 (HQ-Insane)...

Any ideas?

[henryho_hk]

Older versions of xvid_encraw show its own xvid encoding time and exclude the AVS processing efforts. Try the newest avs2qxvid package with xvid_encraw_20070203.exe. squid_80 has revised the fps calculation method to show the overall encoding speed.

---------------

Added: "-qpel" and especially "-vhqmode 4" are quite slow. For ">58%" and ">90%" presets, you may try the FAST version and see if you can tell the difference.

[vanger]

and how about speed of encoding?

[squid_80]

The encoding speed is fine - the raw encoding speed is 112 fps and 35 fps as reported by xvid_encraw. The filtering (fft3dfilter) is what is slowing it down so much.

[Revgen]

I currently cannot encode anything over 10000kbps with Xvid Encraw. I tried the latest version and the latest Xvid core form CD (March 10th version) and no luck. The VFW version works fine above 10000kbps, so I can't see why it would be xvid itself.

[henryho_hk]

This is a "feature" of xvid_encraw. Any -bitrate over 10000 is considered bps (but not kbps). If you wanna specify >=10000kbps, specify *1024, e.g. -bitrate 10240000

[Revgen]

Thanks. That's one step in the right direction.

Now I have another problem. The encode is still undersized. My VFW encode is perfectly encoded at 17000kbps while the encraw one is about 14800kbps. Is their a reason why this is happening?

Here's my .bat file.

E:
cd\MeGUIUpdated\tools\xvid_encraw
xvid_encraw -i G:\kong.avs -type 2 -w 1920 -h 832 -quality 6 -max_bframes 2 -framerate 23.976 -bitrate 17408000 -pass1 -max_key_interval 240 -qtype 1 -qpel -vhqmode 4 -bvhq -par 1 -noclosed_gop -nopacked -turbo -threads 2 -mkv G:\kongx.mkv
xvid_encraw -i G:\kong.avs -type 2 -w 1920 -h 832 -quality 6 -max_bframes 2 -framerate 23.976 -bitrate 17408000 -pass2 -max_key_interval 240 -qtype 1 -qpel -vhqmode 4 -bvhq -par 1 -noclosed_gop -nopacked -threads 2 -mkv G:\kongx.mkv

Thanks.

[Brother John]

Saturation maybe?
Xvid VfW uses min quant 1 to waste more bitrate in that case while Xvid_Encraw uses min quant 2 and produces a smaller file.

If you want to mimic VfW's behaviour add this to your 2nd pass cmdln:
-imin 1 -pmin 1 -bmin 1
Lowering VHQ or b-frame quantization or using a more detailled matrix makes a lot more sense, though.

[Revgen]

Thanks, that did the trick. I'll fiddle with the VHQ, B-frame quants, and the matices later and see what happens.

[Daemon404]

Hurm, here's a version witht he latest cvs code (May 25th) with matroska support:

http://www.highdeffguild.com/daemon404/enc_builds/xvid_encraw-cvs-may-25.7z

And, since apparently people are having a hard time finding the missing matroska.cpp and matroska.h files (no idea why they aren't in cvs), here:

http://www.highdeffguild.com/daemon404/enc_builds/xvid_matroska.7z

[celtic_druid]

When was it last updated in the CVS though?

[squid_80]

And, since apparently people are having a hard time finding the missing matroska.cpp and matroska.h files (no idea why they aren't in cvs), here:
I can provide an answer for that: I wrote them and they're a horrible mess of kludges and hacks so they weren't absorbed into xvid's CVS. Really the matroska handling should be completely rewritten from scratch but since they work as is I can't really can't be bothered.

[Daemon404]

When was it last updated in the CVS though?

April 28th, iirc. However, I really dont see any recent builds with matroska support.

I can provide an answer for that: I wrote them and they're a horrible mess of kludges and hacks so they weren't absorbed into xvid's CVS. Really the matroska handling should be completely rewritten from scratch but since they work as is I can't really can't be bothered.

I figured it was something like that. :P

[celtic_druid]

http://cvs.xvid.org/cvs/viewvc.cgi/xvidcore/examples/xvid_encraw.c?view=log
Not updated since Jan. My point was that builds with Xvid dynamically linked just need a new xvidcore.dll unless encraw itself is updated.

[Daemon404]

Hmm yea, dynamically linking would probably be a smarter idea.

And by Apr 28, i meant that's when xvidcore was last updated.

[Brother John]

I just stumbled one more time over the bit/kbit issue with the -bitrate switch. What about changing the help text to clarify this?
-bitrate [integer] : target bitrate (700)
values <= 10000 are treated as kbit/s
values > 10000 are treated as bit/s

[Selur]

btw. if current encraw supports -alt2pass would be nice to mention it in the 'help'

[squid_80]

This is what my build does:
Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-size integer : target size in kilobytes
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-altpass2 [filename] : twopass mode (2nd pass alt)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-zones : see xvid_encraw -help zones
-max_key_interval integer : maximum keyframe interval (300)
I agree -bitrate could do with some clarification, but don't see the problem with -altpass2 not being mentioned?

[squid_80]

A new build is available: http://members.optusnet.com.au/squid_80/xvid_encraw.zip

See if you can spot the new feature (hint: you need a multi-core machine).

[Sharktooth]

eh:) i didnt get it yet, but if what you're talking about it's an automatic threading feature (without the need of a commandline switch), i will kiss your.... :D

[squid_80]

Old build:
>xvid_encraw -i test.avs -progress -frames 1000
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [warn]: Single pass defaulting to quant 4.00
xvid [info]: Input is 528 x 576, 25.000fps (25/1), starting from frame 0
xvid [info]: Number of frames to encode: 1000
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 TSC
xvid [info]: Detected cpus = 4, threads requested = 4, threads in use = 4
1001 frames(100%) encoded, 38.50 fps, Average Bitrate = 1340kbps
Tot: enctime(ms) =26000.00, length(bytes) = 6709775
Avg: enctime(ms) = 25.95, fps = 38.54, length(bytes) = 6696
I frames: 15 frames, size = 19720/ 295814, quants = 4 / 4.00 / 4
P frames: 427 frames, size = 11520/ 4919399, quants = 4 / 4.00 / 4
B frames: 558 frames, size = 2678/ 1494562, quants = 7 / 7.00 / 7
new build:
>xvid_encraw -i test.avs -progress -frames 1000
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003


Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [warn]: Single pass defaulting to quant 4.00
xvid [info]: Input is 528 x 576, 25.000fps (25/1), starting from frame 0
xvid [info]: Number of frames to encode: 1000
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 TSC
xvid [info]: Detected cpus = 4, threads requested = 3, threads in use = 3
xvid [info]: Threaded input reading active
1001 frames(100%) encoded, 57.66 fps, Average Bitrate = 1340kbps
Tot: enctime(ms) =17359.00, length(bytes) = 6709776
Avg: enctime(ms) = 17.32, fps = 57.72, length(bytes) = 6696
I frames: 15 frames, size = 19720/ 295814, quants = 4 / 4.00 / 4
P frames: 427 frames, size = 11520/ 4919399, quants = 4 / 4.00 / 4
B frames: 558 frames, size = 2678/ 1494563, quants = 7 / 7.00 / 7

[Sharktooth]

ok, i suppose i have to do it then...:D
in the meanwhile it got its way into megui auto-update...

[DarkZell666]

Pretty neat :D

But something else changed too :
B frames: 558 frames, size = 2678/ 1494562, quants = 7 / 7.00 / 7
becomes
B frames: 558 frames, size = 2678/ 1494563, quants = 7 / 7.00 / 7

I don't give a damn for 1 byte honestly, but I'm curious :p Is there any part of the code that's undeterministic ? Or is it a side effect of your new feature ?

[squid_80]

Look closer and you'll see why - using threaded input reduces the number of threads used for encoding by 1. Different number of threads = slightly different output.

[kypec]

@squid_80
Just curious, how did you achieve that performance boost?
I thought that xvid_encraw is just an interface to xvidcore.dll, isn't it?
Thus the encoding performance should be entirely (well, from 99% in our real PC world then) dependent ONLY on Xvid encoder engine installed, or not?

[Brother John]

You're right when you speak about pure enocder perfomance without any overheads, decoding, filtering etc. And multiple encoding threads is a feature of xvidcore.dll.

[squid_80]

xvidcore handles the encoding, but xvid_encraw is responsible for feeding it frames to encode. If those frames are coming from an avisynth script with lots of filtering they can take a while to be retrieved. Each time a frame is fed to the encoder encraw fires off a new thread to fetch the next source frame. Thus when the encoder is ready to encode another frame, the source frame is already waiting in a buffer ready to go.

[kypec]

Yeah, I see now where the performance boost comes from :)
Thanks for explanations squid_80 & Brother John

[plugh]

Is altpass2 working 'ok' for people in the multi-threading environment? There was a bit of code I added in there I was concerned about - ie coordination of variable 'vbvfill_before' in routines rc_2pass2_before and rc_2pass2_after.

Things were confusing enough with the b-frame re-ordering, I wasn't sure what would happen with multiple threads and don't have a multi-core machine to test with...

And for that matter, there was another code segment in rc_2pass2_before that didn't look quite right to me with frame reordering, let alone multiple threads - the |last_quant- new_quant| <= 2 bit - but I assumed the original author(s) knew what they were doing...

[squid_80]

Is altpass2 working 'ok' for people in the multi-threading environment? There was a bit of code I added in there I was concerned about - ie coordination of variable 'vbvfill_before' in routines rc_2pass2_before and rc_2pass2_after.

Things were confusing enough with the b-frame re-ordering, I wasn't sure what would happen with multiple threads and don't have a multi-core machine to test with...

I'm pretty sure it's fine, frames are encoded one at a time and the plugin before and after functions are always executed in order.

[plugh]

Thanks for the reply; I've been somewhat following the recent 'multi-threaded xvid' posts, and now understand that the threads are used _within_ a frame's encode, so that doesn't affect the plugins.

However, at least with the 1.1.2 core, what debug prints revealed to me was that while 'after' is called in presentation order (ie IBBP), 'before' is not.

[MacAddict]

I'm impressed using this on my DualCore system for the past few months. I'm confused why the 2nd pass only appears to use one core and averages about 51% utilization though. My content isn't filtered with anything but ColorMatrix and isn't resized. First pass usually averages 99-103fps while the 2nd pass is around 12-15fps. Anyone else seeing this behavior?

[TheRyuu]

I'm impressed using this on my DualCore system for the past few months. I'm confused why the 2nd pass only appears to use one core and averages about 51% utilization though. My content isn't filtered with anything but ColorMatrix and isn't resized. First pass usually averages 99-103fps while the 2nd pass is around 12-15fps. Anyone else seeing this behavior?

I am as well. I'm wondering if this is slowing down the encoding at all or if that's just as fast as it's gonna get.

I'm pretty sure it's because xvid_encraw is only using 1 thread for xvid and the other for the frame serving or whatever it's called (which is what it's suppose to do). I'm wondering if this is slowing down the second pass?

[squid_80]

If you want to force it to behave differently, use -threads 3 (2 for encoding, 1 for reading input) or -threads 2 and -nothreadedinput. I think you'll find it slower though.

[TheRyuu]

If you want to force it to behave differently, use -threads 3 (2 for encoding, 1 for reading input) or -threads 2 and -nothreadedinput. I think you'll find it slower though.

The best balance seems to be 2 threads for the first pass, and 3 for the second. That way, both passes seem the fastest to me. (~100fps on first, ~30fps on second).

Dunno if it works the same way on quad core though.

[20-40]

It's excellent! Many thanks to Kopernikus and squid_80… Thank you.

Last week I have recompressed many music video clips from my collection, and because of their number and lack of free time, I've dropped-out any thoughts about two-pass re-encoding, but all of them must fit one DVD - and CQ was out of the picture… Reading this thread, I've played a little, and as I have been very satisfied by results, decided to go on…
This is the simple avisynth script I've used:

AVISource("G:\input.avi")
Crop(0,86,512,212)
MDeblock()
Undot()
LanczosResize(640,268)
ResampleAudio(48000)
SoundOut(output="MP3",mode=2,cbrrate=128,filename="D:\HVS.mp3",autoclose=true,showprogress=true)

Best way to achive desired final file-size was frame size.
Hvs doesn't like any blackness around, so I've cropped any borders if necessary (I have almost all clips in 512 or 576 width in 4/3 AR with borders :-( )
For batch re-encoding, I wrote this drag&drop HVS_AQxvid_enc.bat:

@rem ###################################################################################
@rem # DRAG&DROP working Avisynth script on this .bat (but...ADJUST ALL PATHS HERE!!!) #
@rem ###################################################################################
@echo off
@set fname=%1%
@set out1=%fname%_.avi
echo %fname%
echo %out1%
@rem
@rem
@rem
"C:\xvid_encraw\xvid_encraw_hvs_061119.exe" -i %fname% -avi %out1% -framerate 25 -nochromame -cq 4 -max_bframes 2 -bvhq -bquant_offset 200 -bquant_ratio 100 -hvs_aq " quant lc_lum gl_lum 2 / - theta 255 gl_lum - 3 * 4 / lc_lum - theta + 2 / - " -threads 2 -max_key_interval 250 -progress 10
@rem
@rem
@rem #####################################################################
@rem ## IF YOU DON'T HAVE or use soundout.dll plugin in Avisynth script ##
@rem #####################################################################
@rem wavi.exe %fname% - | lame.exe -m s -b 128 - D:\HVS.mp3
@rem #####################################################################
@rem ## IF YOU PREFER rather old avimux.exe instead of Avimux_GUI ##
@rem ## (un-rem following & delete lines afterwards) ##
@rem #####################################################################
@rem "C:\xvid_encraw\avimux.exe" -apre=60 -inter=100 %out1% %fname%_muxed.avi "D:\HVS.mp3"
@rem #####################################################################
@rem ## Avimux_GUI section ##
@rem #####################################################################
set PATH=E:\Progra~1\AviMux;%PATH%
setlocal ENABLEDELAYEDEXPANSION ENABLEEXTENSIONS
@rem #####################################################################
@rem ################# set IN/OUT, besides... it #############
@rem ###### must be ABSOLUTE path for mp3 from/for soundout.dll !!! ######
@rem #####################################################################
set TMPAMG=%fname%.amg
set SRCVFN=%out1%
set SRCAFN=D:\HVS.mp3
set MUXFLN=%fname%_muxed.avi
@rem ######################################################################
@rem ####### following lines create Avimux_GUI script #########
@rem ######################################################################
echo CLEAR > "!TMPAMG!"
echo LOAD !SRCVFN!>> "!TMPAMG!"
echo LOAD !SRCAFN!>> "!TMPAMG!"
echo SELECT FILE 1 >> "!TMPAMG!"
echo ADD VIDEOSOURCE >> "!TMPAMG!"
echo SET INPUT OPTIONS >> "!TMPAMG!"
echo WITH SET OPTION >> "!TMPAMG!"
echo AVI FORCE MP3VBR 0 >> "!TMPAMG!"
echo MP3 VERIFY CBR NEVER >> "!TMPAMG!"
echo MP3 VERIFY RESDLG 0 >> "!TMPAMG!"
echo END WITH >> "!TMPAMG!"
echo SET OUTPUT OPTIONS >> "!TMPAMG!"
echo WITH SET OPTION >> "!TMPAMG!"
echo OVERWRITEDLG 0 >> "!TMPAMG!"
echo CLOSEAPP 1 >> "!TMPAMG!"
echo DONEDLG 0 >> "!TMPAMG!"
echo MAXFILES OFF >> "!TMPAMG!"
echo STDOUTPUTFMT AVI >> "!TMPAMG!"
echo AVI ADDJUNKBEFOREHEADERS 0 >> "!TMPAMG!"
echo AUDIO INTERLEAVE 1 FR >> "!TMPAMG!"
echo PRELOAD 60 >> "!TMPAMG!"
echo AVI HAALIMODE 0 >> "!TMPAMG!"
echo OPENDML 0 >> "!TMPAMG!"
echo LEGACY 0 >> "!TMPAMG!"
echo RECLISTS 0 >> "!TMPAMG!"
echo END WITH>> "!TMPAMG!"
echo START !MUXFLN!>> "!TMPAMG!"
@rem #####################
@rem ## Finaly ... mux ##
@rem #####################
AVIMux_GUI.exe %fname%.amg

After that, all was piece of cake, well… it took two day for all clips, but quality is great and one pass (my time) per clip was saved.

One thing has left - 1-hour Sade concert captured from TV a long time ago. For fun, I've decided to do two pass encoding and final size should be defined… so:


@echo off
@set fname=%1%
@set out1=%fname%_.avi
Echo %fname%
Echo %out1%
@rem
@rem
@rem
"C:\xvid_encraw\xvid_encraw.exe" -i %fname% -avi %out1% -framerate 23.976 -nochromame -pass1 "xvid.stats" -size 614512 -max_bframes 2 -bvhq -bquant_offset 200 -bquant_ratio 100 -hvs_aq " quant lc_lum gl_lum 2 / - theta 255 gl_lum - 3 * 4 / lc_lum - theta + 2 / - " -threads 2 -max_key_interval 250 -progress 10
"C:\xvid_encraw\xvid_encraw.exe" -i %fname% -avi %out1% -framerate 23.976 -nochromame -pass2 "xvid.stats" -size 614512 -max_bframes 2 -bvhq -bquant_offset 200 -bquant_ratio 100 -hvs_aq " quant lc_lum gl_lum 2 / - theta 255 gl_lum - 3 * 4 / lc_lum - theta + 2 / - " -threads 2 -max_key_interval 250 -progress 10
@rem
@rem
@rem
"C:\MingW\bin\mencoder.exe" %out1% -audiofile %fname% -oac mp3lame -lameopts cbr=128 -ovc copy -o %fname%_muxed.avi

This 2-pass_1cd_HVS.bat has done all job by drag&drop simple avisynth script ( similar to aforementioned, but without SoundOut in it).
Kopernik's encraw has been overwhelmed by input and new squid's encraw has jumped in.

Thank you guys, once again.

[20-40]

Better late then never. OK, I use XviD 1.2-127 (CVS) and Avisynth 2.6 (also CVS). Somehow, I've noticed "longer-then-usual" encoding times with xvid_encraw (then it should be with reported FPS). I have, back then; hundreds of encodings on queue, all clips were short… and… I didn't care too much about time, I guess.
Then, longer encodings came… well, if someone has 90000 frames to encode, he can expect 900 seconds if reported fps by xvid_encraw was - 100 fps, right?

HOWEVER - encoding took 9000 seconds, which is almost close to three hours compared to "expect" fifteen minutes.
Then, pretty freaked out, I have compiled xvid again - with gcc, with vc71 and vc80. Then with icl 9.0 and then 9.1. Then encraw exe, turning optimizations on and off. With all these compilers. Different sources of encraw - original and squid80's. Mine. Everything stays - bogus fps on screen…

Well, next time I will fire-up stopwatch.exe, rather then seeing-believing. I humbly suggest that to all concerned.
http://free-kr.t-com.hr/zeman/xvid_encraw-fps-sw.jpg

[jethro]

Did you feed an avisyth script to Xvid_encraw? If yes, it could be that 'enctime' is the CPU time which xvid_encraw itself spent on compressing frames and the missing time comes from avisynth processing.

[squid_80]

Run xvid_encraw -h and tell me the date and time it was built. I think you're using an older version which only counts the time taken for encoding, not including time taken to fetch the input frame i.e. avisynth processing. But it was over a year ago that I changed that behaviour.

[20-40]

As I've mentioned, it's about three versions of source code of encraw. As I've said, it doesn't matter which specific version. As I've mentioned, all encraws exhibit the same error - bogus fps during and after encoding (when it reports some data). Also, it doesn't matter by which compiler is compiled - same as by who - same is valid for exes compiled by you or anyone else.
"FPS" spitted by any encoder is very valid information for anyone, in determining script/filtering options, and in the eyes of the most people, something that is looked at first.
What I'm saying is that "FPS" IS bogus, it MAY be a bogus in god knows how many cases, as it can be subject of easy manipulations.
To clarify myself - I suggest to people to check ACTUAL (as REAL) encoding times, and not to focus on ETAs or FPses alone.
But you've made only ONE zip of source available and one zip with compiled encraw and aviwriter.dll (and I love YOUR version - adding VD routines was very nice idea). There is no other "versions" of your encraw. Kopernikus made available two (but rather obsolete by now). Xvid.org cvs is one only. My own is my gcc and intel modified. Sucus is, it realy doesn't matter.
However:
1.C:\xvid_encraw>xvid_encraw_squid.exe -h >xvid_encraw_squid.txt
xvid_encraw built at 10:22:53 on Aug 31 2007

2.C:\xvid_encraw>xvid_encraw_vc71.exe -h
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampe
rt 2002-2003
xvid_encraw built at 10:08:29 on Feb 14 2008

3.C:\xvid_encraw>xvid_encraw_hvs_061119.exe -h
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampe
rt 2002-2003

xvid_encraw built at 16:38:19 on Nov 19 2006

4.C:\xvid_encraw>xvid_encraw_orig.exe -h
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampe
rt 2002-2003

xvid_encraw built at 15:38:53 on Feb 13 2008
(2. and 4. are CVS XviD.org, 2. is with added your aviwriter code and 4. is CVS but hvs modified version for my gcc-4.3 CYBORG)

P.S. It reminds me on old, built-in prank in all Windows: "Estimated time: 2 minutes left".
Two minutes later is" Estimated time: 6 minutes left". Time CAN flow in - negative. :-)

[henryho_hk]

But it was over a year ago that I changed that behaviour.

Is it now a simple (end time - start time) over the whole clip (or finished portion)? Otherwise, many factors may come in.

[Too busy on my work for the last year.... no time for on my own batch script]

[20-40]

ANY encoder is the last in the encoding chain, it means that it doesn't matter what is input. Encoding application must-should-only could report actual (REAL) number of encoded frames by second. Fluctuations during encoding are expected, as they depend on complexity of input, by filling/emptyng caches/buffers, by writing the output… just to mention a few…
Nevertheless, at least FINAL encoding time should be right, right?
BUT - it's not.
It looks simple, simple end_of_encoding minus start_of_encoding equal total_encoding_time, but it is not - as you can see from the screenshot I've provided dozen inches beforehand.
It seems that people have mixed-up few things.

In the case of XviD codec and encraw, fps is stored and represented in/as "%6.2f". It has nothing to do with avisynth. This value is reported by codec ALSO for VirtualDub, Avidemux and any other application we use. If there were an error in ANY other part of the chain (avisynth with filters, codec itself) - such error would be manifested by/in them too, right? However, it is not the case.
And - in my case "%6.2f" is %6.2f/(ARG_THREADS + 1), but it doesn't fix the error.

[squid_80]

J:\ts-temp2>copy con enc_test.bat
xvid_encraw -h 2> help.txt > nul
find "built" < help.txt
time < nul
xvid_encraw -i lost.avs -frames 50000 2>nul | find "Tot:"
time < nul
^Z
1 file(s) copied.

J:\ts-temp2>enc_test.bat

J:\ts-temp2>xvid_encraw -h 2>help.txt 1>nul

J:\ts-temp2>find "built" 0<help.txt
xvid_encraw built at 10:22:53 on Aug 31 2007

J:\ts-temp2>time 0<nul
The current time is: 21:56:51.29
Enter the new time:
J:\ts-temp2>xvid_encraw -i lost.avs -frames 50000 2>nul | find "Tot:"
Tot: enctime(ms) =5591690.00, length(bytes) = 1729539969

J:\ts-temp2>time 0<nul
The current time is: 23:30:04.39
Enter the new time:
Clock end time - clock start time = ~5593 seconds.
xvid_encraw reports 5591.690 seconds.
Using my build, as stated by the -h option.
I don't see any problems here.

[20-40]

Combination of xvidcore-vfw-ax compiled with gcc-4.3CYBORG and your source (squid80's) encraw compiled with 2003 with your's source of aviwriter dll compiled with 2005 on two cores/threads works error free!
I'm still puzzled with all this, but - everything is fine now.

However, squid80, thank you for reactions.

[Lenchik]

Hope thgis is right thread to ask.

I am trying to create a series of .bat files for batch processing of videos from my photo.
Previous .bat files create series of .avs files. For now problem is to create .bat for video encoding using xvid_encraw.
Here goes code:
setlocal ENABLEDELAYEDEXPANSION ENABLEEXTENSIONS

for %%i in (*.avs) do (
start "%%~ni - Pass 1/2" /b /wait /belownormal xvid_encraw.exe -type 2 -i "%%~dpi%%i" -pass1 "%%~dpi%%~ni.stats" -bitrate 2000 -kboost 100 -chigh 30 -clow 15 -overhead 0 -max_key_interval 100 -nopacked -vhqmode 4 -closed_gop -lumimasking -notrellis -imin 1 -pmin 1 -max_bframes 1 -bvhq -bquant_ratio 162 -bquant_offset 0 -bmin 1 -par 1:1 -threads 3 -vbvsize 3145728 -vbvmax 4854000 -vbvpeak 8000000

start "%%~ni - Pass 2/2" /b /wait /belownormal xvid_encraw.exe -type 2 -i "%%~dpi%%i" -pass2 "%%~dpi%%~ni.stats" -bitrate 2000 -kboost 100 -chigh 30 -clow 15 -overhead 0 -max_key_interval 100 -nopacked -vhqmode 4 -closed_gop -lumimasking -notrellis -imin 1 -pmin 1 -max_bframes 1 -bvhq -bquant_ratio 162 -bquant_offset 0 -bmin 1 -par 1:1 -threads 3 -avi "%%~dpi%%~ni-video.avi" -vbvsize 3145728 -vbvmax 4854000 -vbvpeak 8000000
)

The problem is that sometimes .avs files are not opened correctly (may be because of directshowsource aplied to .mov files). Anyways, i want to create script that will start first pass encoding of current .avs again and again if first attempt fails (if i am doing this manually for jobs in megui or virtualdub - it works fine), and then in same way for second pass of each .avs. How can i do it? may be some errorcheck or error output of xvid_encraw? I am not so good at .bat or at other programming to find out this by myself alone (that's why i have taken some parts of code from avs2qxvid).
By the way, any other suggestions about script are welcome.
Hope you understood my idea and my english.

[buzzqw]

use ffmpegsource and not directshowsource :)

BHH

[henryho_hk]

squid_80, would you mind making a win64 compile of the latest version? ^_^

I have just installed Vista64 SP1 and I was trying the 64bit compiles of Avisynth, xvid_enraw with 64bit XviD. With a simple script (DVD mpeg2source+tdeint+mipsmooth), they are using 1.7GB ram per xvid_encraw process. In the same machine (Q6600 + 8GB RAM) under the same OS, the 32-bit counterparts are using 512MB only. Another interesting point is the speed. At thread=1 or 5, the 64-bit combo is running at similar speed of 25~30 mins for a 18-min clip (cq3, vaq, max-b=1, vhq1, quality6, bvhq, trellis, etc.) For the 32bit combo, it's 14 mins for thread=1 and 24 mins for thread=5. Seems we still have a long way to go for 64bit platform.

[squid_80]

- The pc I use to compile xvid_encraw has been in pieces since xmas, so it's a bit hard to make new builds at the moment. But I think the 64-bit build should be up-to-date, it doesn't need to be recompiled to use new xvidcore builds.
- Always use setmemorymax when checking memory usage for programs that use avisynth scripts. Different avisynth versions may have different default memory usage limits.
- If threading in the 64-bit xvid isn't working properly, talk to the people who compiled xvidcore.dll. For me it works fine.
- 32-bit version takes 14mins with 1 thread but 24 mins with 5 threads? Again, talk to whoever made your xvidcore because that's not right at all.

[henryho_hk]

I recall the 64bit Avisynth from you was 2.55. I will try tweaking setmemorymax to see if it makes any difference. For threading, I was hoping for a 64bit xvid_encraw build of your greatly improved version with threaded input (http://forum.doom9.org/showpost.php?p=1039344&postcount=696). I would not it on the net. Perhaps I should try compiling it myself.

I am sorry that I seem to have made some typos in the speed figures in my last post (it should be something like "32-bit version takes 24mins with 1 thread but 14 mins with 5 threads"). I will test it again tonight.

[hellrasinbrasom]

Have you tried using Xvid4PSP to deal with encoding problems

:helpful:

[Kurtnoise]

I've a question about -zq vs -zw vs -zones switches : the first two are redundant, aren't they ? We can safety use the last one for the same thing, correct ?

[juGGaKNot]

Does it like 4 threads ?

I use

echo %NUMBER_OF_PROCESSORS% > "%mypath%\proc.txt"
set /p trd=<"%mypath%\proc.txt"
set /A trd2=%trd%+1
del "%mypath%\proc.txt"

And it works fine for single and dual core but on a quad xeon it does not work ( 3 threads not 5 )

[Kurtnoise]

you should read previous comments instead of asking this...

[juGGaKNot]

you should read previous comments instead of asking this...

Aham so 3 threads for all cores ?

[halsboss]

I'm confused as to which versions are available ... I read here and there about squid_80's version (whatever that is) but don't know where to download squid_80's own build ...

Also is it possible, similar to HCenc, for pass1 to produce a temporary decoded file to be used as input to pass2 so that pass2 doesn't have to process a glacially slow avisynth script a second time ?



ps go the crows, squid_80 :)

[halsboss]

Jeepers. About 30% of the time xvid_encraw crashes for me with a dialog box
"xvid_encraw has encountered a problem and needs to close."
When I click on more info it says "ModName avisynth.dll Modver 2.5.7.0 Offset 0000adbb".
Any ideas ?

Seems to happen randomly on either pass1 or pass2. Settings and commandline are

"C:\software\XVID_ENCRAW\xvid_encraw.exe" -type 2 -i "G:\HDTV\2\test.avs" -max_bframes 2 -bitrate 1500 -pass2 "G:\HDTV\2\test.stats" -stats -framerate 25 -max_key_interval 125 -asm -quality 6 -vhqmode 4 -bvhq -packed -closed_gop -lumimasking -imin 1 -bmin 1 -pmin 1 -clow 10 -overhead 0 -vbvsize 1835008 -vbvmax 8000000 -threads 3 -progress 25
"C:\software\XVID_ENCRAW\xvid_encraw.exe" -type 2 -i "G:\HDTV\2\test.avs" -max_bframes 2 -bitrate 1500 -pass2 "G:\HDTV\2\test.stats" -stats -framerate 25 -max_key_interval 125 -asm -quality 6 -vhqmode 4 -bvhq -packed -closed_gop -lumimasking -imin 1 -bmin 1 -pmin 1 -clow 10 -overhead 0 -vbvsize 1835008 -vbvmax 8000000 -threads 3 -progress 25 -o "G:\HDTV\2\test.m4v" -kboost 50

using .avs
SetMTmode(mode=5,threads=4) # start with mode=5 for source http://forum.doom9.org/showthread.php?p=1067216#post1067216
SetMemoryMax(512)
LoadPlugin("C:\SOFTWARE\DGindex\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-zzz\tdeint.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-zzz\yadifmod.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-zzz\NNEDI.dll")
mpeg2source("G:\HDTV\2\test.d2v",cpu=4,ipp=true)
AssumeFPS(25)
AssumeTFF()
SetMTmode(mode=2,threads=4)
yadifmod(mode=0,order=1,edeint=nnedi(field=1,threads=4)) # mode=0=normal deinterlace
spline36resize(640,360)
SetPlanarLegacyAlignment(True)
Distributor()

[Sharc]

@halsboss
I also experienced a number of crashes with xvid_encraw (OS is Vista). The crashes seem to be caused by the -qpel function for certain sources.
Workaround was to disable -qpel. :(

[henryho_hk]

Same experience here... 720p + QPel ... during early 2nd pass

[henryho_hk]

squid_80, may we have a x64 compile of your 20070831 version of xvid_encraw?

[MOS-Marauder]

Hi xD
Question.. i saw that XVid ENcraw writes a custom pixel shape (1:1) instead of SQUARE in the bitstream. Any chance of fixing ?

Chris

[Sharktooth]

it's not a bug.
SQUARE is NOT written... is ASSUMED.
while "Custom Shape" is a wrong term used by avinaptic that means an AR is specified in the bitstream...

in practice avinaptic ALWAYS report Square Pixels for streams that have NO AR SIGNALED while it ALWAYS report Custom Shape for ALL the streams with any signaled AR (even 1:1).
that's an avinaptic limitation.

[Gew]

Hi!

I'm just curious. As I've understood it, xvid_encraw uses the same default settings as xvidvfw.dll, hence they should output the same?

However, when I load an .avs in VirtualDubMod, and encode it with XviD/VfW, and then encode the very same .avs with xvid_enraw.exe, the output is not entirely the same. In fact, even the bitrate differs a few kbps. Am I missing out on something? I have a theory that the codec settings are in fact similar, but there's a difference in how the source (.avs) is being projected/fed into the codec, hence the non-identical results.

Is this correct?

Ty in adv~
Regards~

[Sharktooth]

VFW bitstream writing depends on the host application while encraw has bitstream writing.
also there are other differencies... so the output will always be different.

[Gew]

Ty for your clarification Sharktooth. I think I got it.
It's like comparing drinking something straight out of the bottle with from a glass.
It'll be roughly the same result, but still not bit-identical.

:thanks:

[Gew]

What's the difference between the three reported quants?
I.e. when pulling a quick 250 frames encode using -bitrate 50 I get:

I quants = 4 / 9.00 / 14
P quants = 4 / 18.03 / 25
B quants = 7 / 25.18 / 31

What are the three different values reporting? Which one is the actual "overall quantizer" being used..?
Or are the different figures three different ways of telling the exact same thing?
Or any other relationship in between them, something I'm not aware of?

Ty in adv~

:stupid:

[Brother John]

Those are minimum/average/maximum quants used for the respective frame types. Average is a calculated value from all frames; you might call this the »overall quantizer«.

Btw: Xvid quants can only be integer number (from 1–31). Whenever you encounter a fractional number it can only be some average value, but never an actual »real« quant.

[Gew]

Ty a lot Brother John.
Your explanation was well-defined and makes perfectly good sense.
Sorry for "bothering" thread with silly newbish question.
I tried Google for answers before posting but no strings gave any result.

Anyways..

:goodpost::thanks:

Cheers~

[Gew]

Hmm. Compatibility issue with recent xvidcore.dll or something?

I'm using:
xvid_encraw built at 10:22:53 on Aug 31 2007
xvid [info]: xvidcore build version: xvid-1.3.2

Just tried an encode with an .AVS for source. Final bitrate is ~half what I set.
Also, now when I watch my result, the ~2/3 bottom of the video is all just a smear.

Any ideas?


EDIT:
Confirmed! Just tried same encode using an older xvidcore.dll
xvid [info]: xvidcore build version: xvid-1.2.2
No picture/framerate issues there. Any chance for a quick fix? ;)

[Jawor]

Yes, squid_80's version doesn't work correctly with xvidcore.dll 1.3.x (it has been mentioned somewhere in other topics). The reasons lie somewhere in changes made to the Xvid source code, but AFAIK they haven't been identified (otherwise a fix would appear). squid_80's last build comes from August 2007 (that's almost 4 years), so some incompatibilities are to be expected....

[Gew]

Hi (again)...

Now it's been more than seven months. Perhaps foolish of me, but I'm still curious. Are there any versions (builds by others perhaps?) of xvid_encraw available that will handle recent xvidcore.dll without any hassle? I'd be grateful. Let's say I have my reasons why I prefer the xvid_encraw (over eg. ffmpeg/mencoder) for batch encodes.

Thank you in advance~
Regards~
Cheers~

[madhatter300871]

Hi

If you just download the latest xvid from the xvid website, it comes with xvidencraw, it works for me with no problems. All I did was copy the xvidencraw.exe and xvidcore.dll (from system32) into my own folder, it works no problems.

[kalehrl]

Does it use only one core in the second pass?

[madhatter300871]

Does it use only one core in the second pass?

I think so, but I'll double check on my next encode.

[Gew]

Hi

If you just download the latest xvid from the xvid website, it comes with xvidencraw, it works for me with no problems. All I did was copy the xvidencraw.exe and xvidcore.dll (from system32) into my own folder, it works no problems.

Gosh, I never noticed that Xvid had/has started inbedding xvid_encraw.exe in their Win32 bundles. I just downloaded latest stable (Xvid-1.3.2-20110601.exe). Haven't tried it just yet, but I doubt I will run into any problems.

The issue I had with my xvid_encraw.exe (dated 2007 or something) was that if I used an xvidcore.dll from the 1.3.x series I had severely messed up output. Half the screen was "floating" somehow. Therefor I have kept an old Xvid 1.2.x xvidcore.dll just for use with xvid_encraw. This issue is mentioned a couple of pages up in this very thread.

Anyways, big thanks.
Now I will probably be able to kick it with latest Xvid! :D

Cheers!

:thanks:

Update!

Not only did it work with excellent result; average encoding speed raised from ~77 FPS to ~107 FPS (just ran side-by-side test using same input). This (I guess) a result of my old xvid_encraw.exe / xvidcore.dll combination only used 3 out of 4 cores on my CPU (xvid [info]: Detected cpus = 4, threads requested = 3, threads in use = 3), whereas this new set simply yields "Detected 4 cpus, using 4 threads". Awesome. One thing that made me worry was that at first my AVI would not play; MPC gave me "Cannot render the file". After some further digging, I noticed that the stream was actually not wrapped into a AVI container, but more or less raw. I then noticed that the "-avi" switch puts output in AVI, whereas "-o" (which I have always used before) does this raw-isch result. Anyways, like I said, works like a charm now!

[madhatter300871]

That's great.

I checked my last encode and it does indeed use all 4 cores on the second pass, it outputs the message "detected cpus=4, threads in use=4", (or something like that).

I'm getting about 200fps on the first pass, and about 30fps on the second pass, this is with a 1920x1080 input and a 720 x whatever output. Good enough for me :)

[kalehrl]

I don't get one thing.
When I use 1.3.2 xvid_encraw.exe in MeGUI, I get around 60 fps whereas with the default 1.2.2+VAQ xvid_encraw.exe I get around 85 fps.
I tried using Jawor's, generic and encraw from xvid.ru but it is the same - always much slower that the 1.2.2.
When I encode using vdubmod and the 1.3.2 system xvid, I get normal fps (around 85).