xvid_encraw - Patched with AviSynth input support [Archive] - Page 3

holzi

19th July 2006, 13:15

I don't know if the problem was already posted coudn't find anything.
I get a strange result when encoding in 2 pass mode.
It's done by megui:
Here the log:

encoder commandline:
-i "O:\DVD-out\Temp\fight.avs" -pass1 "O:\DVD-out\Temp\fight.stats" -bitrate 1076 -kboost 100 -overhead 0 -max_key_interval 250 -vhqmode 4 -closed_gop -threads 1
successfully started encoding
Processing ended at 01:33:44
----------------------------------------------------------------------------------------------------------

Log for job job1

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

Tot: enctime(ms) =2545641.00, length(bytes) = 821595521
Avg: enctime(ms) = 12.72, fps = 78.62, length(bytes) = 4104
I frames: 2508 frames, size = 17424/43701787, quants = 2 / 2.00 / 2
P frames: 73684 frames, size = 8340/614550880, quants = 2 / 2.00 / 2
B frames: 123908 frames, size = 1318/163342488, quants = 4 / 4.00 / 4
N frames: 46 frames, size = 7/ 366
Trying to retrieve width and height from input header
Input colorspace is YV12
xvidcore build version: xvid-1.2.0-dev
Bitstream version: 1.2.-127
Detected CPU flags: ASM MMX MMXEXT SSE 3DNOW 3DNOWEXT TSC
Detected 1 cpus, using 1 threads.

----------------------------------------------------------------------------------------------------------
Job completed successfully and deletion of intermediate files is activated
Starting job job2 at 01:33:44
encoder commandline:
-i "O:\DVD-out\Temp\fight.avs" -pass2 "O:\DVD-out\Temp\fight.stats" -bitrate 1076 -kboost 100 -overhead 0 -max_key_interval 250 -vhqmode 4 -closed_gop -threads 1 -avi "O:\DVD-out\Temp\fight.avi"
successfully started encoding
Processing ended at 04:42:15
----------------------------------------------------------------------------------------------------------

Log for job job2

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

Trying to retrieve width and height from input header
Input colorspace is YV12
xvidcore build version: xvid-1.2.0-dev
Bitstream version: 1.2.-127
Detected CPU flags: ASM MMX MMXEXT SSE 3DNOW 3DNOWEXT TSC
Detected 1 cpus, using 1 threads.

Tot: enctime(ms) =5988780.00, length(bytes) = 770489159
Avg: enctime(ms) = 29.92, fps = 33.42, length(bytes) = 3849
I frames: 2508 frames, size = 17244/43249140, quants = 2 / 2.00 / 2
P frames: 73684 frames, size = 7516/553839680, quants = 2 / 2.00 / 2
B frames: 123953 frames, size = 1398/173400333, quants = 4 / 4.00 / 4
N frames: 1 frames, size = 6/ 6
desired video bitrate of this job: 1076 kbit/s - obtained video bitrate (approximate): 774 kbit/s

Why doesn't encraw use the hole bitrate of 1076?

My avs script I used:

DGDecode_mpeg2source("O:\DVD-out\Temp\VTS_01_1.d2v",info=3)
ColorMatrix(hints=true)
#Not doing anything because the source is progressive
crop( 0, 74, 0, -76)

LanczosResize(576,240) # Lanczos (Sharp)
Undot() # Minimal Noise

foxyshadis

19th July 2006, 14:10

I frames: 2508 frames, size = 17244/43249140, quants = 2 / 2.00 / 2
P frames: 73684 frames, size = 7516/553839680, quants = 2 / 2.00 / 2
B frames: 123953 frames, size = 1398/173400333, quants = 4 / 4.00 / 4

You're already maxing out the codec quality, anything more is just throwing bits away. Assuming you'd rather increase quality than simply add a hundred megs of padding, at the very least use MPEG matrix, or check out the xvid presets thread and load the presets into megui. Use one of the higher ones.

MatMaul

22nd July 2006, 00:33

lol I ever have problem with the output of xvid_encraw.

If I use the direct mkv output, problem to play the file in vlc (big macroblock on the video).

If I do m4v->mp4->mkv, no problem with vlc BUT problem with the ordered chapters function in mkv,an example : I want to display the first 30s of a video, then I want to see the last 30s of the video (the video is longer than 1mn) and I do that with ordered chapters (=>the timeline have a length of 1mn and the real video have a length > 1mn). No problem with the first 30s, but when the player seek to the last 30s of the video, big freeze of the video.
I have test with some couple of mp4box/mkvmerge, same problem.
And I don't have this problem with mkv direct output.

any idea to have a correct display in vlc and ordered chapters function ?

Do you think anyone can modify mkv output to have a good picture in vlc, or it's a vlc problem ?

Thanks !

Adub

29th July 2006, 04:18

Yo, Squidy, whats with the new version that you updated today?

bug fixes, enhancements, what?

squid_80

29th July 2006, 06:11

Neater error reporting and dynamic linking with xvidcore.dll.

MatMaul

29th July 2006, 10:50

@squid : any news for my mkv problem in the new version ?

squid_80

29th July 2006, 11:38

Is your problem mainly that the mkv output from xvid_encraw is unplayable in vlc? What if you use mkvmerge to remux it to a new file?

MatMaul

29th July 2006, 12:09

the mkv output is playable in vlc but with big and horrible macroblocks. Same problem if I remux the mkv in a new mkv with mkvmerge.

If I do m4v->mp4->mkv, I don't have this problem but an other problem, but I think this new problem is related to mkvmerge (I have report the problem in the mkvtoolnix thread).

squid_80

29th July 2006, 13:20

Are you using GMC on your clips?

MatMaul

29th July 2006, 14:51

no.
I use the preset 58hq :
pass1 : xvid_encraw.exe -i video.avs -type 2 -pass1 xvid.stats -max_key_interval 250 -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -vhqmode 1 -qtype 1 -qmatrix Didees-SixOfNine.cqm -nopacked -quality 5 -nochromame -turbo -zones 0,q,3,O
pass2 : xvid_encraw.exe -i video.avs -type 2 -mkv video.mkv -pass2 xvid.stats -bitrate 2500 -max_key_interval 250 -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -vhqmode 4 -bvhq -qpel -qtype 1 -qmatrix Didees-SixOfNine.cqm -nopacked -imin 2 -imax 4 -bmin 2 -bmax 5 -pmin 2 -pmax 5 -quality 6 -chigh 10 -clow 3 -ostrength 0 -zones 0,w,1,O

squid_80

29th July 2006, 15:18

VLC seems to be buggy in the way it handles native mpeg4 in mkv. Any advanced options such as qpel, gmc, mpeg quantization etc. seem to produce artifacts. I've checked by converting other avi files (not created with xvid_encraw) to mkv with mkvmerge using --engage native_mpeg4 and they show the same errors when played in VLC.

MatMaul

29th July 2006, 15:59

But do you know why it works good if I do m4v->mp4->mkv ?
I have report this problem in the videolan forum and the videolan dev list with a sample but no answers.

squid_80

29th July 2006, 17:06

When you convert from mp4 to mkv do you use --engage native_mpeg4?

MatMaul

30th July 2006, 00:22

When you convert from mp4 to mkv do you use --engage native_mpeg4?

No I don't use --engage native_mpeg4 but mp4 uses raw mpeg4 data, without any hacks, so I think when I remux a mp4 to mkv, it uses native mpeg4 data.
I test --engage native_mpeg4 tomorrow to be sure.

squid_80

30th July 2006, 06:10

When I tested mp4->mkv using mkvmerge, it gave me a vfw mkv file (with the frames in the wrong order around I frames - I believe bond reported this long ago). When I used --engage native_mpeg4, it gave me a native mkv file but there was something strange; the VOL information was appended to each P frame preceeding an I frame. In a native mpeg4 matroska file that information should only occur once in the CodecPrivate section. If I had to guess, I would say VLC isn't reading this at all and is relying on the information being present in the stream which is the wrong way to do it.

MatMaul

30th July 2006, 15:53

OK after test you are right, mkvmerge use vfw mode if I do m4v->mp4->mkv, so the problem is vlc with native mpeg4 stream.

Can you post this technical details in this thread (http://forum.videolan.org/viewtopic.php?t=21189) on videolan forum please ?

EDIT : an other question, how do you compile xvid_encraw with mkv support ?

I have all the dependencies but I don't know how compile a c file with c++ file included in it.

kypec

31st July 2006, 11:11

...and dynamic linking with xvidcore.dll.

Sorry, but what exactly does it mean to me as a plain user of xvid_encraw?
Do I still need to install some XviD build (like 1.2) separately?
@squid_80
BTW, can you confirm that this download link of dunstan's (http://okejl.dk/dunstan/xvid_encraw.zip) contains the most up-to-date release of your builds?
I'm asking this because your FTP server is not working again :mad:

P.S. thanks for your great effort put into this CLI of XviD
I just like to write my scripts/batches once and use them million times:cool:

squid_80

31st July 2006, 11:16

EDIT : an other question, how do you compile xvid_encraw with mkv support ?

I have all the dependencies but I don't know how compile a c file with c++ file included in it.
Compile it as a c++ file. :) You'll also need to link it with libmatroska and libebml.

Sorry, but what exactly does it mean to me as a plain user of xvid_encraw?
Do I still need to install some XviD build (like 1.2) separately?Yes, it just means it should work properly with any version of xvidcore.dll. You still need it installed from somewhere.
BTW, can you confirm that this download link of dunstan's contains the most up-to-date release of your builds?
Yes.

MatMaul

31st July 2006, 11:53

thanks !

bond

6th August 2006, 15:33

mod

22nd August 2006, 12:30

I'm trying to make a change but I can't find (or maybe I'm blind ok..) the source files needed by matroska.h, could anyone please give me a link to an updated page with all source files?
Thanks!

squid_80

22nd August 2006, 13:30

Get libmatroska and libebml from matroska.org.

mod

22nd August 2006, 13:41

Thanks A LOT :)

squid_80

25th August 2006, 09:38

MatMaul: native asp support in videolan is buggy

vlc is not able to play correct native asp files created by xvidencraw or haalis muxer (yeah, xvidencraw creates correct native asp files!!!).
it also doesnt play native asp files created by mkvmerge (via .mp4 source (avi/raw source doesnt work))

in all cases vlc shows artefacts. its a known problem i have told robux4 a long time ago already, but which was never fixed
Despite certain people in the videolan forums declaring asp-in-mkv "proprietary" and "non-standard", VLC has been patched and when I downloaded the latest nightly snapshot it played my test files without problems. So much for having to petition all the major players and codec vendors blah blah blah...

bond

25th August 2006, 17:33

great :)

Brother Darrell

26th August 2006, 11:15

How about a quick post on proper commandline syntax using Xvid_encraw? So far, the only thing I have found are snippets I have copied from Bond or Squid or anyone else who seemed to be getting ANY result different from mine.
I spent a couple of hrs trying to determine why my command line would only do a 1 pass 4quant encode with b-frames. (No b frames in the stats file). As it turns out, it's because I was using "-type 2" before the avs file instead of after it. So I think a small guide on the proper syntax for the command line would be great.

Edit: Duh!!...If I had only read past 2 more threads, I would have found it...You would have thought I could have narrowed it down a great deal using the search funtion for the forums but nooo, As much as I love this site, I think the search function for the forums Is WAY too limited..(Speaking of syntax).

MatMaul

27th August 2006, 00:09

Despite certain people in the videolan forums declaring asp-in-mkv "proprietary" and "non-standard", VLC has been patched and when I downloaded the latest nightly snapshot it played my test files without problems. So much for having to petition all the major players and codec vendors blah blah blah...

yes the problem of blocky images is solved, but I actually can't seek in the file, and the length of the video isn't recognized.

squid_80

27th August 2006, 03:34

As it turns out, it's because I was using "-type 2" before the avs file instead of after it.
Can you elaborate a bit? Like what was the failing command line and what you ended up using to make it work? There is a *very minor* bug with -type but I don't know how it would cause a lack of b frames.
yes the problem of blocky images is solved, but I actually can't seek in the file, and the length of the video isn't recognized.
Remux the file. I never fixed encraw to write cues correctly which probably explains the lack of seeking, I don't know why the duration is missing as well.

MatMaul

27th August 2006, 12:15

ok thanks, no problem after remux.

bond

27th August 2006, 16:55

are cues mandatory for seeking?

squid_80

27th August 2006, 18:46

I don't think so, they just help.

bond

27th August 2006, 18:51

I don't think so, they just help.so vlc is not yet perfect?

Brother Darrell

28th August 2006, 04:55

Can you elaborate a bit? Like what was the failing command line and what you ended up using to make it work? There is a *very minor* bug with -type but I don't know how it would cause a lack of b frames.

Sorry. I've tried to reproduce the problem. Now I can't. Damn thing won't break now that I want it to. I can tell you I think it was an earlier version of encraw I was using.(From February I think). Once I got the syntax correct & installed a newer version, I was good to go. Sorry man.. I will continue to try & break it though. :D

Brother Darrell

29th August 2006, 07:59

Uh oh. My file came out great but with absolutely no header info. I figured out how to use mencoder for the fourcc and bitrate issue (actually, mencoder took care of the bitrate issue). But it says my resolution is 0x0. How do I tell it W/H of the video? I thought encraw did it but obviously not. this is my command line:

mencoder.exe "movie1.m4v" -ovc copy -mc 0 -noskip -ofps 23.976 -ffourcc XVID -o "test.avi"

I have tried aspect=16/9 I have tried -ow 720 -oh 432. I have also tried verbal abuse & bribery. What's the key?

squid_80

29th August 2006, 08:05

Are you trying to use mencoder to mux an elementary stream into avi? Why not just output .avi directly from encraw?

Brother Darrell

29th August 2006, 10:14

Tried that as well. I have played with it enough , so I think I have down the syntax. MediaInfo sees everything I have added (info-wise) except the width & hieght. Avi is seen as 0x0 as well. I can't imagine this being a bug. No one else is complaining. So it has got to be my command-line. Recoding with the following commandline using encraw:

"xvid_encraw.exe " -i "D:\Blah.avs" -quality 6 -vhqmode 1 -imin 2 -imax 31 -pmin 2 -pmax 5 -overhead 650 -framerate 23.976 -pass2 "D:\blah.pass" -qmatrix "D:\hvc matrix.txt" -w 720 -h 432 -par 5 -o "D:\movie1.m4v" -size 2073344 -progress 5

squid_80

29th August 2006, 10:25

So if you use -avi "d:\movie1.avi" instead of -o "d:\movie1.m4v", the avi file has dimensions of 0x0?

mod

29th August 2006, 10:31

I think the line must be:

"xvid_encraw.exe " -i "D:\Blah.avs" -type 2 -quality 6 -vhqmode 1 -imin 2 -imax 31 -pmin 2 -pmax 5 -overhead 650 -framerate 23.976 -pass2 "D:\blah.pass" -qmatrix "D:\hvc matrix.txt" -w 720 -h 432 -par 5 -o "D:\movie1.m4v" -size 2073344 -progress 5[/QUOTE]

squid_80

29th August 2006, 11:01

-type 2 is implied if the input file has an .avs or .avi extension. Width and height should be retrieved automatically, but I think if you manually set them smaller than the actual values the output will be cropped (don't set them bigger, I can't remember what will happen but it certainly won't enlarge the image to fit).

mod

29th August 2006, 11:19

I prefer to set -type 2 ("I sleep better" ^^).
Hmm.. In my experience it has never failed getting the correct dimensions of the video, so I don't see any reason to use it, unless, as you just wrote, to perform a crop (if this is what happens btw). Maybe it's a bit slower (dunno) but I wouldn't mind of that.

squid_80

29th August 2006, 11:43

Mainly -w and -h are for setting width and height when the input is raw yv12 (or stdin).

mod

29th August 2006, 11:46

Ok, didn't know (I never used it with that input). Thanks for the info.

Brother Darrell

29th August 2006, 16:05

AS far as an AVI file, that is correct, at least on my (very short) test clip. I am trying again (m4v) with the command line above. It may very well be I'm using the wrong switches. So I'm looking as always for proper commandline syntax. Is there anything else that would build the header for me, other than mencoder?
And isn't the output I'm producing a raw file? I thought it was as my very 1st attempt gave me NOTHING for header information(mediainfo).

Edit: Well, I got a file which so far seems to be correct in all aspects. MediaInfo sees the dimensions & par correctly. I haven't run it thru mencoder yet but I suspect it will see the info as well. I sometimes tend to get impatient & stop the encode simply to view the results. This may be my issue all along. Mencoder & MediaInfo won't see it for a very good reason. I broke the file.
(Plausible..yes?)

squid_80

30th August 2006, 01:39

If you forcibly close encraw before it is finished then the avi file will be broken.

henryho_hk

3rd September 2006, 07:15

The xvid_encraw compile at http://celticdruid.no-ip.com/xvid/ seems to be the most recent (Jul 06) but the result are always changed to 25fps.

squid_80

3rd September 2006, 07:47

You're right; in my builds I made the default framerate 0 so it would report an error instead of assuming 25fps, since I found this to be undesirable behaviour. The code in xvid's CVS has reverted to a default of 25fps.

henryho_hk

3rd September 2006, 12:01

squid_80, shouldn't it detect the framerate from the source?

squid_80

3rd September 2006, 12:15

It should, but making 25fps the default breaks it. The intended mode of operation (as I coded it) is like this:

Set framerate=0 (default)
Check command line for -framerate parameter. Adjust framerate to given value if found.
If framerate==0, set it to the framerate of the input source if possible (not possible if input is stdin or pgm).
If framerate==0, show an error and abort.

henryho_hk

3rd September 2006, 12:38

Yeah, that version crashes if I put -framerate 0

Brother Darrell

6th September 2006, 04:44

So far I have managed to create several -avi files with no issue. Still can't get the W/H to be seen in mencoder if original file is a raw stream. BUT!, moving on...Avi is wonderful. I LOVE the fact I can do multiple 1st passes WHILE doing a second pass. :D ..
I am an n-pass junkie, sad but true.
A question though. I don't see an option for B frame sensitivity. Is there such an option & I am just missing it?

kurt

6th September 2006, 04:54

it's part of the zones, eg
-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG
--> first zone got a bframe sens. of 25, 2nd 0, 3rd 25 again...

Brother Darrell

6th September 2006, 16:17

-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG

I don't know this switch.
so...-zones
1=the zone or starting frame
w(weight) ,(ok to put a delimiter here?).05,250 (I noticed you used an 'oh', instead of 'zero' but it is zero, correct?)
And to use it globally, I simply change the weight to 1 and leave it as is. -zq I guess will work as well as -zw.
Thanks! I'll give it a try.
Oh. 1 more thing...Why is -stats suppressed (if being written to a log file) if the -progress switch is used?
Like this: -stats>> "stats.log" -progress 10

Same result regardless of where the -progress switch is located. ie: before the -stats switch or after the -stats switch..doesn't matter. It will still not output. Take the -progress switch out though & my log file is produced.

squid_80

6th September 2006, 16:55

The -zones parameter works like described here: http://forum.doom9.org/showthread.php?p=702759#post702759
except there's no parameter for the end frame; a zone runs until the next zone or the end of the stream. In kurt's example it is in fact the letter "O", for chroma optimizer.
As for the stats/progress thing... -progress turns off all frame-by-frame output, otherwise there's not much point in having it (unless stdout is redirected, like you're doing). If you really want the progress update and stats to happen at the same time I guess I could add an optional filename parameter for -stats, to specify the file to output to.

Brother Darrell

7th September 2006, 03:31

The only reason I was wanting a stats output was because of the PSNR values. It's just something for me to play with, not something essential.
If it's not alot of trouble to do so then YES! I would LOVE to have that feature incorporated! I would buy you many cold beers for the effort! That is, if you are open to such blatant bribery.

:thanks:
To you & Kurt for the info about zones. The only switches I know are the ones from -help.
Are there any other switches I can't get from the -help option?

Brother Darrell

8th September 2006, 16:49

I found them listed in one place.
For anyone else is having an issue finding the parameters not listed with -help..
look here->http://forum.doom9.org/showthread.php?p=784465#post784465
It includes syntax and options.

Brother Darrell

11th September 2006, 04:16

Is there an alternate way to set chapter points on the 1st or 2nd pass without using the -zones parameter to force an I-frame? Or editing the pass file directly?

Lets say I have 40 chapter points. That's ALOT of typing!
So I was hoping there was an easier way, such as feeding (porting?) a text file with the frames needed, into the command line, forcing an I-frame where I think I need it.
I realize MkvToolnix can mux an xml or ogg file into the finished product..but what if the chapter point falls on a b-frame? Heck...what if I just want to split the stream at the chapter point(s)?
For now, I'll either do it with the -zones or directly edit my pass file.
I can't help but think there must be (or should be) an easier/proper way. Too much to hope for?

Sharktooth

11th September 2006, 04:24

The chapter start is always set at the beginning of a new scene and the codec will almost likely detect the scene change and place an I-Frame.
MeGUI supports chapters creation and muxing, so it should be ok as soon as this version of xvid will be supported (not until the patches go official...)

Brother Darrell

13th September 2006, 04:21

Does anyone recognise this specifically?
.0 0. ax + + 1 - bn

I was running a batch file using Encraw with the following command line:

xvid_encraw -i "Versus.avs" -pass2 "5th.pass" -max_key_interval 0 -qmatrix matrix.txt -framerate 23.976 -max_bframes 120 -bquant_offset 000 -bquant_ratio 162 -imin 2 -imax 3 -pmin 2 -pmax 3 -bmin 3 -bmax 5 -nopacked -quality 6 -vhqmode 4 -avi "VsBframes2.avi" -size 1706540 -pass1 "6th.pass" -stats>>"6thpass.stats" -progress 10.

I started running it in a command window when I left home this morning. When I got home, the file was finished, but the above er..characters appeared immediately afterwards, as though it was part of my original command line, with the exception of course of XP not recognising anything as a batch process or command...The last part however, -bn, did seem to process, at least XP didn't say "not recognized as an internal or external command,operable program or batch file".
Any clues? Did I break something?

Edit: sorry, the matrix file I used was HvsBest.txt under the name Matrix.txt

squid_80

13th September 2006, 05:54

If the command window has focus while processing and you type something, it will appear at the c:\> prompt when processing finishes. Is that what happened?

Brother Darrell

13th September 2006, 15:52

That's entirely possible. I didn't think about that. So something I was running at the time may have simply dumped garbage. Maybe something that happens all the time but would normally not be seen. That sounds reasonable. I was concerned because it looked a little like code. As far as me typing something while the command window has focus, It certainly would not have been that.

henryho_hk

14th September 2006, 03:16

was that something in your batch file?

Brother Darrell

14th September 2006, 04:00

Not explicitly. I don't even recognize the -bn as a parameter I use for anything. This all appeared after the batch completed.
I would have blown it off as a catwalk over the keyboard, except my keyboard was on top of my monitor. My cat is FAR too old to jump 5 feet onto the top of my monitor, land on the keyboard, type something for the ax register and jump off without knocking down the keyboard.
(Ok, maybe she can...but there is still the fact she never passed her computer basics class...waste of money, that).
Hell.. I don't know what it was. But no harm seems to have come to my file or computer. So I ain't gonna sweat it. I just wanted to make sure it didn't have to do with the batch file doing something Xvid_encraw didn't like.

henryho_hk

18th September 2006, 14:18

Is there an alternate way to set chapter points on the 1st or 2nd pass without using the -zones parameter to force an I-frame?

I have made a batch file implementing Teegedeck's quality presets as well as zone input thru a plain text file. Please refer to http://forum.doom9.org/showthread.php?p=874332#post874332
BTW, there is basically no documentation :p :p :p .

henryho_hk

26th September 2006, 03:30

Champs

27th September 2006, 19:01

Hi, first of all thank you squid_80 for taking over and improving this cli program, I really like it a lot.

I've spent a few days going through every single page of this thread and found several posts by squid_80 and others about encoded outputs set to 25fps without setting the fps explicitly regardless of the what the input avisynth script is serving. I read them but I can't really grasp what is really implied.

My question is, is it simply not possible to change the source code so that the default output framerate is the same as what the avs is serving, or will such change be implemented in the future? I guess I have no problem with preparing a set of batch scripts, each with different fps set explicitly and also sorting avs scripts into different folders by framerates, but I just feel xvid_encraw.exe should be able to do what x264.exe can do.
(I know next to nothing about programming so please forgive me if I'm saying something stupid.)

Apart from this I have nothing to say and want to express my deepest gratitude!

squid_80

27th September 2006, 19:18

squid_80, I found that zone quant. settings seem to be reseting the b-frame max-consec., ratio and offset settings. I need to put the zone parameters before the b-frame parameters so as to use something like 1/1.62/0. Is it intended?
Not sure if it's encraw related or xvidcore, can you give me a complete example command line and brief description of what output you expect?

I've spent a few days going through every single page of this thread and found several posts by squid_80 and others about encoded outputs set to 25fps without setting the fps explicitly regardless of the what the input avisynth script is serving.
My builds of xvid_encraw set the output framerate based on the input. xvid_encraw builds made from xvid's cvs code are forced to 25fps. My latest available compiles should be available from http://members.optusnet.com.au/squid_80/

I may not be available in this forum for a while, so please have patience if reporting problems.

Champs

28th September 2006, 15:05

squid_80, thank you so much for a quick reply! I was using the 12th of July version that I got off this thread and having the afformentioned trouble. Just downloaded your 13th of Sept version now and tried it. Yes it does indeed take the correct fps from the input avs! I'm sincerely sorry for not having been aware of this more recent version, and thanks again!!

squid_80

4th October 2006, 18:34

I have made a new build available for testing: http://members.optusnet.com.au/squid_80/xvid_encraw.zip

Changes:

-o can be used for avi and mkv files, it decides what format to write based on file extension.
-stats takes an optional filename parameter, to specify a file to write the stats to. I guess if you want no output at all (no progress display or stats) you can use -stats nul.
The avi writing code is all new (borrowed from virtualdub), this should fix the > 2GB problems people were having. It may contain large bugs, particularly if you run out of space.
MKV writer interface was changed a bit, should be transparent (but may have introduced bugs).
If ctrl-c is used to terminate the program it should exit gracefully (output files should be flushed, statistics shown).

The new avi writing code is a big change, if there are any problems with the avi files it creates please me know ASAP.

Keep aviwriter.dll with xvid_encraw.exe, so if I change it (likely) it can be overwritten without too much hunting around on your hard drive.

If you get errors about missing msvc dlls, get the vs2005 redistributable pack from here:
http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=32BC1BEE-A3F9-4C13-9C99-220B62A191EE
I know it's big (2.6mb) but most people will probably already have it and if you don't you'll need it sooner or later, unless you never plan on installing any new software.

@henryho_hk: Any news on your problem? I can't see anything wrong with it here. The numbers for -bquant_ratio and -bquant_offset have to be multiplied by 100 i.e. 162 for 1.62, that's about the only thing I can think the problem would be.

henryho_hk

5th October 2006, 01:23

@henryho_hk: Any news on your problem? I can't see anything wrong with it here.

I cannot reproduce the problem again. I think I had done something wrong in the parameters.

henryho_hk

8th October 2006, 14:02

squid_80, AVIMux is complaining on the output avi files when the framerate is not an integer.

xvid_encraws 20060621 gives:

Version: 1.17.7, Aug 8 2006
-----------------------------------------------
AVI-Type: standard
Writing-App : n/a
Title : n/a
resolution : 640x480
FourCC : xvid
biCompression value : XVID
framerate (strh): 29.97 fps
framerate (avih): 29.9697 fps
number of frames : 501 (0:00:16.717)
thereof keyframes : 8
thereof deltaframes : 493
thereof dropped frames : 0
strh[0].dwLength : 501

MainAVIHeader.dwTotalFrames : 501
real value : 501
suggested buffer size : 7809

Flags set: : 0x00000810
AVIF_HASINDEX
AVIF_TRUSTCKTYPE
size of video stream : 208,705 bytes
video data rate : 12 kByte/s

xvid_encraws 20061005 gives:

Version: 1.17.7, Aug 8 2006
-----------------------------------------------
AVI-Type: standard
Writing-App : n/a
Title : n/a
resolution : 640x480
FourCC : xvid
biCompression value : XVID
framerate (strh): 29.97 fps
framerate (avih): 29.9706 fps
The framerate value has appearently been written by a broken program!
number of frames : 501 (0:00:16.717)
thereof keyframes : 8
thereof deltaframes : 493
thereof dropped frames : 0
strh[0].dwLength : 501

MainAVIHeader.dwTotalFrames : 501
real value : 501
suggested buffer size : 0

Flags set: : 0x00000110
AVIF_HASINDEX
AVIF_ISINTERLEAVED
size of video stream : 208,705 bytes
video data rate : 12 kByte/s

squid_80

8th October 2006, 14:48

If you use virtualdub to do the processing, it'll say the same thing about the output file. Basically it's being picky; the framerate in the stream header(strh) is specified as a rational number (rate/scale) while the framerate in the avi file header is given in Microseconds per frame. 29.97fps is normally represented as 30000/1001 in the stream header, which gives 0.0333666.... microseconds per frame. Virtualdub seems to truncate this value rather than rounding it and avimux_gui complains because it's not as accurate as possible. But applications shouldn't be using the avih framerate value anyway since the strh framerate is more accurate.

Blue_MiSfit

12th October 2006, 00:01

Hey everyone...

So, I haven't ever trid XviD through MeGUI before, but I tried giving it a whirl last night and came up with some problems.

Isolated it to the CQM. I remember some talk of xvid_encraw not supporting CQMs, is that the case?

I got the latest xvid_encraw from squid80's optusnet site, pointed MeGUI at it, and built a profile that matches my usual high bitrate virtualdub settings:

C:\Program Files\megui\tools\xvid_encraw\xvid_encraw.exe -i "F:\Movies Work\City of God\God.avs" -pass2 ".stats" -bitrate 700 -kboost 100 -overhead 0 -max_key_interval 240 -nopacked -vhqmode 4 -qpel -qmatrix "D:\XviD Matricies\Sharktooth's Matricies\eqm_v3hr (gp hb).xcm" -closed_gop -lumimasking -max_bframes 3 -bvhq -threads 1 -avi "F:\Movies Work\City of God\God.avi"

script:

LoadPlugin("D:\Archives\Video Software\AviSynth Plugins\Common\RemoveGrainSSE3.dll")

DGDecode_mpeg2source("F:\Movies Work\City of God\VTS_01_1.d2v",info=3)
ColorMatrix(hints=true)

RemoveGrain(mode=5)

It's automated 2 pass, so the bitrate should get overridden..

Anyway, when I try to encode my AviSynth script that works perfectly in virtualdub or x264.exe, it crashes... I get the unexpectedly closed box, and it references xvidcore.dll.

Maybe my xvidcore.dll is out of date? It's 856,064b and in the same folder as xvid_encraw... Am I just really n00bing it and totally missing some crucual step here?

Sorry if I'm asking about a totally common thing, I did some search and didn't find anything..

Also tried a 1 pass CQ encode and had the same problem.

Suggestions?
~MiSfit

henryho_hk

12th October 2006, 01:43

Why is "-overhead 0"? And I think XviD wants us to do a pass1 and then 1 pass2 manually.

squid_80

12th October 2006, 04:46

Isolated it to the CQM. I remember some talk of xvid_encraw not supporting CQMs, is that the case?

I use Sharktooth's cqms a lot with no problems. Does xvid_encraw give any text output at all before it crashes?

Blue_MiSfit

12th October 2006, 06:04

Well the megui log says:

Generating jobs. Desired size: 2306867200 bytes
No audio encoding. Calculating desired video bitrate directly.
Encoded audio file is present: F:\Movies Work\City of God\VTS_01_1 T01 3_2ch 448Kbps DELAY 0ms.mp4 It has a size of 228491910 bytes.
Setting video bitrate for the video jobs to 2131 kbit/s
Setting desired size of video to 2074358784 bytes
Starting job job1-1 at 9:03:29 PM
encoder commandline:
-i "F:\Movies Work\City of God\God.avs" -pass1 "F:\Movies Work\City of God\God.stats" -bitrate 2131 -kboost 100 -overhead 0 -max_key_interval 240 -nopacked -vhqmode 4 -qpel -qmatrix "D:\XviD Matricies\Sharktooth's Matricies\eqm_v3hr (gp hb).xcm" -closed_gop -lumimasking -max_bframes 3 -bvhq -threads 1
successfully started encoding

and then I get the generic "xvid_encraw.exe has encountered a problem and needs to close..." error box.

~MiSfit

Brother John

12th October 2006, 12:35

You isolated it to the CQM? Then with a standard matrix everything works fine?

Corrupted XCM file? Can you use it with XviD VfW?
Just to rule out path name problems: Try a path with only letters and no spaces like "D:\eqm_v3hr.xcm". Spaces shouldn't be a problem, but for some weird reason the ' might.
Which version of xvid_encraw do you use? Run "xvid_encraw.exe -help" to find out.

squid_80

12th October 2006, 15:12

Probably easiest to just post the matrix file, or a link to it. encraw checks to make sure it's 128 bytes long but that's about all it looks for. There's no string manipulation on the filename so spaces and slashes shouldn't be an issue. (Yes it freads in binary mode, in case anyone wondered.)

Blue_MiSfit

13th October 2006, 05:30

My CQM definately works in VFW XviD.

I'm using the October 5 2006 build (from 'xvid_encraw.exe -help')

http://rapidshare.de/files/36538364/eqm_v3hr__gp_hb_.xcm.html

Pretty sure it's not the matrix itself, as it works in vfw just fine..

Tried again, with a no space name (called it eqm_v3hr.xcm and put it in the root of my D: drive...

Very strange indeed..

squid_80

13th October 2006, 06:51

There's something wrong with that matrix file. I started xvid vfw's config and went to the custom matrix section, when I loaded your matrix it showed a 121 and 0 at the end of the second row of the inter matrix. When OK is selected the VFW config corrects the 0 into a 1, xvid_encraw isn't that smart so it crashes.

Blue_MiSfit

13th October 2006, 21:30

How goofy.

I will re-download Sharktooth's CQM pack. Thanks for the help squid_80...

Did that and everything works perfectly. Nice bit of code gentlemen!! No more vdub encoding for me :)

~MiSfit

MetalPhreak

23rd November 2006, 23:08

I either found a bug or I'm doing something wrong:
Using the latest encraw by squid_80 I cannot specify a negative b-frame sensititvity in -zones. Eg. -zones 1000,q,2,-20 the "-" gets ignored and encraw uses the integer entered as a positive value.
Any help would be greatly appreciated.

--EDIT--
Maybe I should also add this is with XviD v1.1.2.
Also I'm going away tomorrow so I won't be able to reply for about a week (if more info should be needed)

henryho_hk

24th November 2006, 02:16

Try inserting e.g., "K" (start with keyframe), O (chroma optimizer), etc. and see if it is parsed correctly.

-zones 0,q,3,KO-3/1,q,3,KO-10

squid_80

24th November 2006, 04:10

Using the latest encraw by squid_80 I cannot specify a negative b-frame sensititvity in -zones. Eg. -zones 1000,q,2,-20 the "-" gets ignored and encraw uses the integer entered as a positive value.

I can't reproduce your results (i.e. it works fine here), can you post your full command line in case it's something else?
G:\vid1>xvid_encraw -i sources.avs -frames 200 -progress
I frames: 3 frames, size = 43890/ 131671, quants = 4 / 4.00 / 4
P frames: 76 frames, size = 24035/1826677, quants = 4 / 4.00 / 4
B frames: 121 frames, size = 5087/ 615613, quants = 7 / 7.00 / 7

G:\vid1>xvid_encraw -i sources.avs -frames 200 -progress -zones 0,q,4,-20
I frames: 3 frames, size = 43890/ 131671, quants = 4 / 4.00 / 4
P frames: 124 frames, size = 20997/2603670, quants = 4 / 4.00 / 4
B frames: 73 frames, size = 3940/ 287642, quants = 7 / 7.00 / 7

MetalPhreak

24th November 2006, 08:28

Try inserting e.g., "K" (start with keyframe), O (chroma optimizer), etc. and see if it is parsed correctly.

Already tried that, eg. forcing keyframes work but same result with b-frame bias.

I can't reproduce your results (i.e. it works fine here), can you post your full command line in case it's something else?

xvid_encraw -single -cq 2 -zones 50,q,2,K-30 -quality 6 -vhqmode 4 -bvhq -qpel -gmc -qtype 0 -max_bframes 2 -bquant_ratio 100 -bquant_offset 100 -i "C:\GARDEN_STATE\VIDEO_TS\gs.avs" -mkv "F:\xvid_encraw\test.mkv" -max_key_interval 250 -par 64:45 -progress 10

--EDIT--
Nevermind, just tried again and now it works perfectly. I was probably half asleep the last time I tried and messed something up. Thanks henryho_hk and squid_80 for your help anyway.

Adub

6th December 2006, 03:18

Yo, Squidy!
Any new developments in the works? Some crazy optimizations or insane quality modes that blow x264 out of the water? ;p

squid_80

6th December 2006, 04:28

encraw's only a frontend for xvid. If you want some new options to play with, check this thread: http://forum.doom9.org/showthread.php?t=114811

henryho_hk

6th December 2006, 06:55

squid_80, it seems both the stat file output and the AVI output routines are quite stable. Will you consider sending the source patch to xvid-devel?

SeeMoreDigital

6th December 2006, 11:12

Hi guys....

Can somebody please confirm where we got to concerning XvidEncRAW and Packed Bit-stream.

Can XvidEncRAW be configured to generate MPEG-4 part-2 streams with PBS. And if so, is it really neccessary to include such an implementation?

Cheers

squid_80

6th December 2006, 12:10

SeeMoreDigital

6th December 2006, 12:29

Default setting with .avi output is to use packed bitstream, if there is no .avi output no packed bitstream.
It is also possible to force it on/off with -packed/-unpacked. This does not affect matroska output.I see... So when generating and outputting RAW "MPEG-4_ASP.M4V" streams, there is no packed bit-stream... Excellent :)

Pardon my ignorance... But may I ask why it was felt necessary to develop XvidEncRAW to include PBS at all, even when outputting to AVI?

squid_80

6th December 2006, 13:30

AVIs without packed bitstream are hard to edit due to the lag between the frame being displayed and the frame being decoded. For example with virtualdub the image that is displayed on screen does not necessarily come from the frame indicated by the slider. Also don't some SAPs actually require packed bitstream?

It's good to have it as an example of how to use the xvid API as well, since that's where the functionality really comes from. xvid_encraw simply passes a flag to the encoder to tell it to deliver a packed bitstream or not.

708145

6th December 2006, 13:43

AVIs without packed bitstream are hard to edit due to the lag between the frame being displayed and the frame being decoded.

Why use avi at all then? If it is not even useful for editing?

SeeMoreDigital

6th December 2006, 13:44

It's good to have it as an example of how to use the xvid API as well, since that's where the functionality really comes from. xvid_encraw simply passes a flag to the encoder to tell it to deliver a packed bitstream or not.I see....

And as a back-up, for all those people who find their SAP's need PBS, but forgot to include it. There is always Moitah's excellent MPEG4 Modifier application - which was recently upgraded to add PBS to MPEG-4 (in AVI) streams too ;)

squid_80

6th December 2006, 13:47

Why use avi at all then? If it is not even useful for editing?
It's fine with packed bitstream. That's all I'm going to say since I don't want this to turn into another avi/vfw war thread.

708145

6th December 2006, 14:10

It's fine with packed bitstream. That's all I'm going to say since I don't want this to turn into another avi/vfw war thread.

OK then.

weaver4

6th December 2006, 14:49

I see that xvid-encraw has picked up a lot of momentum. So, I want to learn a little more about it. Are there any guides or manuals that I could look at.

squid_80

6th December 2006, 14:57

xvid_encraw -help and this thread from about page 11 onwards.

elguaxo

7th December 2006, 18:35

I am using the xvid_encraw version that comes with MeGUI. I don't remember when it was last updated, but the Greyscale (-grey) option is not available:

D:\megui\tools\xvid_encraw>xvid_encraw -help
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

xvid_encraw built at 17:33:35 on Apr 30 2006
Usage : xvid_encraw [OPTIONS]

Input options:
-i string : input filename (stdin)
-type integer: input data type (yuv=0, pgm=1, avi/avs=2)
-w integer: frame width ([1.2048])
-h integer: frame height ([1.2048])
-frames integer: number of frames to encode

Output options:
-dump : save decoder output
-save : save an Elementary Stream file per frame
-o string : save an Elementary Stream for the complete sequence
-avi string: save an AVI file for the complete sequence
-mkv string: save a MKV file for the complete sequence

BFrames options:
-max_bframes integer: max bframes (2)
-bquant_ratio integer: bframe quantizer ratio (150)
-bquant_offset integer: bframe quantizer offset (100)

Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-size integer : target size in kilobytes
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-max_key_interval integer : maximum keyframe interval (300)

Single Pass options:
-reaction integer : reaction delay factor (16)
-averaging integer : averaging period (100)
-smoother integer : smoothing buffer (100)

Second Pass options:
-kboost integer : I frame boost (10)
-kthresh integer : I frame reduction threshold (1)
-kreduction integer : I frame reduction amount (20)
-ostrength integer : overflow control strength (5)
-oimprove integer : max overflow improvement (5)
-odegrade integer : max overflow degradation (5)
-chigh integer : high bitrate scenes degradation (0)
-clow integer : low bitrate scenes improvement (0)
-overhead integer : container frame overhead (24)
-vbvsize integer : use vbv buffer size
-vbvmax integer : vbv max bitrate
-vbvpeak integer : vbv peak bitrate over 1 second

Other options
-noasm : do not use assembly optmized code
-turbo : use turbo presets for higher encoding speed
-quality integer : quality ([0..6]) (6)
-vhqmode integer : level of R-D optimizations ([0..4]) (1)
-bvhq : use R-D optimizations for B-frames
-qpel : use quarter pixel ME
-gmc : use global motion compensation
-qtype integer : quantization type (H263:0, MPEG4:1) (0)
-qmatrix filename : use custom MPEG4 quantization matrix
-interlaced [integer] : interlaced encoding (BFF:1, TFF:2) (1)
-nopacked : Disable packed mode
-noclosed_gop : Disable closed GOP mode
-lumimasking : use lumimasking algorithm
-stats : print stats about encoded frames
-debug : activates xvidcore internal debugging output
-vop_debug : print some info directly into encoded frames
-nochromame : Disable chroma motion estimation
-notrellis : Disable trellis quantization
-imin integer : Minimum I Quantizer (1..31) (2)
-imax integer : Maximum I quantizer (1..31) (31)
-bmin integer : Minimum B Quantizer (1..31) (2)
-bmax integer : Maximum B quantizer (1..31) (31)
-pmin integer : Minimum P Quantizer (1..31) (2)
-pmax integer : Maximum P quantizer (1..31) (31)
-drop integer : Frame Drop Ratio (0..100) (0)
-start integer : Starting frame number
-threads integer : Number of threads
-progress [integer] : Show progress updates every n frames (10)
-par integer[:integer] : Set Pixel Aspect Ratio.
1 = 1:1
2 = 12:11 (4:3 PAL)
3 = 10:11 (4:3 NTSC)
4 = 16:11 (16:9 PAL)
5 = 40:33 (16:9 NTSC)
other = custom (width:height)
-help : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as needed.

I downloaded the latest version available here (http://members.optusnet.com.au/squid_80/), but it is not there either:

c:\temp>xvid_encraw -help
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

xvid_encraw built at 02:09:09 on Oct 5 2006
Usage : xvid_encraw [OPTIONS]

Input options:
-i string : input filename (stdin)
-type integer: input data type (yuv=0, pgm=1, avi/avs=2)
-w integer: frame width ([1.2048])
-h integer: frame height ([1.2048])
-frames integer: number of frames to encode

Output options:
-dump : save decoder output
-save : save an Elementary Stream file per frame
-o string : save an Elementary Stream for the complete sequence
-avi string: save an AVI file for the complete sequence
-mkv string: save a MKV file for the complete sequence

BFrames options:
-max_bframes integer: max bframes (2)
-bquant_ratio integer: bframe quantizer ratio (150)
-bquant_offset integer: bframe quantizer offset (100)

Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-size integer : target size in kilobytes
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-max_key_interval integer : maximum keyframe interval (300)

Single Pass options:
-reaction integer : reaction delay factor (16)
-averaging integer : averaging period (100)
-smoother integer : smoothing buffer (100)

Second Pass options:
-kboost integer : I frame boost (10)
-kthresh integer : I frame reduction threshold (1)
-kreduction integer : I frame reduction amount (20)
-ostrength integer : overflow control strength (5)
-oimprove integer : max overflow improvement (5)
-odegrade integer : max overflow degradation (5)
-chigh integer : high bitrate scenes degradation (0)
-clow integer : low bitrate scenes improvement (0)
-overhead integer : container frame overhead (24)
-vbvsize integer : use vbv buffer size
-vbvmax integer : vbv max bitrate
-vbvpeak integer : vbv peak bitrate over 1 second

Other options
-noasm : do not use assembly optmized code
-turbo : use turbo presets for higher encoding speed
-quality integer : quality ([0..6]) (6)
-vhqmode integer : level of R-D optimizations ([0..4]) (1)
-bvhq : use R-D optimizations for B-frames
-qpel : use quarter pixel ME
-gmc : use global motion compensation
-qtype integer : quantization type (H263:0, MPEG4:1) (0)
-qmatrix filename : use custom MPEG4 quantization matrix
-interlaced [integer] : interlaced encoding (BFF:1, TFF:2) (1)
-nopacked : Disable packed mode
-noclosed_gop : Disable closed GOP mode
-lumimasking : use lumimasking algorithm
-stats : print stats about encoded frames
-debug : activates xvidcore internal debugging output
-vop_debug : print some info directly into encoded frames
-nochromame : Disable chroma motion estimation
-notrellis : Disable trellis quantization
-imin integer : Minimum I Quantizer (1..31) (2)
-imax integer : Maximum I quantizer (1..31) (31)
-bmin integer : Minimum B Quantizer (1..31) (2)
-bmax integer : Maximum B quantizer (1..31) (31)
-pmin integer : Minimum P Quantizer (1..31) (2)
-pmax integer : Maximum P quantizer (1..31) (31)
-drop integer : Frame Drop Ratio (0..100) (0)
-start integer : Starting frame number
-threads integer : Number of threads
-progress [integer] : Show progress updates every n frames (10)
-par integer[:integer] : Set Pixel Aspect Ratio.
1 = 1:1
2 = 12:11 (4:3 PAL)
3 = 10:11 (4:3 NTSC)
4 = 16:11 (16:9 PAL)
5 = 40:33 (16:9 NTSC)
other = custom (width:height)
-help : prints this help message

NB: You can define 64 zones repeating the -z[qw] option as needed.

Where can I find a version that includes the -grey option?

buzzqw

7th December 2006, 18:51

while i perfectly agree... i never seen too -grey... but i will prefer the greyscale from avisynth

BHH

kurt

7th December 2006, 19:03

greyscale is part of the zones and it's missing in MeGUI. What you can do is to use a custom commandline:

eg:
-zones 1,w,0.5,25O/359,w,1,O/161519,w,0.04,25OG
--> last zone starts with frame 161519, has chroma optimizer and greyscale enabled, bframe sensivity of 25 and a weight of 0.04...

here are the complete zones parameters:

Rate control:
Weight 2.01 => ,w,2.01
Quantizer 5.01 => ,q,5.01

Begin with keyframe => ,K
Greyscale encoding => ,G
Chroma optimizer enabled => ,O
Cartoon Mode => ,C
BVOP sensivity 45=> ,45

elguaxo

7th December 2006, 19:03

I am pretty sure it was there sometime. It is even an option in MeGUI's XviD Configurations.

Edit: Great! Thanks kurt!!!

squid_80

14th December 2006, 14:41

New build available (for the second time this week): http://members.optusnet.com.au/squid_80/xvid_encraw.zip

Changes include plugh's alternative 2 pass algorithm (thread (http://forum.doom9.org/showthread.php?t=118419)) and Kopernikus's HVS mods (thread (http://forum.doom9.org/showthread.php?t=114811)). Also some SSIM options from xvid's CVS.

henryho_hk

17th December 2006, 13:55

I am using Celtic Druid's 2006.12.08 CVS compile of XviDcore.dll on a Core 2 Duo E6300 1.86GHz. The encoding FPS reported by xvid_encraw seems incorrect. A single pass encode of a 2hr50m 29.9fps was reported to be encoding at 34fps; however, it took 4hr40m totally, way slower than the reported speed. Or have I misunderstood the encoding fps?

MatMaul

17th December 2006, 14:06

henryho_hk

17th December 2006, 14:08

Do you have fft3dGPU in your avs script?

I am just using mpeg2source(), colormatrix(), tdeint(), degrainmedian() and crop().

squid_80

17th December 2006, 14:24

Or have I misunderstood the encoding fps?
It's encoding fps; it only measures the time taken to encode frames. It doesn't take into account the time taken to get the input frames i.e. avisynth processing.

henryho_hk

17th December 2006, 15:34

I don't get it. Playing back the AVS in media player classic takes about 25% CPU load only. How can it contribute for 16fps lost in encoding speed?

squid_80

18th December 2006, 00:03

So it's a simple enough script that can playback in realtime without lagging? I'd assumed it was more complicated. If you really want to benchmark the avs script open it with virtualdub, set video processing mode to direct stream copy, leave video compression set to none and run a video analysis pass.
Another thing is to make sure the script is outputting yv12, if it's not then some vfw codec might be used for the conversion.

henryho_hk

18th December 2006, 01:32

It's a rather simple script. I can play it realtime at 25% CPU load and a direct stream copy of VirtualDubMod runs at about 50fps. VirtualDubMod says "YV12" and "ATI YVU12 4:2:0 Planar".

vf="test.d2v"
mpeg2source(vf)
colormatrix(d2v=vf)
tdeint(order=1,mtnmode=0)
crop(8,0,704,480)
removegrain(mode=5)

I am now testing "-threads 3" (#core + 1) and see if it helps.

mixanobios

18th December 2006, 03:03

i have experienced the same problem with henryho_hk. if you want i can search for my megui logs

squid_80

18th December 2006, 08:20

I'm in the middle of building a dual core machine (literally putting it together), when it's properly setup I will add threaded code for the input stage and fix the encoding timers (and hopefully remember once and for all to also fix the size variables so they don't wrap to negatives with large files).

G_M_C

18th December 2006, 10:00

Do you have fft3dGPU in your avs script ?
because when I use it, wrong fps is reported by xvid_encraw and x264.

Could you please report that in the FFT3DGPU thread please ?
http://forum.doom9.org/showthread.php?t=89941&page=28

You may have to add a script-example + a clip example to make ... reproduce the error.

I ask this because i use fft3dgpu very often, and making it bug-free is offcourse imporatnt enough for all of us that do too.

MatMaul

18th December 2006, 13:56

I don't think it's a bug of fft3dGPU but just the way fps is calculated in xvid_encraw.

Brother John

21st December 2006, 23:50

Hi, squid_80.

Everytime I look at "xvid_encraw -help" I stumble over the outdated zones options. Isn't it time to document the zones feature properly? To do this you should remove the obsolete -zq and -zw lines from "Rate control options" and replace the note at the very end with a more detailed explanation. I put together the following text. What do you think about it?

Zones options:
You can define up to 64 zones using the -zones option as described below.

-zones start,mode,value[,options][/start,mode,value[,options]]...

Parameters of a zones use the comma (,) as delimiter. Multiple zones are
separated by a slash (/). The end of each zone is defined by either the start
of the following zone or the last frame of the input file.

start : Start frame of the zone.
mode : weight zone = w, quantizer zone = q
value : Depending on mode either the zone's weight or quantizer.
options : Enables certain encoder features for the zone. Each feature is
represented by a single letter. An integer number stands for
b-frame sensitivity. To enable multiple features at the same time
combine the appropriate symbols without any delimiting characters.
K = Begin with keyframe
O = Enable chroma optimizer
G = Greyscale encoding
C = Cartoon mode
integer = B-Frame sensitivity

Example:
To create a first zone starting at frame 0 with weight 1.0, all options
enabled and b-frame sensitivity -5, and a second zone starting at frame 1000
with constant quant 4 and no options enabled you would use the -zones option
like this:

-zones 0,w,1.0,-5KOGC/1000,q,4

squid_80

22nd December 2006, 17:51

The old options (-zq and -zw) are still useful so I'll leave them in, but I'll add a small description to be displayed with -help and your longer description can be displayed with "-help zones" or if an error is detected in the -zones parameters.

With regards to the fps bugs, I am seeing very irregular things happening with my dual core machine and I have a hunch that it's due to the main thread jumping between cores. I remember reading that separate cores don't always have their clocks synchronised exactly, hence if the start time is taken from one core and the finish time is taken from the other the result isn't valid. This may be BS but I'm not sure how else to explain the fact that I'm seeing frames pop out with negative encoding times.

foxyshadis

23rd December 2006, 01:32

Maybe it's one of those things that only works right if you start windows with /usepmtimer. If you already have it, then... dunno. What do you use? GetSystemTickCount?

Brother John

10th January 2007, 18:13

I’ve been wondering for some time now about a specific function in encraw, but search didn’t turn up anything useful.

In Xvid VfW you can set 1-pass target quant in the main dialog and 1-pass constant quant via the zones window. However in Xvid_Encraw I only find -cq for constant quant. No target quant at all. Am I blind? Is target quant not supported in encraw? Or is the VfW GUI misleading?

Also (if it does exist), would it be fair to say that Xvid’s target quant is a similar function to x264’s constant rate factor? Very roughly speaking, maybe. Just to get the general idea.

Kopernikus

11th January 2007, 00:42

There is no equivalent to x264s crf in xvid, IIRC target quant is equivalent to constant quant. If the target is not a integer, the quants are distributed to achieve a average quant near the target.

henryho_hk

21st January 2007, 10:01

Is target quant not supported in encraw?

Both are supported: -cq and -zones

Xvid’s target quant is a similar function to x264’s constant rate factor?

I am afraid not. But you can take a look at the thread " Isn't it time for a new compression mode in XviD?".

elfurbe

2nd February 2007, 10:56

So, it seems that no matter how many threads I specify, xvid_encraw never spawns more than one, at least according to task manager. I've got an A64 4400+ dual-core, so I used -threads 3 (cause I've found one more threads than cores is ideal for most tasks) but Task Manager shows it at 1 thread which is disturbing slow (5.6fps) on pass2 and pretty damn slow on pass1 (~20fps). The AVS framesize is 960x544, down-resing from 720p HDTV content.

What am I doing wrong? Here's my avs file and encode lines:

AVS:
LoadPlugin("c:\plugins\dgdecode.dll")
LoadPlugin("c:\plugins\TIVTC.dll")
mpeg2source("C:\file.d2v")
SelectEven()
Tdecimate()
LanczosResize(960,544,0,0,1280,720)
UnDot()

Pass 1:
xvid_encraw -bitrate 2300 -pass1 -threads 3 -progress -turbo -i file.avs -mkv file.mkv

Pass 2:
xvid_encraw -bitrate 2300 -pass2 -threads 3 -progress -qpel -gmc -i file.avs -mkv file.mkv

foxyshadis

2nd February 2007, 11:24

How many fps do you get if you load the script into virtualdub and hit "run video analysis pass"? That's the maximum upper bound. Using the multithreaded avisynth and appropriate syntax can help. I suspect most of the speed hit comes from TDecimate, but there's not much you can do about that. And of course gmc - didn't I read somewhere that gmc wasn't multithreaded?

If speed is of the utmost importance, you can always do a lossless pre-render.

squid_80

2nd February 2007, 11:43

xvid_encraw reports the xvidcore version and the number of threads being used. Can you post this information?

elfurbe

2nd February 2007, 17:16

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

xvid [warn]: Can't find xvid_plugin_ssim, ssim calculations will be disabled
Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [info]: Input is 960 x 544, 23.976fps (24000/1001), starting from frame 0
xvid [info]: Number of frames to encode: 62979, Bitrate = 2235kbps
xvid [info]: xvidcore build version: xvid-1.1.0
xvid [info]: Bitstream version: 1.1.0
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
xvid [info]: Detected 0 cpus, using 3 threads.

That's what she outputs on start, but in Task Manager it shows 1 thread.

I don't use virtual dub for anything in my encoding chain, so I wouldn't even know how to go about doing what you're talking about.

I ran another encode this morning, and it looks like I underestimated my -turbo pass, it's getting about 32fps which isn't terrible, but pass two is still super slow.

squid_80

2nd February 2007, 17:32

xvid [info]: xvidcore build version: xvid-1.1.0
xvid [info]: Detected 0 cpus, using 3 threads.
You need xvid 1.2.0 for multithreading.

(Also, note to self: Try changing "using x threads" to "requesting x threads" and append "x threads being used", if possible. Also remove ssim warning unless ssim calculations are specifically requested.)

elfurbe

2nd February 2007, 17:53

Well, that could certainly do it, I suppose. Is this:
http://www.koepi.org/XviD-1.2.-127-25022006.exe
what I need to install?

I guess I thought the xvid encoder was rolled into the xvid_encraw binary, a-la x264.exe. Is that not the case?

squid_80

2nd February 2007, 18:01

Well, that could certainly do it, I suppose. Is this:
http://www.koepi.org/XviD-1.2.-127-25022006.exe
what I need to install?Yep, that should work.
I guess I thought the xvid encoder was rolled into the xvid_encraw binary, a-la x264.exe. Is that not the case?No, it uses whatever xvidcore.dll it can find. It helps keep the filesize down a bit and makes it possible to update the core encoder without updating xvid_encraw.exe.

elfurbe

2nd February 2007, 18:09

So if I put xvidcore.dll in the same folder as xvid_encraw, would that be the default one it uses or does it check system first?

squid_80

2nd February 2007, 18:14

Yes, if xvidcore.dll is in the same directory as xvid_encraw.exe it will be used. At least I'm pretty sure that's how windows dll loading rules work.

elfurbe

2nd February 2007, 18:43

Alright, threw the xvidcore.dll from xvid 1.2 in there, and rolled it out, got this on start:
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

xvid [warn]: Can't find xvid_plugin_ssim, ssim calculations will be disabled
Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [info]: Input is 960 x 544, 23.976fps (24000/1001), starting from frame 0
xvid [info]: Number of frames to encode: 62994, Bitrate = 2234kbps
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 TSC
xvid [info]: Detected 2 cpus, using 3 threads.

This is on my computer at work, Core Duo so no 3DNOW, but I've found encoding speeds are similar to my 4400+. Anyway, I'm seeing better first pass (40fps at the moment), but the thread count still stays at 1. I do occasionally see it jump to three, but then right back to one instantly. Is that normal behavior?

squid_80

2nd February 2007, 18:51

(I never noticed task manager had a thread count column available till now, duh.)

Yes that's normal behaviour. x264 does the same sort of thing.

elfurbe

2nd February 2007, 21:17

Neat. My second pass encode speed is up to 12fps or so, which is obviously much better. I suppose my expectations are a little off, as I usually deal with anamorphic DVD conversions to x264, but man this seems slow. Does this seem like a reasonable speed for a 1.8GHz Core Duo on pass 2 for this frame size?

squid_80

3rd February 2007, 15:24

New build is available (http://members.optusnet.com.au/squid_80/xvid_encraw.zip).

Cosmetic changes only:
- fps figures should be more accurate (the time taken to fetch the input frame is included)
- ssim warning should only be displayed if ssim options are used and they are unavailable (in which case it is treated as an error)
- zones help, available via "-help zones". Also displayed if an error is detected in the -zones string
- frame/file sizes are 64-bit integers so there should be no more negative numbers when the output filesize goes over 2gb
- changed the help text for -o to reflect that it can be used for .avi and .mkv
- changed the xvid[info] threads text, still needs a little tweak to distinguish between how many threads are requested vs. how many are actually being used (hint: if detected cpus is 0, multi-threading is probably not available)

henryho_hk

3rd February 2007, 16:39

Excellent. ^_^ I recall that I once did a 20G encode. The numbers went positive, negative, positive, negative .....

Is your next move putting the input and output in separate threads?

squid_80

3rd February 2007, 17:05

It's one of the ideas I have. I don't know if there will be much improvement since cpu usage normally already hits 100% if you're using MT avisynth. Elfurbe's posts raise the point that the extra threads for multithreading are created/destroyed for each frame; I'm wondering if that's causing an overhead penalty that can be avoided.

elfurbe

3rd February 2007, 19:00

Didn't know there was a multithreaded AviSynth, downloaded and installed for the win. Interesting thing though. I added a multithread on the TDecimate filter, since I expected that it was the heaviest non-resizing filter in my chain, and it spawns more threads, now my thread count sits around 3 the whole time, but my cpu usage only barely climbs above 50% indicating that it's not really getting into the second core at all. It basically doesn't matter what I do with thread counts in both places, I can't get more than 53% CPU usage. Wondering what you make of that?

Also, I threw the new version of xvid_encraw in and my framerate went to the toilet. I went from doing 30-40 for this file (with/without MT avis) to doing 12. Switching back to the last version puts the framerates right back up. Is it just a display bug?

squid_80

3rd February 2007, 19:23

I went from doing 30-40 for this file (with/without MT avis) to doing 12. Switching back to the last version puts the framerates right back up. Is it just a display bug?
The difference between the old/new versions is that the old version only measured the time taken by xvid to encode the frame, whereas the new version measures the time taken to retrieve the input frame and encode it. So (given your script) the time reported by the new version is encoding time + mpeg2 decoding + selecteven (negligible?) + tdecimate + resizing + undot. What you could do is try short encodes (e.g. try adding -frames 2000) with both versions and manually time them (don't rely on the times reported by xvid_encraw). The actual time taken should be the same, and the fps reported by the new xvid_encraw should prove more accurate if you work it out by hand.

SealTooGreat

5th February 2007, 00:09

Can someone suggest me up-to-date GUI for xvid_encraw?!

Adub

5th February 2007, 01:39

MeGUI.

squid_80

5th February 2007, 04:34

Virtualdub and xvid's vfw interface. The idea of encraw was to make all the vfw options available via command line, making a gui for it seems a bit backwards.

elguaxo

17th February 2007, 22:44

I want to run encraw with the -ssim_file option, but I'm getting this error: Can't find xvid_plugin_ssim

Do I need a newer xvidcore.dll? Where can I get one that supports this? Thanks!

squid_80

18th February 2007, 04:16

Get an updated build from celtic_druid: http://ffdshow.faireal.net/mirror/XviD/
Go for the build with the most recent date. :)

elguaxo

18th February 2007, 05:12

Perfect. Thanks!

smok3

14th March 2007, 15:31

squid_80: what is the correct latest file?

henryho_hk

19th March 2007, 02:47

squid_80, I have recently “upgraded” my Core2Duo (1.86GHz) + 865PE + 1GB DDR-I-400 to Core2Duo (@3GHz) + P965 + 2GB DDR-II-800@860. You know what; the encoding speed does not get increased! Same script (mpeg2source + colormatrix + tdeint + bicubicresize + degrainmedian => 25fps ) I must have messed something up….

squid_80

19th March 2007, 10:57

That's weird. Try using -threads 1 on both machines and see if there's a difference.

henryho_hk

19th March 2007, 13:00

No, I can't... It's the same CPU. :devil:

I have installed Celtic Druid's 2007/03/10 XviD compile in the new setup. I would try the last version and see if it's better.

squid_80

19th March 2007, 13:10

No, I can't... It's the same CPU. :devil:
You mean it's the same chip overclocked to 3ghz? It might be getting speed throttled due to overheating. Try checking the temps when it's encoding.

henryho_hk

19th March 2007, 14:28

It might be getting speed throttled due to overheating.

It's staying at 58~60 C at full load. Even if it's throttling, it's at 6x @ 2.5G, still way faster than 1.86G. That's why i am puzzled.

squid_80

22nd March 2007, 14:09

Try running avsutil on the script (the newest version that reports the total time taken to play the script) at the new clock speed then drop to a lower clock speed and see if the time taken increases. If the time measurements are the same it's an avisynth problem; if the avsutil times are proportional to cpu clock speed(as they should be) then there's a problem with xvid.

henryho_hk

23rd March 2007, 06:41

avsutil's clock time seems to be consistent with the bus speed (hence cpu & ram). It looks like a xvid related. I will try to test the encoding speed in Virtualdub tonight.
---------------
I tested avs2avi instead. At the same settings (TG's comp test, #threads=1), avs2avi is about 50% faster than xvid_encraw. That's far more than I would ever expect.
---------------
Seems the "threads=2" parameter of Colormatrix 2.1 is causing slowness in xvid_encraw. After I change it to "threads=1", the xvid_encraw is just a few fps slower than avs2avi.
---------------
An sincere apology to squid_80. After testing long encodes (>1hr movie, Colormatrix threads=1), the encoding speed difference further diminishes to 2~3fps. Yet the Colormatrix speed issue (threads=2) still remains.

Brother John

4th April 2007, 14:05

Here’s a strange thing a have been noticing for some time.

Situation: A video encoded with newest xvid_encraw and Matroska output (-mkv parameter). Xvid settings and 1pass/2pass don’t affect the problem.
When I try to play such a video, the player accesses the hard drive for quite a while. It looks like it’s buffering the video. After a while (about 20 secs for a 2 GB movie) it plays and seeks ok. This happens with all DirectShow players, all the common splitters and decoders as well as with VLC.
When I remux the file with MKVMerge (no special settings, simple blocks disabled because I assume encraw doesn’t use them) it plays immediately without the »buffering« effect.

It’s not a huge issue because remuxing is necessary anyway to add audio, but I’m still wondering: could this be a bug in encraw’s MKV handling?

708145

4th April 2007, 14:12

It’s not a huge issue because remuxing is necessary anyway to add audio, but I’m still wondering: could this be a bug in encraw’s MKV handling?

I had the same effect with an old version of mkvmerge.
So it might be a setting with the mkv dictionary.

foxyshadis

4th April 2007, 14:23

I think the large read comes from an insufficient cue contents. (Similar to the avi index.) mkvmerge has three options, none, key-frames, or all frames, and currently defaults to keyframes; most other muxers will write no cues, because they aren't known ahead of time. Without cues, the splitter will read the whole file to extract keyframes, or seeking would be impossible. (It could read on the first seek, but they don't....)

(That's not entirely true. In 2pass mode encraw has all i-frames known, and the locations can be filled with dummy values to be filled in at the end of the encode. mkvmerge probably works that way. Up to squid or someone else to consider implementing that though.)

Simpleblock can still help for xvid, btw. It's the audio tracks that really get the gain from using it, especially AC3 and DTS.

squid_80

4th April 2007, 14:24

It’s not a huge issue because remuxing is necessary anyway to add audio, but I’m still wondering: could this be a bug in encraw’s MKV handling?
It's not really a bug, xvid_encraw just doesn't put cues (used for quick seeking) in files that it outputs. They're still valid mkv files and I never put in the extra effort since as you pointed out remuxing is normally necessary.

Brother John

5th April 2007, 20:09

Thx for the explanation. As you said, it's not too important. Maybe some time, if you get really bored... ;)

henryho_hk

7th April 2007, 16:20

squid_80, have you tried the 01/04 (hope it's not a joke) GCC Core2 compile for XviD (http://ffdshow.faireal.net/mirror/XviD/gcc/xvidcore.Core2.7z)?

The encoding time and the fps are shown as -1:

25701 frames( 99%) encoded, -1.#J fps, Average Bitrate = 1808kbps
Tot: enctime(ms) = -1.#J, length(bytes) = 193906924
Avg: enctime(ms) = -1.#J, fps = -1.#J, length(bytes) = 7532

kurt

7th April 2007, 18:05

Had the same issue and posted on celtic_druids forum (http://celticdruid.no-ip.com/phpBB2/viewtopic.php?p=785#785).

for me the P4 & Prescott builds work fine...

chipzoller

8th April 2007, 04:19

I noticed that encoding with xvid_encraw sets the fourcc in the output file as MP4V (I think, anyway). Since I really haven't messed much with xvid_encraw I was just wondering why this isn't setting the fourcc to XVID or something. I wanted to try some experiments on decoding this output file (640x480, 60fps) using ffdshow and divx to see how each affected the playback on this system. I used the >30% comp. check (hq) in megui to encode the file.

henryho_hk

8th April 2007, 09:52

Do you mean xvid_encraw's MKV output?

chipzoller

8th April 2007, 19:56

Do you mean xvid_encraw's MKV output?
Yes. I just noticed I can force a fourcc when I remux in mkvmerge, so maybe that's just the simplest answer. I was just curious as to why xvid_encraw's MKV output set that fourcc.
EDIT: Evidently if using MKV as the output in encraw, remuxing in mkvmerge won't correct/replace the fourcc. Evidently this has to be done only with avi output.

Brother John

8th April 2007, 23:26

Xvid_Encraw creates MKVs with native MPEG-4 video streams. The ID system for those doesn't identify encoders but video formats (MPEG-4 ASP, AVC, MPEG-2, WMV etc). The FourCC is only used for VfW compatibility mode streams, like those created by the Xvid VfW encoder (e.g. through VDub).

Xvid, DivX, 3ivx and some others are all MPEG-4 ASP encoders and are identified accordingly.

chipzoller

8th April 2007, 23:38

Then is the only way to test decoders ffdshow and DivX by having xvid_encraw output to AVI then change fourCC when muxing to MKV (or leave alone if already set to XVID)? Because it seems DivX will not decode content if the fourCC is set to MP4V. Disabling the decoding options for both xvid and MPEG-4 generic in ffdshow renders the stream undecodable even with DivX is installed and generic Mpeg-4 playback set in the decoder options.

henryho_hk

9th April 2007, 07:06

Is there any way to extract these native MPEG-4 video streams from the MKV file and mux it in an avi?

squid_80

9th April 2007, 10:27

Because it seems DivX will not decode content if the fourCC is set to MP4V. Sounds like a DivX issue to me. Go bug them to support generic mpeg-4 asp in their decoder. Or just output to avi instead of mkv and change the fourcc.
Is there any way to extract these native MPEG-4 video streams from the MKV file and mux it in an avi?I think you have to do mkv->native stream->mp4->avi. Not too sure since I haven't done it myself but I think Bond has posted how to do it.
squid_80, have you tried the 01/04 (hope it's not a joke) GCC Core2 compile for XviD ?

The encoding time and the fps are shown as -1:That used to happen with ANY xvidcore build, when I first fixed the negative bitrate/time issues; I had to re-arrange the code to get it to work correctly. Seems like a missing emms instruction or possibly a non-volatile register not being saved, I just need time to look into it.
I thought there was also a post here about disabling N-VOPs thru xvid_encraw, I'm not sure if that's possible but will look into it.

vanger

16th April 2007, 11:48

I have strange problem with avs2qxvid...
Preset to use = TGHQ-58

start "TGHQ-58 - Pass 1/2" /b /wait /belownormal "D:\avs2qxvid\bin\xvid_encraw_2
0061213.exe" -zones 0,q,3,KO -threads 2 -progress 100 -max_key_interval 250 -pac
ked -quality 5 -vhqmode 1 -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -qty
pe 1 -qmatrix "D:\avs2qxvid\matrix\Didees-SixOfNine.cqm" -nochromame -turbo -pas
s1 "D:\avs2qxvid\tmp\zxxxz2.pass" -type 2 -i "D:\avs2qxvid\zxxxz2.avs"

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [info]: Input is 512 x 336, 25.000fps (25/1), starting from frame 0
xvid [info]: Number of frames to encode: 73744
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
xvid [info]: Detected 2 cpus, using 2 threads.
73701 frames( 99%) encoded, 112.14 fps, Average Bitrate = 1367kbps
Tot: enctime(ms) =657434.00, length(bytes) = 503621549
Avg: enctime(ms) = 8.91, fps = 112.17, length(bytes) = 6829
I frames: 556 frames, size = 28837/16033540, quants = 3 / 3.00 / 3
P frames: 26237 frames, size = 11261/295456472, quants = 3 / 3.00 / 3
B frames: 46951 frames, size = 4092/192131537, quants = 4 / 4.00 / 4

start "TGHQ-58 - Pass 2/2" /b /wait /belownormal "D:\avs2qxvid\bin\xvid_encraw_2
0061213.exe" -zones 0,w,1.0,KO -threads 2 -progress 100 -max_key_interval 250 -p
acked -quality 6 -vhqmode 4 -qpel -max_bframes 2 -bquant_ratio 162 -bquant_offse
t 0 -qtype 1 -qmatrix "D:\avs2qxvid\matrix\Didees-SixOfNine.cqm" -imin 3 -imax 4
-pmin 3 -pmax 5 -bmin 3 -bmax 5 -chigh 10 -clow 3 -bvhq -bitrate 1000 -pass2 "D
:\avs2qxvid\tmp\zxxxz2.pass" -stats "D:\avs2qxvid\tmp\zxxxz2.stats" -type 2 -i "
D:\avs2qxvid\zxxxz2.avs" -avi "D:\avs2qxvid\zxxxz2_TGHQ-58_br_1000.avi"

xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

xvid [info]: outputting statistics to D:\avs2qxvid\tmp\zxxxz2.stats
Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [info]: Input is 512 x 336, 25.000fps (25/1), starting from frame 0
xvid [info]: Number of frames to encode: 73744, Bitrate = 1000kbps
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 3DNOW 3DNOWEXT TSC
xvid [info]: Detected 2 cpus, using 2 threads.
73701 frames( 99%) encoded, 35.35 fps, Average Bitrate = 1000kbps
Tot: enctime(ms) =2085700.00, length(bytes) = 368580465
Avg: enctime(ms) = 28.28, fps = 35.36, length(bytes) = 4997, psnr y = 42.92
, psnr u = 49.28, psnr v = 50.30
I frames: 556 frames, size = 25511/14184197, quants = 3 / 3.64 / 4
P frames: 26237 frames, size = 8172/214430966, quants = 3 / 3.81 / 5
B frames: 46951 frames, size = 2981/139965302, quants = 4 / 4.79 / 7

All completed..............

16.04.2007 0:46:20,03 - Checking Source
16.04.2007 0:46:24,01 - Loading Preset
16.04.2007 0:46:24,12 - Encoding Video
16.04.2007 0:46:24,12 - Encoding Pass 1
start "TGHQ-58 - Pass 1/2" /b /wait /belownormal "D:\avs2qxvid\bin\xvid_encraw_20061213.exe" -zones 0,q,3,KO -threads 2 -progress 100 -max_key_interval 250 -packed -quality 5 -vhqmode 1 -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -qtype 1 -qmatrix "D:\avs2qxvid\matrix\Didees-SixOfNine.cqm" -nochromame -turbo -pass1 "D:\avs2qxvid\tmp\zxxxz2.pass" -type 2 -i "D:\avs2qxvid\zxxxz2.avs"
16.04.2007 2:37:16,64 - Encoding Pass 2
start "TGHQ-58 - Pass 2/2" /b /wait /belownormal "D:\avs2qxvid\bin\xvid_encraw_20061213.exe" -zones 0,w,1.0,KO -threads 2 -progress 100 -max_key_interval 250 -packed -quality 6 -vhqmode 4 -qpel -max_bframes 2 -bquant_ratio 162 -bquant_offset 0 -qtype 1 -qmatrix "D:\avs2qxvid\matrix\Didees-SixOfNine.cqm" -imin 3 -imax 4 -pmin 3 -pmax 5 -bmin 3 -bmax 5 -chigh 10 -clow 3 -bvhq -bitrate 1000 -pass2 "D:\avs2qxvid\tmp\zxxxz2.pass" -stats "D:\avs2qxvid\tmp\zxxxz2.stats" -type 2 -i "D:\avs2qxvid\zxxxz2.avs" -avi "D:\avs2qxvid\zxxxz2_TGHQ-58_br_1000.avi"
16.04.2007 4:51:54,85 - All Completed
The real fps for 1st pass is ~11fps, for 2nd - ~9fps.

the avs script is:
DGDecode_mpeg2source("D:\capture\video\wsa.d2v",info=3).trim(0,73743)
ColorMatrix(hints=true)

crop(14,36,702,504)

FFT3DFilter(sigma=2,plane=4)

LanczosResize(512,336)

I think on my config (AMD X2 3800+ (2500Mhz)) it is too slow...
I have the same fps with the same avs-script on 2nd pass with x264 (HQ-Insane)...

Any ideas?

henryho_hk

16th April 2007, 14:26

Older versions of xvid_encraw show its own xvid encoding time and exclude the AVS processing efforts. Try the newest avs2qxvid package with xvid_encraw_20070203.exe. squid_80 has revised the fps calculation method to show the overall encoding speed.

---------------

Added: "-qpel" and especially "-vhqmode 4" are quite slow. For ">58%" and ">90%" presets, you may try the FAST version and see if you can tell the difference.

vanger

17th April 2007, 06:27

and how about speed of encoding?

squid_80

17th April 2007, 14:35

The encoding speed is fine - the raw encoding speed is 112 fps and 35 fps as reported by xvid_encraw. The filtering (fft3dfilter) is what is slowing it down so much.

Revgen

14th May 2007, 11:07

I currently cannot encode anything over 10000kbps with Xvid Encraw. I tried the latest version and the latest Xvid core form CD (March 10th version) and no luck. The VFW version works fine above 10000kbps, so I can't see why it would be xvid itself.

henryho_hk

14th May 2007, 16:37

This is a "feature" of xvid_encraw. Any -bitrate over 10000 is considered bps (but not kbps). If you wanna specify >=10000kbps, specify *1024, e.g. -bitrate 10240000

Revgen

14th May 2007, 22:09

Thanks. That's one step in the right direction.

Now I have another problem. The encode is still undersized. My VFW encode is perfectly encoded at 17000kbps while the encraw one is about 14800kbps. Is their a reason why this is happening?

Here's my .bat file.

E:
cd\MeGUIUpdated\tools\xvid_encraw
xvid_encraw -i G:\kong.avs -type 2 -w 1920 -h 832 -quality 6 -max_bframes 2 -framerate 23.976 -bitrate 17408000 -pass1 -max_key_interval 240 -qtype 1 -qpel -vhqmode 4 -bvhq -par 1 -noclosed_gop -nopacked -turbo -threads 2 -mkv G:\kongx.mkv
xvid_encraw -i G:\kong.avs -type 2 -w 1920 -h 832 -quality 6 -max_bframes 2 -framerate 23.976 -bitrate 17408000 -pass2 -max_key_interval 240 -qtype 1 -qpel -vhqmode 4 -bvhq -par 1 -noclosed_gop -nopacked -threads 2 -mkv G:\kongx.mkv

Thanks.

Brother John

15th May 2007, 01:16

Saturation maybe?
Xvid VfW uses min quant 1 to waste more bitrate in that case while Xvid_Encraw uses min quant 2 and produces a smaller file.

If you want to mimic VfW's behaviour add this to your 2nd pass cmdln:
-imin 1 -pmin 1 -bmin 1
Lowering VHQ or b-frame quantization or using a more detailled matrix makes a lot more sense, though.

Revgen

15th May 2007, 09:55

Thanks, that did the trick. I'll fiddle with the VHQ, B-frame quants, and the matices later and see what happens.

Daemon404

26th May 2007, 00:40

Hurm, here's a version witht he latest cvs code (May 25th) with matroska support:

http://www.highdeffguild.com/daemon404/enc_builds/xvid_encraw-cvs-may-25.7z

And, since apparently people are having a hard time finding the missing matroska.cpp and matroska.h files (no idea why they aren't in cvs), here:

http://www.highdeffguild.com/daemon404/enc_builds/xvid_matroska.7z

celtic_druid

26th May 2007, 08:44

When was it last updated in the CVS though?

squid_80

26th May 2007, 17:23

And, since apparently people are having a hard time finding the missing matroska.cpp and matroska.h files (no idea why they aren't in cvs), here:
I can provide an answer for that: I wrote them and they're a horrible mess of kludges and hacks so they weren't absorbed into xvid's CVS. Really the matroska handling should be completely rewritten from scratch but since they work as is I can't really can't be bothered.

Daemon404

26th May 2007, 18:51

When was it last updated in the CVS though?

April 28th, iirc. However, I really dont see any recent builds with matroska support.

I can provide an answer for that: I wrote them and they're a horrible mess of kludges and hacks so they weren't absorbed into xvid's CVS. Really the matroska handling should be completely rewritten from scratch but since they work as is I can't really can't be bothered.

I figured it was something like that. :P

celtic_druid

26th May 2007, 20:14

http://cvs.xvid.org/cvs/viewvc.cgi/xvidcore/examples/xvid_encraw.c?view=log
Not updated since Jan. My point was that builds with Xvid dynamically linked just need a new xvidcore.dll unless encraw itself is updated.

Daemon404

26th May 2007, 20:50

Hmm yea, dynamically linking would probably be a smarter idea.

And by Apr 28, i meant that's when xvidcore was last updated.

Brother John

13th August 2007, 14:27

I just stumbled one more time over the bit/kbit issue with the -bitrate switch. What about changing the help text to clarify this?
-bitrate [integer] : target bitrate (700)
values <= 10000 are treated as kbit/s
values > 10000 are treated as bit/s

Selur

13th August 2007, 14:51

btw. if current encraw supports -alt2pass would be nice to mention it in the 'help'

squid_80

13th August 2007, 16:53

This is what my build does:
Rate control options:
-framerate float : target framerate (25.0)
-bitrate [integer] : target bitrate in kbps (700)
-size integer : target size in kilobytes
-single : single pass mode (default)
-cq float : single pass constant quantizer
-pass1 [filename] : twopass mode (first pass)
-full1pass : perform full first pass
-pass2 [filename] : twopass mode (2nd pass)
-altpass2 [filename] : twopass mode (2nd pass alt)
-zq starting_frame float : bitrate zone; quant
-zw starting_frame float : bitrate zone; weight
-zones : see xvid_encraw -help zones
-max_key_interval integer : maximum keyframe interval (300)
I agree -bitrate could do with some clarification, but don't see the problem with -altpass2 not being mentioned?

squid_80

31st August 2007, 01:41

A new build is available: http://members.optusnet.com.au/squid_80/xvid_encraw.zip

See if you can spot the new feature (hint: you need a multi-core machine).

Sharktooth

31st August 2007, 02:48

eh:) i didnt get it yet, but if what you're talking about it's an automatic threading feature (without the need of a commandline switch), i will kiss your.... :D

squid_80

31st August 2007, 03:09

Old build:
>xvid_encraw -i test.avs -progress -frames 1000
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [warn]: Single pass defaulting to quant 4.00
xvid [info]: Input is 528 x 576, 25.000fps (25/1), starting from frame 0
xvid [info]: Number of frames to encode: 1000
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 TSC
xvid [info]: Detected cpus = 4, threads requested = 4, threads in use = 4
1001 frames(100%) encoded, 38.50 fps, Average Bitrate = 1340kbps
Tot: enctime(ms) =26000.00, length(bytes) = 6709775
Avg: enctime(ms) = 25.95, fps = 38.54, length(bytes) = 6696
I frames: 15 frames, size = 19720/ 295814, quants = 4 / 4.00 / 4
P frames: 427 frames, size = 11520/ 4919399, quants = 4 / 4.00 / 4
B frames: 558 frames, size = 2678/ 1494562, quants = 7 / 7.00 / 7
new build:
>xvid_encraw -i test.avs -progress -frames 1000
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampert 2002-2003

Trying to retrieve width and height from input header
xvid [info]: Avisynth detected
xvid [info]: Input colorspace is YV12
xvid [warn]: Single pass defaulting to quant 4.00
xvid [info]: Input is 528 x 576, 25.000fps (25/1), starting from frame 0
xvid [info]: Number of frames to encode: 1000
xvid [info]: xvidcore build version: xvid-1.2.0-dev
xvid [info]: Bitstream version: 1.2.-127
xvid [info]: Detected CPU flags: ASM MMX MMXEXT SSE SSE2 TSC
xvid [info]: Detected cpus = 4, threads requested = 3, threads in use = 3
xvid [info]: Threaded input reading active
1001 frames(100%) encoded, 57.66 fps, Average Bitrate = 1340kbps
Tot: enctime(ms) =17359.00, length(bytes) = 6709776
Avg: enctime(ms) = 17.32, fps = 57.72, length(bytes) = 6696
I frames: 15 frames, size = 19720/ 295814, quants = 4 / 4.00 / 4
P frames: 427 frames, size = 11520/ 4919399, quants = 4 / 4.00 / 4
B frames: 558 frames, size = 2678/ 1494563, quants = 7 / 7.00 / 7

Sharktooth

31st August 2007, 03:15

ok, i suppose i have to do it then...:D
in the meanwhile it got its way into megui auto-update...

DarkZell666

31st August 2007, 09:27

Pretty neat :D

But something else changed too :
B frames: 558 frames, size = 2678/ 1494562, quants = 7 / 7.00 / 7
becomes
B frames: 558 frames, size = 2678/ 1494563, quants = 7 / 7.00 / 7

I don't give a damn for 1 byte honestly, but I'm curious :p Is there any part of the code that's undeterministic ? Or is it a side effect of your new feature ?

squid_80

31st August 2007, 09:35

Look closer and you'll see why - using threaded input reduces the number of threads used for encoding by 1. Different number of threads = slightly different output.

kypec

31st August 2007, 11:21

@squid_80
Just curious, how did you achieve that performance boost?
I thought that xvid_encraw is just an interface to xvidcore.dll, isn't it?
Thus the encoding performance should be entirely (well, from 99% in our real PC world then) dependent ONLY on Xvid encoder engine installed, or not?

Brother John

31st August 2007, 11:45

You're right when you speak about pure enocder perfomance without any overheads, decoding, filtering etc. And multiple encoding threads is a feature of xvidcore.dll.

squid_80

31st August 2007, 11:59

xvidcore handles the encoding, but xvid_encraw is responsible for feeding it frames to encode. If those frames are coming from an avisynth script with lots of filtering they can take a while to be retrieved. Each time a frame is fed to the encoder encraw fires off a new thread to fetch the next source frame. Thus when the encoder is ready to encode another frame, the source frame is already waiting in a buffer ready to go.

kypec

4th September 2007, 08:54

Yeah, I see now where the performance boost comes from :)
Thanks for explanations squid_80 & Brother John

plugh

4th September 2007, 20:11

Is altpass2 working 'ok' for people in the multi-threading environment? There was a bit of code I added in there I was concerned about - ie coordination of variable 'vbvfill_before' in routines rc_2pass2_before and rc_2pass2_after.

Things were confusing enough with the b-frame re-ordering, I wasn't sure what would happen with multiple threads and don't have a multi-core machine to test with...

And for that matter, there was another code segment in rc_2pass2_before that didn't look quite right to me with frame reordering, let alone multiple threads - the |last_quant- new_quant| <= 2 bit - but I assumed the original author(s) knew what they were doing...

squid_80

27th September 2007, 07:06

Is altpass2 working 'ok' for people in the multi-threading environment? There was a bit of code I added in there I was concerned about - ie coordination of variable 'vbvfill_before' in routines rc_2pass2_before and rc_2pass2_after.

Things were confusing enough with the b-frame re-ordering, I wasn't sure what would happen with multiple threads and don't have a multi-core machine to test with...

I'm pretty sure it's fine, frames are encoded one at a time and the plugin before and after functions are always executed in order.

plugh

27th September 2007, 15:44

Thanks for the reply; I've been somewhat following the recent 'multi-threaded xvid' posts, and now understand that the threads are used _within_ a frame's encode, so that doesn't affect the plugins.

However, at least with the 1.1.2 core, what debug prints revealed to me was that while 'after' is called in presentation order (ie IBBP), 'before' is not.

MacAddict

2nd November 2007, 10:53

TheRyuu

13th December 2007, 22:36

I'm impressed using this on my DualCore system for the past few months. I'm confused why the 2nd pass only appears to use one core and averages about 51% utilization though. My content isn't filtered with anything but ColorMatrix and isn't resized. First pass usually averages 99-103fps while the 2nd pass is around 12-15fps. Anyone else seeing this behavior?

I am as well. I'm wondering if this is slowing down the encoding at all or if that's just as fast as it's gonna get.

I'm pretty sure it's because xvid_encraw is only using 1 thread for xvid and the other for the frame serving or whatever it's called (which is what it's suppose to do). I'm wondering if this is slowing down the second pass?

squid_80

14th December 2007, 04:05

If you want to force it to behave differently, use -threads 3 (2 for encoding, 1 for reading input) or -threads 2 and -nothreadedinput. I think you'll find it slower though.

TheRyuu

14th December 2007, 06:39

If you want to force it to behave differently, use -threads 3 (2 for encoding, 1 for reading input) or -threads 2 and -nothreadedinput. I think you'll find it slower though.

The best balance seems to be 2 threads for the first pass, and 3 for the second. That way, both passes seem the fastest to me. (~100fps on first, ~30fps on second).

Dunno if it works the same way on quad core though.

20-40

18th January 2008, 19:55

It's excellent! Many thanks to Kopernikus and squid_80… Thank you.

Last week I have recompressed many music video clips from my collection, and because of their number and lack of free time, I've dropped-out any thoughts about two-pass re-encoding, but all of them must fit one DVD - and CQ was out of the picture… Reading this thread, I've played a little, and as I have been very satisfied by results, decided to go on…
This is the simple avisynth script I've used:

AVISource("G:\input.avi")
Crop(0,86,512,212)
MDeblock()
Undot()
LanczosResize(640,268)
ResampleAudio(48000)
SoundOut(output="MP3",mode=2,cbrrate=128,filename="D:\HVS.mp3",autoclose=true,showprogress=true)

Best way to achive desired final file-size was frame size.
Hvs doesn't like any blackness around, so I've cropped any borders if necessary (I have almost all clips in 512 or 576 width in 4/3 AR with borders :-( )
For batch re-encoding, I wrote this drag&drop HVS_AQxvid_enc.bat:

@rem ###################################################################################
@rem # DRAG&DROP working Avisynth script on this .bat (but...ADJUST ALL PATHS HERE!!!) #
@rem ###################################################################################
@echo off
@set fname=%1%
@set out1=%fname%_.avi
echo %fname%
echo %out1%
@rem
@rem
@rem
"C:\xvid_encraw\xvid_encraw_hvs_061119.exe" -i %fname% -avi %out1% -framerate 25 -nochromame -cq 4 -max_bframes 2 -bvhq -bquant_offset 200 -bquant_ratio 100 -hvs_aq " quant lc_lum gl_lum 2 / - theta 255 gl_lum - 3 * 4 / lc_lum - theta + 2 / - " -threads 2 -max_key_interval 250 -progress 10
@rem
@rem
@rem #####################################################################
@rem ## IF YOU DON'T HAVE or use soundout.dll plugin in Avisynth script ##
@rem #####################################################################
@rem wavi.exe %fname% - | lame.exe -m s -b 128 - D:\HVS.mp3
@rem #####################################################################
@rem ## IF YOU PREFER rather old avimux.exe instead of Avimux_GUI ##
@rem ## (un-rem following & delete lines afterwards) ##
@rem #####################################################################
@rem "C:\xvid_encraw\avimux.exe" -apre=60 -inter=100 %out1% %fname%_muxed.avi "D:\HVS.mp3"
@rem #####################################################################
@rem ## Avimux_GUI section ##
@rem #####################################################################
set PATH=E:\Progra~1\AviMux;%PATH%
setlocal ENABLEDELAYEDEXPANSION ENABLEEXTENSIONS
@rem #####################################################################
@rem ################# set IN/OUT, besides... it #############
@rem ###### must be ABSOLUTE path for mp3 from/for soundout.dll !!! ######
@rem #####################################################################
set TMPAMG=%fname%.amg
set SRCVFN=%out1%
set SRCAFN=D:\HVS.mp3
set MUXFLN=%fname%_muxed.avi
@rem ######################################################################
@rem ####### following lines create Avimux_GUI script #########
@rem ######################################################################
echo CLEAR > "!TMPAMG!"
echo LOAD !SRCVFN!>> "!TMPAMG!"
echo LOAD !SRCAFN!>> "!TMPAMG!"
echo SELECT FILE 1 >> "!TMPAMG!"
echo ADD VIDEOSOURCE >> "!TMPAMG!"
echo SET INPUT OPTIONS >> "!TMPAMG!"
echo WITH SET OPTION >> "!TMPAMG!"
echo AVI FORCE MP3VBR 0 >> "!TMPAMG!"
echo MP3 VERIFY CBR NEVER >> "!TMPAMG!"
echo MP3 VERIFY RESDLG 0 >> "!TMPAMG!"
echo END WITH >> "!TMPAMG!"
echo SET OUTPUT OPTIONS >> "!TMPAMG!"
echo WITH SET OPTION >> "!TMPAMG!"
echo OVERWRITEDLG 0 >> "!TMPAMG!"
echo CLOSEAPP 1 >> "!TMPAMG!"
echo DONEDLG 0 >> "!TMPAMG!"
echo MAXFILES OFF >> "!TMPAMG!"
echo STDOUTPUTFMT AVI >> "!TMPAMG!"
echo AVI ADDJUNKBEFOREHEADERS 0 >> "!TMPAMG!"
echo AUDIO INTERLEAVE 1 FR >> "!TMPAMG!"
echo PRELOAD 60 >> "!TMPAMG!"
echo AVI HAALIMODE 0 >> "!TMPAMG!"
echo OPENDML 0 >> "!TMPAMG!"
echo LEGACY 0 >> "!TMPAMG!"
echo RECLISTS 0 >> "!TMPAMG!"
echo END WITH>> "!TMPAMG!"
echo START !MUXFLN!>> "!TMPAMG!"
@rem #####################
@rem ## Finaly ... mux ##
@rem #####################
AVIMux_GUI.exe %fname%.amg

After that, all was piece of cake, well… it took two day for all clips, but quality is great and one pass (my time) per clip was saved.

One thing has left - 1-hour Sade concert captured from TV a long time ago. For fun, I've decided to do two pass encoding and final size should be defined… so:

@echo off
@set fname=%1%
@set out1=%fname%_.avi
Echo %fname%
Echo %out1%
@rem
@rem
@rem
"C:\xvid_encraw\xvid_encraw.exe" -i %fname% -avi %out1% -framerate 23.976 -nochromame -pass1 "xvid.stats" -size 614512 -max_bframes 2 -bvhq -bquant_offset 200 -bquant_ratio 100 -hvs_aq " quant lc_lum gl_lum 2 / - theta 255 gl_lum - 3 * 4 / lc_lum - theta + 2 / - " -threads 2 -max_key_interval 250 -progress 10
"C:\xvid_encraw\xvid_encraw.exe" -i %fname% -avi %out1% -framerate 23.976 -nochromame -pass2 "xvid.stats" -size 614512 -max_bframes 2 -bvhq -bquant_offset 200 -bquant_ratio 100 -hvs_aq " quant lc_lum gl_lum 2 / - theta 255 gl_lum - 3 * 4 / lc_lum - theta + 2 / - " -threads 2 -max_key_interval 250 -progress 10
@rem
@rem
@rem
"C:\MingW\bin\mencoder.exe" %out1% -audiofile %fname% -oac mp3lame -lameopts cbr=128 -ovc copy -o %fname%_muxed.avi

This 2-pass_1cd_HVS.bat has done all job by drag&drop simple avisynth script ( similar to aforementioned, but without SoundOut in it).
Kopernik's encraw has been overwhelmed by input and new squid's encraw has jumped in.

Thank you guys, once again.

20-40

14th February 2008, 11:10

Better late then never. OK, I use XviD 1.2-127 (CVS) and Avisynth 2.6 (also CVS). Somehow, I've noticed "longer-then-usual" encoding times with xvid_encraw (then it should be with reported FPS). I have, back then; hundreds of encodings on queue, all clips were short… and… I didn't care too much about time, I guess.
Then, longer encodings came… well, if someone has 90000 frames to encode, he can expect 900 seconds if reported fps by xvid_encraw was - 100 fps, right?

HOWEVER - encoding took 9000 seconds, which is almost close to three hours compared to "expect" fifteen minutes.
Then, pretty freaked out, I have compiled xvid again - with gcc, with vc71 and vc80. Then with icl 9.0 and then 9.1. Then encraw exe, turning optimizations on and off. With all these compilers. Different sources of encraw - original and squid80's. Mine. Everything stays - bogus fps on screen…

Well, next time I will fire-up stopwatch.exe, rather then seeing-believing. I humbly suggest that to all concerned.
http://free-kr.t-com.hr/zeman/xvid_encraw-fps-sw.jpg

jethro

14th February 2008, 16:16

Did you feed an avisyth script to Xvid_encraw? If yes, it could be that 'enctime' is the CPU time which xvid_encraw itself spent on compressing frames and the missing time comes from avisynth processing.

squid_80

14th February 2008, 16:19

Run xvid_encraw -h and tell me the date and time it was built. I think you're using an older version which only counts the time taken for encoding, not including time taken to fetch the input frame i.e. avisynth processing. But it was over a year ago that I changed that behaviour.

20-40

14th February 2008, 17:29

As I've mentioned, it's about three versions of source code of encraw. As I've said, it doesn't matter which specific version. As I've mentioned, all encraws exhibit the same error - bogus fps during and after encoding (when it reports some data). Also, it doesn't matter by which compiler is compiled - same as by who - same is valid for exes compiled by you or anyone else.
"FPS" spitted by any encoder is very valid information for anyone, in determining script/filtering options, and in the eyes of the most people, something that is looked at first.
What I'm saying is that "FPS" IS bogus, it MAY be a bogus in god knows how many cases, as it can be subject of easy manipulations.
To clarify myself - I suggest to people to check ACTUAL (as REAL) encoding times, and not to focus on ETAs or FPses alone.
But you've made only ONE zip of source available and one zip with compiled encraw and aviwriter.dll (and I love YOUR version - adding VD routines was very nice idea). There is no other "versions" of your encraw. Kopernikus made available two (but rather obsolete by now). Xvid.org cvs is one only. My own is my gcc and intel modified. Sucus is, it realy doesn't matter.
However:
1.C:\xvid_encraw>xvid_encraw_squid.exe -h >xvid_encraw_squid.txt
xvid_encraw built at 10:22:53 on Aug 31 2007

2.C:\xvid_encraw>xvid_encraw_vc71.exe -h
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampe
rt 2002-2003
xvid_encraw built at 10:08:29 on Feb 14 2008

3.C:\xvid_encraw>xvid_encraw_hvs_061119.exe -h
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampe
rt 2002-2003

xvid_encraw built at 16:38:19 on Nov 19 2006

4.C:\xvid_encraw>xvid_encraw_orig.exe -h
xvid_encraw - raw mpeg4 bitstream encoder written by Christoph Lampe
rt 2002-2003

xvid_encraw built at 15:38:53 on Feb 13 2008
(2. and 4. are CVS XviD.org, 2. is with added your aviwriter code and 4. is CVS but hvs modified version for my gcc-4.3 CYBORG)

P.S. It reminds me on old, built-in prank in all Windows: "Estimated time: 2 minutes left".
Two minutes later is" Estimated time: 6 minutes left". Time CAN flow in - negative. :-)

henryho_hk

15th February 2008, 02:57

But it was over a year ago that I changed that behaviour.

Is it now a simple (end time - start time) over the whole clip (or finished portion)? Otherwise, many factors may come in.

[Too busy on my work for the last year.... no time for on my own batch script]

20-40

15th February 2008, 10:35

ANY encoder is the last in the encoding chain, it means that it doesn't matter what is input. Encoding application must-should-only could report actual (REAL) number of encoded frames by second. Fluctuations during encoding are expected, as they depend on complexity of input, by filling/emptyng caches/buffers, by writing the output… just to mention a few…
Nevertheless, at least FINAL encoding time should be right, right?
BUT - it's not.
It looks simple, simple end_of_encoding minus start_of_encoding equal total_encoding_time, but it is not - as you can see from the screenshot I've provided dozen inches beforehand.
It seems that people have mixed-up few things.

In the case of XviD codec and encraw, fps is stored and represented in/as "%6.2f". It has nothing to do with avisynth. This value is reported by codec ALSO for VirtualDub, Avidemux and any other application we use. If there were an error in ANY other part of the chain (avisynth with filters, codec itself) - such error would be manifested by/in them too, right? However, it is not the case.
And - in my case "%6.2f" is %6.2f/(ARG_THREADS + 1), but it doesn't fix the error.

squid_80

15th February 2008, 13:50

J:\ts-temp2>copy con enc_test.bat
xvid_encraw -h 2> help.txt > nul
find "built" < help.txt
time < nul
xvid_encraw -i lost.avs -frames 50000 2>nul | find "Tot:"
time < nul
^Z
1 file(s) copied.

J:\ts-temp2>enc_test.bat

J:\ts-temp2>xvid_encraw -h 2>help.txt 1>nul

J:\ts-temp2>find "built" 0<help.txt
xvid_encraw built at 10:22:53 on Aug 31 2007

J:\ts-temp2>time 0<nul
The current time is: 21:56:51.29
Enter the new time:
J:\ts-temp2>xvid_encraw -i lost.avs -frames 50000 2>nul | find "Tot:"
Tot: enctime(ms) =5591690.00, length(bytes) = 1729539969

J:\ts-temp2>time 0<nul
The current time is: 23:30:04.39
Enter the new time:
Clock end time - clock start time = ~5593 seconds.
xvid_encraw reports 5591.690 seconds.
Using my build, as stated by the -h option.
I don't see any problems here.

20-40

16th February 2008, 14:13

Combination of xvidcore-vfw-ax compiled with gcc-4.3CYBORG and your source (squid80's) encraw compiled with 2003 with your's source of aviwriter dll compiled with 2005 on two cores/threads works error free!
I'm still puzzled with all this, but - everything is fine now.

However, squid80, thank you for reactions.

Lenchik

5th January 2009, 18:45

Hope thgis is right thread to ask.

I am trying to create a series of .bat files for batch processing of videos from my photo.
Previous .bat files create series of .avs files. For now problem is to create .bat for video encoding using xvid_encraw.
Here goes code:
setlocal ENABLEDELAYEDEXPANSION ENABLEEXTENSIONS

for %%i in (*.avs) do (
start "%%~ni - Pass 1/2" /b /wait /belownormal xvid_encraw.exe -type 2 -i "%%~dpi%%i" -pass1 "%%~dpi%%~ni.stats" -bitrate 2000 -kboost 100 -chigh 30 -clow 15 -overhead 0 -max_key_interval 100 -nopacked -vhqmode 4 -closed_gop -lumimasking -notrellis -imin 1 -pmin 1 -max_bframes 1 -bvhq -bquant_ratio 162 -bquant_offset 0 -bmin 1 -par 1:1 -threads 3 -vbvsize 3145728 -vbvmax 4854000 -vbvpeak 8000000

start "%%~ni - Pass 2/2" /b /wait /belownormal xvid_encraw.exe -type 2 -i "%%~dpi%%i" -pass2 "%%~dpi%%~ni.stats" -bitrate 2000 -kboost 100 -chigh 30 -clow 15 -overhead 0 -max_key_interval 100 -nopacked -vhqmode 4 -closed_gop -lumimasking -notrellis -imin 1 -pmin 1 -max_bframes 1 -bvhq -bquant_ratio 162 -bquant_offset 0 -bmin 1 -par 1:1 -threads 3 -avi "%%~dpi%%~ni-video.avi" -vbvsize 3145728 -vbvmax 4854000 -vbvpeak 8000000
)

The problem is that sometimes .avs files are not opened correctly (may be because of directshowsource aplied to .mov files). Anyways, i want to create script that will start first pass encoding of current .avs again and again if first attempt fails (if i am doing this manually for jobs in megui or virtualdub - it works fine), and then in same way for second pass of each .avs. How can i do it? may be some errorcheck or error output of xvid_encraw? I am not so good at .bat or at other programming to find out this by myself alone (that's why i have taken some parts of code from avs2qxvid).
By the way, any other suggestions about script are welcome.
Hope you understood my idea and my english.

buzzqw

5th January 2009, 20:37

use ffmpegsource and not directshowsource :)

BHH

henryho_hk

20th February 2009, 18:25

squid_80, would you mind making a win64 compile of the latest version? ^_^

I have just installed Vista64 SP1 and I was trying the 64bit compiles of Avisynth, xvid_enraw with 64bit XviD. With a simple script (DVD mpeg2source+tdeint+mipsmooth), they are using 1.7GB ram per xvid_encraw process. In the same machine (Q6600 + 8GB RAM) under the same OS, the 32-bit counterparts are using 512MB only. Another interesting point is the speed. At thread=1 or 5, the 64-bit combo is running at similar speed of 25~30 mins for a 18-min clip (cq3, vaq, max-b=1, vhq1, quality6, bvhq, trellis, etc.) For the 32bit combo, it's 14 mins for thread=1 and 24 mins for thread=5. Seems we still have a long way to go for 64bit platform.

squid_80

28th February 2009, 04:57

- The pc I use to compile xvid_encraw has been in pieces since xmas, so it's a bit hard to make new builds at the moment. But I think the 64-bit build should be up-to-date, it doesn't need to be recompiled to use new xvidcore builds.
- Always use setmemorymax when checking memory usage for programs that use avisynth scripts. Different avisynth versions may have different default memory usage limits.
- If threading in the 64-bit xvid isn't working properly, talk to the people who compiled xvidcore.dll. For me it works fine.
- 32-bit version takes 14mins with 1 thread but 24 mins with 5 threads? Again, talk to whoever made your xvidcore because that's not right at all.

henryho_hk

2nd March 2009, 04:07

I recall the 64bit Avisynth from you was 2.55. I will try tweaking setmemorymax to see if it makes any difference. For threading, I was hoping for a 64bit xvid_encraw build of your greatly improved version with threaded input (http://forum.doom9.org/showpost.php?p=1039344&postcount=696). I would not it on the net. Perhaps I should try compiling it myself.

I am sorry that I seem to have made some typos in the speed figures in my last post (it should be something like "32-bit version takes 24mins with 1 thread but 14 mins with 5 threads"). I will test it again tonight.

hellrasinbrasom

2nd March 2009, 20:53

Have you tried using Xvid4PSP to deal with encoding problems

:helpful:

Kurtnoise

16th May 2009, 15:18

I've a question about -zq vs -zw vs -zones switches : the first two are redundant, aren't they ? We can safety use the last one for the same thing, correct ?

juGGaKNot

20th May 2009, 16:12

Does it like 4 threads ?

I use

echo %NUMBER_OF_PROCESSORS% > "%mypath%\proc.txt"
set /p trd=<"%mypath%\proc.txt"
set /A trd2=%trd%+1
del "%mypath%\proc.txt"

And it works fine for single and dual core but on a quad xeon it does not work ( 3 threads not 5 )

Kurtnoise

20th May 2009, 17:32

you should read previous comments instead of asking this...

juGGaKNot

21st May 2009, 07:37

you should read previous comments instead of asking this...

Aham so 3 threads for all cores ?

halsboss

22nd August 2009, 12:00

I'm confused as to which versions are available ... I read here and there about squid_80's version (whatever that is) but don't know where to download squid_80's own build ...

Also is it possible, similar to HCenc, for pass1 to produce a temporary decoded file to be used as input to pass2 so that pass2 doesn't have to process a glacially slow avisynth script a second time ?

ps go the crows, squid_80 :)

halsboss

23rd August 2009, 07:40

Jeepers. About 30% of the time xvid_encraw crashes for me with a dialog box
"xvid_encraw has encountered a problem and needs to close."
When I click on more info it says "ModName avisynth.dll Modver 2.5.7.0 Offset 0000adbb".
Any ideas ?

Seems to happen randomly on either pass1 or pass2. Settings and commandline are

"C:\software\XVID_ENCRAW\xvid_encraw.exe" -type 2 -i "G:\HDTV\2\test.avs" -max_bframes 2 -bitrate 1500 -pass2 "G:\HDTV\2\test.stats" -stats -framerate 25 -max_key_interval 125 -asm -quality 6 -vhqmode 4 -bvhq -packed -closed_gop -lumimasking -imin 1 -bmin 1 -pmin 1 -clow 10 -overhead 0 -vbvsize 1835008 -vbvmax 8000000 -threads 3 -progress 25
"C:\software\XVID_ENCRAW\xvid_encraw.exe" -type 2 -i "G:\HDTV\2\test.avs" -max_bframes 2 -bitrate 1500 -pass2 "G:\HDTV\2\test.stats" -stats -framerate 25 -max_key_interval 125 -asm -quality 6 -vhqmode 4 -bvhq -packed -closed_gop -lumimasking -imin 1 -bmin 1 -pmin 1 -clow 10 -overhead 0 -vbvsize 1835008 -vbvmax 8000000 -threads 3 -progress 25 -o "G:\HDTV\2\test.m4v" -kboost 50

using .avs
SetMTmode(mode=5,threads=4) # start with mode=5 for source http://forum.doom9.org/showthread.php?p=1067216#post1067216
SetMemoryMax(512)
LoadPlugin("C:\SOFTWARE\DGindex\DGDecode.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-zzz\tdeint.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-zzz\yadifmod.dll")
LoadPlugin("C:\Program Files\AviSynth 2.5\plugins-zzz\NNEDI.dll")
mpeg2source("G:\HDTV\2\test.d2v",cpu=4,ipp=true)
AssumeFPS(25)
AssumeTFF()
SetMTmode(mode=2,threads=4)
yadifmod(mode=0,order=1,edeint=nnedi(field=1,threads=4)) # mode=0=normal deinterlace
spline36resize(640,360)
SetPlanarLegacyAlignment(True)
Distributor()

Sharc

1st September 2009, 07:58

@halsboss
I also experienced a number of crashes with xvid_encraw (OS is Vista). The crashes seem to be caused by the -qpel function for certain sources.
Workaround was to disable -qpel. :(

henryho_hk

2nd September 2009, 08:33

Same experience here... 720p + QPel ... during early 2nd pass

henryho_hk

20th April 2010, 02:13

squid_80, may we have a x64 compile of your 20070831 version of xvid_encraw?

MOS-Marauder

5th July 2010, 17:06

Hi xD
Question.. i saw that XVid ENcraw writes a custom pixel shape (1:1) instead of SQUARE in the bitstream. Any chance of fixing ?

Chris

Sharktooth

5th July 2010, 17:34

it's not a bug.
SQUARE is NOT written... is ASSUMED.
while "Custom Shape" is a wrong term used by avinaptic that means an AR is specified in the bitstream...

in practice avinaptic ALWAYS report Square Pixels for streams that have NO AR SIGNALED while it ALWAYS report Custom Shape for ALL the streams with any signaled AR (even 1:1).
that's an avinaptic limitation.

Gew

19th August 2010, 12:13

Hi!

I'm just curious. As I've understood it, xvid_encraw uses the same default settings as xvidvfw.dll, hence they should output the same?

However, when I load an .avs in VirtualDubMod, and encode it with XviD/VfW, and then encode the very same .avs with xvid_enraw.exe, the output is not entirely the same. In fact, even the bitrate differs a few kbps. Am I missing out on something? I have a theory that the codec settings are in fact similar, but there's a difference in how the source (.avs) is being projected/fed into the codec, hence the non-identical results.

Is this correct?

Ty in adv~
Regards~

Sharktooth

19th August 2010, 15:20

VFW bitstream writing depends on the host application while encraw has bitstream writing.
also there are other differencies... so the output will always be different.

Gew

19th August 2010, 17:17

Ty for your clarification Sharktooth. I think I got it.
It's like comparing drinking something straight out of the bottle with from a glass.
It'll be roughly the same result, but still not bit-identical.

:thanks:

Gew

22nd September 2010, 01:04

What's the difference between the three reported quants?
I.e. when pulling a quick 250 frames encode using -bitrate 50 I get:

I quants = 4 / 9.00 / 14
P quants = 4 / 18.03 / 25
B quants = 7 / 25.18 / 31

What are the three different values reporting? Which one is the actual "overall quantizer" being used..?
Or are the different figures three different ways of telling the exact same thing?
Or any other relationship in between them, something I'm not aware of?

Ty in adv~

:stupid:

Brother John

22nd September 2010, 01:32

Those are minimum/average/maximum quants used for the respective frame types. Average is a calculated value from all frames; you might call this the »overall quantizer«.

Btw: Xvid quants can only be integer number (from 1–31). Whenever you encounter a fractional number it can only be some average value, but never an actual »real« quant.

Gew

22nd September 2010, 01:45

Ty a lot Brother John.
Your explanation was well-defined and makes perfectly good sense.
Sorry for "bothering" thread with silly newbish question.
I tried Google for answers before posting but no strings gave any result.

Anyways..

:goodpost::thanks:

Cheers~

Gew

26th June 2011, 14:53

Hmm. Compatibility issue with recent xvidcore.dll or something?

I'm using:
xvid_encraw built at 10:22:53 on Aug 31 2007
xvid [info]: xvidcore build version: xvid-1.3.2

Just tried an encode with an .AVS for source. Final bitrate is ~half what I set.
Also, now when I watch my result, the ~2/3 bottom of the video is all just a smear.

Any ideas?

EDIT:
Confirmed! Just tried same encode using an older xvidcore.dll
xvid [info]: xvidcore build version: xvid-1.2.2
No picture/framerate issues there. Any chance for a quick fix? ;)

Jawor

26th June 2011, 22:02

Yes, squid_80's version doesn't work correctly with xvidcore.dll 1.3.x (it has been mentioned somewhere in other topics). The reasons lie somewhere in changes made to the Xvid source code, but AFAIK they haven't been identified (otherwise a fix would appear). squid_80's last build comes from August 2007 (that's almost 4 years), so some incompatibilities are to be expected....

Gew

4th January 2012, 18:58

Hi (again)...

Now it's been more than seven months. Perhaps foolish of me, but I'm still curious. Are there any versions (builds by others perhaps?) of xvid_encraw available that will handle recent xvidcore.dll without any hassle? I'd be grateful. Let's say I have my reasons why I prefer the xvid_encraw (over eg. ffmpeg/mencoder) for batch encodes.

Thank you in advance~
Regards~
Cheers~

madhatter300871

11th January 2012, 14:29

kalehrl

11th January 2012, 18:28

Does it use only one core in the second pass?

madhatter300871

11th January 2012, 18:48

Does it use only one core in the second pass?

I think so, but I'll double check on my next encode.

Gew

14th January 2012, 15:55

Hi

If you just download the latest xvid from the xvid website, it comes with xvidencraw, it works for me with no problems. All I did was copy the xvidencraw.exe and xvidcore.dll (from system32) into my own folder, it works no problems.

Gosh, I never noticed that Xvid had/has started inbedding xvid_encraw.exe in their Win32 bundles. I just downloaded latest stable (Xvid-1.3.2-20110601.exe). Haven't tried it just yet, but I doubt I will run into any problems.

The issue I had with my xvid_encraw.exe (dated 2007 or something) was that if I used an xvidcore.dll from the 1.3.x series I had severely messed up output. Half the screen was "floating" somehow. Therefor I have kept an old Xvid 1.2.x xvidcore.dll just for use with xvid_encraw. This issue is mentioned a couple of pages up in this very thread.

Anyways, big thanks.
Now I will probably be able to kick it with latest Xvid! :D

Cheers!

:thanks:

Update!

Not only did it work with excellent result; average encoding speed raised from ~77 FPS to ~107 FPS (just ran side-by-side test using same input). This (I guess) a result of my old xvid_encraw.exe / xvidcore.dll combination only used 3 out of 4 cores on my CPU (xvid [info]: Detected cpus = 4, threads requested = 3, threads in use = 3), whereas this new set simply yields "Detected 4 cpus, using 4 threads". Awesome. One thing that made me worry was that at first my AVI would not play; MPC gave me "Cannot render the file". After some further digging, I noticed that the stream was actually not wrapped into a AVI container, but more or less raw. I then noticed that the "-avi" switch puts output in AVI, whereas "-o" (which I have always used before) does this raw-isch result. Anyways, like I said, works like a charm now!