xvid_encraw.exe affects encoding result?

[kypec]

I observed something strange while encoding with xvid 1.2.1

I'm using xvid-1.2.1-vaq-x86.7z (http://www.xvidvideo.ru/component/option,com_docman/task,doc_download/gid,808/) package downloaded from russian server.
I have unpacked and installed the files contained within:
xvid.ax 73728 13. 12. 2008 00:12:48
xvidcore.dll 617984 13. 12. 2008 00:10:10
xvidvfw.dll 178688 13. 12. 2008 00:12:16
xvid_encraw.exe 142336 13. 12. 2008 00:10:12
xvid.inf 2864 27. 11. 2008 17:42:00

My source is VOB file (2nd episode of FRIENDS DVD PAL R2) with the following Avisynth script# Load all the plugins that will be used in the script
LoadCPlugin("T:\Plugins\yadif.dll")
LoadPlugin("T:\DGMPGDec\DGDecode.dll")
LoadPlugin("T:\Plugins\ColorMatrix.dll")
# Open source video clip
MPEG2Source("E02.d2v",info=3)
# Transform to colour space which is expected by Xvid encoder
ColorMatrix("Rec.709->Rec.601",hints=true)
# Deinterlace video
Yadif()
# Resize video
Spline36Resize(512,384)
#~ # Helps to locate blank alias completely black frames
#~ Histogram(mode="levels")
# Trim only the desired frames
Trim(50,32874)

When I run the encoding process via included xvid_encraw.exe with the following 2-pass commandsT:\xvid\xvid_encraw.exe -i "E02.avs" -avi "E02_preliminary.avi" -pass1 "E02_xvid.stats" -max_key_interval 250 -vhqmode 4 -bvhq -lumimasking -progress 25 -stats >"E02_pass1.log"
T:\xvid\xvid_encraw.exe -i "E02.avs" -avi "E02.avi" -pass2 "E02_xvid.stats" -bitrate 860 -par 1:1 -max_key_interval 250 -vhqmode 4 -bvhq -lumimasking -progress 25 -stats >"E02_pass2.log"
the results look like this:
E02_pass1.logTot: enctime(ms) =247716.00, length(bytes) = 198546081
Avg: enctime(ms) = 7.55, fps = 132.52, length(bytes) = 6048, psnr y = 43.07, psnr u = 45.13, psnr v = 46.24
I frames: 378 frames, size = 25124/9496988, quants = 2 / 2.00 / 2
P frames: 11982 frames, size = 11899/142575894, quants = 2 / 2.00 / 2
B frames: 20465 frames, size = 2270/46473199, quants = 4 / 4.00 / 4

E02_pass2.logTot: enctime(ms) =695318.00, length(bytes) = 141089913
Avg: enctime(ms) = 21.18, fps = 47.21, length(bytes) = 4297, psnr y = 42.54, psnr u = 44.79, psnr v = 45.93
I frames: 378 frames, size = 21221/8021764, quants = 2 / 2.47 / 3
P frames: 11982 frames, size = 7760/92981743, quants = 2 / 2.69 / 3
B frames: 20465 frames, size = 1958/40086406, quants = 4 / 4.43 / 5


However, when I replace xvid_encraw.exe with one provided by squid_80 (http://members.optusnet.com.au/squid_80/xvid_encraw.zip) then there are some tiny differences at B-frames in the 1st pass and at I- & P-frames in the 2nd pass:
E02_pass1.logTot: enctime(ms) =865935.00, length(bytes) = 198546056
Avg: enctime(ms) = 26.38, fps = 37.91, length(bytes) = 6048, psnr y = 43.07, psnr u = 45.13, psnr v = 46.24
I frames: 378 frames, size = 25124/ 9496988, quants = 2 / 2.00 / 2
P frames: 11982 frames, size = 11899/ 142575894, quants = 2 / 2.00 / 2
B frames: 20465 frames, size = 2270/ 46473174, quants = 4 / 4.00 / 4

E02_pass2.logTot: enctime(ms) =1452702.00, length(bytes) = 141100709
Avg: enctime(ms) = 44.25, fps = 22.60, length(bytes) = 4298, psnr y = 42.54, psnr u = 44.79, psnr v = 45.93
I frames: 378 frames, size = 21185/ 8007997, quants = 2 / 2.47 / 3
P frames: 11982 frames, size = 7762/ 93008277, quants = 2 / 2.69 / 3
B frames: 20465 frames, size = 1958/ 40084435, quants = 4 / 4.43 / 5


How is such discrepancy possible when both processes used the same xvid core and scripts?:confused:

[Dark Shikari]

Did you check to see if it's consistent if you run the same xvid_encraw twice?

[kypec]

OK, I just started second run via russian version of xvid_encraw and will post results in few minutes...
Well, 1st pass finished with identical results as I expected.
Then I ran squid_80's version with -nothreadedinput and the 1st pass yielded the same results as russian's version including LOG and STATS files except the fact that the encoding speed of the latter reaches ~132 fps compared to only ~31 fps for squid_80. Both processes reported 2 CPU's available (Athlon 64 X2) and 2 threads used for encoding.
Resulting AVI file contains equally long video stream but russian's header is somehow corrupted, at least AVInaptic says so (suspicious details are highlighted):[ About file ]

Name: E02_preliminary.avi
Date: 22/01/2009 09:39:08
Size: 199,346,688 bytes (190.112 MB)

[ Generic infos ]

Play duration: 00:21:53 (1313 s)
Container type: AVI
Number of streams: 1
Type of stream nr. 0: video
Audio streams: 0
Corrupted AVI sections: îdh


[ Relevant data ]

Resolution: 512 x 384
Width: multiple of 32
Height: multiple of 32

[ Video track ]

FourCC: xvid/XVID
Resolution: 512 x 384 (384 x 512)
Frame aspect ratio: 4:3 = 1.333333
Pixel aspect ratio: 1:1 = 1
Display aspect ratio: 4:3 = 1.333333
Framerate: 25 fps
Number of frames: 32825
Stream size: 198,546,081 bytes
Bitrate: 1209.724789 kbps
Qf: 0.246119
Key frames: 378 (0; 135; 222; 268; 317; ... 32804)
Null frames: 0
Min key int: 1
Max key int: 250
Avg key int: 86.838624
Delay: 0 ms

[ About MPEG4 encoding ]

User data: DivX503b1393p
User data: XviD0050
Packed bitstream: Yes
QPel: No
GMC: No
Interlaced: No
Aspect ratio: Square pixels
Quant type: H.263

[ Profile compliancy ]

Profile to check: MTK PAL 6000
Resolution: Ok
Framerate: Ok
Warning: If you need a more complete report, then click on "DRF analysis"

This report was created by AVInaptic (18-11-2007) on 22 gen 2009, h 13:15:56


Target AVI (from 2nd pass) remuxed in AVIMux-GUI is being reported fine in AVInaptic but the frame resolution values @ offset 0000A0h are still wrong I think (384x512 instead of 512x384) :(

[kypec]

Until squid_80 will turn around and kindly explain why is his version of xvid_encraw much slower than russian's one though utilizing 2 threads as well here is my solution:
All my encodes are being finally remuxed with AVIMux-GUI (http://www.alexander-noe.com/video/amg/) which fixes most of the garbage and mess originally produced by russian's (alias DAMN FAST) xvid_encraw output.
Problematic info about the frame width and height still remains though :mad:
in the struct AVIStreamInfo header (VIDEO stream part) rcFrame.Left = 0
rcFrame.Top = 0
rcFrame.Right = 384
rcFrame.Bottom = 512
In order to swap improperly written values of frame resolution I've used free HEX editor (http://www.chmaas.handshake.de/delphi/freeware/xvi32/xvi32.htm) which supports command line scripting. Simple script takes care of replacing the necessary 4 words in the sequence:
fix_AVIheader.xscREM Go to start of file
ADR $0000
REM Change AVI Stream Header structure elements (as short 16-bit signed)
REM by keeping rcFrame.Left=0, rcFrame.Top=0
REM and swapping rcFrame.Right=512, rcFrame.Bottom=384
REPLACE 00 00 00 00 80 01 00 02 BY 00 00 00 00 00 02 80 01

Hope to see original xvid_encraw AVI writer fixed or squid_80's performance improved, whichever comes first.:rolleyes:

:::EDIT:::
Oops, now I realized that there been some time ago a long discussion (http://forum.doom9.org/showthread.php?p=1100137#post1100137) about how encoding FPS and time measuring was made in xvid_encraw. According to those figures the FPS reported by russian's version is still VERY INNACURATE. So in the end they are probably equally fast...

[alexins]

xvid_encraw my assembly, based on the original code cvs (xvid.org) (http://cvs.xvid.org/cvs/viewvc.cgi/xvidcore/), it does not contain any modifications, which means it was built as it is. xvid_encraw built squid_80, has been modified, it is associated with aviwriter.dll. And this is the reason for differences in the results of coding.

[kypec]

xvid_encraw built squid_80, has been modified, it is associated with aviwriter.dll. And this is the reason for differences in the results of coding.
The results of both encoders are SAME as long as they both run either as single-threaded or as 2 threads (of course the AVI header is not taken into account for this comparison). However, there are few slight differences, mostly in B-frames, depending whether the source has been encoded by single-threaded xvid core or by two-threaded engine. The speed benefit of using multi-threaded processing is NONE on my Pentium4@1.2GHz HT enabled system, or better said, is NEGATIVE because at least 1st pass is faster by few seconds (with threaded input function turned on).
I'll stick with squid's version of xvid_encraw, it's faster (or NOT slower at least) than original with threaded input enhancement, it produces proper AVI header and reports true processing times/fps.:p

[squid_80]

Until squid_80 will turn around and kindly explain why is his version of xvid_encraw much slower than russian's one though utilizing 2 threads as well here is my solution:
Simple answer for the speed difference:
As you've found already the build made by Alexins based on xvid's CVS measures only the time taken to compress the frame, not taking into account the time taken to fetch the original frame and write to the output file. Instead of relying on the FPS figures reported on screen, use another method to time them and they should be roughly the same. (NEVER state in public that program A is much slower than program B based solely on the statistics produced by said programs.)
To explain the bitstream differences:
Different number of threads being used. Threaded input takes up a core for itself, reducing the number of encoding threads by one. I ran a lot of tests when implementing threaded-input and found this was always the best configuration.
To explain the differences between my source and CVS:
Originally xvid_encraw was in xvid's CVS as an example on how to use xvid's API. I added .avi input, .mkv output and most of the missing options that are found in the VFW config. The modified source got added back to xvid's CVS (minus the mkv modules), but MeGUI had started using xvid_encraw for xvid encodes and it was still buggy (including the AVIStreamInfo bug (http://forum.doom9.org/showthread.php?p=821507#post821507)). So I changed from using avifile for output to aviwriter to support files >2GB, fixed the fps timing and added threaded-input. These changes were never adopted into CVS, and I don't really expect them to ever be added (except maybe the AVIStreamInfo fix); xvid_encraw has grown way beyond a simple example program and frankly could do with a complete rewrite.

[kypec]

Instead of relying on the FPS figures reported on screen, use another method to time them and they should be roughly the same. (NEVER state in public that program A is much slower than program B based solely on the statistics produced by said programs.)
Thanks for reminding me, I'll do that for every application from now on ;) And sorry again for my hasty statements about your program, I must remember also "read twice, post once!" :thanks: