Custom No N-VOP XViD?

[crasus]

Good afternoon! :)


Some time ago, while browsing the forums I came across a custom modded xvid with N-VOPs removed. However, that version is quite old - 1.1.2 if I remember correctly.

Does anyone know about a similar custom compile for the latest version? (I'm currently using Koepi's compile, and even though I have B-Frames enabled and there's no drop ratio, I still get the damn NVOPs which jamm my player)

Thank you!

[Nic]

Have you got packed bitstream turned on? If so, you'll get N-VOPs for DivX 5 compatibility (or similar...from memory). If not then that's strange. If you do need it on, but don't need the N-VOPs...that's strange :)

-Nic

ps
Weird, I had no idea what build without N-VOPS you were talking about...looks like I made one back in the day tho:
http://forum.doom9.org/showthread.php?t=93263
I have no recollection of that at all :)

[crasus]

Thanks for the reply, Nic. Packed Bitstream is ALWAYS disabled on my encodes. As is, QPEL & GMC.

Yes, it was your compile :D Quite usefull back then. I've tryied several builds (koepi's latest, xvidvideo.ru's latest - both 1.2.x & 1.3.0 beta) on different sources (with different profiles including Advanced Profile @L5 & MTK NTSC). Same thing happens. And it's annoying.

Clean system, with only xvid & ac3 installed - actual encoding done via VirtualDubMod. Any ideas, please? :)

Thanks,
Crasus

[SeeMoreDigital]

Just in-case you are interested. A while ago, I wanted to create some test cards for use with "stand-alone players" and required an Xvid build that was able to generate encodes that contained "real" N-VOP's (null-frames), which have nothing to do with packed bitstream.

Anyway, here's a link to some of the discussions: -

http://forum.doom9.org/showthread.php?p=770260#post770260

http://forum.doom9.org/showthread.php?p=896080#post896080


Cheers

[crasus]

SeeMoreDigital, I've already read those topics - but thanks for the notice.

Maybe Nic or celtic_druid will be so nice to make a build of the latest xvid with n-vops complitely disabled when frame drop ratio is set to 0, if that's not too difficult? :)

Oh, I forgot to mention something that I didn't really understand.

On some sources, when I set the I-Frame interval to 300 this resulted in 0 N-VOPs. However, if I did set the same source to 240 or 250, N-VOPs started to appear. On other sources, no matter what I did, N-VOPs appear.

[sneaker_ger]

I suggest changing xvid in a way, so that setting "-1" will deactivate N-VOPs for both VfW and xvid_encraw. That would make us able to deactivate N-VOPs in all builds yet to come.

[Nic]

Interesting, as I can't see in the code how you could be getting a N-VOP. I've only looked quick, but there's:

In enc_encode (encoder.c)
if ((pEnc->mbParam.global_flags & XVID_GLOBAL_PACKED && pEnc->bframenum_tail > 0))
Although they may have meant to have the ')' after XVID_GLOBAL_PACKED, precedence means it shouldn't call BitstreamWriteVopHeader with vop_coded == 0 without packed bitstream on.

The only other place is:

In FrameCodeP (encoder.c)
if ( current->sStat.kblks + current->sStat.mblks <= (pParam->frame_drop_ratio * mb_width * mb_height) / 100 &&
( (pEnc->bframenum_head >= pEnc->bframenum_tail) || !(pEnc->mbParam.global_flags & XVID_GLOBAL_CLOSED_GOP)) )

I suppose that maybe should be a '<' rather than '<=' ?

But at this late hour, that's all I can think of :)

Cheers,
Nic

EDIT:
Interesting, looking in the code again, that <= is the issue, but it's not an issue :) The code already does as sneaker_ger suggests - i.e. sets frame_drop_ratio = -1 if it wants to force no N-VOPs. But frame_drop_ratio = MAX(create->frame_drop_ratio, 0) in enc_create forces a minimum of 0 from the user.

EDIT2:
Replace Koepi's xvidcore.dll (probably in c:\windows\system32) with the one from:
http://nic.dnsalias.com/xvidcore_nvop.zip
See if that helps.

[crasus]

And Nic saves the day! :D

Your modified core did the trick. No more unwanted N-VOPs (I've tested on 3 different sources). Maybe you could report this to the XViD team, so they will release an update?

Thank you for taking the time to look into the problem.

[Nic]

Maybe i'll mail the dev list - it's quite quiet nowadays so hopefully they won't mind the noise....

For old times sake, I updated the ol' website build:
http://nic.dnsalias.com
(it's a newer version than the xvidcore listed above)

Had to write that nsi script on the quick side (lost the original), not very tested. But seems to work :)

Cheers,
Nic

[crasus]

I owe you a nice, cold beer! Whenever you're around, drop a sign - first pint is on me :D

[crasus]

Dear Nic,

After extensive tests with the build you posted on your website, I noticed something. Altough I set a bitrate of 1850kbps, the result has a bitrate of 1955kbps.

Tested on 3 different sources. Any hints to what's causing this?

Thank you :)

[Nic]

Strange - not sure why that could be - unless something's changed in CVS....
....Have you got any more details? Like all your settings? Does it fail to hit bitrate in single pass and 2 pass modes? etc

-Nic

[crasus]

It fails to hit the bitrate on the 2pass mode.

Profile : Advanced Simple@L5
Quantization : MPEG
Adaptive Quantization : OFF
B-VOPs : 1/1.62/0.00
Packed Bitstream/QP/GMC : OFF
Zones : None
Motion Search : 6 - Ultra High
VHQ Mode : 4 - Wide Search
Use VHQ for B-frames
Use Chroma Motion
Frame drop : 0
Maximum I-Frame : 240
Quantizer restrictions : 1/31/1/31/1/31
Trellis quantization

AviSynth Script :

TFM(slow=2).TDecimate()
Crop(18,8,-14,-8)
BicubicResize(640,480,0,0.5)
greyscale()
DeGrainMedian(limitY=2,limitUV=3,mode=1)
DeGrainMedian(limitY=2,limitUV=3,mode=1)
FFT3DFilter(sigma=3)

Encoding via VirtualDubMod.

[Nic]

Are you definitely not getting the same sizes with Koepi's latest build (www.koepi.info)? - I get exactly the same size output as Koepi's build....

Strange!

-Nic

[crasus]

It seems you are right! :)

When using koepi's regular build with its files, the bitrate comes out as configured aka 1850.

The issue seems to appear only after replacing Koepi's xvidcore with your compile. Same thing when using a clean install with the setup from your website.

[crasus]

Dear Nic,

I did a test encode via xvid_encraw, using the first xvidcore you uploaded. Exactly the same settings. The result came a little different, closer to the requested bitrate but still off.

On my first encodes via VirtualDubMOD, I got a bitrate of 1955. On this latest encode, I got 1919.

[crasus]

Ok, now I'm really puzzled. I took it step by step and checked everything.

Apparently, the oversized bitrate is caused by the denoising filters - no matter what xvid build I use. Which is weird, as I used the same combination on some other different ocassions.

If I remove them, the bitrate comes out as required on any xvid build. Tryied with koepi's clean build, your modified version & meGUI's.

Does that makes any sense at all?

Update : The problem appears only after enabling FFT3DFilter. Without it, all is good.

Moving the discussion to the Avisynth Usage section of the forum.

[Didée]

Did you do a fresh 1st-pass in all cases, or did you reuse a once made stats file?

In any case, try again with quantizer restrictions of 2/31, not 1/31. FFT3DFilter usually makes the video more compressible, not less. It could be that Xvid can't reach destination rate with q2, hence resorts to q1 to avoid undersizing ... and those q1 frames are so unproportional huUuge that rate control might fail to compensate. (requires more aggressive "overflow treatment" settings, i.e. bigger numbers than 5/5/5).

[Nic]

Hi Crasus,

Thanks for persisting with your testing! I tried lots of things to re-create the issue you were having, but couldn't get anywhere....I didn't think to try your exact avisynth script.

Cheers,
Nic

[crasus]

Dear Nic,

It's my pleasure to help. After all, it's my interest as well :D

A short update, after more tests. I tryied Didee's sugestions - but that didn't fix the problem. Applying a 2/31 quantizer restriction resulted in an undersized rip (~800kbps). More agresive overflow treatment didn't help either.

5 minutes ago, I did another DVDRip with the build from your website. A simple deinterlace with TDeint came with a proper bitrate. However, I noticed that it was a PAL-to-NTSC transcode - so I used srestore to get it back to 25fps. The new rip came, once again, oversized. I used the same XViD settings I mentioned before with no denoiser.

TDeint(mode=1, mtnmode=1)
srestore()
Crop(2,60,-2,-68)
BicubicResize(672,368,0,0.5)

Somewhere, somehow...there's a problem and I don't think is filter related.

Thank you,
Crasus

[Didée]

Oops, I had forgotten about this thread ... sorry!

...that didn't fix the problem. Applying a 2/31 quantizer restriction resulted in an undersized rip (~800kbps).
Actually, that is a clear indication that my suspicion was correct: Xvid couldn't reach target rate with Q2's. And when that's the case, Xvid does use Q1. And Q1's do hardly help, but they tend to cause oversize problems.

There's lots of possibilities to react to that problem ... use higher resolution ... use a "high bitrate" matrix ... use lower bframe settings (like multiply=1.0, offset=1) ... use less Bframes (only 1 consecutive, not 2) ... or no Bframes at all ... a bit more sharpening in the avisynth script ...

Lots of possibilities, and every single one probably is better than using Q1's to prevent and undersized rip.

(And BTW, who says that undersizing is a bad thing per se? It could very well be that the result is visually adequate, and that the source just does not necessarily need all the bitrate that you asked for...)


More agresive overflow treatment didn't help either.
Perhaps they were not aggressive enough? With "higher than 5/5/5" I did not mean "6/6/6" or "7/7/7", but more in ranges like "25/25/25" to "50/50/50".

(Not that such overflow treatment setting were recommended ... it's better to not use Q1 frames and thus not needing such extreme settings, than to allow those silly Q1 frames and then having to compensate for their unproportional oversizing.)

[BigDid]

Hi,

Just re-read this thread and would like to give my, user-only, 2cts ;)

If I have it right, the n-vop problem is solved but in the meantime new problems have arisen:
1/ oversized
2/ undersized

1/ oversized related to or after using 1/31 quant instead of 2/31..
Didée (and others) have justified to not use 1/31
So how can a 2/31 encode be oversized?
- I, now, make a simple 1 pass encode (no filtering) at 75% to get a very good approximation of the final 2 pass size so usually no oversize.
- Beside above,my only experience (with vdubmod) is mis-using the 1st pass log, or confusing with log from another encode :eek:

2/ undersized encodes with 2/31 quant are more frequent;
I agree to all suggestions given by Didée; to be more specific my testlist would be:
- Try no B-frames; 20/25% more size
- Try high bitrate matrix; 5/10% more size ; if concerned about SAP compatibility, sharktooth cqm for xvid like Eqm V2 sharp, Eqm V3 HR (modded) can be used. There is an old thread on cqm for ASP: http://forum.doom9.org/showthread.php?s=&threadid=83125
- Try no-VAQ; up to 20/25% more size; this option is proposed last, because I consider VAQ as a major improvement (even if a bit tricky to use sometimes)

Sum of the three options can raise size to 45/60% :cool:

Numbers are coming from tests (and only tests, no real and sustained encodes):
http://forum.doom9.org/showthread.php?p=1250506#post1250506

Did

[Jawor]

I guess there's no point in starting a new thread...

I uploaded new builds (1.2.2 Final & 1.3.-127 CVS-Head 5.11.2009) to http://jawormat.republika.pl/xvid.html.
Frame Drop Ratio can now be set to "-1" to completely disable "real" N-VOPs. I chose "-1" because it's the value used by Xvid API to disable N-VOPs. Behavior of "0" is not changed from vanilla builds.

Builds include MTK and DivX profiles (as usual). In the 1.2.2 build VAQ replaces the old AQ (which wasn't very good anyway). The 1.3.-127 build has a dropdown list to choose the AQ algorithm (it was put there by Xvid developers, not by me).

I created EXE installers with Inno Setup. They work under Windows 2000/XP/Vista. I didn't have access to Windows 7 to test them there.

There's no DirectShow decoder in the packages. For playback I suggest using ffdshow-tryouts (http://ffdshow-tryout.sourceforge.net/), but any MPEG-4 ASP-capable decoder will do.

It's best to uninstall any previous Xvid builds before installing one of these.