strange behaviour of cabac

[loki03]

hy, i got a problem with cabac: my encodes play fine, but when i use cabac (= not use --no-cabac) the movie plays back but always on the same positions the image is vertically "stretched", ie the lower black bar of a letterboxed movie is filled either witch a stretched version of the last few lines of the picture or it is filled with big blocks consisting of the last picture lines. when this happens the movie gets jerky and sometimes vlc crashes. playback doesnt recover. what can i do to avoid this? thanks

i use x264 408m sharktooth, vlc 084a; same behaviour with newest ffdshow, xp 2100, a7v133.


command line example:
1)error:

x264.exe --qp 18 --ref 5 --mixed-refs --no-fast-pskip --bframes 2 --b-rdo --weightb --nf --subme 7 --analyse all --8x8dct --me esa --progress --no-psnr

no error:

x264.exe --qp 18 --ref 5 --no-cabac --mixed-refs --no-fast-pskip --bframes 2 --b-rdo --weightb --nf --subme 7 --analyse all --8x8dct --me esa --progress --no-psnr


2)
error:
x264 --bitrate 3000 --nf --subme 5 --analyse p8x8,b
8x8,i4x4,p4x4 --me hex --progress --no-psnr --output


no error:
x264 --bitrate 3000 --nf --subme 5 --no-cabac --analyse p8x8,b
8x8,i4x4,p4x4 --me hex --progress --no-psnr --output

[Sirber]

what if you remove --nf?

[loki03]

no change..

cabac.jpg:
x264 --bitrate 3000 --nf --subme 5 --analyse p8x8,b8x8,i4x4,p4x4 --me hex --progress --no-psnr --output "D:\snapstream-recordings\x264\L'opération Corned Beef-xvid_fr_sub.mp4" "D:\snapstream-recordings\x264\L'opération Corned Beef-xvid_fr_sub.avs"

cabac2.jpg
without --nf

and exactly the same effect with mefui "hq-slow" profile

[Manao]

Did you try with another decoder ( vlc and ffdshow shares the same decoder ), such as nero or coreavc ?

Did you overcloked your PC, or do you have any reason to think some part of your hardware could be buggy ?

Does it always happen when cabac is on ? or only with some settings ?

[loki03]

yes, only cabac. no buggy hardware. sometimes i get the following error:
visual c++ runtime library
assertion failed!
file: libavcodec/h264.c
line:2610
expression: pic->data[0]
..
retry cancel ignore


showtime shows the same behaviour on the same frames, with crash/freeze
will try coreavc
i tried to recode it to mpeg2 and serve the reencoded version, because i thought some frames might be corruptet, but didnt change

where to download coreavc? thx

[Manao]

Ok, so obviously your version of x264 is producing an invalid file.

Right now, you're the only user reporting such an issue, so I'm inclining to think the issue lies somewhere with your system/hardware, or x264 build. Can you try another build ( especially, one of x264.nl ) ? Also, you didn't tell me if your computer was overclocked.

[loki03]

ok, i´ll do so. no overclocking. cce,xvid etc run fine, so if, it must be a strange hardware issue, but of course possible

[loki03]

x264-112-install core44

x264 --bitrate 3000 --nf --subme 5 --no-cabac --analyse p8x8,b8x8,i4x4,p4x4 --me hex --progress --no-psnr

same thing, but it starts earlier in the stream, but after that at the same positions.



x264.exe --qp 18 --ref 5 --mixed-refs --no-fast-pskip --bframes 2 --b-rdo --weightb --nf --subme 7 --analyse all --8x8dct --me esa --progress --no-psnr

zoomplayer: access violation in ffdshow, vlc: error but "blockier", showtime: little error, then recovers, movie halts in the middle

[Manao]

You're saying it happens often at the same places. Is it systematical ? If so, the possibility that your hardware is guilty is becoming more remote. Does it happens also on other sources ?

[DarkZell666]

Just asking :

Are the picture dimensions divideable by 16 ? ;)

Something like : 640*480 or 720*576 ? ^^

The bug in itself shouldn't happen anyway even with non mod-16 resolution since that issue has been worked around (I think ?), but still, just double check it :)

[bond]

x264-112-install core44112? are you using a totally outdated x264 version?
try this:
http://mirror01.x264.nl/x264/revision413/x264-413-install.exe

i guess the only way to check this is that you upload a small source file, give us your avisynth script and settings to use and we try to reproduce the problem

[loki03]

sorry, typo, second version i tested was from x264.nl core44 svn412.

picture dimensions are divisible by 16 and the error is always reproducable at one scene, at another mostly and on other ~two scenes it isnt always there (one encoding shows it, another one not, but if its there it shows at every playback)

build 413 shows the same behaviour in libav+showtime.



avs:
# Created with Gordian Knot
#
# http://gknot.doom9.org

# PLUGINS
LoadPlugin("C:\Programme\GordianKnot\DGMPGDec\DGDecode.dll")
#LoadPlugin("C:\Programme\GordianKnot\AviSynthPlugins\decomb.dll")
#LoadPlugin("C:\Programme\GordianKnot\AviSynthPlugins\KernelDeInt.dll")
LoadPlugin("C:\Programme\GordianKnot\AviSynthPlugins\UnDot.dll")
#LoadPlugin("C:\Programme\GordianKnot\AviSynthPlugins\dgbob.dll")
#LoadPlugin("C:\Programme\GordianKnot\AviSynthPlugins\Convolution3d.dll")
#LoadPlugin("C:\Programme\GordianKnot\AviSynthPlugins\FluxSmooth.dll")
#LoadPlugin("C:\Programme\GordianKnot\AviSynthPlugins\TomsMoComp.dll")
#LoadPlugin("C:\Programme\GordianKnot\AviSynthPlugins\VSFilter.dll")
#LoadPlugin("C:\Programme\GordianKnot\AviSynthPlugins\SimpleResize.dll")
mpeg2source("D:\snapstream-recordings\x264\L'opération Corned Beef-xvid_fr_sub.d2v")
#ConvertToYV12()
# TRIM
trim(1500,2000)

# IVTC
#Telecide(order=1,guide=1).Decimate()
# or use
#IVTC(44,11,95)
#GreedyHMA(1,0,4,0,0,0,0,0)

# DEINTERLACING (1)
#FieldDeinterlace()
#FieldDeinterlace(blend=false)
#TomsMoComp(1,5,1)

# DEINTERLACING (2)
#KernelDeInt(order=1,sharp=true)
# or maybe
#DGBob(order=1,mode=0)

# DEINTERLACING (3) - special requests
#GreedyHMA(1,0,0,0,0,0,0,0)
#Telecide()
#SeparateFields()

# CROPPING
crop(12,126,702,412)

# SUBTITLES
#VobSub("FileName")

# RESIZING
LanczosResize(704,384)

# DENOISING: choose one combination (or none)
Undot()

# 1) little noise
#Temporalsoften(2,3,3,mode=2,scenechange=6)
#mergechroma(blur(1.3))
#FluxSmoothST(5,7)

# 2) medium noise
#Temporalsoften(3,5,5,mode=2,scenechange=10)
#Convolution3d("moviehq")
#FluxSmoothST(7,7)

# 3) heavy noise
#Temporalsoften(4,8,8,mode=2,scenechange=10)
#Convolution3d("movielq")
#FluxSmoothST(10,15)

# BORDERS
#AddBorders(left,top,right,bottom)

# COMPRESSIBILITY CHECK
# !!!!Snip Size now has to be 20 for use in GKnot!
#SelectRangeEvery(400,20)

# FOOL CCEnc
#empty = BlankClip()
#AudioDub(last,empty)
#converttoyuy2()

[loki03]

the mp4 file, avs and the source file are available at
http://www.offenelistebludenz.at/files/

[ChronoCross]

I can't reprodude the error. I'm using revision 410.

x264.exe --pass 2 --bitrate 2930 --stats ".stats" --ref 16 --mixed-refs --bframes 3 --b-pyramid --b-rdo --bime --weightb --filter -2,-1 --subme 7 --trellis 2 --analyse all --8x8dct --qpmin 1 --me umh --merange 32 --progress --no-psnr --output

AVS
mpeg2source("mpeg.d2v")
crop(10,126,700,322)
LanczosResize(640,256)

[loki03]

i encoded on a laptop and it plays fine on this machine, i encoded the same on this machine and it had the c++ error....so i guess its a hardware issue! thank you for your help!