Automatic video glitch tester

[new_age]

Hello!

Lately I'm encoding a lot with x264 and sometimes I have glitches in the video /small blocks in a few frames/. (I'm trying to figure out the cause right now with memtest etc)
I've viewed the video with media players on pc and popcorn and with a samsung B650 tv. The h264 file is muxed into MKV container.

Until I figure out the reason is there any way to automatically check the encoded h264 file whether it has video glitches/problems/bad frames/etc or not?

thanks in advance

NA

[sneaker_ger]

Can't really answer your question but you might want to take a look at this thread. (http://forum.doom9.org/showthread.php?t=148956)

[LoRd_MuldeR]

First of all you should update to the latest revision to x264, because a bug was fixed recently. A bug that only showed up with QPRD (SubME 10). Then you should check your source AVS script carefully in a tool like VirtualDub and see if there are any errors in the input video, because this may be a decoding problem in fact! If the problem is definitely not in the input data, but shows up in the encoded video, then please post a sample clip (unprocessed source!) that can be used to reproduce the problem. And also post your exact x264 commandline.

BTW: I think there is no tool that automatically checks for glitches, because the encoded video is expected to be different from the source, unless you are using x264 in lossless mode (QP=0). How should the tool distinguish between "regular" compression artifacts and glitches that shouldn't appear?

[new_age]

I'm using megui with latest updates it uses. (x264: 1183 Jeeb's patched build)

AVS scripts are always simple: avcsource/mpegsource and crop and/or trim

About automatic checking: the theory is simple.
The glitches I'm hunting have always the same properties:
- appear suddenly
- shape is rectangle or multiple rectangles
- position of the rectangle is multiple of N (8, 16, 32)

Good examples are in the thread sneaker_ger mentioned.

From these properties a half automatic half manual checking can be made I believe. A prg with some tolerance settings checks _only_ the output video and after the suspicious frames were found displays and you can check it: is it ok or not.

The other idea is much more easier to implement if you have the input video. Comparing the input and output makes it more easier.

Perhaps there are lot of problems implementing these simple ideas. Eg. these rectangles have not solid colors inside; they are visible to the eye becase they have different color compared to the neighbour area.

I'm talking about HQ compressions with HQ sources. LQ source and compression makes it impossible I think just as LoRd_MuldeR mentioned.

So I thought someone had the same idea and had some programming skills to already make such a tool.

[laserfan]

Until I figure out the reason is there any way to automatically check the encoded h264 file whether it has video glitches...I had similar issues and like LM believe there is no tool to find same. Sometimes there would be only one glitch in a 2 hour movie.

All my problems completely disappeared when I built a new computer with an Nvidia video card, and started using neuron2's DGIndexNV toolset. This tells me that my original pixellation problems were decoding problems, completely fixed by using nvidia GPU. FWIW.

[LoRd_MuldeR]

I'm using megui with latest updates it uses. (x264: 1183 Jeeb's patched build)

Please update to x264 r1222, as suggested before! Preferably an unpatched build.

Good examples are in the thread sneaker_ger mentioned.

The problem mentioned in that thread was already fixed.

I encoded the "Authors.avs" example with latest x264 and the settings in question. There were no glitches anymore.

So I thought someone had the same idea and had some programming skills to already make such a tool.

There probably is no way to reliably distinguish such glitches from "regular" compression artifacts.

Maybe some heuristic could be implemented to detect a specific kind of glitches, but you better use your time to track down the cause of the problem...

[new_age]

memtest 24h without problems
now pirme95 is running
trying to track down the problems
but I still beleive that such a tool can be made

[LoRd_MuldeR]

Believe it or not, why don't you simply provide a sample clip (unprocessed source!) and your complete x264 commmand-line, so we can try to reproduce the problem?

[new_age]

Believe it or not, why don't you simply provide a sample clip (unprocessed source!) and your complete x264 commmand-line, so we can try to reproduce the problem?

I'm working on that. Need lot of time until I can get the original source again.

[LoRd_MuldeR]

I'm working on that. Need lot of time until I can get the original source again.

Please explain what your original source is and where you got it...

[new_age]

Probably that would be against forum rules so we need to wait until I found another glitch in another movie or I get the source...
But anyway thanks for your efforts.

[new_age]

FYI: I've made my simple tool and works just fine. The idea & algorithm is very simple.

I found out that all my latest encodes (seems like only 1080p but not the DVD 720x576) have video glitches.

I've downloaded r1222 and I'll reencode these movies and run the comparer again. If I'll have glitches again I'll post my x264 command line (basically it is what megui creates with DXVA-HD-HQ mode) and some samples.
I'll also use CoreAVC (directshowsource) as frameserver instead of libavccodec (DGAVCDec).

I'm considering an Corei7 + some NV card. :mad:

[Chevy787]

funky, I'm getting the same problems as you are (I have the newest x264 build)...doing some hardware tests myself at the moment.

[LoRd_MuldeR]

funky, I'm getting the same problems as you are (I have the newest x264 build)...doing some hardware tests myself at the moment.

Are you 100% sure that those "glitches" are NOT in your source? And I don't mean the source file! I mean the output of the source Avisynth script ;)

You should load up the very same AVS file that you were feeding into x264 in VirtualDub and check it thoroughly for glitches.

If any glitches are visible in your source AVS, then this is not x264 related! Only if you are 100% that there are no glitches in the input AVS, then x264 is to blame.

In that case please post a short sample to re-produce the problem plus the exact commandline you were using...

[Chevy787]

Yes, I'm sure since I've ran it multiple times and encountered the same glitches in different sections of the video. As well, I checked it it with virtual dub as advised.
I'll upload a lossless clip tonight of the first minute and half or so of the video which errors have always occurred in.

[LoRd_MuldeR]

Yes, I'm sure since I've ran it multiple times and encountered the same glitches in different sections of the video.

Which doesn't mean that a decoder problem can be excluded...

As well, I checked it it with virtual dub as advised.

Did you check the x264-encoded video -or- the source AVS file you are encoding from?

[Chevy787]

The source AVS

[Dark Shikari]

Yes, I'm sure since I've ran it multiple times and encountered the same glitches in different sections of the video. As well, I checked it it with virtual dub as advised.
I'll upload a lossless clip tonight of the first minute and half or so of the video which errors have always occurred in.How about a clip containing the actual problems...?

[Chevy787]

How about a clip containing the actual problems...?

http://www.megaupload.com/?d=QX1551TP
about a 14 second lossless video containing 3 different clips from the borked x264 encode.

Ah, I should include this....

Pass 1----------------
32x264.exe --level 5.1 --pass 1 --stats 720-D-TRIMZ.avs.264stat --bitrate 1850
--profile high --bframes 16 --deblock 0:0 --subme 7 --merange 8 --keyint 300
--b-adapt 2 --b-pyramid --psy-rd 0.4:0 --aq-strength 0.6 --chroma-qp-offset 2
--no-mbtree --min-keyint 10 --weightb --partitions none --me umh --direct auto
--threads auto --thread-input --fps 23.976 --output NUL 720.avs
Pass 2----------------
32x264.exe --level 5.1 --pass2 --stats 720-D-TRIMZ.avs.264stat --bitrate 1850
--profile high --bframes 16 --deblock 0:0 --subme 10 --merange 24 --keyint 300
--ref 16 --trellis 2 --b-pyramid --psy-rd 0.4:0 --b-adapt 2 --aq-strength 0.6
--no-dct-decimate --chroma-qp-offset 2 --no-mbtree --min-keyint 10 --me tesa
--partitions p8x8,b8x8,i4x4,i8x8 --8x8dct --weightb --mixed-refs --no-fast-pskip
--direct auto --threads auto --thread-input --fps 23.976 --output output.mkv 720.avs


Another thing to note, from what I've watched of my newest encode so far, no errors have occurred...
Changed in that encode...1. Lowered CPU voltages (just incase the problem was hardware related) 2.Included --no-fast-pskip in the first pass 3. Used subme 9
I haven't gone through too much of it yet, so it may just be a false alarm....

[Dark Shikari]

http://www.megaupload.com/?d=QX1551TP
about a 14 second lossless video containing 3 different clips from the borked x264 encode.... I need the actual video stream, not a lossless re-encode of it.

[Chevy787]

Here is the entire video.
http://www.megaupload.com/?d=B5UT969Q

[Dark Shikari]

Found one on frame 480. Can I have:

1) The shortest possible source sample which produces the problem consistently when encoded.
2) The encoding settings used for 1).

?

[Chevy787]

Weird, I am now unable to reproduce the problem. I even tried re-encoding the entire video with what I'm almost certain are the same settings and the same .avs script.
I can only think that is was my cpu acting up, and that by lowering the voltages I was able to fix it (since the error occurred on 3 encodes of the same source prior to this).

[Dark Shikari]

That would make sense. Here is what makes the problem so bizarre:

1. It exactly matches an 8x16 region of the frame (two chroma blocks).

2. x264 is aware of the error (it doesn't propagate), so the error could not occur in anything done by the decoder (MC, dequant, idct).

3. The error involves writing extra data to the bitstream, so it cannot be caused by omitting data. DCT decimation is off, so the early termination hack in chroma is disabled as well, so that can't be it either.

As a result, the problem can almost surely only be in the forward transform or quantization, both of which are absurdly simple functions that have been tested to the ends of the earth.

[skynetman]

Weird, I am now unable to reproduce the problem. I even tried re-encoding the entire video with what I'm almost certain are the same settings and the same .avs script.
I can only think that is was my cpu acting up, and that by lowering the voltages I was able to fix it (since the error occurred on 3 encodes of the same source prior to this).

Chevy, i had similar problems in the past. unrar two files at two different moments and sometimes they gave corruption sometimes not for example...
Computer are an exact science, it's impossible doing the same exact thing two times in a raw with different results: it's a HARDWARE PROBLEM.

So before hitting your head against the wall try ANOTHER COMPUTER with same exact settings!

In my case, my 1997 computer had a faulty Abit MB that caused random errors with 2 ram modules installed, my MSI motherboard two years ago had problems with on board RAID with random errors in read/write....
Prime95 is not hard enough on the cpu to test it fully.
If you have an intel processor please try this IntelBurnTest 2.3 test at least at HIGH settings.
http://www.ultimate-filez.com/files/IntelBurnTest.zip

This way you will be sure at least that cpu/ram combo is ok (hard disk/controller problems in read write still possible though).
After checking with this test i had to slow down my overclock cause my system bsodded every time and this explained why some apps were giving errors or why after a week always on i had to reboot vista64....
Bye ;)

[laserfan]

Thank you skynetman (I think!) for pointing-out this tool which I've not seen before. On the PC I am using now, a general purpose one, I tried it and while it got a perfect score I heard some noises from it I've never heard before! The fan(s) revving like race cars! Can't wait to try it now on my two AV computers... :)

[skynetman]

Thx laserfan, it's quite useful indeed. See this description form the author:

"Tool that helps Intel(R) Core 2(tm)-based (or others) test for system stability in a relatively short time (greatly shorter than Prime95).
The program utilizes Intel(R)'s very own CPU stress testing binaries for the most stressful, and speedy testing.
This software will put even the highest end Intel(R) Core 2(tm) Extreme rigs to their knees, and shows how REALLY stable the system is.
It is highly recommended to use a 64-bit Windows operating system (XP x64 preferred) for the most accurate testing and using maximum amount of free system memory. It will work on even Intel(R) Pentium III processors, but unfortunately, Intel(R) designed the binaries in a way that makes it not run at all on any AMD CPUs. Donations are appreciated!
Caution: Not for beginning overclockers. This program WILL stress the CPU, Memory, North Bridge, and other system components to their highest extent! Make sure your PC is properly cooled and ventilated!"

[skynetman]

Sorry for these OT but i spent 10 hours trying to find the problem and i want to avoid anyone of you the same loss.
There is a bug in latest intelburntest that on vista 64bit gives ALWAYS A FAILURE.
So don't crash your pc, it's a false alarm.

Who wants to fully test his system can use a similar program called Linx 0.6.2 that works and uses updated Intel Libs.
Download info here:
http://www.xtremesystems.org/forums/showthread.php?t=201670

Bye