Automated pixel-by-pixel 2 clip comparison?

[shae]

Is there a script to automatically compare two clips and output a list of frames that are different?

[poisondeathray]

Is there a script to automatically compare two clips and output a list of frames that are different?

Did you want identical vs. not, or some fudge factor for noise /compression differences?

WriteFileIf()

Something like this


a=whateversource("a.ext")
b=whateversource("b.ext")

WriteFileIF(last, "text.log", "LumaDifference(a,b)>0.1", "current_frame")


Completely identical would show 0.0 (ie no difference). So that 0.1 has some "leeway" , adjust as you see fit maybe higher or lower in your scenario

[shae]

Thanks.

Looking to do an exact comparison, to see that my intermediate format is the same.

[StainlessS]

What is your intermediate format ?

UT_Video and MagicYUV should be totally lossless [EDIT: and HuffYUV], Lagarith (so I learnt not so very long ago) is not always completely lossless.

EDIT: 'Totally lossless' assuming no colorspace conversions occur.

[poisondeathray]

Thanks.

Looking to do an exact comparison, to see that my intermediate format is the same.


You're probably ok using it like that with >zero, but LumaDifference() only calculates the Y plane. So technically there could be U,V plane differences (I'm assuming YUV source), and you could add ChromaUDifference(), ChromaVDifference() . You add "AND" operator with "&&"






UT_Video and MagicYUV should be totally lossless [EDIT: and HuffYUV], Lagarith (so I learnt not so very long ago) is not always completely lossless.

What was the issue with lagarith?

Lagarith should be lossless too in the same colorspace if the recieving application handles it correctly - I mean in the absence of something rare and funky like encoding errors (which can affect other "lossless" codecs too)

[StainlessS]

What was the issue with lagarith?

Arithmetic encoders are not always reliable (apparently), I think it came up about 2 years ago, think RaffRiff was involved, not sure.

EDIT:

Originally Posted by Dark Shikari View Post
Lagarith relies on floating point math in the arithmetic coder, which nearly guarantees that errors like this will occur eventually, depending on the phase of the moon and so forth. The format should generally be avoided if possible.


*This* is my issue with Lagarith. I get too many glitches.

He's referring to the "random" glitches (artifacts) that some people seem to be getting with Lagarith.

AFAIK, Lagarith uses arithmetic coding, which - in its most simple form - uses infinite-precision real numbers. Computers don't have that. So, in practice, we usually use a "range coder" instead of a plain arithmetic coder. It is essentially one and the same thing, yes. But the "range coder" works entirely on integer math (rather than using real numbers). This makes it more suitable for implementation on a real-world computer. Also ensure that the results are always deterministic.

Lagarith, on the other hand, seems to implement arithmetic coding based on limited-precision floating point math. This is kind of "delicate" with respect to rounding errors and stuff. And this is what may cause nondeterministic behavior :eek:

Here tis:- http://forum.doom9.org/showthread.php?p=1656154#post1656154

[StainlessS]

Shae,

Could substitute RT_FrameDifference instead of LumaDifference in PDR script,
[EDIT: for additional chroma, or RGB, Standard 2.6 AVS colorspaces only]

Below from RT_Stats v2Beta (not exactly same as v1.43)

RT_FrameDifference(clip c,clip c2,int "n"=current_frame,int "n2"=current_frame,Float "ChromaWeight"=1.0/3.0,
int "x"=0,int "y"=0,int "w"=0,int "h"=0,int "x2"=x,int "y2"=y,bool "Altscan"=false,bool "ChromaI"=false,
Int "Matrix"=(Width>1100||Height>600)?3:2)
Returns difference between clip c frame n area x,y,w,h and clip c2 frame n2 area x2,y2,w,h. (range 0.0 -> 255.0).

Args:-
c, c2, clips to difference.
n, default = current_Frame. Frame from clip c.
n2, default = current_Frame. Frame from clip c2.
ChromaWeight, default 1.0/3.0. Range 0.0 -> 1.0.
Y8, returns same as LumaDifference. ChromaWeight ignored.
YUV,
Weighting applied YUV chroma:- (1.0 - ChromaWeight) * Lumadif + ChromaWeight * ((Udif + Vdif)/2.0).
RGB,
If ChromaWeight > 0.0, then returns same as RT_RGBDifference() [ie average RGB pixel channel difference].
If ChromaWeight == 0.0, then returns same as RT_LumaDifference() using Matrix arg to convert RGB to YUV-Y Luma.
x,y,w,h. All Default 0 (full frame), for clip c.
x2,y2. x2 defaults x, y2 defaults y, for clip c2.
Altscan, default false. If true then scan only every other scanline starting at clip c y coord, and clip c2 y2 coord.
ChromaI, default false. If YV12 and ChromaI, then do YV12 interlaced chroma scanning.
Ignored if not YV12. Both YV12 clips must be same, cannot difference one that is progressive and one that has interlaced chroma.
Matrix, default (c.Width>1100||c.Height>600||c2.Width>1100||c2.Height>600)?3:2
Conversion matrix for conversion of RGB to YUV-Y Luma. 0=REC601 : 1=REC709 : 2 = PC601 : 3 = PC709.
Default = (c.Width > 1100 OR c.Height>600 OR c2.Width > 1100 OR c2.Height>600) then 3(PC709) else 2(PC601). YUV not used
Matrix only Used if RGB and ChromaWeight == 0.0, where returns same as RT_LumaDifference().
Note, 'x2' and 'y2' default to 'x' and 'y' respectively.
v2.0, c and c2 need not be same dimensions but BEWARE, x2 and y2 default to x and y also w and h default to 0
which is converted to c.width-x and c.height-y, may be best to provide x2, y2, w and h where c2 not same dimensions as c.

[poisondeathray]

Arithmetic encoders are not always reliable (apparently), I think it came up about 2 years ago, think RaffRiff was involved, not sure.

Here tis:- http://forum.doom9.org/showthread.php?p=1656154#post1656154

Yes I remember that -

Did you read the author's replies?

Maybe RR should buy a lotto ticket :D

[StainlessS]

Did you read the author's replies?

Nope, had not. I was quite disappointed when I first read that, Lags can produce miniscule files in some cases, guess I may un-delete Lags again.

"Maybe RR should buy a lotto ticket.",
Nah, he looks smug enough already.

[raffriff42]

Which author's reply are you guys talking about?

[poisondeathray]

Which author's reply are you guys talking about?

lagarith's author

[raffriff42]

linky plz? I searched and could not find Lagarith's author's post (he only speaks Japanese AFAIK)

[poisondeathray]

Tin man posted the link above in post#6. Just scroll down to read the replies. Lagarith's author is "SirLagsalot"



What is "too many glitches" ?

Rare random glitches do occur for some people - but they do with every codec, even UT, huffyuv. Go back in history and there are actually more random glitches reported for huffyuv than lagarith. (Maybe there is a skewed statistical explanation; ie. maybe there were more huffyuv users, I would imagine so because it was less resource intensive back in the single core days.) So glitches happens - for whatever reason. But to some people only. It's easy to jump to conclusions and assume AC is what is causing it for lagarith . But "causation" requires a stronger burden of proof.

UT has become more popular, heavy usage on Adobe forums, for example. But there have been reports of glitches too (some of them could be explained from other causes), but others remain unexplained. But I think we are seeing at least partially some statistical effect here too - more users, more reported cases of glitches. Go back, way back to early days, with very few users, there were no reports of anything like that.

But I can emphasize with the "once bitten..." mentality



lagarith can still be useful in some workflows - it has higher compatibility in some programs, and yes, lags can produce ultra miniscule files in certain scenarios with the null frames option, but FFV1 or x264 lossless in log gop modes usually have better compression ratio in most general cases, at the expense of very sluggish seeks .

[raffriff42]

>But I can emphasize with the "once bitten..." mentality
Thanks, seriously. That's all I ask. Opinions are funny things.

[feisty2]

why not just go with the uncompressed binary if it's just the intermediate result anyways, zero possibility of getting into some kind of random occasional failure

[poisondeathray]

why not just go with the uncompressed binary if it's just the intermediate result anyways, zero possibility of getting into some kind of random occasional failure

Not zero - write failures, memory failures etc...

Some of those reported glitches from lossless codecs eventually turned out to some of those - like a bad memory stick etc...

Also other obvious "cons" like huge storage requirements



(It's "empathize" :D , not "emphasize")

RR - If you're seeing "too many" glitches (implying more than one) when using lagarith in the past, I would probably consider exploring other explanations if you haven't already

I would say I'm a fairly heavy user of lossless codecs. Huffyuv in the past, lagarith, but now more commonly UT, ffv1 and even x264. I've never been bitten (I just jinxed myself....:D ....but I've never won the lotto either, or at least not the big one)

[feisty2]

Also other obvious "cons" like huge storage requirements



also other obvious "pros" like faster processing speed (no need for decompressing)

[poisondeathray]

also other obvious "pros" like faster processing speed (no need for decompressing)

Or slower processing speed when you hit the storage I/O bottleneck because your files are too huge :D


Rare is the situation when something like UT video becomes a decoding bottleneck

[feisty2]

Or slower processing speed when you hit the storage I/O bottleneck because your files are too huge :D


Rare is the situation when something like UT video becomes a decoding bottleneck

not if you got raid0


(It's "empathize" , not "emphasize")

sorry? u Canadians don't do "empathise" and "emphasise", eh?

[poisondeathray]

not if you got raid0



I guess it depends on what type of processing. Knowing you, it's going to be some slow a$$ filter chain anyways so it doesn't matter anyways :D

But how many drives ? For pure speed UT will usually still be faster , unless you have a monster array .

You can barely ensure 1x speed with a 2 drive array for 1920x1080p30 (you need >100MB/s sustained). UT decodes at 400-600fps on a regular quadcore desktop with a single drive (thats 10-20x speed)

[jmac698]

Don't people have SSD's now? Anyhow, when I used harddrives, I always processed from one drive to another, that always worked out. Sometimes with median, I'd have the copies on different drives, it was really fast then.
The most I've ever needed for a project was 350GB, but you should easily be able to get away with a small (in 2017) work SSD of 256GB, which is <$200 USD.

[StainlessS]

I remember something from some years back, M$ said of their drive compression software that if you had a 130MHz Pentium (think that figure correct),
then it was faster to use compression than to not use it (general files, circa 1997).

EDIT: It may have been a bit less than 130MHz (~110->120MHz), I was on 130MHz at the time and was in the 'compress is better' category.
It was a little less efficient on compression but more so on decompress [likely used more often than compress stage].

EDIT: Initial compression was done quickly with only light compression, you would then later do a batch re-compress
at a higher compression level, much like people do file defragmentation when free time allows it.
https://en.wikipedia.org/wiki/DriveSpace

[shae]

What is your intermediate format ?
I was toying with UTVideo, but it's occurred to me that Gradient prediction is a new feature, so bugs are possible.

I'll check if RT_FrameDifference() is better than Luma/U/YDifference() combined.

why not just go with the uncompressed binary if it's just the intermediate result anyways
Maybe that's the way to go. For some reason I'm always uncomfortable with large unverified writes, but with uncompressed, random damage will actually be very difficult to tell. In lossless codecs, I guess at the worst case a flipped bit could ruin a whole frame, or more.

I do wonder why HDD makers never improve their error rate beyond the perennial 10^-14, or maybe 10^-15 in select models.

Or a better option: have encode break/resume implemented in x264. :)

[poisondeathray]

Don't people have SSD's now? Anyhow, when I used harddrives, I always processed from one drive to another, that always worked out. Sometimes with median, I'd have the copies on different drives, it was really fast then.
The most I've ever needed for a project was 350GB, but you should easily be able to get away with a small (in 2017) work SSD of 256GB, which is <$200 USD.


In 2017, SSD's maybe ok for uncompressed small SD projects. SSD's are still cost prohibitive for most people. 1hr of 1920x1080 woud be about 400-450GB (depends on pixel type, bit depth). Forget about UHD.



I remember something from some years back, M$ said of their drive compression software that if you had a 130MHz Pentium (think that figure correct),
then it was faster to use compression than to not use it (general files, circa 1997).

EDIT: It may have been a bit less than 130MHz (~110->120MHz), I was on 130MHz at the time and was in the 'compress is better' category.
It was a little less efficient on compression but more so on decompress [likely used more often than compress stage].


Yes there is overhead when decoding compressed formats, but when using uncompressed, there is also CPU overhead on large storage transfers (other RAID modes are worse, the more redundancy, the more the overhead, but still significant with RAID-0) . Onboard "cheap" RAID controllers are worse then external PCI-E controllers (where it's partially offloaded), but CPU usage increases proportionally - the faster the transfer , the higher the usage. Writes are usually worse than reads.




I was toying with UTVideo, but it's occurred to me that Gradient prediction is a new feature, so bugs are possible.

I'll check if RT_FrameDifference() is better than Luma/U/YDifference() combined.


Maybe that's the way to go. For some reason I'm always uncomfortable with large unverified writes, but with uncompressed, random damage will actually be very difficult to tell. In lossless codecs, I guess at the worst case a flipped bit could ruin a whole frame, or more.

I do wonder why HDD makers never improve their error rate beyond the perennial 10^-14, or maybe 10^-15 in select models.

Or a better option: have encode break/resume implemented in x264. :)


Depends on the "type" of error. If you have a "brown out" or some electrical issue, the whole file could be corrupt when doing it mid transfer (the entire AVI or MOV corrupted). Some people use image sequences for that reason, especially on very slow CG renders and projects (I'm talking many times slower than QGTMC) - so hours/days/weeks of work isn't lost. Image sequences enable a "resume" of sorts.

Other methods people use to check file integrity/validity are per frame PSNR/SSIM testing. You can batch process with ffmpeg for example (It prints metrics every frame, not sure how to print only if something isn't lossless). FFV1 was supposed to have some sort of secondary CRC check per frame in the development timeline, not sure what the status on that is.

[shekh]

Maybe this can help:
ffmpeg.exe -i test.avi -f framemd5 test.txt
You can generate frame checksums for 2 versions of video and then do text compare to find errors.

FFV1 can have ebmedded checksums, it can help to detect if a file gets corrupt. Not sure if there is any other use for that feature.

[shae]

Interesting. There are also other hash types (https://ffmpeg.org/ffmpeg-formats.html#framehash-1).

https://trac.ffmpeg.org/wiki/framemd5%20Intro%20and%20HowTo

[StainlessS]

Who's life is it that depends upon this ?, or maybe you can't say.

[shae]

If you mean why bother verifying it, don't you verify your pipeline before doing a lot of encoding?

[StainlessS]

Strikes me like the university lecturer (MIT or maybe Berkley) that used to check his spreadsheets by hand during his dinner break
(eating his sandwiches on a park bench on campus). One day he discovered an error, and the FDIV bug in first gen Pentium.
I'm just wondering how many calculations he's checked in the 20+ years since then, musta been very disappointing.

I'm quite happy for others to do the fault finding in other peoples code, there is always someone like you that has
plenty of free time :)

[shae]

I'd rather do some extra checks initially than find out in the end I have to redo 200-250 hours of encoding.
And that's on my daily use computer, which isn't dedicated to encoding.

There are other reasons as well I want to make sure it's fine.

[johnmeyer]

Strikes me like the university lecturer (MIT or maybe Berkley) that used to check his spreadsheets by hand during his dinner break
(eating his sandwiches on a park bench on campus). One day he discovered an error, and the FDIV bug in first gen Pentium.
I'm just wondering how many calculations he's checked in the 20+ years since then, musta been very disappointing.

I'm quite happy for others to do the fault finding in other peoples code, there is always someone like you that has
plenty of free time :)I remember that Excel/Pentium bug. It got a lot of press at the time. I did not remember that there was a professor involved, but you are absolutely correct:

Pentium FDIV bug (https://en.wikipedia.org/wiki/Pentium_FDIV_bug)

[StainlessS]

I reported an error in [EDIT: 32 bit] integer long division in Lattice compiler libraries on Commodore Amiga, many moons ago (Motorola MC68000).
I was doing some big number arithmetic routines at the time (variable number of bits, eg 4096 bits, or more).
[EDIT: Whilst debugging my code,] I accidentally stumbled into the compiler library code, I could have exited quickly but decided to step through and see how it compared to the routines I was writing (in C). At one point I found that the numbers were not being properly normalized, and at first assumed that it was some kind of optimization, but decided to try and contrive a combo of numbers that would reveal a bug. The combo did indeed produce an error and I reported it to Lattice, dont think that they even bothered to reply (probably thought that I was a nuisance, as I reported in the region of a dozen and a half bugs (at least about 6 in the linux style date/time routines).

Here, the fixed MC68000 assembler source (disassembled).


; --------------------------------------------------------------
; Lattice v5.61 had error in normalise and produced wrong result for
; eg $FFFE000 / $20007
; Have fixed it in lcr.lib.
; --------------------------------------------------------------
SECTION TEXT
XDEF __CXD33
XDEF __CXD22
; --------------------------------------------------------------
NSHIFT_1 EQU $0100 ; ERROR WAS $0080
NSHIFT_2 EQU $1000 ; ERROR WAS $0800
NSHIFT_3 EQU $4000 ; ERROR WAS $2000
; --------------------------------------------------------------
__CXD33: tst.l d0
bpl.b L1a
neg.l d0
tst.l d1
bpl.b L12
neg.l d1
bsr.b __CXD22
neg.l d1
rts
L12: bsr.b __CXD22
neg.l d0
neg.l d1
rts
L1a: tst.l d1
bpl.b __CXD22
neg.l d1
bsr.b __CXD22
neg.l d0
rts
; --------------------------------------------------------------
__CXD22: move.l d2,-(a7)
swap d1
move.w d1,d2
bne.b L4c
swap d0
swap d1
swap d2
move.w d0,d2
beq.b L3c
divu.w d1,d2
move.w d2,d0
L3c: swap d0
move.w d0,d2
divu.w d1,d2
move.w d2,d0
swap d2
move.w d2,d1
move.l (a7)+,d2
rts
L4c: move.l d3,-(a7)
moveq #$10,d3
cmpi.w #NSHIFT_1,d1
bcc.b L5a
rol.l #8,d1
subq.w #8,d3
L5a: cmpi.w #NSHIFT_2,d1
bcc.b L64
rol.l #4,d1
subq.w #4,d3
L64: cmpi.w #NSHIFT_3,d1
bcc.b L6e
rol.l #2,d1
subq.w #2,d3
L6e: tst.w d1
bmi.b L76
rol.l #1,d1
subq.w #1,d3
L76: move.w d0,d2
lsr.l d3,d0
swap d2
clr.w d2
lsr.l d3,d2
swap d3
divu.w d1,d0
move.w d0,d3
move.w d2,d0
move.w d3,d2
swap d1
mulu.w d1,d2
sub.l d2,d0
bcc.b L96
subq.w #1,d3
add.l d1,d0
L96: moveq #$00,d1
move.w d3,d1
swap d3
rol.l d3,d0
swap d0
exg d0,d1
move.l (a7)+,d3
move.l (a7)+,d2
rts
; --------------------------------------------------------------
END


EDIT: __CXD33 is signed 32 bit division, __CXD22 is unsigned. Divide register d0.L by d1.L

EDIT: I think for above unsigned error combo $FFFE000 / $20007,
produced something like: Result=True_Result-1 : Remainder=True_Remainder+$20007,
ie the result was off by 1, and the remainder was greater than the divisor.

EDIT: Normalize[assuming that divisor is non zero]:- So long as most significant bit of Divisor is zero, shift Dividend and Divisor left one bit .
Something like above, not done exactly like that in assembler.

EDIT: I think I found above bug within about 12 months after FDIV bug, so about 1995 I think.

[shae]

You should've sent a fix to Knuth. At least you'd get something in return. :)

[StainlessS]

No need, I was following Knuth at that time for the rapid divide algo, so long as numbers properly normalized, then first estimated divide result
could be no more than 1 in error, so if proper normalized and error, then add 1 to result and reduce remainder by divisor to give correct answer.
If not normalized proper, the single correction may fail as it may need additional corrections.
Leastwise, think it works something like that, twas a long time ago.
(Either Semi-numerical Algorithms [vol 2] or Fundamental Algorithms [vol 1], TAOCP [The Art Of Computer Programming, Donald Erwin[I think] Knuth]).