GMSD and SSIM Quality Metrics [Archive] - Page 5

Iron_Mike

20th March 2019, 09:25

another thing I just saw via HEVC browser...

re (1): the encode places all 5x IDR keyframes @ 100 frames interval (as specified)...

re (2) the encode places the 4x simple I frames (following the only real IDR keyframe) in incorrect position, they are a few frames off

re (3) the encode places the 1x simple I frame (following the only real IDR keyframe) in correct position

HEVC browser stats of the 3 files I uploaded - note: frame count starts at 1, not at 0 in the below stats

re (1) - x265 w/ closed-gop and custom keyframes intval 100 - IDR keyframes at positions: 1, 101, 201, 301, 401 | simple (non-IDR) I frames: none

re (2) - x265 w/ open-gop and custom keyframes intval 100 - IDR keyframes at positions: 1 | simple (non-IDR) I frames at positions: 97, 197, 297, 397

re (3) - x265 w/ open-gop and no custom keyframes intval - IDR keyframes at positions: 1 | simple (non-IDR) I frames at positions: 247

not sure how ffms2 works, but if it reads file (2), then sees the specified GOP size with keyintval of 100 (in the encoding settings), and then does not find the IDR frames in the correct positions, or matter of fact does not find any IDR frames at all (b/c ffmpeg w/ open-gop only places simple I frames), then that may be the reason why it acts up...

ChaosKing

20th March 2019, 09:50

There are new ffms2 / lsmash builds up. But seems to behave the same.
And I tested the mp4 with the old ffms2 and it has no seeking problems. https://github.com/FFMS/ffms2/releases/download/2.23/ffms2-2.23.1-msvc.7z
(This is the second file what has no problems with the old version)

I tested also a build by HolyWu (ffmpeg 4.1.1) https://forum.doom9.org/showthread.php?p=1866411#post1866411
-> Seeking issues

sneaker_ger

20th March 2019, 10:29

not sure how ffms2 works, but if it reads file (2), then sees the specified GOP size with keyintval of 100 (in the encoding settings), and then does not find the IDR frames in the correct positions
It doesn't work like that. No decoder cares about keyint in custom/optional "encoding settings" SEI that x264/x265 write.

poisondeathray

20th March 2019, 15:27

It's ok display order wise.

The offset position and encoding order does not necessarily reflect the display order of the frames. That utility does not display the GOP number , display frame number, encoded frame number , but hey it's free

There is another analyzer, GitlHEVCAnalyzer on sourceforge but it only supports 8bit HEVC . But you guys are using 8bit here, but it will display a graph like elecard with "I" slices in red. But you can see they are placed correctly. Another method, with avisynth ffms2 is to use FFInfo() and it will display the pic type. But again, it does not distinguish between IDR types , or non IDR "i". Just simple I,P,B slices . Or commercial analyzers ($) .

ChaosKing

20th March 2019, 16:11

It's ok display order wise.

with avisynth ffms2 is to use FFInfo() and it will display the pic type. But again, it does not distinguish between IDR types , or non IDR "i". Just simple I,P,B slices . Or commercial analyzers ($) .

VS version: core.ffms2.Source(source=r"D:\Download\crowd_run_1080p50.y4m").text.FrameProps(props=['_PictType'])

or just text.FrameProps() to show all frame props

WorBry

20th March 2019, 16:14

@WorBry - for x265 it's --frame-threads . I guess ffmpeg -threads get passed as libx265 frame threads. A value of 6 would correspond to a 32 core machine . You can see on the chart
https://x265.readthedocs.io/en/latest/threading.html

The frame thread effect is smaller than with x264, but it's the reason why your values were slightly higher, because it should have been 2 with a 6C/6T machine. But if you specified -threads 6 , and that got passed to libx265 as frame-threads, then I would have expected the scores to be lower...

It was really oversight on my part :o I have my commands on a text file - copied another script for the file paths and neglected to delete the -threads 6.

So, I've re-encoded the x265 files omitting the "-threads 6". MediaInfo reports 'frame-threads=2'.

These are the SSIM:GMSD (Downsample=False) obtained with Wolfberry's ffms2 (default seekmode=1)

crowd_run_1080p50_y4m_x265_CRF28_v2.mp4:

stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170

crowd_run_1080p50_y4m_x265_CRF28_KeyInt100_v2.mp4:

stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637

Compared with the x265 files that were encoded with '-threads 6' (MediaInfo reports 'frame-threads=6')

crowd_run_1080p50_y4m_x265_CRF28.mp4:

stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170
stop 436.0761918885030863357599170 83.36467719049633404160282170

crowd_run_1080p50_y4m_x265_CRF28_KeyInt100.mp4:

stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637

So it actually made no difference to the results.

In fact the two crowd_run_1080p50_y4m_x265_CRF28.mp4 files are byte identical and the crowd_run_1080p50_y4m_x265_CRF28_KeyInt100.mp4 files differ by only 5 bytes.

Still, here are the re-encoded files (omitting -threads 6):

crowd_run_1080p50_y4m_x265_CRF28_v2 (https://drive.google.com/file/d/1j0A4ImFL2c1xV9a1RX-KdOB-8pS6KlT8/view?usp=sharing)

poisondeathray

20th March 2019, 16:54

So it actually made no difference to the results.

Maybe not enough vertical motion to exhibit the frame threads effect in this shot. No idea why your values where consistently slightly higher than Iron_Mikes' (frame threads would have been a nice way to explain it) . Maybe some more inconsistencies between runs or systems :(

But beyond that , look at the bitrate / filesize between your uploads and his. Way different. Some settings are actually different too looking at mediainfo . psy-rdoq is 0 for his 1 for yours for the keyint 100 encode, rd4 vs 2 etc... . Need to compare apples to apples . Or at least look at the same physical file

WorBry

20th March 2019, 17:30

Iron_Mike

20th March 2019, 23:21

He modified the 'original' test encode settings for those uploads - changed -preset slow to fast and added -rc-lookahead=100:

https://forum.doom9.org/showthread.php?p=1869402#post1869402

Why, I don't know.

So he should upload x265 files encoded with the 'original' scripts for valid comparison:

https://forum.doom9.org/showthread.php?p=1869390#post1869390

you misread my post (the last post you linked to):

rc-lookahead was ALWAYS a parameter I included if I specified custom keyframes... but I did ONE TEST to see if specifically the rc-lookahead parameter was the problem for ffms2, hence I removed that param in that ONE encode...

which is stated in my original post (https://forum.doom9.org/showpost.php?p=1869188&postcount=128), and as you can see in point (2) in that post - which you removed from the quote - I have used the rc-lookahead param...

All files uploaded by me were done w/ the rc-lookahead param (if custom keyframes were specified), only reason I had to re-upload once is b/c I used incorrect syntax for the no-open-gop flag on one of the three files...

so, if a custom keyframes intval was specifed these three params were added to the x265-params: keyint=100:min-keyint=100:rc-lookahead=100

WorBry

21st March 2019, 00:41

WorBry

21st March 2019, 01:24

And I tested the mp4 with the old ffms2 and it has no seeking problems. https://github.com/FFMS/ffms2/releases/download/2.23/ffms2-2.23.1-msvc.7z
(This is the second file what has no problems with the old version)

Trouble is, VS crashes when I try to test it.

Think I'll switch to LWLibavSource for future metric testing.

poisondeathray

21st March 2019, 01:30

@Poisondeathray, I'm interested to know how your results with the files I uploaded compare anyway.

Not repeatable results here , unless seekmode=0 . I only tested your keyint 100, 6 frame thread version . Larger errors on some runs, clearly frame mismatches

Not able to replicate the 12th or whatever tiny decimal error with seekmode=0 over 10 runs . 100% consistent

Tried 2 computers, but both were 4C/8T Intels . I also have a really old AMD (with XP!) lying around but I didn't test on that

EDIT #1: also on the other file, with default settings (no custom keyint) , it was inconsistent too. Iron_Mike reported "big" inconsistent results on some runs only when using custom keyint . I'm seeing inconsistent everywhere with ffms2 default seekmode on these.

EDIT #2: remuxing to mkv with mkvmerge made everything consistent for the keyint 100 file (10 runs matched) with default seekmode, as CK also reported . This makes me think it has something to do with ffms2 reading the timebase and timecodes in the container , or container mux, or interpretation issue

ffmpeg -i input.ext

mkv
Stream #0:0: Video: hevc (Main), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR
16:9], 50 fps, 50 tbr, 1k tbn, 50 tbc (default)

mp4
Stream #0:0(und): Video: hevc (Main) (hev1 / 0x31766568), yuv420p(tv, bt709,
progressive), 1920x1080 [SAR 1:1 DAR 16:9], 15411 kb/s, 50 fps, 50 tbr, 12800 t
bn, 50 tbc (default)

ffmpeg "thinks" the tbn is 12800 for the mp4 . tbn is the container time base

Iron_Mike

21st March 2019, 02:01

No I didn't misread your post. You changed the preset to fast also. First you are following my encode script, as a common ground - now you're not. Have you even taken the trouble to test the files I uploaded and post the results, as I did yours ? Really can't be bothered with this.

@Poisondeathray, I'm interested to know how your results with the files I uploaded compare anyway.

Cheers.
You did misread as your own post clearly proves, b/c I state in detail in my post that I left out rc-lookadhead in one encode...... !?

The preset does NOT matter in any of this - no idea why you say any of that.

I also don't care if you test ffms2 or not. I posted and proved that there are issues w/ seekmode=1, other people confirmed.

Don't care whether you personally test this or not.

The only reason why I even bothered to write here is b/c you spend some time doing some tests and if you ever are subject to this behavior - or other inconsistent behavior we've not yet discovered - then your posted results here would be pointless.

Hence, it's always good if another source confirms general trends of test results of these metrics... which I did.

Iron_Mike

21st March 2019, 02:23

Not repeatable results here , unless seekmode=0 . I only tested your keyint 100, 6 frame thread version . Larger errors on some runs, clearly frame mismatches

Not able to replicate the 12th or whatever tiny decimal error with seekmode=0 over 10 runs . 100% consistent

Tried 2 computers, but both were 4C/8T Intels . I also have a really old AMD (with XP!) lying around but I didn't test on that

EDIT #1: also on the other file, with default settings (no custom keyint) , it was inconsistent too. Iron_Mike reported "big" inconsistent results on some runs only when using custom keyint . I'm seeing inconsistent everywhere with ffms2 default seekmode on these.

tested WorBry's files...

WorBry - crowd_run_1080p50_y4m_x265_CRF28.mp4 - seekmode=1 - GMSD/SSIM

stop 83.36467719049633404160282170 436.0761547550154326735594167
stop 83.36467719049633404160282170 436.0761547550154326735594166
stop 83.36467719049633404160282170 436.0761547550154326735594166
stop 83.36467719049633404160282170 436.0761547550154326735594166
stop 83.36467719049633404160282170 436.0761547550154326735594166

WorBry - crowd_run_1080p50_y4m_x265_CRF28_KeyInt100.mp4 - seekmode=1 - GMSD/SSIM

stop 124.0039187143357256870590535 227.9924869038146211797091216
stop 83.19534706841788751496835637 435.9938219159915119282899813
stop 120.1705765900797210909978223 248.1439760410638495358170989
stop 122.2412107559194712758099402 236.4289261580102229909350346
stop 122.2412107559194712758099402 236.4289261580102229909350346

matches my previously reported behavior...

Iron_Mike

21st March 2019, 02:26

And I tested the mp4 with the old ffms2 and it has no seeking problems. https://github.com/FFMS/ffms2/releases/download/2.23/ffms2-2.23.1-msvc.7z

As I posted before, this old ffms2 version crashes for me w/ your Fatpack... I simply copied the x64 plug into the Fatpack's VS plugins folder... anything else I need to do ?

WorBry

21st March 2019, 03:10

The only reason why I even bothered to write here is b/c you spend some time doing some tests and if you ever are subject to this behavior - or other inconsistent behavior we've not yet discovered - then your posted results here would be pointless.

Which clearly is not the case - unlike your comparison of medium vs fast presets which, if the attachment ever gets approved, must be viewed as suspect in the light of your unresolved issues. How can you post the results of study knowing that your test system is unreliable? Very unscientific ?

ran a couple of tests to see how preset behave for x265...

here I tested medium vs. fast... same range as WorBry CRF 0-30, used the same graph layout

https://forum.doom9.org/attachment.php?attachmentid=16770&stc=1&d=1552905262

OK, are we done ?

WorBry

21st March 2019, 03:18

Iron_Mike

21st March 2019, 03:23

Which clearly is not the case - unlike your comparison of medium vs fast presets which, if the attachment ever gets approved, must be viewed as suspect in the light of your unresolved issues. How can you post the results of study knowing that your test system is unreliable? Very unscientific ?

b/c contrary to you - who has not tested his system and only learned from me that there are potential problems - I've only run all metrics w/ seekmode=0 and confirmed each metric multiple times (consistent, repeatable) before posting them...

yes, you are done, Brian. go take a nap. :D

WorBry

21st March 2019, 03:38

b/c contrary to you - who has not tested his system and only learned from me that there are potential problems ..

Ridiculous statement.

..go take a nap. :D

Very juvenile, but I had that sense already.

Iron_Mike

21st March 2019, 03:47

So ffms2 is unreliable on some machines. What about LWLibavSource ?

well... that took a while. :sly:

as far as I can see, the ONLY one that it has been fully "reliable" w/ seekmode=1 w/ custom keyframes is you...

I posted it, CK confirmed and poisondeathray confirmed...

As I stated pages ago, it would not surprise me if it turns out to be a package difference, and by that I mean a specific VS plugin difference, but w/o a direct, convenient method to list version/release data of all VS plugins involved in the VS chain, it'll be hard to compare...

easiest way to confirm it is to create a minimal VS venv - only what's needed for this ffms2 read test - zip it, upload it and then everybody test in that venv on their machines...

for VS, I created an isolated venv, but I used CK's Fatpack as a base, and there's too many other things included that are not needed for this minimal test...

good news is that seekmode=0 is rock solid on my end (and not slower), and Lsmash as well, so it's not a show stopper...

poisondeathray

21st March 2019, 04:05

good news is that seekmode=0 is rock solid on my end (and not slower), and Lsmash as well, so it's not a show stopper...

If you ignore the long decimal place variation issue , that you reported here with seekmode=0 . That's not a deal breaker, I guess you could "round" it or truncate it . So still some weirdness going on there than I cannot replicate

https://forum.doom9.org/showthread.php?p=1869188#post1869188

And also , remuxing to MKV "fixed" it for me and CK with default seekmode, but not you , is that correct ? So some weirdness and inconsistencies between setups too

I always have like 15 different ffms2 versions handy for AVS and VPY. The way I organize it is keep each one in a separate labelled subfolder. Because each build has different kinds of quirks and bugs . Some with different formats, e.g some drop certain types of MOV support, others will get the fps slightly off with buggy timecodes. Others will double TS frames (reading field rate as frame rate) . Some will have VP9 issues. Some AV1 issues. Some will have seek issues with certain formats. LSmash versions can share these little bugs too , but they are not as frequently released

I would love a "super duper" ffms2 and lsmash version :(

WorBry

21st March 2019, 04:15

well... that took a while. :sly:

You just don't stop do you.

So there is an issue, and clearly this is of interest for others, who may run into the same problems...

Not denying there is an issue, but I'm not seeing it.

WorBry

21st March 2019, 04:23

If you ignore the long decimal place variation issue , that you reported here with seekmode=0 . That's not a deal breaker, I guess you could "round" it or truncate it . So still some weirdness going on there than I cannot replicate

https://forum.doom9.org/showthread.php?p=1869188#post1869188

Nor I.

LSmash versions can share these little bugs too...

What about LWLibavSource ?

Did you test LWLibavSource also ?

poisondeathray

21st March 2019, 04:35

Did you test LWLibavSource also ?

Not on this series

And for MP4/MOV container , you can use LibavSMASHSource without indexing . But that might affect the seeking here , might be worth while to test

Iron_Mike

21st March 2019, 04:36

If you ignore the long decimal place variation issue , that you reported here with seekmode=0 . That's not a deal breaker, I guess you could "round" it or truncate it . So still some weirdness going on there than I cannot replicate

yeah, in some cases it provided more precision, in a perfectly stable, repeatable system that should not happen

And also , remuxing to MKV "fixed" it for me and CK with default seekmode, but not you , is that correct ?

I've not done that yet, if you tell me exactly which tools involved I can test it... or can I just copy the streams over into an mkv container via ffmpeg ?

poisondeathray

21st March 2019, 04:40

I've not done that yet, if you tell me exactly which tools involved I can test it... or can I just copy the streams over into an mkv container via ffmpeg ?

I used mkvmerge (mkvtoolnix is the GUI)

CK reported this finding here
https://forum.doom9.org/showthread.php?p=1869333#post1869333

Sorry I got mixed up; it was WorBry that still got seek errors after remuxing to MKV with several methods

Iron_Mike

21st March 2019, 04:51

@ChaosKing

could you assist in putting together a stripped down, minimal venv for a dedicated ffms2 test ?

In the vein of your Fatpack just extremely slimmed down - only to read vid files via ffms2... so:

> Py 3.7
> VS
> ffms2 Wolfberry
> all ffms2 dependencies

this should not be much more than 120MB... we could zip it and distrbiute it for testing...

read in your thread that you simply "threw everything in one folder", so if u lmk any caveats or your problems I can do it or maybe you can do it...

or can we just remove all plugs from your latest Fatpack and use that as a base... ?

Wolfberry

21st March 2019, 04:54

I always have like 15 different ffms2 versions handy for AVS and VPY. The way I organize it is keep each one in a separate labelled subfolder. Because each build has different kinds of quirks and bugs . Some with different formats, e.g some drop certain types of MOV support, others will get the fps slightly off with buggy timecodes. Others will double TS frames (reading field rate as frame rate) . Some will have VP9 issues. Some AV1 issues. Some will have seek issues with certain formats. LSmash versions can share these little bugs too , but they are not as frequently released

I would love a "super duper" ffms2 and lsmash version :(

Read: it may have nothing to do with the encoder or the decoder, but about the ability of FFMS2 to use the FFmpeg API to parse the bitstream in the correct way, and certain encoder/decoder combinations expose errant behavior in the parser. This would explain why ffplay (or mpv) works fine on the same sample and the same build of libdav1d that FFMS2 uses, but FFMS2 exhibits problems.

I think that the reason behind the different kinds of quirks and bugs is the different FFmpeg version that FFMS2 / L-SMASH is built with (like 3.x or 4.x)

A newer FFmpeg version can fix/add support for some formats but can also introduce new bugs in some other formats in FFMS2 / L-SMASH due to their ability to use the FFmpeg API to parse the bitstream in the correct way.

WorBry

21st March 2019, 04:55

And for MP4/MOV container , you can use LibavSMASHSource without indexing . But that might affect the seeking here , might be worth while to test

https://forum.doom9.org/showthread.php?p=1869332#post1869332

WorBry

21st March 2019, 04:57

it was WorBry that still got seek errors after remuxing to MKV with several methods

With the seek-test script yes, but not in the metric tests.

poisondeathray

21st March 2019, 05:21

With the seek-test script yes, but not in the metric tests.
But for Iron_Mike, didn't he say the seek-test correlated with the other metric tests? The gist I got was that it was predictive for him - when one one failed, the other failed ?

Basically you guys were getting something different from those results compared to each other . And I'm getting a 3rd different :(

Seek test fails both MP4 and MKV with default seekmode, but metrics inconsistent with MP4, but consistent with MKV

Another question is why does this vpy script request frames out of order, instead of linearly? You can use the same decoder, the same ffms2 source filter, and use -f vapoursynth in Wolfberry's ffmpeg and the ssim / psnr there and it's consistent 10/10 times because frames are requested linearly in order . I realize from WorBry's earlier discussion the ssim calcs are slightly different; but ideally this should be a linear operation.

In terms of difficulty the synthetic seek test is like a torture, worse case sceneario .

WorBry

21st March 2019, 06:38

With the seek-test script yes, but not in the metric tests.

But for Iron_Mike, didn't he say the seek-test correlated with the other metric tests? The gist I got was that it was predictive for him - when one one failed, the other failed ?

Basically you guys were getting something different from those results compared to each other .

Right, and this is why I looked at the alignment of the reference and test file output frames when applying random frame searches vs playback from frame #0 in VSEditor:

https://forum.doom9.org/showthread.php?p=1869323#post1869323

So it is only random seeking with ffms2 (default, seekmode=1), which is what the seek-test script does, that gives these frame offsets, not sequential output from frame #0, as is the case when the metric test scripts are run with Benchmark to generate the score log files.

This perfectly explains why the seek-test script gave errors with the encoded x265 file and ffms2 mode #1 (seekmode=1), yet the results of the SSIM:GMSD metric tests with ffms2 (default, seekmode=1) were in perfect agreement (aggregate and per frame scores) with those produced with LWLibavSource - in my tests at least.

And I'm getting a 3rd different :(

Seek test fails both MP4 and MKV with default seekmode, but metrics inconsistent with MP4, but consistent with MKV

The difference between me and thee being that I also get consistent metrics with mp4 and ffms2 in default seekmode=1

Iron_Mike

21st March 2019, 06:42

But for Iron_Mike, didn't he say the seek-test correlated with the other metric tests? The gist I got was that it was predictive for him - when one one failed, the other failed ?

correct, the seek script predicted things correctly for me.

alright, copied video stream of the mp4 encode w/ custom keyframes and open-gop into mkv container via ffmpeg

<ffmpeg> -i encode_with_keyfranmes_open_gop.mp4 -c copy encode_with_keyfranmes_open_gop.mkv

(lmk if that is not the correct way to create the mkv file)

> 1st metrics test run: MDSI - mkv - seekmode=1

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 202.9107845033357604980217559

> 2nd metrics test run: MDSI - mkv - seekmode=1 (same script)

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> 3rd metrics test run: MDSI - mkv - seekmode=1 (same script, just more reps)

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> 4th metrics test run: MDSI - mkv - seekmode=1 (same script, just more reps)

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 194.8927784337109077039684736
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> 5th metrics test run: GMSD/SSIM - mkv - seekmode=1

stop 105.8009960111949401551090232 321.3940602997202934110188006
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 114.6468917102554510301093416 275.2515876977237659073427521
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524

> seek-test.py from CK, test #1 on mkv: seek errors, 3 frames off

> the corresponding .mp4 file (encode w/ custom keyframes and open-gop) - seekmode=1

> MDSI:

stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986
stop 158.6441298300117068720282986

> GMSD/SSIM:

stop 123.9634227532677362149104996 227.1636231975790896908673488
stop 122.1886438717397119491092642 235.7664674057195217937721310
stop 123.9120154727625848778416187 228.0649240752797068665369550
stop 123.6926367223807698425552817 228.4161432110821759733365609
stop 123.7779981257600182276146230 228.0649240752797068665369550

--> so why is the MDSI metric much more stable (although not perfect) w/ seekmode=1 ?

ChaosKing

21st March 2019, 09:37

As I posted before, this old ffms2 version crashes for me w/ your Fatpack... I simply copied the x64 plug into the Fatpack's VS plugins folder... anything else I need to do ?
Nope, all you need is to replace the "old" ffms2.dll. You probably missing a vs runtime.
You could try downloading the 64 bit version of these files: https://www.dll-files.com/msvcp_win.dll.html, https://www.dll-files.com/msvcrt.dll.html and see if it helps. Copy the runtime dlls also inside the plugins folder where ffms2 is located.
And maybe all msvcp*.dlls from VapourSynth64 folder just in case :D But I think msvcrt.dll should fix it.

I always have like 15 different ffms2 versions handy for AVS and VPY. The way I organize it is keep each one in a separate labelled subfolder. Because each build has different kinds of quirks and bugs . Some with different formats, e.g some drop certain types of MOV support, others will get the fps slightly off with buggy timecodes. Others will double TS frames (reading field rate as frame rate) . Some will have VP9 issues. Some AV1 issues. Some will have seek issues with certain formats. LSmash versions can share these little bugs too , but they are not as frequently released

I would love a "super duper" ffms2 and lsmash version :(

How about a test folder with some video files and a folder with different ffms2/lsmash dlls + a test script wich just tests all files with all dlls via seek-test? Like an automated mass seek-test if you will. Easy to make... the results could be written into a txt or csv file.

@ChaosKing

could you assist in putting together a stripped down, minimal venv for a dedicated ffms2 test ?

In the vein of your Fatpack just extremely slimmed down - only to read vid files via ffms2... so:

> Py 3.7
> VS
> ffms2 Wolfberry
> all ffms2 dependencies

this should not be much more than 120MB... we could zip it and distrbiute it for testing...

read in your thread that you simply "threw everything in one folder", so if u lmk any caveats or your problems I can do it or maybe you can do it...

or can we just remove all plugs from your latest Fatpack and use that as a base... ?

Sure why not. Will upload it a bit later.

Another question is why does this vpy script request frames out of order, instead of linearly?
Because the requests are made in parallel wich makes your script run faster.

Wolfberry

21st March 2019, 10:57

You could try downloading the 64 bit version of these files: https://www.dll-files.com/msvcp_win.dll.html, https://www.dll-files.com/msvcrt.dll.html and see if it helps. Copy the runtime dlls also inside the plugins folder where ffms2 is located.
And maybe all msvcp*.dlls from VapourSynth64 folder just in case :D But I think msvcrt.dll should fix it.

Well, I don't think downloading arbitrary runtime dlls is a good way to solve runtime dependency problems.

If you are lazy you can just install an AIO pack (https://repacks.net/viewtopic.php?t=247) as mentioned in this thread (http://forum.doom9.org/showthread.php?t=176026).

ChaosKing

21st March 2019, 11:55

Correct. It's just to quickly confirm that this dll is the culprit. I checked with ldd and dependency walker and msvcrt stood out. Or maybe it's ADVAPI32.dll but I think its a system dll.

EDIT
ok lol, the "old" ffms2 crashes with the y4m file!!! I only tested it via seek-test and the mp4 file in vsedit.

ldd ffms2.dll <--- old ffms2.dll
ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffd32440000)
KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x7ffd2f980000)
KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x7ffd2e5c0000)
apphelp.dll => /cygdrive/c/WINDOWS/SYSTEM32/apphelp.dll (0x7ffd2c780000)
ADVAPI32.dll => /cygdrive/c/WINDOWS/System32/ADVAPI32.dll (0x7ffd2fa60000)
msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x7ffd2fe20000)
sechost.dll => /cygdrive/c/WINDOWS/System32/sechost.dll (0x7ffd2f8e0000)
RPCRT4.dll => /cygdrive/c/WINDOWS/System32/RPCRT4.dll (0x7ffd30380000)
WS2_32.dll => /cygdrive/c/WINDOWS/System32/WS2_32.dll (0x7ffd323a0000)
USER32.dll => /cygdrive/c/WINDOWS/System32/USER32.dll (0x7ffd2fb20000)
Secur32.dll => /cygdrive/c/WINDOWS/SYSTEM32/Secur32.dll (0x7ffd1c190000)
win32u.dll => /cygdrive/c/WINDOWS/System32/win32u.dll (0x7ffd2f410000)
GDI32.dll => /cygdrive/c/WINDOWS/System32/GDI32.dll (0x7ffd32370000)
gdi32full.dll => /cygdrive/c/WINDOWS/System32/gdi32full.dll (0x7ffd2e9d0000)
msvcp_win.dll => /cygdrive/c/WINDOWS/System32/msvcp_win.dll (0x7ffd2eb70000)
ucrtbase.dll => /cygdrive/c/WINDOWS/System32/ucrtbase.dll (0x7ffd2e8b0000)
SSPICLI.DLL => /cygdrive/c/WINDOWS/SYSTEM32/SSPICLI.DLL (0x7ffd2e340000)
IMM32.DLL => /cygdrive/c/WINDOWS/System32/IMM32.DLL (0x7ffd30510000)

$ ldd ffms2_.dll <-- (dll by Wolfberry)
ntdll.dll => /cygdrive/c/WINDOWS/SYSTEM32/ntdll.dll (0x7ffd32440000)
KERNEL32.DLL => /cygdrive/c/WINDOWS/System32/KERNEL32.DLL (0x7ffd2f980000)
KERNELBASE.dll => /cygdrive/c/WINDOWS/System32/KERNELBASE.dll (0x7ffd2e5c0000)
bcrypt.dll => /cygdrive/c/WINDOWS/System32/bcrypt.dll (0x7ffd2f3e0000)
msvcrt.dll => /cygdrive/c/WINDOWS/System32/msvcrt.dll (0x7ffd2fe20000)
USER32.dll => /cygdrive/c/WINDOWS/System32/USER32.dll (0x7ffd2fb20000)
Secur32.dll => /cygdrive/c/WINDOWS/SYSTEM32/Secur32.dll (0x7ffd1c190000)
win32u.dll => /cygdrive/c/WINDOWS/System32/win32u.dll (0x7ffd2f410000)
GDI32.dll => /cygdrive/c/WINDOWS/System32/GDI32.dll (0x7ffd32370000)
gdi32full.dll => /cygdrive/c/WINDOWS/System32/gdi32full.dll (0x7ffd2e9d0000)
msvcp_win.dll => /cygdrive/c/WINDOWS/System32/msvcp_win.dll (0x7ffd2eb70000)
ucrtbase.dll => /cygdrive/c/WINDOWS/System32/ucrtbase.dll (0x7ffd2e8b0000)
WS2_32.dll => /cygdrive/c/WINDOWS/System32/WS2_32.dll (0x7ffd323a0000)
RPCRT4.dll => /cygdrive/c/WINDOWS/System32/RPCRT4.dll (0x7ffd30380000)
SSPICLI.DLL => /cygdrive/c/WINDOWS/SYSTEM32/SSPICLI.DLL (0x7ffd2e340000)
sechost.dll => /cygdrive/c/WINDOWS/System32/sechost.dll (0x7ffd2f8e0000)
IMM32.DLL => /cygdrive/c/WINDOWS/System32/IMM32.DLL (0x7ffd30510000)

WorBry

21st March 2019, 19:20

With the seek-test script yes, but not in the metric tests.

Just to mention that the seek-test script (Wolfberry ffms2 seekmode=1) reports no issues with x264 mp4/mkv

ffmpeg -i {Path}:/crowd_run_1080p50.y4m -vcodec libx264 -preset slow -crf 28 -pix_fmt yuv420p -r 50/1 -x264-params colorprim=bt709:transfer=bt709:colormatrix=bt709 {Path}:/crowd_run_1080p50_y4m_x264_CRF28.mp4

The rav1e.mkv encodes of Crowd Run that ChaosKing uploaded earlier...

https://forum.doom9.org/showthread.php?p=1866894#post1866894

....do give errors (1 frame off) with the seek-test (ffms2 seekmode=1) but the SSIM:GMSD results are consistent on consecutive runs and the log files give the frames in correct order.

Iron_Mike

22nd March 2019, 07:40

ran a couple of tests to see how preset behave for x265...

here I tested medium vs. fast... same range as WorBry CRF 0-30, used the same graph layout

https://forum.doom9.org/attachment.php?attachmentid=16775&stc=1&d=1553236484

more stats - x265 - preset: medium vs. fast

Butteraugli / MDSI per CRF

https://i.imgur.com/6QVs6Ru.png

VMAF / SSIM / GMSD per bitrate (Mbps)

https://i.imgur.com/PM5RWBo.png

Butteraugli / MDSI per bitrate (Mbps)

https://i.imgur.com/EzIjj93.png

WorBry

22nd March 2019, 19:38

Incidentally, in your tests, what results do you get for the different metrics when you perform a 'self test' on your x265 encodes i.e. use the same encode as the reference and test clip as a positive control ?

VMAF is an exception because it gives a component 'motion2' metric score of 0 to the first frame which can skew the per-frame and aggregate VMAF score.

But for the SSIM, GMSD, MDSI and Butteraugli tests, the 'self' scores should be absolute i.e. 1 for SSIM and 0 for the other three. If not, something is amiss.

I ran self tests in all of the studies I carried out earlier, as well as duplicate tests on representative encodes (typically at the highest and lowest quality settings), and encode duplicates, (except for the rav1e series, which I didn't encode) to check for reproducibility before proceeding to test the entire series. And found no inconsistencies.

Iron_Mike

22nd March 2019, 22:32

Incidentally, in your tests, what results do you get for the different metrics when you perform a 'self test' on your x265 encodes i.e. use the same encode as the reference and test clip as a positive control ?
But for the SSIM, GMSD, MDSI and Butteraugli tests, the 'self' scores should be absolute i.e. 1 for SSIM and 0 for the other three. If not, something is amiss.

so run a control test on a x265 encode that has custom keyframes w/ open-gop ? seekmode=0 or seekmode=1 ? or what are we looking for here ?

VMAF is an exception because it gives a component 'motion2' metric score of 0 to the first frame which can skew the per-frame and aggregate VMAF score.

the "first frame" thing is only specific to the crowdrun file, we discussed this a while ago. When I ran extensive VMAF control tests on other files than crowdrun, the score was never 99.999 and it wasn't just the first frame score that threw it off...

as you know, NF does state that control test score of 98.x are to be expected...

WorBry

22nd March 2019, 23:44

so run a control test on a x265 encode that has custom keyframes w/ open-gop ? seekmode=0 or seekmode=1 ? or what are we looking for here ?

If that's how you encoded your x265 files, yes, and using whatever seekmode you used in your metric tests. You don't need to test the entire series.

You should also run a 'self-test' on the source/reference clip though - I assume it was crowd_run_1080p50.y4m.

the "first frame" thing is only specific to the crowdrun file, we discussed this a while ago. When I ran extensive VMAF control tests on other files than crowdrun, the score was never 99.999 and it wasn't just the first frame score that threw it off...

as you know, NF does state that control test score of 98.x are to be expected...

All I was saying is that VMAF is an exception.

Iron_Mike

23rd March 2019, 00:25

If that's how you encoded your x265 files, yes, and using whatever seekmode you used in your metric tests. You don't need to test the entire series.

You should also run a 'self-test' on the source/reference clip though - I assume it was crowd_run_1080p50.y4m.

control - crowd_run_1080p50.y4m - seekmode=0 - GMSD/SSIM

stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500

control - (one example file) x265 CRF 28, preset medium, open-gop, no custom keyframes - seekmode=0 - GMSD/SSIM

--> this is how I encoded my series of encodes for the preset medium|fast comparison

stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500
stop 0 500

WorBry

23rd March 2019, 00:54

That's good then.

ChaosKing

23rd March 2019, 11:05

Ok here's my idea for testing for frame seeking issues. VSNTSF - VapourSynths Next Top Source Filter

Download https://www.dropbox.com/s/illkfly7wq72ay7/VapourSynth64PortableTEST.rar?dl=1

Put your video files in test_chamber\video_test_files
ffms2 & lsmas dlls in test_chamber\ffms2 & test_chamber\lsmas

Edit the bat file if needed
set start=0
set end=300
set tests=ffms2 lsmas lsmasav ffms2seek0

Result is a csv file (open in excel or similar app)
https://i.imgur.com/YP6vAxa.png

Testing only the first 10 frames!
https://i.imgur.com/0Sn66gc.png

EDIT:
- Added LibavSMASHSource() which doesn't create an index, but works only with mp4? (From Avisynth wiki: " Recommended for MP4, MOV, ISO Base Media and its derived file formats.")

set tests=ffms2 lsmas lsmasav ffms2seek0
means:
- ffms2.Source()
- lsmas.LWLibavSource()
- lsmas.LibavSMASHSource()
- ffms2.Source(seekmode=0)