GMSD and SSIM Quality Metrics

[WorBry]

did you use the download of crowd_run_1080p50.y4m or did you use your own encode from the 2160p version as ref file ?

The crowd_run_1080p50.y4m file of course :rolleyes:

[zorr]

Mike, can you take a look at what the log files look like when you get the inconsistent results (like below)?

FFMS2 (Wolfberry) - seekmode=1 (default):
stop 123.93944837285996 227.9372376995322
stop 123.74699115224307 228.86426005799092
stop 82.89543157743563 436.641839735243
stop 121.66415998563879 239.07092253508384
stop 122.1422683865401 236.40679167570886


The error there is huge and it might be revealing to see what the scores are for the individual frames. Another option would be to save the resulting video with parameter showstats = True.

[Iron_Mike]

Not denying there is an issue, but I'm not seeing it.

another pointer is that, on my end, when running CK's test script option #1 and option #5 showed both success if the file was NOT encoded w/ keyframes setting... if it was encoded w/ keyframes, then option #1 (seekmode=0) showed seek errors, always 3 frames off...

meaning: seek errors on my system are repeatable (if triggered), and are both confirmed by CK's test script and the inconsistent GMSD/SSIM results.

you posted that you also had seek errors w/ option #1, yet when specifically setting seekmode=1 when running the GMSD/SSIM tests you get no errors...

[Iron_Mike]

Mike, can you take a look at what the log files look like when you get the inconsistent results (like below)?

The error there is huge and it might be revealing to see what the scores are for the individual frames. Another option would be to save the resulting video with parameter showstats = True.

does that require the latest Zoptilib ?

where do I pass in the showstats parameter ? when I create an instance of the class, e.g.

zopti = Zopti(log_fp, metrics=metrics_list, showstats=True)

is that correct ?

[ChaosKing]

@Iron_Mike, yes it's correct.



The error there is huge and it might be revealing to see what the scores are for the individual frames. Another option would be to save the resulting video with parameter showstats = True.

showstats doesn't affect the score since the text overlay is done after the ssim gsm filters. Plus I just tested it again, just to be sure :D

alt_clip = muv.SSIM(alt_clip, clip, **filter_args).text.FrameProps()
Frameprops are showing me this:
https://i.imgur.com/TsrzNz7.png

The values are much shorter. That means Decimal() is also totally unnecessary!? And we need to round even more?
Or is FrameProps() rounding them?

[WorBry]

you posted that you also had seek errors w/ option #1, yet when specifically setting seekmode=1 when running the GMSD/SSIM tests you get no errors...

Correct. The seek-test script reported errors (the 3 frame off thing) with ffms2 - Wolfberry's and the version included with VS Fatpack portable. But I see no score inconsistencies running GMSD/SSIM with the same.

[Iron_Mike]

Correct. The seek-test script reported errors (the 3 frame off thing) with ffms2 - Wolfberry's and the version included with VS Fatpack portable. But I see no score inconsistencies running GMSD/SSIM with the same.

are you going through Zopti or using muvsfunc directly ?

[WorBry]

Through Zoptilib

[WorBry]

Well I have now the run tests with x265 CRF28 encoded with custom key frame interval, exactly as per your script:

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

And these are the results I get on serial testing with SSIM:GMSD (Downsample=False):

LWLibavSource:

stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786

ffms2- Wolfberry (default, i.e. seekmode=1)
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786
stop 435.99385609567906 83.19534706841786


I added Decimal() for testing purposes https://github.com/theChaosCoder/zoptilib
The metric values are now much longer, sum should always be the same between runs. Please test.

Retested with Wolfberry's ffms2 (default, seekmode=1) and zoptilib with the Decimal update:

stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637
stop 435.9938560956790122302706430 83.19534706841788751496835637

Consistent results on consecutive testing (still), but slightly different scores to previous results when rounded up to 14 decimal places.....not that it really matters.

[ChaosKing]

Correct. The seek-test script reported errors (the 3 frame off thing) with ffms2 - Wolfberry's and the version included with VS Fatpack portable. But I see no score inconsistencies running GMSD/SSIM with the same.

Maybe you have a consistent inconsistency :D
I mean your log is always in perfect order... I tested vspipe with 3 frames, and even then sometimes the output is 0 2 1 (before the update)

[Iron_Mike]

alright, more tests, results are the same (for seekmode=1 on keyframes encoded files) - updated to Zoptilib 1.0.8

this is the .vpy script

import vapoursynth as vs
from zoptilib_108 import Zopti

core = vs.get_core()

# set vars
ref_fp = r'/path/to/crowd_run_1080p50.y4m'
enc_fp = r'/path/to/encode_with_keyframes.mp4'
log_fp = r'/path/to/zopti.log'

metrics = ('gmsd', 'ssim')

# load files using default seekmode=1
vid_ref = core.ffms2.Source(source=ref_fp)
vid_enc = core.ffms2.Source(source=enc_fp)

# initialize output file and chosen metrics
zopti = Zopti(log_fp, metrics=metrics)

# run
zopti.run(vid_ref, vid_enc)



--- VSPipe ---

stop 123.8387661168988139459568034 228.0649240752797068665369550
stop 122.1054561320879287833474084 236.3004045199170526647680646
stop 123.9634227532677362149104996 227.1636231975790896908673488
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 120.4371235439986860815686501 244.7757634970582563960217470


--- VSedit ---

I did not put the script in a loop, but: open script > F7 > press START (every time for a new run)

stop 120.4699959471850678194293496 245.2682722981770835624093506
stop 122.9025251845727162713917127 231.9192097981770835091186457
stop 93.01091081028496621030399182 389.5661511983989193086408413
stop 91.15626372903303242800632232 399.5109242681809407216064747
stop 85.74332191455660112655579041 428.0741357421874995203836523

> deleted the ffindex files, which did not get re-created, 12 more runs of the same script

stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 101.1081413762829624147787655 346.7421019904996144500586296
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 100.0318691978474301140789038 351.6773840784143521642768346
stop 93.44757297579696733724752276 387.2203145194347990720729052
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 87.01267141164586690904592101 421.4470892107928237657610272

--> as u can see sometimes it gets 3 in a row right...



Mike, can you take a look at what the log files look like when you get the inconsistent results (like below)?[/B].

file # 1

0; 0.163505585855354740321132567260065115988254547119140625; 0.89805531442901231908848558305180631577968597412109375;
1; 0.1769067779219471170382149693978135474026203155517578125; 0.85627302758487655598429455494624562561511993408203125;
2; 0.1792310442991052743533231250694370828568935394287109375; 0.8481661723572531119685891098924912512302398681640625;
3; 0.1784714411393954203166600791519158519804477691650390625; 0.84848427854938268666273870621807873249053955078125;
4; 0.1779983423827615485190989375041681341826915740966796875; 0.85007233796296299832562226583831943571567535400390625;
5; 0.170543962458093500966782585237524472177028656005859375; 0.87440025800540122080661831205361522734165191650390625;
6; 0.1773295845776405865734659528243355453014373779296875; 0.85208224826388889727724063050118274986743927001953125;
7; 0.179107019975634884634274612835724838078022003173828125; 0.8447753906249999555910790149937383830547332763671875;
8; 0.17649390964808808757169344971771351993083953857421875; 0.85264449508101847197139022682677023112773895263671875;
9; 0.17707103545260505228498004726134240627288818359375; 0.85212275752314814436516599016613326966762542724609375;
10; 0.1701688806650836494060996528787654824554920196533203125; 0.87342773437499998667732370449812151491641998291015625;
...
490; 0.274298550587420308932706802806933410465717315673828125; 0.28572524836033952322367213128018192946910858154296875;
491; 0.27483107733594713817382171328063122928142547607421875; 0.27932755835262346177927383905625902116298675537109375;
492; 0.27492352759235305637020019275951199233531951904296875; 0.2829961178626543283343153234454803168773651123046875;
493; 0.273530875811458684676580332961748354136943817138671875; 0.300018687307098741445798850691062398254871368408203125;
494; 0.273138560134608965324076734759728424251079559326171875; 0.29670530719521603923993779972079209983348846435546875;
495; 0.2725773231787707029383227563812397420406341552734375; 0.313249662422839525444118180530495010316371917724609375;
496; 0.273262551506504303144851064644171856343746185302734375; 0.301779996141975315193661799639812670648097991943359375;
497; 0.2638890744928075182684779065311886370182037353515625; 0.39238395785108026725396257461397908627986907958984375;
498; 0.245544133215554405413882932407432235777378082275390625; 0.5239230685763889283634853200055658817291259765625;
499; 0.177711940866405015970741487763007171452045440673828125; 0.84061951437114201812761393739492632448673248291015625;
stop 93.44757297579696733724752276 387.2203145194347990720729052

file # 2

0; 0.163505585855354740321132567260065115988254547119140625; 0.89805531442901231908848558305180631577968597412109375;
1; 0.1769067779219471170382149693978135474026203155517578125; 0.85627302758487655598429455494624562561511993408203125;
2; 0.1792310442991052743533231250694370828568935394287109375; 0.8481661723572531119685891098924912512302398681640625;
3; 0.1784714411393954203166600791519158519804477691650390625; 0.84848427854938268666273870621807873249053955078125;
4; 0.1779983423827615485190989375041681341826915740966796875; 0.85007233796296299832562226583831943571567535400390625;
5; 0.170543962458093500966782585237524472177028656005859375; 0.87440025800540122080661831205361522734165191650390625;
6; 0.1773295845776405865734659528243355453014373779296875; 0.85208224826388889727724063050118274986743927001953125;
7; 0.179107019975634884634274612835724838078022003173828125; 0.8447753906249999555910790149937383830547332763671875;
8; 0.17649390964808808757169344971771351993083953857421875; 0.85264449508101847197139022682677023112773895263671875;
9; 0.17707103545260505228498004726134240627288818359375; 0.85212275752314814436516599016613326966762542724609375;
10; 0.1701688806650836494060996528787654824554920196533203125; 0.87342773437499998667732370449812151491641998291015625;
...
490; 0.164946739428928845772048816797905601561069488525390625; 0.871590350115740708503153655328787863254547119140625;
491; 0.173819541766866370036126454579061828553676605224609375; 0.849342146508487605416348742437548935413360595703125;
492; 0.1763885578142398158352222026223898865282535552978515625; 0.84145887586805556690450202950160019099712371826171875;
493; 0.1748831330143497397866525489007472060620784759521484375; 0.84578281732253091096396246939548291265964508056640625;
494; 0.1741334644004921827775689280315418727695941925048828125; 0.846259825906635843040248801116831600666046142578125;
495; 0.1677916978418648452642258916966966353356838226318359375; 0.86350929542824073070761414783191867172718048095703125;
496; 0.173813412948995438256361012463457882404327392578125; 0.84659740306712960578039428582997061312198638916015625;
497; 0.1727639159829111592525663354535936377942562103271484375; 0.84863715277777773682288398049422539770603179931640625;
498; 0.17653796953829969940130695249536074697971343994140625; 0.8404566936728394832556432447745464742183685302734375;
499; 0.177711940866405015970741487763007171452045440673828125; 0.84061951437114201812761393739492632448673248291015625;
stop 85.74332191455660112655579041 428.0741357421874995203836523

had to trim the files as the post ended up having too many chars, but thee two example files are attached (see below)...

[WorBry]

Correct. The seek-test script reported errors (the 3 frame off thing) with ffms2 - Wolfberry's and the version included with VS Fatpack portable. But I see no score inconsistencies running GMSD/SSIM with the same.

Maybe you have a consistent inconsistency :D
I mean your log is always in perfect order... I tested vspipe with 3 frames, and even then sometimes the output is 0 2 1 (before the update)

Wasn't sure if you were joking there, but I've examined this further and I have an explanation.

When the seek-test script is run on the Crowd Run 1080p50 x265 CRF28.mp4 encode (0-499 frames) it reports errors (3 frames off) with ffms2 mode #1 (seekmode=1) but no errors with ffms2 mode# 5 (seekmode=0) or L-Smash Works.

The source/reference crowd_run_1080p50.y4m gives no errors with any of these modes, as you would expect, being raw, intra-frame video.

Logically then you would expect to see frame mismatches between the x265 encode and the reference clip when imported with ffms2 in default seekmode=1.

So I ran a script to compare the output frames.

import vapoursynth as vs
core = vs.get_core()
Ref = core.ffms2.Source(source=r'X:/crowd_run_1080p50.y4m')
Ref = core.std.Crop(Ref, left=0,top=0,right=960,bottom=0)
Test = core.ffms2.Source(source=r'X:/crowd_run_1080p50_x265_CRF28_KeyInt100.mp4')
Test = core.std.Crop(Test, left=0,top=0,right=960,bottom=0)
Stack = core.std.StackHorizontal([Ref, Test])
Stack.set_output()

Note I used the x265 encode with the custom key interval (100) here.

Opened the script in VSEditor > Preview and made random frame selections with the mouse cursor (left click) on the timeline and also using the frame selection box (to the left of the timeline). Up to frame #99 there were no frame mismatches, but from #100 there were indeed frame offsets.

However, when I reset to frame #0, closed and re-opened Preview, played the video (from frame #0) and paused at any point, the frames always matched.

With ffms2 set to seekmode=0, there were no frame mismatches on random frame seeking or playback, and likewise with LWLibavSource as the import filter.

So it is only random seeking with ffms2 (default, seekmode=1), which is what the seek-test script does, that gives these frame offsets, not sequential output from frame #0, as is the case when the metric test scripts are run with Benchmark to generate the score log files.

This perfectly explains why the seek-test script gave errors with the encoded x265 file and ffms2 mode #1 (seekmode=1), yet the results of the SSIM:GMSD metric tests with ffms2 (default, seekmode=1) were in perfect agreement (aggregate and per frame scores) with those produced with LWLibavSource - in my tests at least.

Clearly, if there are issues with the indexed frame ordering then it's going to complicate matters further.

[WorBry]

As for Iron_Mike's case - purely a stab in the dark, but maybe try using LibavSMASHSource, which doesn't create an index file. It won't open the reference crowd_run_1080p50.y4m as such, but try it for importing the x265 test clip:

clip = core.lsmas.LibavSMASHSource(source=r'{Path}:/video')

And use LWLibavSource for the reference crowd_run_1080p50.y4m clip.

In my tests at least, using LibavSMASHSource for the test clip produces exactly the same metric scores as LWLibavSource. But YMMV.

Otherwise, try converting the crowd_run_1080p50.y4m clip to lossless x264 Intra, use that as the source for your x265 encode, and run the metric tests with LibavSMASHSource as the import filter for both files.

ffmpeg -i {Path}:/crowd_run_1080p50.y4m -vcodec libx264 -preset slow -crf 0 -intra -r 50/1 -pix_fmt yuv420p -x264opts colorprim=bt709:transfer=bt709:colormatrix=bt709 {Path}:/crowd_run_1080p50_x264_lossless_intra.mp4

Incidentally, if you run the seek-test script in ffms2 mode #1 on the crowd_run_1080p50.y4m clip does it report any seeking errors? It shouldn't, being all intra, but if it does, it suggests there's something else going on.

[ChaosKing]

crowd_run_1080p50.y4m is fine with ffms2
ffms2 has only problems with the x265 mp4 encode. Remux it to mkv and all is fine.
lsmash is also fine.

p.s. you should also run more than 100 frames with seek-test with our x265 mp4 file, just to be sure. (edit the bat file or use cmd)

[WorBry]

Remux it to mkv and all is fine.


Actually, I still get seek errors (the 3 frames off) in the seek-test (ffms2 seekmode=1, all 500 frames) with an mkv remux of the x265.mp4 file. And likewise re-encoding crowd_run_2160p50.y4m to x265.mkv.

[ChaosKing]

hmm strange. I used mkvtoolnix v32 to remux the mp4. Just tested the mkv again -> no seeking issues.

[WorBry]

I tried several re-muxing methods - ffmpeg, mkvtoolnix, (TMPGEnc) SmartRenderer 5 - and the mkv files all give seek errors with seek-test and ffms2 (searchmode=1).

[ChaosKing]

I'm out of ideas then. The only factor seems to be the cpu then. AMD FX-6300 vs Ryzen 1700

I used this ffms2 version https://forum.doom9.org/showthread.php?t=176198. And as far as I can see you too...

[Iron_Mike]

I'm out of ideas then. The only factor seems to be the cpu then. AMD FX-6300 vs Ryzen 1700

I used this ffms2 version https://forum.doom9.org/showthread.php?t=176198. And as far as I can see you too...

that's the version I used as well in all my tests.

seekmode=0 and Lsmash give consistent results.

seekmode=1 on a file that was NOT encoded w/ keyframes setting also works just fine.

so, as I stated before:

it's the keyframes intval in the encoded file that throws off ffms2, which evidently starts at frame 100, which is where the first keyframe is (if you used my keyframe setting: keyint=100:min-keyint=100:rc-lookahead=100)


man, it would be desperately needed to be able to easily output the versions of the plugs used via VS.. there may be a difference in sw packages, which would explain why WorBry gets different results

being able to create the exact same dev environments for all peeps involved is key, and with a plethora of packages and setup/config options here, it'll be hard to figure this one out...

[Iron_Mike]

btw, I wanna add since I saw CK mention this a few times:

my Zopti logs - whether there are seeking issues or not via ffms - are always in correct frame order (at least at quick glance) - see the two attached example log files above in the other post (https://forum.doom9.org/showpost.php?p=1869294&postcount=161)...

this machine here (notebook): Win 8.1 x64, Intel i7-4810, 24GB RAM

I get the VS notice "script ran out of memory, consider increasing cache size"... but not sure if this can be a factor since the notice always comes, even when there are no seeking errors

[poisondeathray]

"100" might not be a "real" IDR keyframe , since you have open GOP enabled (default)

You guys should upload your files, because --frame-threads will be different on a 6 core , vs. 4C/8T , vs.8C/16T . That's why Worbry's values are consistently slightly higher. By default he will only have 2 Frame threads from 6 cores; 8T will have 3, 16 will have 5 . The files are slightly different. Does that affect what you are seeing in terms of seeking here ( besides the actual values)

The big errors are clearly frame mismatches. But the 10th ,12 etc.. tiny decimal place errors are something else Iron_Mike report that with seekmode=0 without keyint too - so that's another separate issue that shouldn't happen and is still worrisome

[ChaosKing]

btw, I wanna add since I saw CK mention this a few times:

my Zopti logs - whether there are seeking issues or not via ffms - are always in correct frame order (at least at quick glance) - see the two attached example log files above in the other post (https://forum.doom9.org/showpost.php?p=1869294&postcount=161)...

this machine here (notebook): Win 8.1 x64, Intel i7-4810, 24GB RAM



Because the log is now always written in correct order. https://github.com/theChaosCoder/zoptilib/commit/d99ff1e72d913a89dc99ee05c45b60f39e2148fa


I get the VS notice "script ran out of memory, consider increasing cache size"... but not sure if this can be a factor since the notice always comes, even when there are no seeking errors
This is normal. VS uses a percentage of the total allowed memory. But I don't know what the default % is. You can set it higher with http://www.vapoursynth.com/doc/pythonreference.html?highlight=core#Core.max_cache_size

[Iron_Mike]

"100" might not be a "real" IDR keyframe , since you have open GOP enabled (default)

if the encode was done via a specified key intval setting, then I verify afterwards via ffprobe that all IDR keyframes are in the correct position - and they are on my encodes, so @ frame 100 is a real IDR keyframe (unless these are labeled as IDR keyframes but are not ?)


The big errors are clearly frame mismatches. But the 10th ,12 etc.. tiny decimal place errors are something else Iron_Mike report that with seekmode=0 without keyint too - so that's another separate issue that shouldn't happen and is still worrisome

just ran another 10x tests w/ seekmode=0 on a keyframe encoded file via Zopti 1.0.8 - GMSD/SSIM


stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836523
stop 85.74332191455660112655579041 428.0741357421874995203836524
stop 85.74332191455660112655579041 428.0741357421874995203836523


edit: and another 10x tests w/ seekmode=0 on a non-keyframe encoded file (different encoding settings than file above, results will differ) via Zopti 1.0.8 - GMSD/SSIM


stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585
stop 86.95672484416033712339988421 425.5289550781250008659739585

[WorBry]

"100" might not be a "real" IDR keyframe , since you have open GOP enabled (default)


Elecard StreamEye identifies frame #100 (display sequence, #97 stream sequence) as a 'Key Frame', which according to their definition is an IDR frame:

"...key frames (also referred to as IDR frames; all the other frames positioned in between IDR frames cannot make reference or be dependent to/on the frames outside of this interval)."

http://i.imgur.com/szAS3Rsm.png (https://imgur.com/szAS3Rs)

As for the x265 CRF28.mp4 file encoded at default settings, the first key frame appears at frame #250 in the display sequence, as expected.

[poisondeathray]

What version of streameye ? Note that earlier versions misidentified HEVC IDR frames . It might be corrected in newer versions, not sure.

FFmpeg/FFProbe do not distinguish between them either - they mark them all as "key"

A freebie that correctly identifies them is hevcesbrowser
https://github.com/virinext/hevcesbrowser
https://www.codeproject.com/Tips/896030/The-Structure-of-HEVC-Video


You can check by encoding with --no-open-gop and you will see it's working . (Or not working in the case of FFprobe) .

x265 almost always places "i" frames by default . Very few frames will be IDR unless it's a scenechange or completely different content

Note there are 2 types of IDR frames in HEVC . The true strict type is an IDR_N_LP or "no leading picture"

Does this affect seeking ? Possibly. Open gop's can be problematic in some situation

[WorBry]

You guys should upload your files

No problem:

crowd_run_1080p50_y4m_x265_CRF28 (https://drive.google.com/file/d/1H4cpcE2Q70z-g0SN6FMK88iYDVK4k0Fl/view?usp=sharing)

Zipped folder contains two x265 (CRF28) files, encoded without and with custom key interval (100), as per:


(1) encode w/o keyint, min-keyint, rc-lookahead parameters:

ffmpeg command
<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"


(3) encode w/ keyint, min-keyint but w/o rc-lookahead:

--> wanted to see if it is specifically the rc-lookahead setting that causes the issues w/ seekmode=1

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

[Iron_Mike]

What version of streameye ? Note that earlier versions misidentified HEVC IDR frames . It might be corrected in newer versions, not sure.

FFmpeg/FFProbe do not distinguish between them either - they mark them all as "key"

A freebie that correctly identifies them is hevcesbrowser
https://github.com/virinext/hevcesbrowser
https://www.codeproject.com/Tips/896030/The-Structure-of-HEVC-Video


You can check by encoding with --no-open-gop and you will see it's working . (Or not working in the case of FFprobe) .

x265 almost always places "i" frames by default . Very few frames will be IDR unless it's a scenechange or completely different content

Note there are 2 types of IDR frames in HEVC . The true strict type is an IDR_N_LP or "no leading picture"

Does this affect seeking ? Possibly. Open gop's can be problematic in some situation


x265 encoded with keyframes setting (open gop)

ffprobe positions of all found IDR keyframes: 0, 100, 200, 300, 400 (as expected)
HEVC browser GUI, I Slices found: 5 (1x IDR slice, 4x I slice)


x265 encoded with keyframes setting (using param: no-open-gop)

<ffmpeg> -y -i "/path/tocrowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-x265-params "keyint=100:min-keyint=100:open-gop=0:rc-lookahead=100:colorprim=1:transfer=1:colormatrix=1"
-r 50/1 "/path/to/encode_with_key_frames_closed_gop.mp4"

ffprobe positions of all found IDR keyframes: 0, 100, 200, 300, 400 (as expected)
HEVC browser GUI, I Slices found: 5 (5x IDR slice)

x265 encoded without keyframes setting

ffprobe positions of all found IDR keyframes: 0, 250 (as expected)
HEVC browser GUI, I Slices found: 2 (1x IDR slice, 1x I slice)


re HEVC browser: I assume that an "IDR Slice" is a real IDR keyframe and every other "I slice" is as mere I frame...

[WorBry]

What version of streameye ? Note that earlier versions misidentified HEVC IDR frames . It might be corrected in newer versions, not sure.

StreamEye 4 (demo). Actually, their definition of key frames is given in the context of the leading key frame in a sequence, which is an IDR frame, so it's a bit confusing whether it applies to I-frames designated as 'key frames' downstream. The full quote:

".. Decoding of the whole video sequence, including scenarios at positioning, starts with key frames (also referred to as IDR frames; all the other frames positioned in between IDR frames cannot make reference or be dependent to/on the frames outside of this interval). For instance, video startsplaying only after receiving a key frame at network broadcasting."


A freebie that correctly identifies them is hevcesbrowser
https://github.com/virinext/hevcesbrowser
https://www.codeproject.com/Tips/896030/The-Structure-of-HEVC-Video


Thanks, I'll have a look at that.

[poisondeathray]

You have to use ES (elementary stream) with hevcesbrowser

[Iron_Mike]

and here are 3 encodes from me: https://drive.google.com/file/d/1-Jl8iVYc-PI6PLwob2vwbu4dY-h7Isun/view?usp=sharing

[Iron_Mike]

You have to use ES (elementary stream) with hevcesbrowser

I only tested the GUI, how is this done ?

or are you using the console ?

edit: ah. gotta convert... more PAIN ;-)

[poisondeathray]

ES (elementary stream) - just demux it from the container . Open the "naked" :) hevc stream with the GUI

e.g

ffmpeg -i input.mp4 -c:v copy output.hevc

[poisondeathray]

@Iron_Mike - your "closed" gop mp4 isn't "closed" for some reason . The command wasn't passed. You can verify with mediainfo (view=>text)

[Iron_Mike]

ES (elementary stream) - just demux it from the container . Open the "naked" :) hevc stream with the GUI

e.g

ffmpeg -i input.mp4 -c:v copy output.hevc

ok, edited my original post, see here (https://forum.doom9.org/showpost.php?p=1869391&postcount=177)

num keyframes match what ff probe reports...

how can I determine the frame number of the HEVC reported I slices ?

[Iron_Mike]

and... 10x GMSD/SSIM metrics test on x265 closed-gop encode - Zopti 1.0.8 w/ seekmode=1:


stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194457 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309
stop 85.81188288087124935166194456 427.7750750506365728931257309



since these are all fine, seekmode=1 trips up on x265 open-gop encodes with custom keyframes interval...

[WorBry]

and here are 3 encodes from me: https://drive.google.com/file/d/11IzqeEUl91Bw4XxlN6xIwQcsd6XzgG4Z/view?usp=sharing

I assume the script for the crowdrun_x265_keyframes-100_closed_gop.mp4 encode was:


x265 encoded with keyframes setting (using param: no-open-gop)

<ffmpeg> -y -i "/path/tocrowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-x265-params "keyint=100:min-keyint=100:no-open-gop:rc-lookahead=100:colorprim=1:transfer=1:colormatrix=1"
-r 50/1 "/path/to/encode_with_key_frames_closed_gop.mp4"


But what about the other two - crowdrun_x265_keyframes-100_open_gop.mp4 and crowdrun_x265_keyframes-none.mp4 ?

Were they also encoded with -preset fast and rc-lookahead=100 ? Please post the scripts.

[Iron_Mike]

@Iron_Mike - your "closed" gop mp4 isn't "closed" for some reason . The command wasn't passed. You can verify with mediainfo (view=>text)

hah... all for nothing ;-)

re-doing the tests...

[Iron_Mike]

alright...

> updated correct HEVC browser results, see here (https://forum.doom9.org/showpost.php?p=1869391&postcount=177)

> uploaded a new .zip w/ the three encodes, see here (https://forum.doom9.org/showpost.php?p=1869394&postcount=180)

> updated GMSD/SSIM results for closed-gop encode, see here (https://forum.doom9.org/showpost.php?p=1869401&postcount=185)


seems like @poisondeathray has been right all along that the default open-gop flag on x265 could possibly trip up seekmode=1 when custom keyframe intervals are set... interesting that the default keyframe intval does NOT trip up seekmode=1 on x265...


I assume the script for the crowdrun_x265_keyframes-100_closed_gop.mp4 encode was:
But what about the other two - crowdrun_x265_keyframes-100_open_gop.mp4 and crowdrun_x265_keyframes-none.mp4 ?

Were they also encoded with -preset fast and rc-lookahead=100 ? Please post the scripts.

(1) keyframes none


<ffmpeg> -y -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-r 50/1 "/path/to/encode_keyframes_none.mp4"

(2) custom keyframes intval 100 (2 secs) - default open-gop


<ffmpeg> -y -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-x265-params "keyint=100:min-keyint=100:rc-lookahead=100:colorprim=1:transfer=1:colormatrix=1"
-r 50/1 "/path/to/encode_with_keyframes_open_gop.mp4"

(3) custom keyframes intval 100 (2 secs) - open-gop=0 (closed gop)


<ffmpeg> -y -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -preset fast -crf 28 -pix_fmt yuv420p
-x265-params "keyint=100:min-keyint=100:open-gop=0:rc-lookahead=100:colorprim=1:transfer=1:colormatrix=1"
-r 50/1 "/path/to/encode_with_keyframes_closed_gop.mp4"

[WorBry]

OK, thanks.

Here are the SSIM:GMSD (Downsample=False) results I get with your files using ffms2 (Wolfberry) in default seekmode=1. I created fresh ffindex files for each run.

crowdrun_x265_keyframes-none.mp4

stop 425.5289660493827174070347034 86.95672484416033712339988421
stop 425.5289660493827174070347034 86.95672484416033712339988421
stop 425.5289660493827174070347034 86.95672484416033712339988421
stop 425.5289660493827174070347034 86.95672484416033712339988421
stop 425.5289660493827174070347034 86.95672484416033712339988421

crowdrun_x265_keyframes-100_open_gop.mp4

428.0741578052662025255514777 85.74332191455660112655579041
428.0741578052662025255514777 85.74332191455660112655579041
428.0741578052662025255514777 85.74332191455660112655579041
428.0741578052662025255514777 85.74332191455660112655579041
428.0741578052662025255514777 85.74332191455660112655579041

crowdrun_x265_keyframes-100_closed_gop.mp4

425.8651685474537029740815799 86.64507809503377200321949660
425.8651685474537029740815799 86.64507809503377200321949660
425.8651685474537029740815799 86.64507809503377200321949660
425.8651685474537029740815799 86.64507809503377200321949660
425.8651685474537029740815799 86.64507809503377200321949660

Edit: Oh, I see you uploaded a new set of encodes while I was testing. Ah well.
Edit: But judging from the file sizes (in bytes) it looks like it's just the crowdrun_x265_keyframes-100_closed_gop.mp4 that's different - so I'll only test that one.
Edit:
crowdrun_x265_keyframes-100_closed_gop.mp4 Updated Version

stop 427.7751079041280849457606904 85.81188288087124935166194456
stop 427.7751079041280849457606904 85.81188288087124935166194456
stop 427.7751079041280849457606904 85.81188288087124935166194456
stop 427.7751079041280849457606904 85.81188288087124935166194456
stop 427.7751079041280849457606904 85.81188288087124935166194456

Do I really need to test with LWLibavSource.

[Iron_Mike]

Edit: But judging from the file sizes (in bytes) it looks like it's just the crowdrun_x265_keyframes-100_closed_gop.mp4 that's different - so I'll only test that one.

only the closed gop file has changed, the other two are the same.

[poisondeathray]

Also, I would expect WorBry's frame-threads to be 2 for a 6C/6T . But mediainfo says 6 . Were those encoded on a different computer ?

[WorBry]

only the closed gop file has changed, the other two are the same.

Results added to my last post.

[WorBry]

Also, I would expect WorBry's frame-threads to be 2 for a 6C/6T . But mediainfo says 6 . Were those encoded on a different computer ?

No, same PC. But looking back at the scripts I did actually specify -threads 6. So to be clear:

crowd_run_1080p50_y4m_x265_CRF28.mp4:

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -threads 6 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

crowd_run_1080p50_y4m_x265_CRF28_KeyInt100.mp4:

<ffmpeg path> -i "/path/to/crowd_run_1080p50.y4m" -c:v libx265 -threads 6 -preset slow -crf 28 -pix_fmt yuv420p -x265-params "keyint=100:min-keyint=100:colorprim=1:transfer=1:colormatrix=1" -r 50/1 "/path/to/encode.mp4"

[Iron_Mike]

Results added to my last post.

can you post your .vpy script used in these tests ?

[poisondeathray]

@WorBry - for x265 it's --frame-threads . I guess ffmpeg -threads get passed as libx265 frame threads. A value of 6 would correspond to a 32 core machine . You can see on the chart
https://x265.readthedocs.io/en/latest/threading.html

The frame thread effect is smaller than with x264, but it's the reason why your values were slightly higher, because it should have been 2 with a 6C/6T machine. But if you specified -threads 6 , and that got passed to libx265 as frame-threads, then I would have expected the scores to be lower... You can check with mediainfo (view=>text) to see what parameters were actually used or passed





RE: frame number - unfortunately that utility doesn't keep track of display frame number across a GOP, but within a GOP (a true GOP) , POC (pic order count) pic_order_cnt_lsb should be the display order framenumber within that GOP . That's also one of the ways you can tell a true IDR delimiter , an IDR_N_LP frame. The pic count will be "zero" .

ffmpeg/ffprobe (and thus ffms2) had these sorts of problems with h264/AVC too. hevc is 10x worse because of the different subtypes of NAL units and picture types . I think if they got all that identifying and proper decoding sorted out, then "normal" mode for ffms2 should work (but it seems to work with WorBry ok :) eitherway ) . Until then seekmode=0 (and people usually use threads=1 along with it) - that' s just a lot of experience with ffms2 from avisynth days. There is no reason why vpy would be different. I have crowdrun y4m on a HDD somewhere. I'll test it too for a 3rd set of data later probably tomorrow





interesting that the default keyframe intval does NOT trip up seekmode=1 on x265...


Did you upload that one ? Default would have been 250

The custom keyframe version with open gop is introducing sublayer leading pictures . That might have to do with it, causing ffms2 /ffmpeg/ffprobe to mix things up

[Iron_Mike]

Did you upload that one ?
The custom keyframe version with open gop is introducing sublayer leading pictures . That might have to do with it, causing ffms2 /ffmpeg/ffprobe to mix things up

yes, there were 3 encoded files in the .zip (I've updated the post with a new zip a bit ago, replacing the closed_gop encode, since it was faulty, as you discovered)

(1) crowdrun_x265_keyframes-100_closed_gop.mp4 - custom keyframes intval of 100 (2 secs @ 50fps), with open-gop=0

(2) crowdrun_x265_keyframes-100_open_gop.mp4 - custom keyframes intval of 100 (2 secs @ 50fps), with default open-gop=1

(3) crowdrun_x265_keyframes-none.mp4 - no custom keyframes set, with default open-gop=1, which will result in 2 keyframes

file (3), although open-gop, does NOT trip up ffms2 w/ seekmode=1


edit:

another thing I just saw via HEVC browser...

re (1): the encode places all 5x IDR keyframes @ 100 frames interval (as specified)...

re (2) the encode places the 4x simple I frames (following the only real IDR keyframe) in incorrect position, they are a few frames off

re (3) the encode places the 1x simple I frame (following the only real IDR keyframe) in correct position


so, the ffmpeg encode of (2) is bad...

so how come this does not happen to WorBry using the same ffmpeg version and the same command ?

[poisondeathray]

(3) crowdrun_x265_keyframes-none.mp4 - no custom keyframes set, with default open-gop=1, which will result in 2 keyframes

file (3), although open-gop, does NOT trip up ffms2 w/ seekmode=1

Ok, sorry I misinterpreted that ; I thought "none" meant "infinite" . basically 1 keyframe for any stream. But this is just "default" setting

[poisondeathray]

What about threading on the vpy side ? How is that handled ? I know you can set core = vs.get_core(threads = something) , but what is the "auto" value ? And does this affect script execution or frame requests in that script?

[WorBry]

@WorBry - for x265 it's --frame-threads . I guess ffmpeg -threads get passed as libx265 frame threads. A value of 6 would correspond to a 32 core machine . You can see on the chart
https://x265.readthedocs.io/en/latest/threading.html


It was really oversight on my part :o I have my commands on a text file - copied another script for the file paths and neglected to delete the -threads 6.

I have crowdrun y4m on a HDD somewhere. I'll test it too for a 3rd set of data later probably tomorrow


Otherwise, it's here:

https://media.xiph.org/video/derf/

[WorBry]

can you post your .vpy script used in these tests ?

Well it's nothing special:

import vapoursynth as vs
import muvsfunc as muv
from zoptilib import Zopti
core = vs.get_core()
orig = core.ffms2.Source(source=r'{Path}:/crowd_run_1080p50.y4m')
zopti = Zopti('{Path}:/SSIM_GMSD_NDS_crowdrun_x265_keyframes-100_closed_gop_v2_#5.txt', metrics=['ssim', 'gmsd'])
zopti.addParams('ssim', dict(downsample=False))
zopti.addParams('gmsd', dict(downsample=False))
alternate = core.ffms2.Source(source=r'{Path}:/crowdrun_x265_keyframes-100_closed_gop_v2.mp4')
zopti.run(orig, alternate)