Filter suggestion for avisynth/VirtualDub2 of source with "fata morgana"

[stormy1777]

Got a sample clip:

https://anonfile.com/C7Ffyar1n7/Filter_Movement_mp4

of landscape in "movement" that appears from hot air I think.. sort of "Fata Morgana".. (above is not the source, it is encoded in x264 8bit, which is also my target encode)

would love to get some ideas on what kind of builtin filter (VirtualDub2), or avisynth script can be used to "process" this and reduce file size a bit yet maintain quality.

Thanks for any possible tips.

Source material is:

Format : MPEG-4
Format profile : Base Media / Version 2
Codec ID : mp42 (mp42/avc1/CAEP)
File size : 2.34 GiB
Duration : 5 min 40 s
Overall bit rate : 59.0 Mb/s

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L5.1
Format settings : CABAC / 2 Ref Frames
Format settings, CABAC : Yes
Format settings, RefFrames : 2 frames
Format settings, GOP : M=3, N=24
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 5 min 40 s
Bit rate : 58.9 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 50.000 FPS
Standard : Component
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.568
Stream size : 2.33 GiB (100%)
Language : English
Color range : Full
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
Codec configuration box : avcC

[StainlessS]

Getting warning about a deceptive site from your link, I aint gonna download it.

[Groucho2004]

Getting warning about a deceptive site from your link, I aint gonna download it.I just dl'd it, no problem, no complaints.

[StainlessS]

Warning from fireFox 66.xx, Settings : Security : Deceptive Content and Dangerous Software Protection
(via some Google service about deceptive content). Might just have been because of the AnonFile.com name.

Although I was a little sceptical myself about a 1920x1080@50.0FPS of duration 5:xx minutes (which is what source file seemed to be) and with output file size of about 3.5 MB.

EDIT: Firefox "Learn About" page on the Deceptive content stuff:- https://support.mozilla.org/en-US/kb/how-does-phishing-and-malware-protection-work?as=u&utm_source=inproduct

[stormy1777]

*Stainless*, original source is:

File size : 2.34 GiB

I clipped just a short segment and encoded it down to 3.5MB, just to give you an idea of the "movement" or "flickers" that appear in the sky/wires..

any idea on which filter/avisynth script would handle it best?

[StainlessS]

I went back to have another go, downloaded the named file (which turned out to be an html file, could not find real file name embeded in the html),
went back again for yet another go, and the constant refreshing of page made me shy off from download.
I guess that I'm just not as courageous as the brave yet grouchy one.

https://i.postimg.cc/2L1bV8gL/Deception.png (https://postimg.cc/2L1bV8gL)

Stormy, suggest that you use another file host in future, that one is just a little too scarey.

[I tend to get alarmed whenever anything unusual happens, eg when you try leave site and it will not let you until you
click "Are you sure ?", "Leave Page" button, I tend to pull plug on connection and force kill the browser just to avoid pressing the button that it seems desperate
for me to press. Paranoia rules OK].

[manolito]

I am under WinXP and using good old Opera 12 with a custom hosts file from winhelp2002.mvps.org/hosts.htm.

The only annoyance with this site is that after the first click on the file I get redirected (opening a new tab) to some advertising site. Clicking on the file for a second time downloads it without any issues. So I believe this is a little too much paranoia... :devil:

[mariush]

I made a copy of the video here, for those reluctant: http://savedonthe.net/download/2427/Filter_Movement.html

[stormy1777]

Thanks Mariush :)

so sorry, I run no-script (firefox addon), and so all these noisy things do not really appear, anyways, I can also upload the original SOURCE, not the encoded, but it's pretty much the same video, just much much larger.. the question is there any "encoding" scheme that can "cope" with such almost "watery" or "vapory" movement in the sky :)
Thanks...

[shekh]

Just saw Neat Video post and they claim to deal with "heat haze". https://blog.neatvideo.com/post/nv5-overview
No idea if this is available from plugin.

[StainlessS]

I overcame my cowardice and bravely downloaded, thanx Mariush.

So basically 'Heat Haze'. [I'de never heard the term Fata Morgana, looked Heat Haze up and it got me to Wikipedia Mirage:- https://en.wikipedia.org/wiki/Mirage].
I would not have any idea how to simulate heat haze.

Regarding the wikipedia Mirage link, Fata Morgana paragraph, "atmospheric duct" hilited popup, I think I once experienced that myself.
Back in 1981, I was stayng at a guest house in Bedfordshire countryside, about 5 of us were watching an Italian movie on analogue TV.
Due to weird atmosperic conditions, we could not pick up UK TV, only a single Italian TV station (maybe due to 'atmospheric ducting' which
probably acts similar to microwave WaveGuide used in Radar and microwave communications). the Italian movie of course had no subtitles
(as it was broadcast for Italians, not Brits), so we asked the Swiss girl with us to translate (she spoke Swiss German, Italian, French and English),
and we all sat there transfixed as she translated for a good 10 minutes, at which point she realised that nobody was watching the movie,
we were all fixed upon her. I said something like, "Enough of the French, how about giving it to us in English".
Some additional quite strange linguistic ducting had occurred.

EDIT:
Maybe some kind of +/- random number generation per pixel (vertical I think), and then run some kind of 'centered moving average' (to keep results
coherent), would also need some kind of 'centered moving average' horizontally and temporally too.
When all numbers crunched, then get pixel from Y coord +/- random sourced result.
Not really sure, but something like above (maybe).

EDIT from later post: Oh crap, I thought you wanted to synthesize Heat Haze, my bad.

[stormy1777]

Oh, wait a sec.. Is that "shekh"? the developer of VD2 telling us to purchase a software package to encode.. hmm.. should we be worried ?? :) :)
Thanks for the link, it mentioned "in-camera noise reduction", not sure I can find anything on that, it really appears to move, even to naked eye, so not sure camera is at fault here..

The term "Heat haze" is exactly right.. hopefully someone has some experience with encoding such content; almost looks like water/movement fuzzy shifting :)

Stainless, I think you posted sober this time around (at least BEFORE the EDIT), nice story, and with a happy "ending", did anyone end up marrying the girl :) :) :)

[StainlessS]

did anyone end up marrying the girl
Alas, yep, but not any of us watching TV. [I did keep in touch for a decade or so, and visited her in CH several times, and she re-visited here too].

I think I may try play with daft edit idea, but might take some time before I get around to it.

Tracking particals in meteorology, or as water goes over a waterfall (or even out of a tap/faucet) is an incredibly intractable problem, this is no easier probably (hence daft random number thing).

EDIT: Oh crap, I thought you wanted to synthesize Heat Haze, my bad.

[Groucho2004]

I wouldn't do anything apart from some light de-noising. Just tried with "KNLMeansCL(d = 1, a = 2, h = 2.0, s = 2)" and the bitrate went from 5500 to 3000 @CRF18. At 1080p, this is already a very low bitrate.

If you want more comprehensive advice, upload a piece cut directly from the source.

[shekh]

Oh, wait a sec.. Is that "shekh"? the developer of VD2 telling us to purchase a software package to encode.. hmm.. should we be worried ?? :) :)


It was funny to see the term "heat haze" in 2 unrelated places in one day, I thought it won't hurt to link :)

[stormy1777]

Groucho,

Sorry, wasn't thinking, naively thought answer is easy :)

here is part of original clip: https://anonfile.com/G30bHardn8/Heat_Haze_Original_mp4

roughly 46MB, not too bad... The idea is that the reencode would not be apparent to human eye, or add artifacts that are not in the original already...

Stainless, here's your second chance :) :) no worries, your company is enough sometimes :)

Stormy.

[stormy1777]

It was funny to see the term "heat haze" in 2 unrelated places in one day, I thought it won't hurt to link :)

some say, there is a REASON (for everything), and that we're all really "connected" somehow.. Matrix??

yeah, certainly funny chance to see such a rare phrase appear in two places :) :)

[Groucho2004]

The idea is that the reencode would not be apparent to human eye, or add artifacts that are not in the original already...
The original is 59 Mbps. I encoded your chunk with x264 (CRF 16, tune film) and the resulting bitrate is 9 Mbps. That's 15%. I you use some light de-noising you can reduce it even further.
At CRF 16 I highly doubt that you will see a difference between the original and the re-encoded file.

At CRF 18, the encoded file has 5.5 Mbps (9.3% of the original).

So, how much reduction do you have in mind?

[stormy1777]

thanks so much, i went up to CRF17, but fear to go higher, yeah, 10% is great, was wondering which filter/syntax to use for 'denoising' for this particular clip given the "movement", did not find anything in the provided filters for VirtualDub2 (with noise in the name).. do u still recommend: "KNLMeansCL(d = 1, a = 2, h = 2.0, s = 2)" for this clip? I can give it a try on the full source and see...

[Groucho2004]

was wondering which filter/syntax to use for 'denoising' for this particular clip given the "movement", did not find anything in the provided filters for VirtualDub2 (with noise in the name).. do u still recommend: "KNLMeansCL(d = 1, a = 2, h = 2.0, s = 2)" for this clip? I can give it a try on the full source and see...Give KNLMeansCL a try, it's quite useful in many situations.

Another option:
function MCD(clip video, int "threshold")
{
last = video
threshold = default(threshold, 200)

bs = (width() > 960) ? 16 : 8
sc = MSuper(pel = 2, sharp = 1, hpad = 16, vpad = 16)
backward_vector = MAnalyse(sc, isb = true, delta = 1, blksize = bs, overlap = bs / 2, sadx264 = 4)
forward_vector = MAnalyse(sc, isb = false, delta = 1, blksize = bs, overlap = bs / 2, sadx264 = 4)
backward_vector = MRecalculate(sc, backward_vector, blksize = bs / 2, overlap = bs / 4, thSAD = 100)
forward_vector = MRecalculate(sc, forward_vector , blksize = bs / 2, overlap = bs / 4, thSAD = 100)
MDegrain1(sc, backward_vector, forward_vector, thSAD = threshold)

return last
}

This is basically a function that StainlessS posted some time ago, can't find it right now so I posted my modified version. Try 'threshold' 200 ~ 300.

Also, watch out for banding if you use a de-noiser.

I usually stick with 'less is more' and use filters (particularly de-noisers) as little as possible.

[StainlessS]

These are the two funcs that I like to use


Function MCDegrain(clip c, int "frames")
{ # By Didee, http://forum.doom9.org/showthread.php?p=1508289#post1508289

frames = default(frames, 2)
bs = (c.width>960) ? 16 : 8
super = c.MSuper(pel=2, sharp=1)
backward_vec3 = MAnalyse(super, isb = true, delta = 3, blksize=bs, overlap=bs/2)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=bs, overlap=bs/2)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=bs, overlap=bs/2)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=bs, overlap=bs/2)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=bs, overlap=bs/2)
forward_vec3 = MAnalyse(super, isb = false, delta = 3, blksize=bs, overlap=bs/2)
(frames<=0) ? c :\
(frames==1) ? c.MDegrain1(super, backward_vec1,forward_vec1,thSAD=400) :\
(frames==2) ? c.MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) :\
c.MDegrain3(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400)
return(last)
}

Function MCDegrainSharp(clip c, int "frames", float "bblur", float "csharp", bool "bsrch",bool "Precise") {
# From:- http://forum.doom9.org/showthread.php?p=1737045#post1737045
# Based on MCDegrain By Didee, http://forum.doom9.org/showthread.php?t=161594
# Also based on DiDee observations in this thread: http://forum.doom9.org/showthread.php?t=161580
# "Denoise with MDegrainX, do slight sharpening where motionmatch is good, do slight blurring where motionmatch is bad"
# In areas where MAnalyse cannot find good matches, the blur() will be dominant.
# In areas where good matches are found, the sharpen()'ed pixels will overweight the blur()'ed pixels
# when the pixel averaging is performed.
#
# Mod by StainlessS to add Precise, 3 Sept 2015.
frames = default(frames, 2)
bblur = default(bblur, 0.6)
csharp = default(csharp, 0.6)
bsrch = default(bsrch, true)
Precise=Default(Precise,False) # Use MRecalculate
bs = (c.width>960) ? 16 : 8
c2 = c.blur(bblur)
super = bsrch ? c2.MSuper(pel=2, sharp=1) : c.MSuper(pel=2, sharp=1)
super_rend = c.sharpen(csharp).MSuper(pel=2, sharp=1,levels=1) # Only 1 Level required for sharpened Super (not MAnalyse-ing)
backward_vec3 = MAnalyse(super, isb = true, delta = 3, blksize=bs, overlap=bs/2)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, blksize=bs, overlap=bs/2)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=bs, overlap=bs/2)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=bs, overlap=bs/2)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, blksize=bs, overlap=bs/2)
forward_vec3 = MAnalyse(super, isb = false, delta = 3, blksize=bs, overlap=bs/2)
# If Precise, then recalculate on Prefiltered (blurred) Super (NOT the sharpened render super)
backward_vec3 = (Precise) ? MRecalculate(super, backward_vec3, blksize=bs/2, overlap=bs/4,thSAD=100) : backward_vec3
backward_vec2 = (Precise) ? MRecalculate(super, backward_vec2, blksize=bs/2, overlap=bs/4,thSAD=100) : backward_vec2
backward_vec1 = (Precise) ? MRecalculate(super, backward_vec1, blksize=bs/2, overlap=bs/4,thSAD=100) : backward_vec1
forward_vec1 = (Precise) ? MRecalculate(super, forward_vec1 , blksize=bs/2, overlap=bs/4,thSAD=100) : forward_vec1
forward_vec2 = (Precise) ? MRecalculate(super, forward_vec2 , blksize=bs/2, overlap=bs/4,thSAD=100) : forward_vec2
forward_vec3 = (Precise) ? MRecalculate(super, forward_vec3 , blksize=bs/2, overlap=bs/4,thSAD=100) : forward_vec3
#
(frames<=0) ? c :\
(frames==1) ? c2.MDegrain1(super_rend, backward_vec1,forward_vec1,thSAD=400) :\
(frames==2) ? c2.MDegrain2(super_rend, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400) \
: c2.MDegrain3(super_rend, backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400)
return(last)
}

[stormy1777]

Took this zip: https://github.com/Khanattila/KNLMeansCL/releases/download/v1.1.1/KNLMeansCL-v1.1.1.zip

placed the files in VD2's plugin32/64, restarted VD2, added to script "KNLMeansCL(d = 1, a = 2, h = 2.0, s = 2)", however it fails with:

---------------------------
File open error
---------------------------
Avisynth open failure:
Script error: there is no function named "KNLMeansCL"
(M:\temp\190515-80D-300-2749_MVI.avs, line 3)
---------------------------
OK
---------------------------


searching online, not very clear, but this made that error go away:

LoadPlugin("c:\Programs\VirtualDub2\plugins32\KNLMeansCL.dll")


However script then failed with:

---------------------------
VirtualDub Error
---------------------------
Video compression error: An unknown error occurred (may be corrupt data). (error code -100)
---------------------------
OK
---------------------------


Tried reinstalling avisynth, rebooting (win 10), same errors.

Version shows:

AviSynth 2.60, build:Mar 31 2015 [ 16:38:54]

will try the other code and see.

[StainlessS]

LoadPlugin("c:\Programs\VirtualDub2\plugins32\KNLMeansCL.dll")

Obviously that is an Avisynth plugin, and should not be in virtualDub plugins32 (maybe it is the source of the error in VDub2).

Either put the dll in avisynth plugins (for auto load) OR use LoadPlugin (from some other directory than VDub plugins32).

[stormy1777]

Could not yet resolve the "error code -100", some sort of 'memory' malloc failure:

AVI: Opening file "Y:\test.avs"
[i] AVI: Avisynth detected. Extended error handling enabled. Beginning dub operation.
[i] Dub: Input (decompression) format is: YUV420.
[!] x264 [warning]: --psnr used with psy on: results will be invalid!
[!] x264 [warning]: --tune psnr should be used if attempting to benchmark psnr!
[i] x264 [debug]: using mv_range_thread = 40
[i] x264: using SAR=1/1
[!] x264 [warning]: MB rate (408000) > level limit (245760)
[i] x264: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[i] x264: profile High, level 4.1, 4:2:0, 8-bit
[i] Dub: Output (compression) format is: YUV420.
[E] x264 [error]: malloc of size 6955648 failed
[E] x264 [error]: x264_encoder_encode failed
[i] x264 [debug]: scene cut at 14 Icost:476577 Pcost:452142 ratio:0.0513 bias:0.0560 gop:14 (imb:6289 pmb:1499)
[i] x264 [debug]: scene cut at 17 Icost:542324 Pcost:521823 ratio:0.0378 bias:0.0680 gop:17 (imb:6206 pmb:1582)
[i] x264 [debug]: scene cut at 23 Icost:506481 Pcost:466510 ratio:0.0789 bias:0.0920 gop:23 (imb:5755 pmb:2033)
[E] Error: Video compression error: An unknown error occurred (may be corrupt data). (error code -100) Ending operation.

phy. memory on the win10/64bit laptop, 16gb, current 6gb, plenty of swap/page, that's not really making sense.. it's some sort of mis-configuration, just not sure what.. I'm running virtualdub2 32-bit , could not yet figure how to run all things in 64bit..

Took this code to a working PC, still the only way to use KNLMeansCL is to load ahead using LoadPlugin, this is a sample code:


LoadPlugin("c:\Programs\VirtualDub2\plugins32\KNLMeansCL.dll")
KNLMeansCL(d = 1, a = 2, h = 2.0, s = 2)
return FFVideoSource("y:\test.MP4")


however, this fails with:

---------------------------
File open error
---------------------------
Avisynth open failure:
Script error: Invalid arguments to function "KNLMeansCL"
(y:\test.avs, line 2)
---------------------------
OK
---------------------------


not sure what the issue is..

EDIT: Oh, wait, maybe the script needs to be like this:

LoadPlugin("c:\Programs\VirtualDub2\plugins32\KNLMeansCL.dll")
FFVideoSource("y:\test.MP4")
return KNLMeansCL(d = 1, a = 2, h = 2.0, s = 2)


this seems to be running, very very slow, wow, says 1fps.. that can take hours :) maybe will run faster on the newer laptop that gets the malloc error :)

[Groucho2004]

EDIT: Oh, wait, maybe the script needs to be like this:

LoadPlugin("c:\Programs\VirtualDub2\plugins32\KNLMeansCL.dll")
FFVideoSource("y:\test.MP4")
return KNLMeansCL(d = 1, a = 2, h = 2.0, s = 2)


this seems to be running, very very slow, wow, says 1fps.. that can take hours :)
Such a slow speed would indicate that the filter is running on the CPU instead of the GPU, unless you have a very slow graphics adapter. Check the system requirements for KNLMeansCL (https://github.com/Khanattila/KNLMeansCL/wiki/Requirements).

While your at it, read some Avisynth documentation, at least the basics (the topics under "New to AviSynth – Start Here" on avisynth.nl (http://avisynth.nl/index.php/Main_Page)).

Edit - Try also the MCD function I posted above.

[shekh]

16Gb is mostly irrelevant for 32-bit process
you can enable 4Gb using extra\auxsetup (enable LAA)
reduce memory used by avisynth
reduce memory used by x264 (using less lookahead etc)

[stormy1777]

Groucho2004, sorry, the mixing in syntax was an oversight/typo, got it right after made the post :) will try the MCD soon, the -100 error is what prevents making progress since newer PC can handle a lot better than older (which has no failures, and same VD 32-bit is running in both cases).

shekh, i don't think it's memory related, it's just reporting that, but really "run video analysis pass" work fine without any failure in:

1) Uncompressed
2) FFMPEG / x265
3) FFMPEG / x265 lossless
4) Xvid MPEG-4 codec

Only in the x264 8 or 10 bit, it is failing, as well as in the old x264vfw - H.264/MPEG-4 AVC codec, with:

x264vfw [error]: x264_encoder_encode failed


Above failures are with VD2's latest, unzipped in new directory, added nothing in plugins, and "script" has just *1* line:

FFVideoSource("y:\test.MP4")


Processing the test.MP4 file DIRECTLY in the same instance of the program, works fine in all codecs, including x264/8bits and old x264vfw. Is there any other "x264" setting or environmental aspect aside what's in the runtime directory that can impact?

it sounds like some setup, b/c the other PC has less memory, much older, and yet all these things work fine on it.

Stormy.

[Groucho2004]

the -100 error is what prevents making progress since newer PC can handle a lot better than older (which has no failures, and same VD 32-bit is running in both cases).Run "AVSMeter avsinfo" on that PC and check for errors. Also, run your script with AVSMeter.

[stormy1777]

FYI:

also, deleted this file each time, to rule out anything with that:

test.MP4.ffindex

no help :)

Groucho2004,

I'm honored, seems like you're the author of AVSMeter!! never heard of it :) :) sorry.

Not sure what is an error, some lines are in RED, but otherwise output is identical to working PC.


AVSMeter.exe avsinfo

AVSMeter 2.9.1 (x86) - Copyright (c) 2012-2019, Groucho2004

VersionString: AviSynth 2.60, build:Mar 31 2015 [16:38:54]
VersionNumber: 2.60
File / Product version: 2.6.0.6 / 2.6.0.6
Interface Version: 5
Multi-threading support: No
Avisynth.dll location: C:\WINDOWS\SysWOW64\avisynth.dll
Avisynth.dll time stamp: 2015-03-31, 06:40:58 (UTC)
PluginDir2_5 (HKLM, x86): C:\Program Files (x86)\AviSynth\plugins


[CPP 2.5 Plugins (32 Bit)]
C:\Program Files (x86)\AviSynth\plugins\AddGrainC.dll [1.7.1.0]
C:\Program Files (x86)\AviSynth\plugins\avstp.dll [1.0.3.0]
C:\Program Files (x86)\AviSynth\plugins\dfttest.dll [1.9.4.0]
C:\Program Files (x86)\AviSynth\plugins\dither.dll [2015-12-30]
C:\Program Files (x86)\AviSynth\plugins\LSMASHSource.dll [2017-02-24]

[CPP 2.6 Plugins (32 Bit)]
C:\Program Files (x86)\AviSynth\plugins\DePan.dll [2.13.1.3]
C:\Program Files (x86)\AviSynth\plugins\DePanEstimate.dll [2.10.0.2]
C:\Program Files (x86)\AviSynth\plugins\DirectShowSource.dll [2.6.0.3]
C:\Program Files (x86)\AviSynth\plugins\ffms2.dll [2016-12-29]
C:\Program Files (x86)\AviSynth\plugins\flash3kyuu_deband.dll [2018-05-02]
C:\Program Files (x86)\AviSynth\plugins\masktools2.dll [2.2.18.0]
C:\Program Files (x86)\AviSynth\plugins\mvtools2.dll [2.7.36.0]
C:\Program Files (x86)\AviSynth\plugins\RgTools.dll [0.97.0.0]
C:\Program Files (x86)\AviSynth\plugins\TCPDeliver.dll [2.6.0.7]

[Scripts (AVSI)]
C:\Program Files (x86)\AviSynth\plugins\colors_rgb.avsi [2015-03-30]
C:\Program Files (x86)\AviSynth\plugins\dither.avsi [2015-12-30]
C:\Program Files (x86)\AviSynth\plugins\mt_xxpand_multi.avsi [2010-09-11]
C:\Program Files (x86)\AviSynth\plugins\ShowDiff.avsi [2018-12-03]

[Uncategorized DLLs (32 Bit)]
C:\Program Files (x86)\AviSynth\plugins\libfftw3f-3.dll [2010-06-21]

[Uncategorized files]
C:\Program Files (x86)\AviSynth\plugins\addgrain.avs [2011-10-27]
C:\Program Files (x86)\AviSynth\plugins\ffms2.lib [2016-12-29]
C:\Program Files (x86)\AviSynth\plugins\ffmsindex.exe [2016-12-29]
C:\Program Files (x86)\AviSynth\plugins\version.avs [2018-12-02]


Then running the script directly:

AVSMeter.exe y:\test.avs

AVSMeter 2.9.1 (x86) - Copyright (c) 2012-2019, Groucho2004
AviSynth 2.60, build:Mar 31 2015 [16:38:54] (2.6.0.6)

Number of frames: 24908
Length (hh:mm:ss.ms): 00:08:18.160
Frame width: 1920
Frame height: 1080
Framerate: 50.000 (50/1)
Colorspace: i420

Frame (current | last): 495 | 24907
FPS (cur | min | max | avg): 1.852 | 1.717 | 5.439 | 2.083
Process memory usage: 305 MiB
Thread count: 13
CPU usage (current | average): 15.1% | 14.5%

Time (elapsed | estimated): 00:03:57.611 | 03:19:16.393

Press 'Esc' to cancel the process...


It keeps running, this is a BEAST, CLI for AVS!!! monster!!! love it...but wait, seems like it's not writing anything?, this is just a benchmark tool?, so not so good for my final goal :) to get an MP4 file :)

Need to figure out why Vdub thinks the system is misconfigured and dies, there must be something external to the vdub zip/directory that makes it fail, probably by wrongly trying to allocate more memory than needed..

EDIT: AVSMeter seems to only consume 1 cpu, whereas vdub appears to consume entire box (all cpus), maybe there's a way to tell it to run on more than one, but at this point it is not going to help more than it already did since it doesn't appear to WRITE anything :)

[stormy1777]

can't really test the MCD, it too fails with same error:

---------------------------
VirtualDub Error
---------------------------
Video compression error: An unknown error occurred (may be corrupt data). (error code -100)
---------------------------
OK
---------------------------


went to ctrl-P, configure x264, and LOADED defaults, confirmed OK, and still same failure.

Seems like something to do with avisynth/Vdub2 integration, although version() and ColorBars() functions ARE working OK:

Version() and: ColorBars are both fine..

[stormy1777]

Tried DirectShowSource( instead of FFVideoSource, same -100 failure...

[Groucho2004]

this is just a benchmark tool?Mainly, yes.

AVSMeter seems to only consume 1 cpu, whereas vdub appears to consume entire box (all cpus)Since you're running the script single-threaded, only a fraction of the CPU will be utilised. VDub also runs an encoder which is multi-threaded.

What GPU do you have on the box where you ran the benchmark?


Edit - I'm a bit puzzled that AVSMeter runs the script but VDub crashes. However, VDub also runs the encoder so - maybe a hardware problem? Dunno...

[stormy1777]

There is no GPU on this laptop, it is brand new, no h/w errors , and vdub works 100% fine without AVS scripting, it CAN encode using the same encoders without AVS, i.e. opening the file DIRECTLY from Vdub. I'm not doing any heavy lifting, really, these are 5-10 min clips, nothing huge at all..

[shekh]

This is somehow memory related but finding the reason is not trivial. AFAIK x264 with default settings and 1080 source should fit in 1GB.
What you can do:
start virtualdub and open your avs script, then run vmmap.exe and select virtualdub.exe process. Then use "File->Save as" to save mmp file and share it.

test2:
setup x264 to use ultrafast profile, run analyse pass and if it does not fail - repeat vmmap inspection while analysis is still running.

[stormy1777]

yeah, tried many avisynth scripts, all input calls, including: AviSource fail with very small clips, all scripts are just one liners, and encoder is the x264/8-bit.. xvid, and others compress fine, so it's some internal vdub issue that's somehow misconfigured on this windows 10.. if there anything to re-install x264 to fix it??

[shekh]

x264 is standalone, very hard to break.
link to vmmap: http://downloads.tomsguide.com/SysInternals-VMMap,0301-33273.html

My theory: there is little free usable memory (less than usual).

[stormy1777]

Thanks shekh for your time and interest.

I think it's some sort of mis-configuration on the host (not sure why) that manifests itself as an apparent "memory" issue, but really, it's not a "pure" memory issue, since not using AVS script I can encode massive files on the same laptop using the SAME vdub window, just introducing the AVS script causes this failure.

OK, trimmed a 3 sec clip, and that fails in all cases on analysis run with x264/8-bit (but does not on xvid), under all profiles EXCEPT Ultrafast, in Ultrafast, it never fails.

vmmap when analysis is running OK on the full file, with ultrafast profile (the only one that can analyze) is here:

https://anonfile.com/PesdRdrfn0/VirtualDub_zip

by mis-configuration, i mean, maybe one codec is not installed correctly, or another program that vdub2 is somehow influenced by it.. At the time of running, nothing major is open/running, just basic, browser and command prompt. no photoshop, or any memory hog like that :)

Stormy.

[stormy1777]

x264 is standalone, very hard to break.

In this case, it seems almost impossible NOT to break :) so long as AVS is in the mix :) When you say x264 is "standalone" is there an install that needs to take place or all the files under vdub folder are only running?

I'm 99% convinced that something else on this laptop is causing this standalone to fail when file is opened via AVS, opening the video file directly, OR, using different codec, OR using x264/ultrafast causes the error to go away :)

Stormy.

[shekh]

Please put the archive on some file sharing
x264.vdplugin is just dll with no fancy dependencies, no user space config files. Nothing to install.

[stormy1777]

Please put the archive on some file sharing
x264.vdplugin is just dll with no fancy dependencies, no user space config files. Nothing to install.

sorry, here it is: https://anonfile.com/PesdRdrfn0/VirtualDub_zip

well, then I'm very curious indeed to uncover what's going on, maybe another software/installed/configured is "polluting" the VM address space and then leaving almost no memory for Vdub2... this is win7 updated to win10 (64bit , latest updates from winupdates)

Stormy.

[stormy1777]

Quick update:

1) In an "act of desperation" :) unzipped vdub2 build 42711 (Oct 2018), noticed it processed "video analysis run" just fine in x264/with same AVS file, it seems to write the file too.. hmmm..

2) Ran the latest vdub, side-by-side and noticed it too can run the video analysis run if that is the very FIRST thing after launch (just confirm x264 is selected).

However, subsequent attempts, analysis or writing to disk, gets the -100 error.

These appear to behave the same, it appears that first operation runs OK, but subsequent, even identical write, fails with the -100.

not sure what that means, but at least I got some sort of a "workaround", so now the KNL is running, very slow 2fps, but that's fine, just for educational purposes :)

[Groucho2004]

now the KNL is running, very slow 2fps, but that's fine, just for educational purposes :)
KNLMeansCL needs a decent, dedicated graphics card in order to run at reasonable speeds. With your integrated GPU I would just forget about it and use the "classic" motion-compensated de-noising such as the MCD function (or the more versatile "MCDegrainSharp" StainlessS posted).

[shekh]

From mmp looks like there is too much heap usage. Maybe 500M more than should be. Who is using it? Not possible to know from one log, you need at least two logs to see the difference.

You can continue hunting:
start empty virtualdub.exe (no source files), connect vmmap and note heap size (also good idea to save log at each step)
open avs (avisource), refresh vmmap and note heap size...
run analyze pass (no compression) and note heap size...
open video file directly and run x264 analyze pass...

[Groucho2004]

From mmp looks like there is too much heap usage. Maybe 500M more than should be. Who is using it?Maybe mapped video memory from the integrated GPU? It would be interesting to see without KNLMeansCL.

[stormy1777]

shekh -
Thanks for the support and guidance, hoping this will pay off soon :) Definitely it's the Heap growing as you say!

All the mmap tests were done without KNL, just "return FFVideoSource(.." 1-liner.

ran several times, to be sure and this is roughly state of heap:

Launch 5MB
Loaded 78MB
Video Pass 1.3GB
Compression/write attempt FAILS drops to 650MB, but fails.


running a 'diff' between last and one before last shows thousands of new lines like these below.

Here are all 4 files: https://anonfile.com/g2I4Ubr0nd/Vdub2-heapgrowth_zip

It *seems* as if something on the system triggers x264 to allocate a lot of heap...


<Allocations/></Snapshot>
<Snapshot Timestamp="132025721965550445" PageTableSize="0">
<MemoryRegions>
<Region Address="0" Blocks="0" ShareableWS="0" SharedWS="0" Size="65536" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="65536" UseType="6" Type="Free" Details=""/>
<Region Address="65536" Blocks="1" ShareableWS="4096" SharedWS="4096" Size="65536" Commit="65536" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="4" Storage="262144" UseType="5" Type="Shareable" Details="">
<Region Address="65536" Blocks="0" ShareableWS="4096" SharedWS="4096" Size="65536" Commit="65536" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="4" Storage="262144" UseType="5" Type="Shareable" Details=""/></Region>
<Region Address="131072" Blocks="1" ShareableWS="4096" SharedWS="4096" Size="4096" Commit="4096" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="2" Storage="262144" UseType="5" Type="Shareable" Details="">
<Region Address="131072" Blocks="0" ShareableWS="4096" SharedWS="4096" Size="4096" Commit="4096" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="2" Storage="262144" UseType="5" Type="Shareable" Details=""/></Region>
<Region Address="135168" Blocks="0" ShareableWS="0" SharedWS="0" Size="61440" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="65536" UseType="10" Type="Unusable" Details=""/>
<Region Address="196608" Blocks="1" ShareableWS="0" SharedWS="0" Size="4096" Commit="4096" PrivateBytes="4096" PrivateWS="4096" Id="4294967295" Protection="4" Storage="131072" UseType="4" Type="Private Data" Details="">
<Region Address="196608" Blocks="0" ShareableWS="0" SharedWS="0" Size="4096" Commit="4096" PrivateBytes="4096" PrivateWS="4096" Id="4294967295" Protection="4" Storage="131072" UseType="4" Type="Private Data" Details=""/></Region>
<Region Address="200704" Blocks="0" ShareableWS="0" SharedWS="0" Size="61440" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="65536" UseType="10" Type="Unusable" Details=""/>
<Region Address="262144" Blocks="1" ShareableWS="102400" SharedWS="102400" Size="106496" Commit="106496" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="2" Storage="262144" UseType="5" Type="Shareable" Details="">
<Region Address="262144" Blocks="0" ShareableWS="102400" SharedWS="102400" Size="106496" Commit="106496" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="2" Storage="262144" UseType="5" Type="Shareable" Details=""/></Region>
<Region Address="368640" Blocks="0" ShareableWS="0" SharedWS="0" Size="24576" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="65536" UseType="10" Type="Unusable" Details=""/>
<Region Address="393216" Blocks="3" ShareableWS="0" SharedWS="0" Size="262144" Commit="49152" PrivateBytes="49152" PrivateWS="36864" Id="4294967295" Protection="260" Storage="131072" UseType="1" Type="Thread Stack" Details="">
<Region Address="393216" Blocks="0" ShareableWS="0" SharedWS="0" Size="212992" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="8192" UseType="1" Type="Thread Stack" Details=""/>
<Region Address="606208" Blocks="0" ShareableWS="0" SharedWS="0" Size="12288" Commit="12288" PrivateBytes="12288" PrivateWS="0" Id="4294967295" Protection="260" Storage="131072" UseType="1" Type="Thread Stack" Details=""/>
<Region Address="618496" Blocks="0" ShareableWS="0" SharedWS="0" Size="36864" Commit="36864" PrivateBytes="36864" PrivateWS="36864" Id="4294967295" Protection="4" Storage="131072" UseType="1" Type="Thread Stack" Details=""/></Region>
<Region Address="655360" Blocks="3" ShareableWS="0" SharedWS="0" Size="1048576" Commit="86016" PrivateBytes="86016" PrivateWS="77824" Id="7712" Protection="260" Storage="131072" UseType="1" Type="Thread Stack" Details="">
<Region Address="655360" Blocks="0" ShareableWS="0" SharedWS="0" Size="962560" Commit="0" PrivateBytes="0" PrivateWS="0" Id="7712" Protection="0" Storage="8192" UseType="1" Type="Thread Stack" Details=""/>
<Region Address="1617920" Blocks="0" ShareableWS="0" SharedWS="0" Size="8192" Commit="8192" PrivateBytes="8192" PrivateWS="0" Id="7712" Protection="260" Storage="131072" UseType="1" Type="Thread Stack" Details=""/>
<Region Address="1626112" Blocks="0" ShareableWS="0" SharedWS="0" Size="77824" Commit="77824" PrivateBytes="77824" PrivateWS="77824" Id="7712" Protection="4" Storage="131072" UseType="1" Type="Thread Stack" Details=""/></Region>
<Region Address="1703936" Blocks="1" ShareableWS="12288" SharedWS="12288" Size="16384" Commit="16384" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="2" Storage="262144" UseType="5" Type="Shareable" Details="">
<Region Address="1703936" Blocks="0" ShareableWS="12288" SharedWS="12288" Size="16384" Commit="16384" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="2" Storage="262144" UseType="5" Type="Shareable" Details=""/></Region>
<Region Address="1720320" Blocks="0" ShareableWS="0" SharedWS="0" Size="49152" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="65536" UseType="10" Type="Unusable" Details=""/>
<Region Address="1769472" Blocks="1" ShareableWS="8192" SharedWS="0" Size="8192" Commit="8192" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="2" Storage="262144" UseType="5" Type="Shareable" Details="">
<Region Address="1769472" Blocks="0" ShareableWS="8192" SharedWS="0" Size="8192" Commit="8192" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="2" Storage="262144" UseType="5" Type="Shareable" Details=""/></Region>
<Region Address="1777664" Blocks="0" ShareableWS="0" SharedWS="0" Size="57344" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="65536" UseType="10" Type="Unusable" Details=""/>
<Region Address="1835008" Blocks="1" ShareableWS="0" SharedWS="0" Size="8192" Commit="8192" PrivateBytes="8192" PrivateWS="8192" Id="4294967295" Protection="4" Storage="131072" UseType="4" Type="Private Data" Details="">
<Region Address="1835008" Blocks="0" ShareableWS="0" SharedWS="0" Size="8192" Commit="8192" PrivateBytes="8192" PrivateWS="8192" Id="4294967295" Protection="4" Storage="131072" UseType="4" Type="Private Data" Details=""/></Region>
<Region Address="1843200" Blocks="0" ShareableWS="0" SharedWS="0" Size="57344" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="65536" UseType="10" Type="Unusable" Details=""/>
<Region Address="1900544" Blocks="1" ShareableWS="0" SharedWS="0" Size="4096" Commit="4096" PrivateBytes="4096" PrivateWS="4096" Id="4294967295" Protection="32" Storage="131072" UseType="4" Type="Private Data" Details="">
<Region Address="1900544" Blocks="0" ShareableWS="0" SharedWS="0" Size="4096" Commit="4096" PrivateBytes="4096" PrivateWS="4096" Id="4294967295" Protection="32" Storage="131072" UseType="4" Type="Private Data" Details=""/></Region>
<Region Address="1904640" Blocks="0" ShareableWS="0" SharedWS="0" Size="61440" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="65536" UseType="10" Type="Unusable" Details=""/>
<Region Address="1966080" Blocks="1" ShareableWS="0" SharedWS="0" Size="49152" Commit="49152" PrivateBytes="49152" PrivateWS="49152" Id="4294967295" Protection="2" Storage="131072" UseType="4" Type="Private Data" Details="">
<Region Address="1966080" Blocks="0" ShareableWS="0" SharedWS="0" Size="49152" Commit="49152" PrivateBytes="49152" PrivateWS="49152" Id="4294967295" Protection="2" Storage="131072" UseType="4" Type="Private Data" Details=""/></Region>
<Region Address="2015232" Blocks="0" ShareableWS="0" SharedWS="0" Size="16384" Commit="0" PrivateBytes="0" PrivateWS="0" Id="4294967295" Protection="0" Storage="65536" UseType="10" Type="Unusable" Details=""/>



EDIT: Found that some operations cause heap to DROP, and re-trying several times, one out of 5 or so, it will go through with encode! so doing that now, MCD appears to reduce file size significantly, will wait for final output though..

[shekh]

"Video Pass 1.3GB" - is this analyze pass without compression (selected codec is "raw")? Looks bloated.

[stormy1777]

"Video Pass 1.3GB" - is this analyze pass without compression (selected codec is "raw")? Looks bloated.

sorry, mis-read your original post.. I did not do 'no compression' since that does not fail. the tests above were with x264/compression. if u look at files, there are 1000's of new heap segments, not sure who owns/creates them, but they are inside the vdub process, so it's running out of address space inside the process for some reason.

I can re-run again withOUT compression "Uncompressed RGB/YcBcr", again, that does NOT fail, even after running many times, also opening MP4 file DIRECTLY never fails, only when going through AVS (even one liner) along with x264 that it fails.

Tried xvid again, that does not fail either.
Stormy.

[shekh]

I can re-run again withOUT compression "Uncompressed RGB/YcBcr", again, that does NOT fail, even after running many times, also opening MP4 file DIRECTLY never fails, only when going through AVS (even one liner) along with x264 that it fails.

Tried xvid again, that does not fail either.
Stormy.

Yes it is important to see that test too. Does not matter what fails, the difference compared to other tests will show something new.
Maybe avisynth is consuming memory during decoding. If not, maybe this is fragmentation effect of dueling heaps. Memory allocation is so random it is surprising it works sometimes at all.

[stormy1777]

ok, re-ran, here are fresh results for heap size cmvl

1) empty virtualdub.exe (no source files) 5MB.

2) open avs (avisource), refresh vmmap 40MB.

3) run analyze pass (no compression) ~600MB.

4) Analyze pass with compress x264, FAILS, ~596MB.

5) Open AVI directly, analyze+x264 pass, works, 990MB.

This file has all 5 stages: https://anonfile.com/q5BfW2r4nf/VirtualDub2_-_crash_zip

all above testing is using a single instance/window of vdub....

Maybe it is possible to somehow "instrument" or otherwise mark these segments for later inspecting what kind memory that is, who allocated, etc.

[stormy1777]

sorry, did not anticipate all this AVS issues, I'll be glad to move this debug to vdub forum.

Was not aware that avisynth on 64 is not trivial (doesn't work on my setup at all).

ran another 32bit/vdub2 analysis pass and looked at process explorer for vdub2, noticed

ffdshow.ax is "loaded" in memory map, not sure if that is expected?

also, noticed 'x264vfw.dll' loaded, but the compression chosen is the builtin x264-8bit, NOT the one down at the bottom: x264vfw!!

could there be something in the registry/ffdshow that remaps x264 incorrectly? I can't reinstall the OS, but if there's anything to inspect that's no problems, otherwise all things appear to be OK (i.e. without AVS, never got any failures in months).

see images at: https://imgur.com/a/t2D4tqU

On the actual encode, managed to get MCD going, baseline CRF17/film preset was ~270MB, MCD for what appears to be equal quality comes down to 134MB!! that is HALF the size!! maybe i need to see an eye doctor?.... it's amazing simply amazing that it "removed" so much data, yet it seems just as good, at least to me :) thanks for this tip, i think that is good enough for this encode, will try another clip and will use this MCD in such heat/haze cases.
Stormy.