Need Suggestions for VERY GRAINY source

[Sasovics]

Folks,

I've just encoded another 15000 frames from Transformers (h.264 1080p source - > x264 720p destination) and got more then excellent results!

AVInaptic reported VERY HIGH DRF quality when used Sagekilla's TemporalDegrain filter compared to MEDIUM DRF quality without the filter! Nice work Sagekilla!

The only backdraw is the speed.. At 2fps it would take almost 48h for me to encode the entire movie ... but that's the price paid for this quality boost

[Adub]

Just out of curiosity, were you using MT?

[Sasovics]

Actually no...

I have downloaded the MT version of the avisynth and have put this line into my avs script: SetMTmode(2), but for some reason
my encoding is even slower, and I have no idea why

fyi - I am running Intel Core 2 Duo E6700 2.6GHz


EDIT: Well, according to this post, using MT when you already have 100%CPU usage (which I obviously have as x264 is running), will slow-down encoding even more...

[Sagekilla]

Thanks for the praise, but it should be directed towards Didee; He wrote the core functionality for TD. If you want, you can try TemporalDegrain(HQ=2,ov=2,Degrain=1) for a bit more speed. It doesn't degrain as strongly, but that's the offset of using MVDegrain1 over MVDegrain2.


I also have an extremely simple script that cleaned up 300 fairly well at about 6-8 fps. It's inherently multithreaded because I call MT() for the filter chain, so no need to use SetMTMode(). I'll send it to you if you're interested, but to be honest it's nothing special. Still, it did a fairly good job on degraining 300.

[Sasovics]

I am interested in that simplified script, please upload it to mediafire and I'll post back my feedback!

Thanks in advance!

[Sagekilla]

Here's the script, I couldn't upload it earlier because I was in school still. There should be a noticeable difference between Temporal Degrain and the Fast Degrain script, as well as a difference in sharpness since TD has Contra Sharpening.

Feel free to adjust any settings if you feel it isn't working for you, or you see room for improvement. Just ask that you send me a PM detailing any useful changes that I can add to this Only setting I recommend you be careful of is blksize, as sometimes you might getting blocking at 0 (I noticed this while testing on 300) If this is the case, raise it to 2.

###################################################################
# Fast Degrain by Sagekilla                                       #
#                                                                 #
# Nothing special, just removes noise and grain very quickly and  #
# sharpens the image a bit. Runs at about 8 fps on my machine.    #
# Tweak to your tastes, but defaults perform perfectly fine.      #
#                                                                 #
# Required plugins: HQDn3d.dll / LimitedSharpenFaster.avs         #
#                    MT.dll / mt_masktools.dll / MVTools.dll      #
#                                                                 #
###################################################################

# Fast Degrain Syntax
#
# threads = Number of threads for MT -- Speeds up the program on multicores.
# degrain = Uses MVDegrain 1, 2, or 3. Each is progressively stronger and slower.
# thSAD   = Affects overall degraining strength. Default works fine.
# limit   = Limits maximum change from MVDegrain. Lower from 255 if you experience artifacts.
# pel     = Subpixel accuracy for MVAnalyse. Increasing it raises quality of MVs and lowers speed. 
# ov = 

function FastDegrain( clip input, int "threads", int "degrain", int "thSAD", int "limit", int "pel",  
 \                    int "blksize", bool "sharp", bool "post", int "str", float "ss", int "sft", int "oshot" )
{ 
threads = default( threads,  0 ) # Threads to use for MT
degrain = default( degrain,  2 ) # Degraining method
thSAD   = default( thSAD,  350 ) # MVDegrain thSAD
limit   = default( limit,  255 ) # MVDegrain limit
pel     = default( pel,      1 ) # MVAnalyse pel
ov      = default( ov,       0 ) # MVAnalyse block size
sharp   = default( sharp, true ) # Toggles LSF
post    = default( post,  true ) # Toggles postprocessing
str     = default( str,    150 ) # LSF strength
ss      = default( ss,     1.5 ) # LSF supersampling
sft     = default( sft,     40 ) # LSF soft
oshot   = default( oshot,    0 ) # LSF overshoot
Return(input)
global idx1 = 50

MT("""
idx1  = idx1 + 1
f     = last
o     = f

# Search for motion vectors. No prefiltered clips because it slows things down too much.
b3vec = (degrain>=3) ? o.MVAnalyse(isb=true, delta=3,idx=idx1,pel=pel,ov=ov) : BlankClip
b2vec = (degrain>=2) ? o.MVAnalyse(isb=true, delta=2,idx=idx1,pel=pel,ov=ov) : BlankClip
b1vec = (degrain>=1) ? o.MVAnalyse(isb=true, delta=1,idx=idx1,pel=pel,ov=ov) : BlankClip
f1vec = (degrain>=1) ? o.MVAnalyse(isb=false,delta=1,idx=idx1,pel=pel,ov=ov) : BlankClip
f2vec = (degrain>=2) ? o.MVAnalyse(isb=false,delta=2,idx=idx1,pel=pel,ov=ov) : BlankClip
f3vec = (degrain>=3) ? o.MVAnalyse(isb=false,delta=3,idx=idx1,pel=pel,ov=ov) : BlankClip

# Degraining the video using MVDegrain. Nothing special here.
f     = (degrain==3)  ? f.MVDegrain3(b1vec,f1vec,b2vec,f2vec,b3vec,f3vec,idx=idx1) : f
f     = (degrain==2)  ? f.MVDegrain2(b1vec,f1vec,b2vec,f2vec,idx=idx1) : f
f     = (degrain==1)  ? f.MVDegrain1(b1vec,f1vec,idx=idx1) : f

# Sharpen the video a bit to counter the slight loss of detail and texture.
# Follow this up by post processing to remove stray dancing pixels.
f     = (sharp==true) ? f.LimitedSharpenFaster(strength=str,ss_x=ss,ss_y=ss,soft=sft,overshoot=oshot) : f
f     = (post==true)  ? f.HQDn3D(2,1,6,1) : f

return(f)
""",threads,2)
}

Enjoy.

[Adub]

Want me to add it to the wiki?

[Sasovics]

Where do I get the LimitedSharpenFaster.avs script ?

[Adub]

1. Using search.
2. Here:
http://avisynth.org/mediawiki/LimitedSharpen

[Sagekilla]

Originally Posted by Merlin7777

Want me to add it to the wiki?

You can stick it under the TemporalDegrain one as an alternative, faster degraining method. It uses pretty much the same syntax as TD does.

[Adub]

Cool, will do.
Edit: Done.

[Sagekilla]

Also, I just got my hands on a new Blu-Ray drive so I can finally rip and play with 300 in HD! I'll see if I can do a before and after using 1080p frames to show how well it works, and help me do testing to improve it. If I have enough time, I might even encode it twice to show the bitrate reduction possible.

Edit: How odd, I tried changing the idx on the NR2 MVDegrain from idx=2 to idx=3 and wanna know what my results were? idx=2 (same idx as the last MVdegrain) was more detailed and looked better. It was only MARGINALLY better though. You have to look VERY closely to see if there was any change.

Results from no TD (first one) vs with TD

G:\Movies\300_HD>x264_aq --crf 18 --ref 5 --mixed-refs --no-fast-pskip --bframes
 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --subme 7 --me umh --trellis 1
 --aq-strength 0.3 --threads auto --progress "source.avs" --output "Grainy.264"
--pass 1 --stats "Grainy.log"
avis [info]: 1920x800 @ 23.98 fps (360 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 3DNow!
x264 [info]: slice I:2     Avg QP:15.80  size:402956  PSNR Mean Y:47.30 U:51.04
V:51.57 Avg:48.26 Global:48.26
x264 [info]: slice P:181   Avg QP:18.05  size:224610  PSNR Mean Y:44.65 U:49.72
V:50.04 Avg:45.80 Global:45.78
x264 [info]: slice B:177   Avg QP:20.15  size:152771  PSNR Mean Y:42.90 U:49.27
V:49.64 Avg:44.20 Global:44.18
x264 [info]: mb I  I16..4: 29.0% 45.7% 25.3%
x264 [info]: mb P  I16..4: 29.5% 16.7% 12.0%  P16..4: 18.4% 17.3%  5.1%  0.0%  0
.0%    skip: 0.9%
x264 [info]: mb B  I16..4: 13.4%  7.8%  7.2%  B16..8: 38.1%  6.1%  6.6%  direct:
14.4%  skip: 6.4%
x264 [info]: 8x8 transform  intra:28.5%  inter:17.4%
x264 [info]: ref P  75.9% 12.5%  6.4%  3.2%  2.1%
x264 [info]: ref B  74.2% 19.5%  4.4%  2.0%
x264 [info]: SSIM Mean Y:0.9813726
x264 [info]: PSNR Mean Y:43.801 U:49.506 V:49.853 Avg:45.025 Global:44.928 kb/s:
36497.25

encoded 360 frames, 0.68 fps, 36497.60 kb/s

G:\Movies\300_HD>pause
Press any key to continue . . .

G:\Movies\300_HD>pause
Press any key to continue . . .

G:\Movies\300_HD>x264_aq --crf 18 --ref 5 --mixed-refs --no-fast-pskip --bframes
 16 --bime --weightb --b-pyramid --b-rdo --8x8dct --subme 7 --me umh --trellis 1
 --aq-strength 0.3 --threads auto --progress "source.avs" --output "NotGrainy.26
4" --pass 1 --stats "NotGrainy.log"
avis [info]: 1920x800 @ 23.98 fps (360 frames)
x264 [info]: using cpu capabilities: MMX MMXEXT SSE SSE2 SSE3 3DNow!
x264 [info]: slice I:2     Avg QP:15.45  size:278436  PSNR Mean Y:47.03 U:51.37
V:51.57 Avg:48.07 Global:48.07
x264 [info]: slice P:136   Avg QP:17.75  size:108867  PSNR Mean Y:44.63 U:50.04
V:50.26 Avg:45.81 Global:45.79
x264 [info]: slice B:222   Avg QP:19.84  size: 32984  PSNR Mean Y:43.42 U:50.00
V:50.23 Avg:44.73 Global:44.71
x264 [info]: mb I  I16..4: 21.3% 54.7% 24.0%
x264 [info]: mb P  I16..4:  6.4% 12.5%  3.1%  P16..4: 34.6% 30.1% 11.2%  0.0%  0
.0%    skip: 2.1%
x264 [info]: mb B  I16..4:  0.3%  0.7%  0.2%  B16..8: 38.1%  3.7%  8.2%  direct:
 6.2%  skip:42.6%
x264 [info]: 8x8 transform  intra:57.0%  inter:46.0%
x264 [info]: ref P  70.6% 21.3%  3.3%  2.4%  2.3%
x264 [info]: ref B  85.3% 13.8%  0.6%  0.3%
x264 [info]: SSIM Mean Y:0.9810072
x264 [info]: PSNR Mean Y:43.895 U:50.026 V:50.245 Avg:45.159 Global:45.100 kb/s:
12086.75

encoded 360 frames, 0.26 fps, 12087.09 kb/s

G:\Movies\300_HD>pause
Press any key to continue . . .

[Sagekilla]

Sasovics, did my updated script or the uploaded version work for you at all or do you still have problems?

[Sasovics]

Post moved to separate topic here

[Sagekilla]

Please don't use my FastDegrain -- It's currently b0rked. Also, Sasovics please don't cross post.

[Nikos]

Edit: How odd, I tried changing the idx on the NR2 MVDegrain from idx=2 to idx=3 and wanna know what my results were? idx=2 (same idx as the last MVdegrain) was more detailed and looked better. It was only MARGINALLY better though. You have to look VERY closely to see if there was any change.

According to Didée from post 125 for using 2 mvdegrain (mvdegrain2+mvdegrain1):

Without corrected weightings, it will denoise less, not more. (Unless you split the idx'es, then it will denoise more indeed.)

With the word split he mean different idx in each degrainX?
We need a little more teaching from Didée about idx

[Terranigma]

Originally Posted by Sasovics

I think the guy "Bobby" who encoded the movie, had to use some kind of combination of avs filters as I am using the very same x264 encoder settings and of course the very same source, US retail HDDVD.

I have tried various avs filters, including Sagekilla's TemporalDegrain, FastDegrain, Dark Shikari's GrainOptimizer,
as well the LimitedSharpenFaster alone and many more.

None of them helped to get to the quality of iLL release

Because iLL's like the greatest (real-life source) encoder there is?
They don't use any of these premade scripts, I can tell you that for sure.

[Sasovics]

Originally Posted by Terranigma

Because iLL's like the greatest (real-life source) encoder there is?
They don't use any of these premade scripts, I can tell you that for sure.

Don't understand what you mean with that "real-life source"
.. if they don't use these scripts, what are they using ? How can you tell ?

[Terranigma]

Originally Posted by Sasovics

Don't understand what you mean with that "real-life source"

Real Life source, as in, non-toon sources (like this movie 300 for instance).

Originally Posted by Sasovics

.. if they don't use these scripts, what are they using ? How can you tell ?

I can tell because they've been encoding way before temporaldegrain, mc_spuds, or this thread started, so that shouldn't be so hard to figure out.
(btw, lets leave out encoders or groups for that matter, as it's against the rules per se.)

Lets just say that what they use is actually simpler than you think; btw, it's not so hard to write scripts, just look over the syntax document in avisynth's directory, or take examples from scripts like mc_spuds or temporaldegrain for that matter.

[Sagekilla]

Originally Posted by Terranigma

Real Life source, as in, non-toon sources (like this movie 300 for instance).


I can tell because they've been encoding way before temporaldegrain, mc_spuds, or this thread started, so that shouldn't be so hard to figure out.
(btw, lets leave out encoders or groups for that matter, as it's against the rules per se.)

Lets just say that what they use is actually simpler than you think; btw, it's not so hard to write scripts, just look over the syntax document in avisynth's directory, or take examples from scripts like mc_spuds or temporaldegrain for that matter.

Indeed, TD is fairly simple as far as filter chains go. It doesn't use a lot of complex concepts compared to others imo.