Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion. Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules. |
26th February 2013, 21:05 | #1 | Link |
Registered User
Join Date: Apr 2009
Posts: 153
|
help to increase compressibility a very noisy source
Hi. I'm trying to rip a bluray (without resizing it) that is very difficult to compress.
I usually don't use a lot of avisynth filters, usually i use ffvideosource as input (i remux the bd to mkv to be able to use fffvideosource with make mkv) and smdegrain with very low settings (tr=1, prefilter=0, sometimes contrasharp=true) so i'm not an expert and i don't know what to use in this difficult rip. I tried some filter but the result was overfiltered (removed even too much grain and washed out all the detail) or with not so much gain. Here's a sample of the bd (i took a very noiisy part, cut with mkvmerge from my remux): Code:
http://www.sendspace.com/file/b8brqb Last edited by phate89; 27th February 2013 at 03:03. |
27th February 2013, 15:08 | #3 | Link | |
Registered User
Join Date: Apr 2009
Posts: 153
|
Quote:
I don't think it's possible to preserve it anyways... The compressibility gain is awesome but i don't know if it is worth it.. I will try to reduce the strength.. Last edited by phate89; 27th February 2013 at 18:16. |
|
28th February 2013, 04:12 | #5 | Link | |
Registered User
Join Date: Apr 2009
Posts: 153
|
Quote:
The grain is my priority because if i try to rip it with a decent quality i get almost the original file size, and rip to get the same size it's pointless.. |
|
28th February 2013, 12:41 | #7 | Link | |
Registered User
Join Date: Apr 2009
Posts: 153
|
Quote:
About color matrix, shouldn't be already rec709 from bd h264 sources? And how should i use smoothlevels (or smoothadjust?) to get lower contrast? I tried with only temporaldegrain that already prefilter with fft3dfilter and with default settings (fft3dfilter sigma=16 and mvdegrain 2) it cleans everything with a very washed out result... |
|
28th February 2013, 13:26 | #8 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
I don't see why LemMotlow belabours chroma noise so much? Chroma noise is a rather minor problem here. Also, yes, the shadow parts are somewhat crushed - however there is not awfully much that could be "fixed". Dark areas without detail are dark areas without detail. Unless they are turned into brightened areas without detail. But then - of course, when trying to force some "very strong" levels manipulation on the near-black parts, it is quite possible that thereof some chroma problems come into view.
Regarding compressibility, it's surely the luma noise grain that hurts. x264/CRF18/slow/film comes out at 36000 kbit/s on this sample, which is not little. Would you consider stepping down to 720p? Me, I surely would! The NR-processing required for a compressible result with good detail & sharpness will be VERY slow @ 1080p. Excessive MVTools-usage on FullHD is no fun. 720p is so much easier ... and in case playback is done via PC again, Upsize+FineSharp during playback makes things look quite similar to 1080p, again. _____ BTW, especially with "seasonal series" from the U.S., I have often the impression that some noticeable black-crush is going on. I dunno, but *perhaps* it is related to gamma handling during the mastering chain? In Rec.709, gamma is not a purely Gamma 2.2, but a modification with a linear ramp in the near-black area: (blue = native gamma 2.2, green = Rec.709 Gamma definition) If recording is done "blue", but during processing it is assumed to have-been "green" (or some similar mishandling during processing), this could explain the black crush that is often to be seen.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
28th February 2013, 13:44 | #9 | Link |
Registered User
Join Date: Apr 2009
Posts: 153
|
Encoding time it's not a big deal for me (good pc, time to waste), i usually encode in placebo/subme 10/me_range 32 so i'm used to slow encodings.
Refining during play it is because for now i'm stuck with a cheap htpc. With other tv shows should be ok to simply switch to 720p, but with this i'd prefer to keep it at 1080p... If i can't a good compression improvement i will probably keep the dumped mkv... About the crushed black i saw it in other shows, but not all luckily. I don't understand why tv shows have to get a worst handling than movies (with some exception of course). I tried to take once the itunes version and it is even worse, the colours are completely different, all greenish... Last edited by phate89; 28th February 2013 at 14:08. |
28th February 2013, 14:04 | #10 | Link | |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
OK, then today evening I'll revisit the script I started to fiddle with yesterday. Who knows, perhaps ...
Quote:
BTW, when answering to the post directly above, there's no point in placing a full-quote.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
|
28th February 2013, 14:17 | #11 | Link | |
Registered User
Join Date: Apr 2009
Posts: 153
|
Quote:
Try me, i'm ready. |
|
28th February 2013, 21:13 | #13 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
@phate89 - I'm not really sure what you expect or how you judge denoising results. So, please rate the following result:
To you, is the following a "good" result, or is it more like "no thanks, too much detail loss" : Slow-Denoise (24 MB) Technically: with x264 --preset slow --tune film --CRF 18 : no filtering: 35976 kbit/s (6.30 fps) slow filtering: 11130 kbit/s (0.44 fps) I.e. 69% reduction of bitrate, but 14 times longer to process/encode.
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
1st March 2013, 00:08 | #14 | Link | |
Registered User
Join Date: Apr 2009
Posts: 153
|
Quote:
The compression gain is even more than expected and the temporal noise (is this the definition of that stuff?) is cleaned very well. I tried to play it and in that there's something a little bit strange on the sy, the static ponts (due to the high denoising).. It is because i know it was terrible so i look at it, without knowing that i'd never notice that. After i watched the original and was more unnatural than your rip, so i want definnitely to give it a chance at least with a 5% to see how it works with other parts of the video. Maybe leaving or putting back a little bit of noise will save from that little strange behaviour? Last edited by phate89; 3rd March 2013 at 12:00. |
|
3rd March 2013, 18:06 | #15 | Link |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Yeah, that sort of "static grain" is a very common issue when the source starts with really strong grain, and the main denoising process is temporal filtering. It's hard to come by, unless some sort of spatial denoising/smoothing is thrown into the chain. But with spatial denoising in the game, it is very likely to cause blurring real detail where you don't want to have it.
Another approach: -edit- (Buggy script) -- Split_slowdenoise-3c_CRF18.mkv (31 MB) (Correct script) -- Split_slowdenoise-3d_CRF18.mkv (29 MB) This issues mainly temporal filtering, with additional spatial pre-filtering, masked to target by brightness/chroma masking. Compressibility is somewhat lower than before (now 14500 13570 kbit/s @ CRF18/slow/film, compared to 36000 without filtering). Detail retention is different than before - a little less here, a little more there. In overall balance, I think it is better than before. The used script. It is very slow, and it's possible that you cannot use it in a direct script-->x264 chain when using x264 with "exhaustive" settings (in particular high mbtree lookahead). edit: corrected vector bug (delta value) Code:
SetMemoryMax(1000) # LoadPlugin( ... # # MVTools2.dll # mt_masktools-25.dll # RemoveGrain.dll # Repair.dll # MedianBlur.dll pel1 = 1 # you can use pel=2, pel2 = 1 # but with pel=1 the script is already slow & memory-hungry enough ... dgsource("G:\Split.dgi") changefps(last,last,true) o = last ox=o.width() oy=o.height() pre01 = o.sbr() pre02 = pre01.medianblur(2,0,0) pre03 = o.mt_makediff(mt_makediff(pre01,pre01.removegrain(4,-1)),U=2,V=2).sbr() msk = mt_lutxy(o.utoy(),o.vtoy(),"x 128 - 0 1 - * 128 + y + 2 / 148 - 16 *").greyscale() brite = pre02. levels(80,1.0,160,0,255,false).greyscale() brite = brite.bicubicresize(ox/2,oy/2) brite2 = brite.repair(brite.repair(brite.mt_inpand().mt_inpand().mt_inpand().mt_inpand().mt_expand().mt_expand().mt_expand().mt_expand(),1,0),12,0) bmask = mt_lutxy(brite2.mt_inpand(),msk.mt_expand(),"x y -").mt_expand().removegrain(20).greyscale() \ .bicubicresize(ox,oy,1,0) sss1 = mt_lutxy(pre02,o,"x x y - abs 1 1.22 / ^ 0.51 * x y - x y - abs 0.001 + / * -",U=2,V=2) sss2 = sss1.temporalsoften(2,12,12,20,2).merge(pre02,0.14) sss3 = sss1.mt_merge(sss2,bmask,luma=true,U=3,V=3) sup0 = sss3.msuper(pel=pel1) sup1 = o.msuper(pel=pel1,levels=1) bv02 = sup0.manalyse(isb=true, delta=2,blksize=16,overlap=8 , search=5)#,searchparam=8,DCT=5) bv01 = sup0.manalyse(isb=true, delta=1,blksize=16,overlap=8 , search=5)#,searchparam=8,DCT=5) fv01 = sup0.manalyse(isb=false,delta=1,blksize=16,overlap=8 , search=5)#,searchparam=8,DCT=5) fv02 = sup0.manalyse(isb=false,delta=2,blksize=16,overlap=8 , search=5)#,searchparam=8,DCT=5) mdg1 = o.mdegrain2(sup1,bv01,fv01,bv02,fv02,thsad=400) mdg1a = mdg1.mt_makediff(mt_makediff(mdg1,o).bicubicresize(ox/4,oy/4).bicubicresize(ox,oy,1,0),U=2,V=2) mdg1aD = mt_makediff(o,mdg1) pre02oD= mt_makediff(o,pre02) xDD = pre02oD.repair(mdg1aD,12) spat2 = o.mt_makediff(xDD,U=2,V=2).minblur(0,2) spat2 = mt_lutxy(spat2,o,"x 1 + y < x 2 + x 1 - y > x 2 - y ? ?",U=2,V=2) spat2 = o.mt_merge(spat2,bmask,luma=true,U=3,V=3) sup10 = mdg1.msuper(pel=pel2) sup11 = spat2.FSx(mode=-1,sstr=2.0,cstr=-2.0,xstr=0.0).msuper(pel=pel2,sharp=1,levels=1) bv4 = sup0.manalyse(isb=true, delta=4,blksize=16,overlap=8 , search=4,searchparam=1,DCT=5) bv3 = sup0.manalyse(isb=true, delta=3,blksize=16,overlap=8 , search=5,searchparam=8,DCT=5) bv2 = sup0.manalyse(isb=true, delta=2,blksize=16,overlap=8 , search=5,searchparam=8,DCT=5) bv1 = sup0.manalyse(isb=true, delta=1,blksize=16,overlap=8 , search=5,searchparam=8,DCT=5) fv1 = sup0.manalyse(isb=false,delta=1,blksize=16,overlap=8 , search=5,searchparam=8,DCT=5) fv2 = sup0.manalyse(isb=false,delta=2,blksize=16,overlap=8 , search=5,searchparam=8,DCT=5) fv3 = sup0.manalyse(isb=false,delta=3,blksize=16,overlap=8 , search=5,searchparam=8,DCT=5) fv4 = sup0.manalyse(isb=false,delta=4,blksize=16,overlap=8 , search=4,searchparam=1,DCT=5) spat2.removegrain(11,0).mdegrain3(sup11,bv1,fv1,bv2,fv2,bv4,fv4,thsad=800) mdg23 = last mt_makediff(mt_makediff(last,spat2).bicubicresize(ox/4,oy/4).bicubicresize(ox,oy,1,0),U=2,V=2) mt_lutxy(o,"x 4 + y < x 3 + x 4 - y > x 3 - x 0.49 * y 0.51 * + ? ?",U=3,V=3) last.mt_merge(mdg23,bmask,luma=true,U=3,V=3) mt_lutxy(sbr(),"x 4 + y < x 3 + x 4 - y > x 3 - x 0.51 * y 0.49 * + ? ?",U=2,V=2) return(last) # ========================================= # ========================================= # Helper functions function sbr(clip o) { rg11=o.removegrain(11) rg11D=mt_makediff(o,rg11) rg11DD=mt_makediff(rg11D,rg11D.removegrain(11)).mt_lutxy(rg11D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?") o.mt_makediff(rg11DD,U=2,V=2) } function MinBlur(clip clp, int r, int "uv") { uv = default(uv,3) uv2 = (uv==2) ? 1 : uv rg4 = (uv==3) ? 4 : -1 rg11 = (uv==3) ? 11 : -1 rg20 = (uv==3) ? 20 : -1 medf = (uv==3) ? 1 : -200 RG11D = (r==0) ? mt_makediff(clp,clp.sbr(),U=uv2,V=uv2) \ : (r==1) ? mt_makediff(clp,clp.removegrain(11,rg11),U=uv2,V=uv2) \ : (r==2) ? mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20),U=uv2,V=uv2) \ : mt_makediff(clp,clp.removegrain(11,rg11).removegrain(20,rg20).removegrain(20,rg20),U=uv2,V=uv2) RG4D = (r==1) ? mt_makediff(clp,clp.repair(clp.removegrain(4,rg4),14),U=uv2,V=uv2) \ : (r==2) ? mt_makediff(clp,clp.medianblur(2,2*medf,2*medf),U=uv2,V=uv2) \ : mt_makediff(clp,clp.medianblur(3,3*medf,3*medf),U=uv2,V=uv2) DD = mt_lutxy(RG11D,RG4D,"x 128 - y 128 - * 0 < 128 x 128 - abs y 128 - abs < x y ? ?",U=uv2,V=uv2) clp.mt_makediff(DD,U=uv,V=uv) return(last) } function FSx(clip c, int "mode", float "sstr", float "cstr", float "xstr", float "lstr", float "pstr") { mode = default(mode, 1 ) # 1 to 3, weakest to strongest. When negative -1 to -3, a broader kernel for equalisation is used. sstr = default(sstr, 2.0 ) # strength of sharpening, 0.0 up to ?? _cstr = spline(sstr, 0,0, 0.5,0.1, 1.0,0.6, 2.0,0.9, 2.5,1.09, 3.0,1.15, 3.5,1.19, 4.0,1.249, 8.0,1.5, 255.0,2.0) cstr = default(cstr, _cstr) xstr = default(xstr, 0.19 ) lstr = default(lstr, 4.0 ) pstr = default(pstr, 1.6) str1 = sstr str2 = cstr SSTR = string(abs(sstr)) CSTR = string(abs(cstr)) LSTR = string(abs(lstr)) PSTR = string(abs(pstr)) rg=mode>0?11:20 b = (abs(mode)==1) ? c.removegrain(11,-1).removegrain(4,-1) \ : (abs(mode)==2) ? c.removegrain(4,-1).removegrain(11,-1) \ : (abs(mode)==3) ? c.removegrain(4,-1).removegrain(11,-1).removegrain(4,-1) : c shrpD = mt_lutxy(c,b,"x y - abs "+LSTR+" / 1 "+PSTR+" / ^ "+LSTR+" * "+SSTR+" * x y - x y - abs 0.001 + / * x y - 2 ^ x y - 2 ^ "+SSTR+" 4.0 * + / * 128 +") shrp = (str1<0.01) ? c : c.mt_adddiff(shrpD,U=2,V=2) bzzD = (str2>0.0) ? shrpD.mt_lut("x 128 - "+CSTR+" * 128 +").removegrain(rg,-1) \ : shrpD.mt_lut("x 128 - "+CSTR+" * 128 +").removegrain(11,-1).removegrain(20*1,-1) shrp = (abs(str2)<0.01) ? shrp : shrp.mt_makediff(bzzD,U=2,V=2) shrp = (xstr<0.01) ? shrp \ : mt_lutxy(shrp,shrp.removegrain(20,-1),"x x y - 9.9 * +").repair(shrp,12,0).mergeluma(shrp,1.0-xstr) return(shrp) }
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) Last edited by Didée; 4th March 2013 at 09:40. |
3rd March 2013, 21:05 | #16 | Link |
Registered User
Join Date: Mar 2002
Location: Krautland
Posts: 903
|
@Didée:
I've looked at your second take. Great improvement in the sky area and overall. But......as you've mentioned, the script is slow as hell. On my Core2Duo it would have take ages til the end A resized version is acceptabel, so: Thanks for sharing. It's something for my toolbox. |
3rd March 2013, 22:44 | #17 | Link | |
Registered User
Join Date: Apr 2002
Location: Germany
Posts: 5,391
|
Well ... especially when considering there's A BUG in the script - in the 2nd MDegrain instance two vectors are identical when they should be different. Copy+Paste mistake.
Also, I need to include one more helper - I've used a slightly modified FineSharp script. The given parameters are bad when used with the official version. Correction to come, I'm about to start encoding the sample again. edit - ah, with correct vectors, sample is 13570 instead of 14500 kbps. Quote:
But anyway ... with 1080p, it's almost suicidal to use a 2nd instance of Manalyse/MDegrain that is referencing a 1st MDegrain instance as basis/source. Well, I can't help ... I fiddle together the sort of processing that seems capable to solve a problem. If it turns out the chain is too slow, ... then it is like it is. [*shrugs shoulders*]
__________________
- We´re at the beginning of the end of mankind´s childhood - My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!) |
|
5th March 2013, 03:01 | #19 | Link |
Registered User
Join Date: Apr 2009
Posts: 153
|
I tried the script on 5% of an episode.
I have to say, the result is great, a lot better than i expected. it loses a little bit of detail but it improves drastically the video quality and leaves a little bit of normal grain that helps to hide defects (if someone wants to look at it i have a crf 23 placebo to upload, crf 18 takes too much time at 40 kbps) I understand very few of what exactly the script does but it works very well with my problem. I think i'm gonna launch the first full episode now, and if i decide it's worth it i will start the full season. If i can't manage to do it with your script i'll give a chance to 720p at least. It's all unother thing with a smaller resolution right? It's easier to clean well and preserve detail? Last edited by phate89; 5th March 2013 at 03:04. |
3rd April 2013, 03:49 | #20 | Link |
Registered User
Join Date: Apr 2009
Posts: 153
|
This week I finally find the time to look at the result of the rip of the full episode (it took me 5 days!). The result is very good but doing it for 13 episodes and probably for the next season is simply too much.. So I'm gonna do some quick try with 720p and if the visual result is not enough I'll keep the remux on the disk..
Is it easier in your opinion to get a good 720p or I still need something very complicated like the old script for 1080p? |
Thread Tools | Search this Thread |
Display Modes | |
|
|