Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 7th December 2007, 18:42   #41  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,394
Quote:
this idea you came up with
The idea in itself is not exactly new - Fizick once made a basic script with multi-freq filtering, and IIRC that was before his first implementation of FFT3DFilter. It's been a while since then.

Quote:
are you saying it'd be more beneficial to use this idea you came up with, to remove low-frequencies, will work better if it's used before mo-comp. denoising?
It's still the basic method of using a pre-filter before doing the motion search. This can be done in several different ways, and this here was just one of them that can be used. E.g., if one is using a pure spatial prefilter, the effect is the opposite: it will take out the hi-freq's, but leave the flicker mostly intact, therefore still irritating the ME engine.
Fact is, with strong grain there is so much uncertainty at the pixel level that there is hardly any "this is the right way to do". There are plenty of different possible points to break into the circle of catch-22 ... but there's no "correct" one.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 7th December 2007, 19:05   #42  |  Link
Terranigma
*Space Reserved*
 
Terranigma's Avatar
 
Join Date: May 2006
Posts: 953
Ah I see, I understand now. Thanks for clarifying.
Terranigma is offline   Reply With Quote
Old 8th December 2007, 05:48   #43  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Personally, I prefer Didee's script over most plugins I've tried for removing grain since it doesn't blur the image at all, and when it comes to the dancing grain like in 300 it stabilizes it so it doesn't look horrendous after encoding. Really, my biggest beef is how soft the image becomes after most denoise/grain filters so Didee's script is really what I've been looking for.

I really wasn't expecting too much in the bitrate reduction department after I read his explanation, but just the sheer increase in video quality (to my eyes) from not having dancing grain is more then enough.
Sagekilla is offline   Reply With Quote
Old 8th December 2007, 08:33   #44  |  Link
Pookie
Registered User
 
Join Date: Apr 2005
Posts: 1,339
Another one for the cookbook. Thanks (again), Didée.
Pookie is offline   Reply With Quote
Old 10th December 2007, 20:49   #45  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by Pookie View Post
Another one for the cookbook. Thanks (again), Didée.
ok all I finally did the final encode which took 3 days to crunch
Please download and take a look I am curious as to all your thoughts!

The clips

I learned A LOT from this thread. Nothing worked straight up to the level I wanted and the best by far was to use a little of everything I learned in this thread and test test test then tweak and test some more lol I hope you all check out the final result clip I think you will be very surprised at just how awesome it came out and IMHO trounces the original in all aspects.

Some Notes and things I learned about grain.

File Size. Grain kills the ability to compress more than I thought. I used x.264 and to reach a Q=16 average the original needed a 9000 bitrate and mine without grain only needed a 4000 bitrate and mine IMHO looks better and has in some areas more detail. (get to that in a minute)

the codec it self plays a huge part on detail loss even with massive bitrate. In this case I learned x.264 blows out detail no matter how much bitrate you give it because of the inloop filters so if you have the bitrate turn them off. If you do not then yeah leave them on as they help a ton with blocking etc....

grain and dancing grain (dancing grain is what really annoys me) really is only noticeable to the point of distraction on walls, and sky and wheat fields and skin (not face as much like say a women's smooth tummy, i.e. not so detailed skin areas). Where there is lots of detail it gets lost in that detail so that made it easy to find the level of detail and mask then only remove A LOT of grain from the low level detail areas, some from middle detail areas and almost none from high detail areas so we can keep as much detail a possible since the filtering will always get rid of some. Down side is SLOW SLOW SLOW.

Where you place filters in the chain is so important when it comes to what I feel is re-mastering type of filtering. an example is my huge filter chain washed out the colors some and lightened up everything so I had to add more filters to adjust that and when you look at the original you even see that they had the same problem and I was able to make it look even better than the original in that area also like on the shields where the highlights were so bright and huge it made the them look fake but my tweaks added back the bronze color and toned down the contrast so it looked real and the color more rich and vibrant. The thing is if I place the filters at the end of chain it just sorta gave the whole frame a boost and it looked fake since some things shouldn't get the boost. However, if I used it at the start of chain then let all the removal and detail enhancement at it, they knocked it back down to not fake levels and they did it selectively. A bronze shield that needed a touch of more yellow-bronze got it but a silver sword did not else it would have turned purple or blue and not stayed silver looking. very interesting results here.


Anyway, if you watch the clips note the wheat behind the bad guy while he says "the hot gates" huge difference in dancing pixels between the clips. A ton to none. After that the sky is next place that looks so much better. Even the kings son's face looks better because in this example the dancing grain was drawing your eye away from detail. This is what I meant above by how some detail is even better with the grain removed. Another is the pull back shot with wheat all around. The original has so much dancing grain it makes it hard to see each stalk since the grain is jumping on and off them and since each grain was as large or larger that the stalk width it hid them but with it gone you can see each stalk and the fine lines. The wheat has more detail now and so does the mountain range in the distance or perhaps I should say it has more perceived detail. :P


Anyway I am very pleased with the results and the only thing I may do differently is to run it again with more bitrate

Last edited by Zep; 10th December 2007 at 22:05.
Zep is offline   Reply With Quote
Old 10th December 2007, 21:14   #46  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Quote:
Originally Posted by Zep View Post
because it is not really doing much. At least for me it only reduces the amount of grain by a very small amount and what is left is dancing around almost as much.

if you want to see a fast and really good grain removal use ffdshow noise removal and turn on Denoise3d but only use the HQ setting and max it out. Very fast and very good.


Now mix that with your avi synth script and tweak from there.


I think you see better results because you are not coming from the Bluray but are using the lower rez version which has far less grain and most of these plugs are geared to low rez DVD. Removegrain just can't cut it on high rez Bluray source and this is where removegrainHD is better and why it was written.
The point for me is not to touch the grain in 300 since it's very difficult to remove grain and retain sharpness. I refuse to degrain if it means my image is becoming smoother, and I've already encoded the video and quite like how the results have come out.

Edit: To clarify, none of the degrain scripts I've tried have offered a decent way to get rid of the dancing grain (Which is my goal really) while maintaining sharpness, and Didee's script is what really does this for me. Plus, I don't use ffdshow at all so I can't do postprocessing that way.

Last edited by Sagekilla; 10th December 2007 at 21:17.
Sagekilla is offline   Reply With Quote
Old 10th December 2007, 21:52   #47  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by Sagekilla View Post
The point for me is not to touch the grain in 300 since it's very difficult to remove grain and retain sharpness. I refuse to degrain if it means my image is becoming smoother, and I've already encoded the video and quite like how the results have come out.

Edit: To clarify, none of the degrain scripts I've tried have offered a decent way to get rid of the dancing grain (Which is my goal really) while maintaining sharpness, and Didee's script is what really does this for me. Plus, I don't use ffdshow at all so I can't do postprocessing that way.

you say all that without looking at my final clip (for now anyway odds are I will run more testing next weekend) or reading my last post I take it.

My clip has almost no grain and almost no loss of detail and in some areas more detail and I do it at half the bitrate of the original. If I had to choose which one to watch I would take mine by miles and I bet most would and that is right now. I could easily make it even better if I wanted to spend more time and use the same bitrate as the original. (I would get the Q average in x.264 down to 12 I bet and could turn off the in loop filters hahaha)

The bit rate issue alone is such a huge plus you could really go to town with more filtering like mega masking for selective removal as well as selective sharpening. Furthermore, IIRC you said you do not have the Bluray. if you did you would see that Didee's script doesn't work on it that well. Dancing grain was almost as bad and there was loss of detail on top of that. My guess is the same reason removegrainHD was made. With higher rez source you need go beyond just 9 pixel box to catch the grain. Anyway, that script may work on low rez stuff but it didn't do much on direct from the Bluray source. Trust me I wish it did as it would have saved me a lot of time.

Anyway, download my latest clip and compare as an example the start of the clip the kings face VS the original. The detail is still there. (read my post above about MASKING and you will know why it is still there)

Last edited by Zep; 10th December 2007 at 22:00.
Zep is offline   Reply With Quote
Old 10th December 2007, 23:47   #48  |  Link
StifflerStealth
Registered User
 
Join Date: Feb 2004
Location: Somewhere Over the Rainbow!
Posts: 141
Zep, could you post your final AVS script that you used to make that final build? I would love to learn off of it. I have other movies that I own that are grainy and it ticks me off each time I watch them. They are as bad as 300, especially the first two Harry Potter movies. It's like they cone back to Video Tape days.

And I too have learned a lot about grain in this thread. The tips posted here are invaluable.
StifflerStealth is offline   Reply With Quote
Old 11th December 2007, 00:34   #49  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Quote:
Originally Posted by Zep View Post
you say all that without looking at my final clip (for now anyway odds are I will run more testing next weekend) or reading my last post I take it.

My clip has almost no grain and almost no loss of detail and in some areas more detail and I do it at half the bitrate of the original. If I had to choose which one to watch I would take mine by miles and I bet most would and that is right now. I could easily make it even better if I wanted to spend more time and use the same bitrate as the original. (I would get the Q average in x.264 down to 12 I bet and could turn off the in loop filters hahaha)

The bit rate issue alone is such a huge plus you could really go to town with more filtering like mega masking for selective removal as well as selective sharpening. Furthermore, IIRC you said you do not have the Bluray. if you did you would see that Didee's script doesn't work on it that well. Dancing grain was almost as bad and there was loss of detail on top of that. My guess is the same reason removegrainHD was made. With higher rez source you need go beyond just 9 pixel box to catch the grain. Anyway, that script may work on low rez stuff but it didn't do much on direct from the Bluray source. Trust me I wish it did as it would have saved me a lot of time.

Anyway, download my latest clip and compare as an example the start of the clip the kings face VS the original. The detail is still there. (read my post above about MASKING and you will know why it is still there)
Yeah but there's exactly that: You have HD vs my SD. I don't know how well my source would hold up to your script. I'd love to test your script on my SD source but I haven't found a script (or didn't look hard enough, will later when I'm not so busy)
Sagekilla is offline   Reply With Quote
Old 11th December 2007, 13:38   #50  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,394
Quote:
Originally Posted by Zep View Post
Please download and take a look I am curious as to all your thoughts!
In a nutshell:

- Grain removed, at the price of:

- faces often times look soapy
- some detail is emphasised, some detail is lost or blurred
- there is edge ringing and mosquito noise / texture echoes caused by FFT3DFilter

Quote:
Anyway I am very pleased with the results and the only thing I may do differently is to run it again with more bitrate
For what you have there, the bitrate is fully sufficient. More bitrate won't help for the downsides introduced by your filterchain.

It's by all means a respectable try, but "remastering" (your words) needs a bit more than this.
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)
Didée is offline   Reply With Quote
Old 11th December 2007, 22:55   #51  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by StifflerStealth View Post
Zep, could you post your final AVS script that you used to make that final build? I would love to learn off of it. I have other movies that I own that are grainy and it ticks me off each time I watch them. They are as bad as 300, especially the first two Harry Potter movies. It's like they cone back to Video Tape days.

And I too have learned a lot about grain in this thread. The tips posted here are invaluable.

so I take it you watched the clips and like my final result? if you compare them side by side the difference really is huge and at half the bitrate I am very happy with the result.

so which one would you watch? hahaha

yes I will post the script first chance I get.
Zep is offline   Reply With Quote
Old 11th December 2007, 23:01   #52  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by Sagekilla View Post
Yeah but there's exactly that: You have HD vs my SD. I don't know how well my source would hold up to your script. I'd love to test your script on my SD source but I haven't found a script (or didn't look hard enough, will later when I'm not so busy)
yeah my script may not work so well on lower rez stuff. I admit I spent hours and hours tweaking to the point it may only work on this movie and its grain for all I know.

What is even worse the speed. Even pass 1 in x.264 was only like 10 FPS and I normally get like 40 FPS when do just basic filtering.

Pass 2 was like 1.7 FPS


Did you get a chance to look at the final encode? Just curious as to your thoughts about it and if you felt it kept enough detail.
yes there was and will always be a little loss of detail but I tried very hard to keep the loss in the detail areas to a minimum via masking. it is by no means perfect.
Zep is offline   Reply With Quote
Old 12th December 2007, 00:15   #53  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by Didée View Post
In a nutshell:

- Grain removed, at the price of:

- faces often times look soapy
- some detail is emphasised, some detail is lost or blurred
- there is edge ringing and mosquito noise / texture echoes caused by FFT3DFilter

For what you have there, the bitrate is fully sufficient. More bitrate won't help for the downsides introduced by your filterchain.

It's by all means a respectable try, but "remastering" (your words) needs a bit more than this.
well if you look at the original there is just as much mosquito noise and even more in some scenes which I could upload if you really wanted to see more. And I could get rid of it with more tweaking and a mosquito noise filter in the chain. heck even adding undot may fix it since it is so light mosquito noise but I haven't gotten that far yet and I doubt I ever will.

"It's by all means a respectable try" thank you and I can improve and may give it a go some more in the areas you noted. I too saw the cons but it is slow going to test. I need a 8 core box for sure haha


Please do not twist context/meaning of what I said. I certainly did not say it is "remastering" in the hard core way you are trying to make it sound like I said.

what I did say however was "I feel is re-mastering type of filtering"

"I feel" as in my opinion

"type of filtering" as in it is not remastering in the classic sense but in the general sense of giving an over all better experience even at the cost of some things taking a slight hit. Also note this really is meant to be seen in motion. That is what I tweaked it for. if you do frame to frame compares only then yes you can see the cons a lot easier and will not even see the dancing grain in stills which was my main goal to get rid of even if it meant other areas had to take a slight degradation. This thread is about grain removal first and foremost after all.

of course even with the cons the big question is, based on just the 1 clip which would you rather watch? Me I can't stand the dancing grain so you know what I would watch
Zep is offline   Reply With Quote
Old 12th December 2007, 00:20   #54  |  Link
Zep
Registered User
 
Join Date: Jul 2002
Posts: 587
Quote:
Originally Posted by Didée View Post

It's by all means a respectable try

BTW - since you do have the original source clip why not give it a go and see what you can do with it and upload here so we can all learn even more. I for one would use anything you come up with to make it even better.

thanks,

Zep
Zep is offline   Reply With Quote
Old 12th December 2007, 17:10   #55  |  Link
dbzgundam
Hates all his encodes
 
dbzgundam's Avatar
 
Join Date: Sep 2003
Posts: 166
Be grateful you guys can at least enjoy the grain as an unmuddied mess in the Blu-Ray version.

The DVD version is not as forgiving

The heavy grain look as of late is just a mystery to me. I understand wanting SOME grain in an image, but film stocks have improved drastically over the last few years in constant efforts to reduce this if anything, and the advent of digital video has only led to nearly-grain-free (Well it is grain free, but has noise now) picture quality... Oh well.

I actually used degrain1 for this beast, and it handled it pretty well. The movie is actually fairly compressible due to the softness of the picture, large amount of slow motion scenes, and while it does have high contrast, a lot of blown out areas make it that much less difficult to compress.
dbzgundam is offline   Reply With Quote
Old 12th December 2007, 22:11   #56  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Quote:
Originally Posted by Zep View Post
yeah my script may not work so well on lower rez stuff. I admit I spent hours and hours tweaking to the point it may only work on this movie and its grain for all I know.

What is even worse the speed. Even pass 1 in x.264 was only like 10 FPS and I normally get like 40 FPS when do just basic filtering.

Pass 2 was like 1.7 FPS


Did you get a chance to look at the final encode? Just curious as to your thoughts about it and if you felt it kept enough detail.
yes there was and will always be a little loss of detail but I tried very hard to keep the loss in the detail areas to a minimum via masking. it is by no means perfect.
Yes I did look at the encode, and personally I think if I were to use your script for encoding 300 it wouldn't be too bad since I'm using a much smaller resolution. (864x368 vs 1920x816) Plus I do use 1-pass crf so wouldn't have to worry about it being slow on a first pass since I am doing only one
Sagekilla is offline   Reply With Quote
Old 12th December 2007, 22:16   #57  |  Link
Zanejin
Registered User
 
Zanejin's Avatar
 
Join Date: Jan 2007
Posts: 24
Quote:
Originally Posted by dbzgundam View Post
The heavy grain look as of late is just a mystery to me. I understand wanting SOME grain in an image, but film stocks have improved drastically over the last few years in constant efforts to reduce this if anything, and the advent of digital video has only led to nearly-grain-free (Well it is grain free, but has noise now) picture quality... Oh well.
Grain is apparently becoming more and more popular and is not as undesired as one may think. It may give the illusion of detail and improve perceived color depth by dithering, or its abundance may simply appear more attractive to the general public. Much of 300's grain was added after filming.
Zanejin is offline   Reply With Quote
Old 12th December 2007, 22:18   #58  |  Link
Sagekilla
x264aholic
 
Join Date: Jul 2007
Location: New York
Posts: 1,752
Quote:
Originally Posted by Zanejin View Post
Grain is apparently becoming more and more popular and is not as undesired as one may think. It may give the illusion of detail and improve perceived color depth by dithering, or its abundance may simply appear more attractive to the general public. Much of 300's grain was added after filming.
In the case of 300 though, us DVD users pretty much get screwed over considering how poor our source quality is to begin with. Oh, what I would do for a Blu-ray player for my computer..
Sagekilla is offline   Reply With Quote
Old 14th December 2007, 09:37   #59  |  Link
Pookie
Registered User
 
Join Date: Apr 2005
Posts: 1,339
Looks like Kassandro has chimed in on this topic-

http://videoprocessing.11.forumer.co...opic.php?t=102
Pookie is offline   Reply With Quote
Old 15th December 2007, 05:47   #60  |  Link
Didée
Registered User
 
Join Date: Apr 2002
Location: Germany
Posts: 5,394
To not get accused of only babbling & critizising, without demonstrating something better, I gave it a go. Note it was quite an effort, since my PC is sub- (sub-sub-sub-...)-par for using demanding filters on & H264-encoding of HD content. You guys have machines that crunch 10 times faster than mine ...

Sample: MegaUpload -or- RapidShare (67MB)
[edit: now with working links...]

Bitrate came out a tad higher than that of Zep's sample (his: 4992 kbit, mine: 5066 kbit). No color correction was done. The grain was processed strictly temporal (spatial processing was only used for probability checks, but not applied in the denoising chain), and straightforward: no kind of area/detail/whatever masking was used.
Also, no arbitrary "pimp-it-up" sharpening was used; what has been used is a "dont-add-more-than-what-was-removed-previously" kind of sharpening. The goal was to keep the result looking the same as the original does ... just without the grain.

The filterchain definetly could benefit from some additional twists, but my pre-Christian Celeron doesn't allow for more. (That's why the result is named "easy".) The x264 encoding surely could be done better too - I'm an old-fashioned Xvid user, tweaking x264 is too complicated for me. (truley, I'm not fully satisfied with what I managed to get from x264. Is someone giving lessons?)


Some screenshots:

(middle= original / left= Zep's / right= my try)

Frame 50: <= =>

Frame 686: <= =>

Frame 910: <= =>

Frame 1274: <= =>

Frame 2100: <= =>


[post break because of image limit]
__________________
- We´re at the beginning of the end of mankind´s childhood -

My little flickr gallery. (Yes indeed, I do have hobbies other than digital video!)

Last edited by Didée; 15th December 2007 at 15:02. Reason: sample links were missing
Didée is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 06:21.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.