DVD Movie Noise

[Oline 61]

This noise seems to be present in most DVD releases, and I am wondering how to get rid of it.
http://img361.imageshack.us/img361/7267/test3vn.png
I have tried
RemoveGrain(mode=2).RemoveGrain(mode=2)
and
VagueDenoiser(threshold=0.8,method=1,nsteps=6,chromaT=0.8)
But I'm not sure which to use. Does anyone have any recommendations, and/or other filter combos to use?
Neither seems to touch the noise of the upper left wall. I don't want to set threshold on Vague too high because I will lose detail.

To add some more information, I am planning on backing up 2 movies to 1 DVD5, so this will be at about 2400kbps. Do I need to remove this grain if I plan on using a bitrate that high (x264, pretty darn high settings too)?

[hartford]

There are lots of filters and combos to try.

For "Three Days of the Condor" I used:

RemoveGrain(mode=1).RemoveGrain(mode=2)
ConvertToYUY2()
Pixiedust(limit=2)
ConvertToYV12()
LimitedSharpen(soft=true,smode=3) # Function by Didée


For "Daredevil" I used:

Removegrain(mode=17)
ConvertToYUY2()
PixieDust(limit=2)


Other functions to try:

LimitedRemoveDust (LRemoveDust)
PixieDustPP


I find that I use PixieDust and some mode of Removegrain on "live"
color movies, then I may sharpen with LimitedSharpen.

I have tried LimitedTemporalSoften (function by Didée), but I was not able
to find settings that didn't produce artifacts and yet cleaned well; artifacts
were only apparent in fast motion, and only when viewed frame-by-frame.

PixieDust can artifact in a rather strange way: at times a "chroma ghost"
will appear in the frame before an object in motion arrives in the next frame.
I've only noticed this in anime.

I can only suggest trying different filters and functions until you find
something that works for you. I've spent up to 4 hours experimenting until
I was satisfied with the result.

Best!

[Oline 61]

Where can I get PixieDust? It's not in any filter locations I usually look for.

[Oline 61]

Nevermind, I found it, but now I have another problem :(.
BTW, thanks hartford, the denoising stuff works great :).
I'm trying to back up Friday Night Lights, and I can't use Force Film in DGIndex because it leaves some interlaced frames behind. The problem is if I use decomb or TIVTC on it I get this:
http://img89.imageshack.us/img89/8773/00000ih.jpg
Original:
http://img97.imageshack.us/img97/8122/00011tr.jpg
Using hybrid processing doesn't help. At 29.97fps it shows the standard 3 progressive then 2 interlaced frames that most clips do. Why must it screw up the bleachers?

[hartford]

I sorry, but I can't help you with that because I've not had that problem using DGIndex to calculate whatever it does to IVTC the source that I put to it.


Sorry, I just don't know.

[Mug Funky]

that scene is throwing off the comb detection.

often when nothing else works, smartdecimate fed with 2 sources bobbed with tdeint(1,1,tryweave=true) will work pretty well (and fast).

if that returns too much combing, you can tweak it a tad. if you read the smartdecimate docs it should become a little clearer.

[edit] of course, turning off postprocessing on telecide should also help. the source is most likely pure film, but encoded without pulldown detection (this often happens, which is a shame because quality suffers a tad), so you might be able to get away without postprocessing telecide.

slightly off topic, but if you're using x264 at greater than 2000kbps you probably wont need any denoising at all. it'll be much faster and you'll get more detail and a more "film like" viewing experience. you can even use original audio too, which will give you a massive audio quality boost (transcoding lossy to lossy will completely pwn the audio)

[Elic]

Oline 61
Try this dumb sequence:doubleweave().pulldown( X, Y )where X and Y must be picked up manually (and see also help about pulldown() in Avisynth docs).

[Oline 61]

Thanks for helping me everyone :).
I've come up with this script for Friday Night Lights:
#loadplugin("C:\Program Files\Avisynth 2.5\plugins\loadpluginex.dll")
#loadplugin("C:\Program Files\Avisynth 2.5\plugins\dustv5.dll")

MPEG2Source("VTS_01_1.d2v")

DoubleWeave().Pulldown(0,3)

ColorMatrix()
Lanczos4Resize(640,480)
#Removegrain(mode=1).Removegrain(mode=2)
#ConvertToYUY2()
#PixieDust(limit=2)
#converttoyv12()
That sequence works great for decombing Friday Night Lights Elic.
I might try once without denoising and once with denoising. The problem is denoising kills my fps, even on first pass turbo I'm only doing 3 FPS. First pass turbo with no denoising usually stays around realtime (~23fps). I'm not sure if I have enough time to encode witht he denoising. If Pixiedust is too slow I might only use it on certain sections of the video, or just stick to 2 iterations of RemoveGrain.
I appreciate the help everyone :goodpost:.

[Oline 61]

Actually after looking throught the film a little more I've discovered that there are a bunch of interlaced frames in the middle of the movie. I have no idea why this happens but it gets really ugly. The movie must have been telecined incorrectly.
Could you elaborate on your method Mug Funky? I didn't try it because I'm not sure how you mean to set up the video for smartdecimate. Could you post a script for it please?

[Mug Funky]

this is a quick hack, and may not work (it's based on a PAL conversion script that i've removed the PAL conversion from).

function IVTCdude (clip c, int "mthreshL", bool "careful")
{

mthreshL = default(mthreshL,12)
careful = default(careful,false)

order = c.getparity()==true ? 1:0

global weaveb = c.tdeint(1,order,tryweave=true,mtnmode=3,type=2,cthresh=4,MI=24)
global bobb = c.tdeint(1,order,tryweave=true,mtnmode=3,type=2,cthresh=4,MI=12)
c

careful==false? smartdecimate(24,60,bob=bobb,tel=0.7) : smartdecimate(24,60,bob=bobb,weave=weaveb,tel=0.7)
}


default should be fine, add "careful=true" if not.

you need the 3rd december 2005 version of tdeint (well, that's the date stamp on my tdeint.dll) and smartdecimate (the latest you can find i guess).

copy pasta the above into the top of your script (before the "mpeg2source" line) and run like so:

mpegsource("blah.d2v")
assumetff() # just to be sure of field-order... it'll probably be top first.

ivtcdude()

your_favourite_denoiser()

btw, on the interlace in the middle of the movie.. this is a pattern change. there's probably an edit in the source somewhere. or if it's a really long movie it might have come on 2 tapes and the join threw the pattern off. it's pretty normal for the pattern to change during a movie.

[Oline 61]

Thanks for helping Mug Funky. That sequence creates a different artifact in the bleacher scene, but there is still significant artifacting. Using careful=true helps a little, but good frames are still interspersed with stuff like this:
http://img99.imageshack.us/img99/5275/00000zt.jpg
I am going to try finding where the pattern change starts, and then creating 2 seperate processes for those areas.

[Mug Funky]

hmm. is the field order right? perhaps tdeint is seeing the stadium seating as bottom-first stuff.

try hacking my script by raising "cthresh" in the second tdeint line (bobb = ...).

it's freaky because the seating looks just like interlace.

[Oline 61]

This movie is absolutely baffling. There are 2 pattern changes, but I stopped after I found the first 1. The video was relatively good for the first 2 patterns, but I noticed some artifacting:
http://img85.imageshack.us/img85/3254/00001hv.jpg
A = MPEG2Source("VTS_01_1.d2v").trim(0,85361).ColorMatrix().DoubleWeave().PullDown(0,3).Lanczos4Resize(640,480)
B = MPEG2Source("VTS_01_1.d2v").trim(85362,211571).ColorMatrix().DoubleWeave().PullDown(0,3).Lanczos4Resize(640,480)

UnalignedSplice(A,B)
Now I am just going to use TFM() and TDecimate() on the whole film and use DoubleWeave()+PullDown() on the portions that artifact. That is the best compromise I can think of, as no single filter can clean up the whole movie.

[foxyshadis]

I bet tdeint will always do that in those shots, you might have to just use trims or applyrange with it.

I have this video that was IVTC'd and nasty combs left all over it, so I've been developing a filter to get rid of them by vertical blurring affected areas (actually using eedi2 on each field). Is that even a decent solution, or should I just pull from another frame? Maybe I can do both...

The actually implementation is a terribly slow mess of Didée-like luts and blurs and merges, motion compensation (even for anime it helps my masks), but right now it manages to very accurately find problematic areas. If the benches are fairly temporally stable you could add a mask such as:


deint=tdeint(....)
bt=mt_logic(mt_lutxy(orig,back,yexpr="x y - abs").blur(0,1.58).blur(0,1.58) \
,mt_lutxy(orig,fore,yexpr="x y - abs").blur(0,1.58).blur(0,1.58),"min").levels(3,1,30,0,255,coring=false).fity2uv
mt_merge(deint,last,bt,chroma="process")


which is part of what I use. Oh, specific question related... is there a really fast vertical-only blur similar to two blur(0,1.58)?

[Oline 61]

Wow, thats very complicated. If you have the source DVD I recommend you just rerip and encode. I have settled on what I will use for mine.
A = MPEG2Source("VTS_01_1.d2v").ColorMatrix().tfm(d2v="VTS_01_1.d2v").tdecimate().trim(0,8417)
B = MPEG2Source("VTS_01_1.d2v").ColorMatrix().DoubleWeave().PullDown(0,3).trim(8418,8483)
C = MPEG2Source("VTS_01_1.d2v").ColorMatrix().tfm(d2v="VTS_01_1.d2v").tdecimate().trim(8484,169256)
UnalignedSplice(A,B,C).Lanczos4Resize(640,480)
I'm not sure if these is a more efficient way to do this or not, but I'm sticking with it because this movie has been a pain in the ass. Most movies just work great with Force Film in DGIndex, but this one had to be a jerk.

[Mug Funky]

same thing, faster:

MPEG2Source("VTS_01_1.d2v").ColorMatrix()

A = last.tfm(d2v="VTS_01_1.d2v").tdecimate()
B = last.DoubleWeave().PullDown(0,3)

A.trim(0,8417)++B.trim(8418,8483)++A.trim(8484,169256)
Lanczos4Resize(640,480)

[Oline 61]

Thanks :).

[Oline 61]

Now I'm trying to do LoTR "The Fellowship of the Ring". Using force film, there are a few spots of interlaced frames leftover, but using "Honor Pulldown Flags" and TFM->Tdecimate or similar there are nasty nasty artifacts in the film.
I came up with this script to get around it. Source d2v is Force Film, with cropping already done in DGIndex. Planning on doing an anamorphic encode with x264 :).
MPEG2Source("VTS_01_1.24.d2v")
ColorMatrix()
AssumeTFF()
TDeint(mode=0)
#Set SAR in Encoder to 52/45
What do y'all think? It looks good to me.

[Oline 61]

Here is an example of the artifacting. Notice the red areas behind the helmets and stuff. This doesn't appear in the Forced Film DGIndex output, only when Film is not Forced and Telecide_>Decimate or TFM->Tdecimate is used.
http://img382.imageshack.us/img382/7718/00006ws.png

[Oline 61]

I found some places where TDeint caused artifacts on edges that already looked fine so.......
In = MPEG2Source("VTS_01_1.24.d2v").ColorMatrix()
Proc = MPEG2Source("VTS_01_1.24.d2v").ColorMatrix().TDeint(mode=0)

In.Trim(0,175226)++Proc.Trim(175227,175228)++In.Trim(175229,256563)

#Set SAR in Encoder to 52/45
Does anyone have a script that will search for and detect interlaced frames, so I don't have to search through the whole damn movie to find all the spots where Force Film left interlaced frames.


Obviously there is something wrong with the DVD spec if all of the movies exist that are improperly telecined. What idiot decided that all content on a DVD had to be a specific framerate? If I had to choose a framerate 23.976 would make the most sense, right? 29.97 is EVIL.


Can anyone explain to me why TFM/Tdecimate or Telecide/Decimate create there bizarre artifacts, and Force Film leaves telecined frames? When I parse the d2v all the frames show that they are in pattern except for the first 50 or so. Shouldn't that mean that everything will work perfectly? I have completely disable postprocessing in Telecide, so how is it generating these red artifacts that simply do not exist in the source?

I appreciate all the help you can and have given to me.

******[EDIT]******
I am now using this script for the Forced Film D2V and it appears to be working great:
UnProc = MPEG2Source("VTS_01_1.24.d2v").ColorMatrix()
Proc = MPEG2Source("VTS_01_1.24.d2v").ColorMatrix().TDeint(mode=0,order=1,)
ConditionalFilter(UnProc,Proc,UnProc,"IsCombed()","=","true")

#Set SAR in Encoder to 52/45
I am still very confused about Telecide though......
If I read the reference manual I get this:
Here's what Telecide does by default. When it receives a request for a frame it gets access to the next frame and the requested one (called the current frame). Here is what he sees (where N=next, C=current, t=top field, b=bottom field):

Ct Nt
Cb Nb

Telecide makes 2 combinations of frames from the available fields. Here are the combinations:

Nt
Cb

Ct
Cb

Now both of these frames are checked for combing. The least combed frame is the output frame (assuming no pattern guidance). It is that simple.

If pattern guidance is enabled, the decision described above can be overridden by a predicted match based on the perceived pattern surrounding the current frame. The prediction is ignored if the difference between the matches is too large, as that implies a possible change of phase of the pattern, to which Telecide() should re-sync.

Consider now a cycle of frames from a 3:2 pulldown sequence (top field first):
a a b c d
a b c c d

Telecide will produce matched output frames as follows:

a b c c d
a b c c d

It can be seen that a duplicate frame has been produced. Decimate removes the duplicate frame by finding and removing the frame most similar to its predecessor. It is easy to see now why Telecide followed by Decimate produces an inverse 3:2 telecine (IVTC) function.
So, it sounds like Telecide only rearranges the content that is put in. I tested this by using Telecide(order=1,guide=0,post=0) only. It should not generate any colors that were not present in the source, right? WRONG! These little clouds of red around the helmets and swords are not present in the source. In the video of the soldiers marching there is no red at all in the source.
Where does this come from?
http://img359.imageshack.us/img359/3223/untitled9ay.png
Is this a bug in Telecide?

[foxyshadis]

Well, there is the possibility that it's Hybrid, that tends to happen with movies with a lot of CG... maybe. http://www.avisynth.org/VariableFrameRateVideo for information on detecting and dealing with it, other than your own.

The crazy chroma shifting I have no idea, although neuron2 and tritical might be interested in a sample.

Another way you could do it without using a conditional filter is:

cm=UnProc.CombMask().bilinearresize(width/2,height/2).bilinearresize(width,height).mt_expand()
mt_merge(UnProc,Proc,cm,chroma="process")

but your code works about as well (this will get smaller combs but is slower).

edit mod: corrected link :)

[Oline 61]

I don't have all the nessecary filters to run that. I grabbed masktools, but not sure what I need for mt_expand.

I have prepared several scripts for that 29.97 fps d2v, and their outputs.

tfm(d2v="VTS_01_1.30.d2v")
tdecimate(hybrid=1)
http://img221.imageshack.us/img221/4120/00009ys.png
-------------------------------------------------------------------------
Telecide(order=1,guide=1)
Decimate()
http://img207.imageshack.us/img207/3569/00004ie.png
-------------------------------------------------------------------------
tfm(d2v="VTS_01_1.30.d2v")
tdecimate()
http://img52.imageshack.us/img52/9599/00008dk.png
-------------------------------------------------------------------------
DoubleWeave()
Pulldown(0,3)
http://img207.imageshack.us/img207/1007/00006mp.png
-------------------------------------------------------------------------
It appears that the weird chroma shift is unique to Telecide, but even using a dumb pulldown or tfm doesn't correctly IVTC the clip. This is quite odd, because the film does follow a 3:2 pattern, and tfm has done well on other films that follow a 3:2 pattern. What is it about this movie specifically that causes so many problems?

EDIT: Here is one particularly bad frame produced with tfm(d2v="VTS_01_1.30.d2v",display=true)
http://img236.imageshack.us/img236/1661/00008zv.png

[Oline 61]

Well, somehow I got rid of the weird chroma bug caused by using Telecide or TFM. Things I did differently:
1. IEEE Reference IDCT instead of SSE2 MMX
2. Crop after IVTC filter
3. ColorMatrix after IVTC filter
4. ReRipped disc w/ DVD Decrypter, without any Stream Processing....

What a pain.

[Oline 61]

Okay, it's official. Putting crop before the IVTC function in my Script causes the artifacts and chroma shifts. If I crop after the IVTC function, everything is nice and pretty. Thanks for helping out everyone.

[foxyshadis]

I have a question, can you try using "align=true" as the last argument to crop, run before the IVTC? If it works, it means they aren't handling unaligned frames properly.

[Oline 61]

Even with align=true the artifacts still show up whenever crop is before IVTC.

I more closely examined Friday Night Lights in DGIndex and partway through the movie the whole damn field order changes. It goes from TFF to BFF at least once, not sure if it goes back to TFF or not. I guess thats what I get for trying to rip a pan and scan 4:3 version of a widescreen movie. (That wasn't my fault, it was a Christmas present, rest assured I would have bought widescreen :)).

The field order flipping doesn't cause the artifacts in the stadium bleachers though, those are caused by something in Telecide and/or TFM.

[Elic]

Oline 61
> Putting crop before the IVTC function in my Script causes the artifacts and chroma shifts.
Does chroma shifts appear when crop upper side by 2,6,10,etc. llines or by 4,8,12,...? Or it doesn't matter?

[Oline 61]

It doesn't matter, whatever I crop by the chroma shift appears.

I figured out a way to rip Friday Night Lights 100% artifact free. Use Mencoder. Here is a script that rips it to high bitrate MPEG2. I will try with X264 once I make a new compile from CVS. Haven't tried on Windows, but works great on Linux.

[Oline 61]

Okay, I feel like I have made a complete ass of myself by now. I figured out what was happening to the bleachers in Friday Night Lights, and Mug Funky pointed me in the right direction back on the first page. Telecide thinks that the bleachers are combed, so it attempts to post process them and makes an awful mess of it. If I use order=1,guide=1,post=0 the entire movie looks great. I think I am finally done bloating the database with this thread. Thanks for all the help everyone gave me :).