Need serious help with poor video

[aNToK]

Ok, hope I'm doing this right. I have a (supposedly) retail dvd set that I'm trying to clean up and add hardsubs to for viewing.

I'm somewhat familiar with avisynth and have used it for quite a while, but this vid has me pulling my hair out and ready to start over. Hopefully some kind soul(s) can help me clean this thing up before I use the disks for target practice.

Following are 2 clips, still interlaced but compressed with divx 6.8 to keep the filesizes low enough to upload. Quality varies widely from scene to scene, with some very blurry, some with major interlacing, and some that seem almost progressive. Help!!

I've currently been using guavacomb, peachsmoother, and yadif, followed by a script using mvoverlap with degrainmedian and fft3d and sharpened with limitedsharpenfaster. Files are watchable, but definitely nowhere near decent quality.

http://rapidshare.com/files/183215297/rawclip2.avi

http://rapidshare.com/files/183218696/rawclip3.avi


Edit: Hmm.. don't seem to have the fileden thing figured out. Uploading clips to rapidshare

[thetoof]

We need an unprocessed sample - DGIndex, select a section, save and demux video, upload the resulting .m2v to mediafire.com

[aNToK]

mediafire seems to have been down since last night for maintenance. I can up to rapidshare for the time being, hope that's OK. I selected a 1:20 section that's about 75 megs. Is that too big?

[neuron2]

Just do it! You don't have to ask permission.

[aNToK]

Heh, cool. Here's the link to the file.

http://rapidshare.com/files/183245125/sample.demuxed.m2v

[setarip_old]

@aNToK

Hi!I have a (supposedly) retail dvd setWhy do you go out of your way to say "supposedly"?

[aNToK]

@aNToK

Hi!Why do you go out of your way to say "supposedly"?

I say supposedly mostly due to the quality of the set. My niche is East Asian drama series, and for all the ones of good quality (YA Entertainment vids, etc), I have several that end up being poor transfers from Indonesia or China, and some that were apparently ripped from tv, shabbily encoded, and put out there as a real "retail" product.

Check the clip and you'll see what I mean. This one and the other one I'm currently losing sleep trying to fix were both legit retail purchases.

A better word to use would probably have been "Professional".

This particular series "Punch!" I've got 3 different sets of, and this one's the best of the lot. A "widescreen" version I picked up turned out to be the same source video with the top and bottom hacked off to get it close to 16:9.

The other one I'm currently trying to clean up is a 70 episode, 18 disk set of another Korean drama series called "Terms of Endearment". First retail DVD set I've purchased with that comes up as progressive instead of interlaced in dgindex. Be nice if the supplier had thought about some proper deinterlacing before mastering (or leaving it interlaced? No clue on that one...), but several vids from this time period from there are almost impossible to find in any format.

So Professional would be a better term. Sorry about the confusion.

[Nightshiver]

MPEG2Source("D:\sample.demuxed.d2v", cpu=0)
animeivtc(mode=2, aa=0, iregion="PAL", oregion="PAL")
source = last
vectors = source.MVAnalyseMulti(refframes=3, pel=2, overlap=4, sharp=1, idx=1)
source.MVDegrainMulti(vectors,thSAD=700,idx=1)
LimitedSharpenFaster(ss_x=1.7, ss_y=1.7, smode=3, strength=170, overshoot=1, undershoot=1)
Tweak(cont=1.0,sat=1.1,bright=0, hue=0)

Get's rid of the ilacing and other things except the blocking. You have some, so tweak the "cpu" parameter to meet your needs.
cpu=2 >>light deblocking
cpu=4 >>medium deblocking
cpu=6 >>heavy deblocking

[aNToK]

@Nightshiver: Thanks for the reply! do you mean adjust the cpu=0 entry in line 1 for deblocking?

I'm running the script atm, is it normal to run at a little under a frame per second on a 4 core cpu?

MPEG2Source("F:\sample.demuxed.d2v", cpu=4)
animeivtc(mode=2, aa=0, iregion="PAL", oregion="PAL")
source = last
vectors = source.MVAnalyseMulti(refframes=3, pel=2, overlap=4, sharp=1, idx=1)
source.MVDegrainMulti(vectors,thSAD=700,idx=1)
LimitedSharpenFaster(ss_x=1.7, ss_y=1.7, smode=3, strength=170, overshoot=1, undershoot=1)
Tweak(cont=1.0,sat=1.1,bright=0, hue=0)

[neuron2]

@Nightshiver: Thanks for the reply! do you mean adjust the cpu=0 entry in line 1 for deblocking? Yes, of course.

I'm running the script atm, is it normal to run at a little under a frame per second on a 4 core cpu? When you run super slow filters like that, yes. Ask yourself if you really need that.

[Nightshiver]

animeivtc is very slow but gives great results, and I also have a 4 core cpu, but that's just the way it is. You can help the speed by doing a rendering pass first.

[aNToK]

Any hints on how to do a rendering pass? Though frankly, if this cleans up as much as it looks like it will, it can run for a day or two to finish for all I care. Been screwing with this video set off and on for over a year.... Too picky and not enough knowledge to do the job right myself.

btw, I normally leave the filtered video in uncompressed rgb until I burn in my subs and stuff. (eventually compress to a manageable size with megui). Are there any advantages to compressing with xvid or divx while filtering? Space is the last thing I have an issue with on this system.

[Sagekilla]

Render pass generally means creating a lossless version of your script. That is, load up the script into VirtualDub, let it save as a huffyuv/lagarith compressed avi, and then feed that into whatever is encoding your source.

There's another way that actually writes a video file as the script runs, so you can run your script and have it create a lossless file and do your first pass, then have the lossless feed into the second pass. Don't remember how that's done though.

[aNToK]

Would that be faster than just loading the script in vdubmod and saving it uncompressed?

[Nightshiver]

That's what he meant. Rendering pass = uncompressed avi

[thetoof]

Actually, not necessarily. By using huffyuv or lagarith lossless compression, you can have the exact same quality with 1/4 of the disk space.
video-compression-lagarith (YV12 mode)
video-fast recompress
file-save as avi

I'd be interested in knowing more about the second alternative Sagekilla mentionned, as I had never heard about it before.

[2Bdecided]

It's an option, labelled "render" or similar, in some 2-pass encoders e.g. HCenc.

Nothing magic - just lets you walk away from the machine and get a 2-pass encode without running the script or waiting for a lossless render in Vdub to finish before throwing that at the 2-pass encoder.

Cheers,
David.

[aNToK]

OK, ran Nightshiver's script on different parts of the vid, adjusting the cpu= value a few times. Cleanup's nothing short of amazing to me. I did find two things common to all of the samples, not sure if they can be corrected or what to tweak. First, the motion appears a little jerky, like it might be dropping frames or something, and second, I always get a warning at the end about a locked thread error. 3-5 warnings, always reported in the same thread (different thread for each encode). Doesn't seem to affect the vid as far as I can see, though.
Any thoughts?

[thetoof]

Thanks David for the info.

1 - You could load mpeg2source("d2v").bob() to see which frames are duplicates and which should be kept to find a pattern.
2 - A short sample of the specific section where it becomes jerky would be good. If it's already in the one uploaded, at what time?

[aNToK]

Thanks David for the info.

1 - You could load mpeg2source("d2v").bob() to see which frames are duplicates and which should be kept to find a pattern.
2 - A short sample of the specific section where it becomes jerky would be good. If it's already in the one uploaded, at what time?


OK, here's a quickie sample showing the jerkiness. I ran the bob script on the original, but I didn't see any repeated frames, though I don't really know what to look for.

Here's a link to a quickie sample. I used the Logarith codec you mentioned with fast recompress. Looks identical to the original one.

http://www.mediafire.com/?sharekey=80d5d5d1b27350c591b20cc0d07ba4d215616e25c143309e

Lastly, here is the error code I get from vdubmod with this script:

dub: I/O thread has not cycled for ten seconds --possible livelock. (Thread location: 7C90E4F4)

repeated 4 times with every encode.

Another silly question: I use vdubmod almost exclusively for running avisynth scripts. are there any better options or have I been on the right path with it? I do sometimes use an msu filter or two.

[Nightshiver]

I generally use MeGUI for encoding things. From the new sample and the jerkyness, you might have a hybrid source and want to do a VFR encode.

[aNToK]

Hmm.... I remember that using the avisynth script tool in megui for this disk detected it as mixed and selected yadiff as the deinterlacer. Most other ones it selected tkerneldeint, etc.

Do you mean use one of the methods like here for VFR http://avisynth.org/mediawiki/VFR ?

Haven't had to deal with much as far as hybrid sources, and the more I'm on this forum, the more I realize how much I have to learn.....

edit: I use megui for my final encodes as well, but since I'm adding subs and stuff, I run them uncompressed for the filtering run and do my cropping, etc. in megui on the filtered file when I encode to mp4. Always open to different and better ways of doing things.

[Nightshiver]

Don't trust the autodetect function in MeGUI for everything, it's more accurate to use your eyes. Are you hardsubbing or softsubbing? If you softsub, then you don't extra files/steps. If you hardsubbing, you still don't as you can simply use the TextSub() function in avisynth to burn the subs in. To the link you just gave, yes, that is VFR. A little more on softsubbing: Use whatever program you like to do subs (ssa, aegisub) and then save the sub files. Encode your video + audio and then use mkvmerge (if you are using the mkv container, which I highly recommend you do) to put in the softsubs, audio and video.

Edit: even more. Looked at your post again and saw that you are using mp4 for the container. I have nothing against mp4, but mkv is much more versatile and would be much easier to do VFR and softsubbing with it.

[aNToK]

On this particular set, I'm hardsubbing. Most of my other stuff is soft-subs. I plan on going all avisynth sooner rather than later, and textsubs's what I've been using in vdubmod anyway.

Would I use one of the methods in the link I gave after filtering it or would I be replacing animeivtc with it? Little confused, as I'm not sure which function does what on the script.

[Nightshiver]

animeivtc can do vfr itself. You would use:
mode=5,pass=1 #load in vdub, file>>run video analysis pass, wait until it finishes. It will generate some text files.
Then replace the above line by animeivtc(mode=5,pass=2) and reload in vdub

For more info check out this thread:
http://forum.doom9.org/showthread.php?t=141433

An optional approach would be to use MeGUI. Load the script (of course using mode=5, pass=1) and then press "Queue Analysis Pass".

[aNToK]

Ah, I get it a little. Running this in Virtualdub. VdubMod doesn't seem to have an analysis pass function that I could find. So after the 3 hours this is expected to run, then I should change pass=1 to pass=2 and add in the bottom part of the script with the mvdegrain, limitedsharpen, etc for the final pass. Is that about right? I seel little text files popping up in the folder.

On plugins, are we supposed to copy all of the .h, .ccp, etc. text files into the avisynth plugin folder, or simply the .dll's and .avs/avsi ones? I just have the latter in mine.

This is running:
MPEG2Source("F:\PUNCH! WORKSPACE\Ep 4\VTS_01_1.d2v", cpu=2)
animeivtc(mode=5, pass=1, aa=0, iregion="PAL", oregion="PAL")

[Nightshiver]

Here's what you do:

MPEG2Source("D:\sample.demuxed.d2v", cpu=4)
animeivtc(mode=5, pass=1, aa=0, iregion="PAL", oregion="PAL")


Load that script into vdub and then go to File>>Run video analysis pass. Once done, change the pass=1 to pass=2 and reload the script into vdub using a lossless codec, like huffyuv or lagarith.
I don't know what you mean by copyig .h or .ccp files to the plugin's folder. Are you talking about the requirements for animeivtc?

[aNToK]

I was just talking about the text files that come with the plugins when you download them. Had to download/update a bunch of them for animeivtc. I tried out the lagarith codec earlier. Like the results.

Still not clear on how the cpu= parameter helps with deblocking. I thought that it was just to set how many threads were being accessed.

[Nightshiver]

no, cpu= deals with deblocking, not with threads. And no, you only move the .dll's and avsi scripts to the plugin's folder.

[aNToK]

That tip will come in real handy on my other little project. Thank you.

Now if this first pass would just finish...

[aNToK]

Hmm... Running a quick analysis in megui gives me that cute little cpu=4 line. The full script it gives reads like this:

DGDecode_mpeg2source("F:\PUNCH! WORKSPACE\Ep 4\ep4.d2v",cpu=4,info=3)
ColorMatrix(hints=true,interlaced=true)


I'm wondering what the info=3 parameter does. It seems to be related to ColorMatrix, as it disappears if I uncheck the mpeg color correction box.

[neuron2]

I'm wondering what the info=3 parameter does. You can find out by reading the DGDecode users manual.

RTFM!

[thetoof]

Check DGDecode manual : info: 0 to 3 (default: 0)

Debug Information.
- 0: Do not generate debug information
- 1: Overlay debug information on the video
- 2: Output debug information via OutputDebugString()
- 3: Output hints in the video (as defined in utilities.cpp/utilities.h)

[saint-francis]

MeGUI sets info=3 to pass along information that helps ColorMatrix make wise decisions.

ColorMatrix(hints=true,interlaced=true)

- 3: Output hints in the video (as defined in utilities.cpp/utilities.h)

[aNToK]

@neuron2: Sorry to be so annoying. I've been on the other side of noobs who don't know anything asking questions that are clearly spelled out elsewhere and I tend to respond similarly.
I'm building and reading a list of tutorials and manuals for the things I'm using as I go along, but I'm a bit overwhelmed at the moment. Just trying to figure out the basics of what does what. I'll be much less of a pain as I go along.

[thetoof]

Wooooohooow! Just had the opportunity to look at your sample

1 - AnimeIVTC(mode=2) in its current state is meant to remove field-blending
2 - By doing a simple bob(), you can see that there is motion in every frame, so your source is interlaced (NOT field blended)
3 - There are some crazy blends that are not restorable in your clip (when the girl is running... something like 4-5 blends in a row with varying weight) because your source seem to be a mix of interlaced and... something else with a LOT of blends that you should keep imo since anything would do more damage than good
4 - AnimeIVTC(mode=5) = field matching + vfr decimation to 23.976 & 29.97fps........ you have interlaced PAL (25i), so there is no way this solution is appropriate for you.

So, you need a deinterlacer, not a field matcher (or field unblender)
I personally like the output of tempgaussmc_beta1mod for your clip, but you can use whatever you want.
About the very slow speed of the script... tgmc already uses mvdegrain, so if you absolutely want to use mvdegrain3 (uses 7 frames for motion compensated degraining ("cleaning")), you can simply set tr2=3 in tgmc and remove the mvxxxmulti lines.

[Nightshiver]

Sorry then. I thought my mode=2 line would have worked, but then you said there were jerks. So then I though of VFR, and according to thetoof, that isn't going to work either. Also, thetoof, he was only using mvdegrain because that was part of the script I gave him, but since tgmc has mvdegrain, he can remove it if he so wishes.

[aNToK]

Hmm... I think this baseball cap's all that's keeping my head from exploding trying to figure these things out. Lesse where I'm at:

Using the script Nightshiver posted with mode=2 looks the best to me frame-by-frame at the moment. Only issue is the jerkiness upon playback. If I could get rid of that, I'd be happy!

Tried it with mode=5 and the results were significantly less sharp and noise removal was less effective.

Ran TempGaussMC with default values and it seemed to be kind of in between the two (also 50fps resulting vid). Is there a tutorial on tweaking it out and what parameters do what? I searched and dug through some threads dealing with it, but I'm more lost now than before I began...

Not ashamed to admit that much of the stuff is beyond me atm.

[aNToK]

Looking at different parts of this vid and some of the other episodes, I swear that they used completely different equipment, settings, you name it. and just stitched them together any quick way that worked. I've seen similar in a few other Korean dramas of this time period. Some scenes that just need cleaning and tweaking, others so dark and/or blurry that it'd be laughable if it weren't so frustrating.

[thetoof]

Also, thetoof, he was only using mvdegrain because that was part of the script I gave him, but since tgmc has mvdegrain, he can remove it if he so wishes.Yeah, precision=3 for mode=2 == tgmc. So he can remove it and use another denoiser if he really wants to.

Using the script Nightshiver posted with mode=2 looks the best to me frame-by-frame at the moment. Only issue is the jerkiness upon playback. If I could get rid of that, I'd be happy!
Ran TempGaussMC with default values and it seemed to be kind of in between the two (also 50fps resulting vid). Is there a tutorial on tweaking it out and what parameters do what? I searched and dug through some threads dealing with it, but I'm more lost now than before I began... Why it looked cleaner was because of the additional degraining, but basically mode=2 is tgmc + vinverse + deblending and decimation, so the only visual difference was caused by ++++ degraining. Many options available : dfttest, fft3dfilter and MANY others.

Looking at different parts of this vid and some of the other episodes, I swear that they used completely different equipment, settings, you name it. and just stitched them together any quick way that worked. I've seen similar in a few other Korean dramas of this time period. Some scenes that just need cleaning and tweaking, others so dark and/or blurry that it'd be laughable if it weren't so frustrating.Have fun ;)
a=source.trim(values).appropriate filtering for that part
b=source.trim(values).appropriate filtering for that part
etc
a+b+.....

[kemuri-_9]

Have fun ;)
a=source.trim(values).appropriate filtering for that part
b=source.trim(values).appropriate filtering for that part
etc
a+b+.....

ApplyRange() could probably make that simpler...

[thetoof]

Sure, depends on your preferences.

[aNToK]

Well, after playing with this and trying to figure out what I could adjust without hosing the whole thing, here's a sample of what the vid looks like atm with the following script. I left the colormatrix info in there and dialled down the yellow and green with the hsv adjust in vdub. I'd like to have more thorough cleaning, but I couldn't figure out how to do what thetoof mentioned here:
"if you absolutely want to use mvdegrain3 (uses 7 frames for motion compensated degraining ("cleaning")), you can simply set tr2=3 in tgmc and remove the mvxxxmulti lines."

Not familiar enough with things yet, I guess. Not sure if adding the limitedsharpen was correct, but it did look sharper and didn't seem to reintroduce much noise. I used the framerate select in megui in the last stage to drop it down to 25fps from 50. I assume that's normal. Any tips on how to improve it further with my limited understanding of avisynth usage?

So far the comments here have been very helpful in making this thing watchable!

DGDecode_mpeg2source("F:\PUNCH! WORKSPACE\Ep 4\ep4.d2v",cpu=4, info=3)
ColorMatrix(hints=true,interlaced=true)
TempGaussMC_beta1mod(tr2=3)
source = last
LimitedSharpenFaster(ss_x=1.7, ss_y=1.7, smode=3, strength=100, overshoot=1, undershoot=1)


Clip is here:

http://www.mediafire.com/?idftiwzdxwr

[thetoof]

What I meant is that when you set tr2=3 in tgmc (like you did), it "activates" this part of the script:stage1b.MDegrain3(stage1bsuper,bvec1,fvec1,bvec2,fvec2,bvec3,fvec3,thSAD=_thSAD2,thSCD1=_SCth1,thSCD2=_SCth2)
which is very similar to vectors = source.MVAnalyseMulti(refframes=3, pel=2, overlap=4, sharp=1, idx=1)
source.MVDegrainMulti(vectors,thSAD=700,idx=1)
since refframes=3 is the same as using mvdegrain3 (simply different versions of the plugin with different syntax)
In other words, some mo-comped degraining is done in the bobbing (deinterlacing) process.
If you want to clean it more, check the denoiser section of the external plugins list @ avisynth.org

[aNToK]

Ah, I see! Thanks!

I added fft3d filter bumped up limitedsharpenfaster a bit. Encoding a clip right now. Looks promising....

Funny, adding fft3d seems to have bumped up the speed of the whole thing significantly. Probably more due to using a different clip I'm guessing.