Batch convert

[jriker1]

Not sure if the AVISynth area is the right place for this but is part of my main normal workflow so thought I would post it here.

I have an enormous amount of files I created back in the day from some DVD's I owned so I could shred them and clear out a closet full of DVD's and VHS tapes. I just got back to them and they have a lot of granularity with them. Call it endearing to the content or not I want to clean it up like I do the rest. That process is basically:


Pull in the MKV with an AVISynth script containing: ffmpegsource2("<MKV FILE>", vtrack = -1)
Convert lossless to AVI in VirtualDub.
Open the file in Premiere Pro and use NeatVideo to create a cleaning profile and output to an interim lossless AVI file

This is the part I need help with. So if I have 300 MKV files and want to do something like this for every one of them, how would I automate this? I will accept some level of imperfection by not creating a unique noise profile for each file individually as obvioiusly that would take forever.

Note I do also have NeatVideo for VirtualDub however it has a first frame problem that I have opted not to deal with when running it thru AVISynth as opposed to using it directly in VirtualDub.

Thoughts?

Thanks.

JR

[creaothceann]

I'd write a batch file:

@echo off
for %%f in (*.mkv) do call :process "%%f"
exit /b


:process
:: %1 = MKV file name
echo Processing: %~1
>"%~n1.avs" echo FFMPEGSource2("%~1", vtrack=-1)
avs2avi "%~n1.avs" "%~n1.avi" -q -l {codec file}
del "%~n1.avs"
exit /b
Replace {codec file} with the codec settings file that you can create with AVS2AVI (download (http://moitah.net/download/latest/AVS2AVI.zip)) (see its commandline documentation by starting it in a console). Then put that batch file in the same directory as the MKV files and start it.

[jriker1]

Thanks creaothceann. Very slick. Never used AVS2AVI but will assume there is no negative to it over manually processing thru VirtualDub. So two questions remain for me:

1. When defining the codec type, how do you tell it what settings to use? Like if I use Lagarith how do I tell it to use YUY2 over RGB which is a pulldown option in the settings?
2. How to deal with NeatVideo for cleanup activities across all these pieces of footage. Asked Vlad who makes the product so not sure what he will say.

Thanks.

JR

EDIT: NeatVideo guy said there is no good way to automate the noise profiling thru their tool. That said, This is painful. I'm thinking of years to manually create noise profiles and process these files one at a time, as opposed to weeks of just letting the servers run thru these automatically. if there was an automatic process for the noise reduction part.

[creaothceann]

AVS2AVI accepts these options: http://i.imgur.com/uA17OJo.gif
You'd use switches -s and -e to configure the codec and save these settings to a file.

I don't know about NeatVideo, maybe you can replicate the functionality in Avisynth somehow?

[johnmeyer]

I've never been a big fan of Neatvideo, but maybe I just don't know how to get a good noise profile.

I've done hundreds of hours of VHS restoration and I find that scripts based on MDegrain2 work really well. I've posted various scripts in this forum many times. Using AVISynth will easily let you batch the process, although you'll want to spend a lot of time experimenting with various parameters before you push he button and let things crank because no noise reduction scheme works well with all source material.

One other advantage of doing everything in AVISynth is that you can add additional restoration steps, like reducing chroma noise, something that is a big problem with many VHS captures.

I would also recommend checking things like levels, although I usually do this in my video editing program.

Finally, lots of old video can benefit from restoring the audio. I use iZotope RX for this, which is out of the price range of most people (I do this professionally). If you can afford it, you can reduce hiss and eliminate various types of audio distortion.

[jriker1]

AVS2AVI accepts these options: http://i.imgur.com/uA17OJo.gif
You'd use switches -s and -e to configure the codec and save these settings to a file.

I don't know about NeatVideo, maybe you can replicate the functionality in Avisynth somehow?

Yeah, there are always other noise removers that are built for AVISynth, just think it's a fair statement none are as good as NeatVideo. If someone has any recommendations within AVISynth that would be automatic and provide "good" results, open to that rather than manually going thru hundreds of videos and creating noise profiles.

JR

[johnmeyer]

Yeah, there are always other noise removers that are built for AVISynth, just think it's a fair statement none are as good as NeatVideo. If someone has any recommendations within AVISynth that would be automatic and provide "good" results, open to that rather than manually going thru hundreds of videos and creating noise profiles.
JRI just gave you the answer in my last post of something that should give you good results. We aren't allowed to say that something is best or better, but I think you will be very happy with the results you can get with the approach I described, maybe even happier than you are now with Neat.

[edit] P.S. One thing I CAN say for sure: the MDegrain approach will be MUCH faster than Neat, especially if you use the MT version of AVISynth.

[goorawin]

Yes I do it all the time using Avisynth batch scripter Version1.01 and VirtualDub Batcher II V1.2.8 by Beppe, and it works like a dream. Both programs are very light and run from anywhere. I'm currently using them of Windows8.1.
You may have to change the settings.ini file in Avisynth Batch to accept additional file types but that should be all.
Another option which allows a bit more flexibility in using different scripts in the one batch, and you do not have to set up a special directory.

[jriker1]

Crap. I tried a few options with the footage I have. Basically old DVD footage that wasn't encoded originally at optimal bitrate and is really old content to boot. Tried:

tr = 6 # Temporal radius
lsb = false # 16-bit
thSAD = 200 # denoising strength
blksize = 16 # block size
overlap = 4 # block overlap
super = MSuper ()
multi_vec = MAnalyse (super, multi=true, blksize=blksize, overlap=overlap, delta=tr)
MDegrainN (super, multi_vec, tr, lsb=lsb, thSAD=thSAD, thSAD2=150)


Which I found on the forums, and also

QTGMC(InputType=1, Preset="slow", EZDenoise=2.0)

Neither dented the noise or the compression artifacts. Not that I expect a noise reducer to go after compression artifacts. That said, I also tried NeatVideo creating a profile manually, and it did a lot better job with the content of issue. Of course that's manually creating a noise profile for a single piece of footage.

JR

[johnmeyer]

The settings in the script (for MDegrain) don't look right. I'd recommend starting with the sample code given in the MVTools2 documentation and working from there.

If you can post a small sample of the video, I can take a shot at it. MDegrain should be able to produce a significant, easily seen reduction in the noise. I'm not sure what it will do with compression artifacts. If you (unfortunately) used a low bitrate or lousy encoder back in the day, the problems introduced by that will be much tougher to reduce. By contrast, most analog noise responds pretty well to various noise reduction techniques.

[jriker1]

Thanks for the offer. Original footage of the first of the 300 clips was done using DIVX and don't have any desire to install that codec on my encoding servers. I cut a segment with ffmpegsource2. For reference the original footage as I'm providing lossless to keep the content untouched:

Divx 3 Low
1,421 kbps video
108 Kbps audio

https://www.dropbox.com/s/9v6s16x82poqova/test.avi?dl=0


Encoding changed over the years from DIVX, to XVID, to MP4.

If it's to large let me know. Didn't want to compress it further and wanted to give you at least 4 or 5 minutes of content to review.

Thanks.

JR

[johnmeyer]

OK, I tried a whole bunch of different parameters using MDegrain. I tried scripts with, and without MRecalculate, which sometimes improves denoising.

I did run into one issue. It looks like you may have, at one time in the past, applied some denoising to this material. As a result, there is noise correlation. Therefore, when additional denoising is applied, you end up getting "clumping" of the noise, for lack of a better way to describe it. However, the noise is most definitely reduced by the script below.

Before I give you the link to the sample I created, I'd like to suggest that you be really selective on which things you decide to spend your time improving. I know what it is like to have a big library of video, having invested $$$ in laserdiscs back in the 1990s. I still have them, and at the time, they represented the best quality video you could get. However, DVDs arrived and many movies were transferred again, and the quality was better. Then HD arrive, and movies were transferred one more time, and the quality was better still. And then finally, digital restoration tools matured, and even if the movie wasn't transferred again, the existing HD transfers were treated to things like stabilization, dust removal, and advanced color correction.

If it were me, I'd simply rent or stream the movie, and it will probably look better than anything you can get with your old sources. This is certainly true of my old laserdisc movies. This particular episode that you provided for your sample is available, both as "The Menagerie," as well as in the form of the original pilot that was never broadcast, but was used for that Menagerie episode. I've seen some of the modern transfers, and they are stunning. No way you can get older sources to look like that.


Here is a sample output from the script below, encoded with a lossless codec (Lagarith) so you won't get any additional encoding artifacts.

Denoised Sample (https://www.mediafire.com/?e7bpdayrm4z4p3s)

Here is the script I used. It is nothing fancy, and is more or less what you see in the MVTools2 documentation.

SetMemoryMax(768)

Loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

SetMTMode(5,6)
#Modify this line to point to your video file
source=AVISource("E:\fs.avi").killaudio().AssumeBFF()
SetMTMode(2)

output=MDegrain2F(source,8,4,400,0)

#stackhorizontal(source,output)
return output

#-------------------------------

function MDegrain2F(clip source, int "blksize", int "overlap", int "strength", int "dct")
{
overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct=default(dct,0) # use dct=1 for clip with light flicker
strength=default(strength,400)

super = source.MSuper(pel=2, sharp=1)
backward_vec2 = super.MAnalyse(isb = true, delta = 1, blksize=blksize, overlap=overlap, dct=dct)
forward_vec2 = super.MAnalyse(isb = false, delta = 1, blksize=blksize, overlap=overlap, dct=dct)
backward_vec4 = super.MAnalyse(isb = true, delta = 2, blksize=blksize, overlap=overlap, dct=dct)
forward_vec4 = super.MAnalyse(isb = false, delta = 2, blksize=blksize, overlap=overlap, dct=dct)
MDegrain2(source,super, backward_vec2,forward_vec2,backward_vec4,forward_vec4,thSAD=strength)
}

[jriker1]

Thanks for the help. I think I'll take you up on your suggestion and obtain newer remastered versions. Question though. If you have some 720p BD content that was encoded at 1,882 Kbps is denoisers going to be useful? Or does most of the noise get blurred / altered compressing them down like that?

Friend of mine has the 720p BD versions of this content and sent me a sample to look at but made a MKV file with the video bitrate at 1,882kbps and I can barely see any difference thru any of the denoiser techniques from the
"original" file he sent me. Oddly I am used to dealing with original source digital content and the noise in there is more prominent and really easy to remove. Perhaps will buy the boxsets that are out now. Don't think renting is an option since they came out a long time ago.

JR

[johnmeyer]

1,882 kbps is an extraordinarily low bitrate for encoding HD content, even if it is 720p rather than 1080i or 1080p. If the content was not pristine prior to compressing to that low bitrate, then I would expect to see encoding artifacts because too many bits will be "used up" trying to chase all those random dots around the screen.

Your question is interesting, at least in an academic way: does a low bitrate effectively filter noise ("does most of the noise get blurred / altered")? While I've never tested this, I think the answer is mostly no.

Having said that, a lot depends on the nature of the noise. One reason there are so many denoising techniques is that there are many different types of noise. Most really clean sources from movie film contain film grain and this form of "noise" has quite a different character than the noise you find on VHS video. And both these types of noise are different than what you find from camera video sensors.

I got my start encoding video doing encodes to VCD, SVCD and XVCD, and I learned a lot about doing really low bitrate encodes. That's where I learned the power of first doing IVTC so that, for movie sources, you are always encoding individual frames, without the pulldown fields. Once you have a really clean 24 fps progressive source, you can get pretty darn good results with some remarkably low bitrates. If you then use a really good encoder that can budget bits by doing multiple passes, you can dramatically improve that low bitrate result.

The other thing I learned, as DVD blanks got cheap, and then hard drives (and now memory sticks) became cheap is that, while you can get good encodes at low bitrates, high bitrate encodes sure look a lot better. Put another way, I learned that encoding at a low bitrate is not worth the effort. Instead, choose a decent bitrate and get a top-quality encode.

My grandfather, who died back in 1957, used to say: "the quality is remembered long after the price is forgotten." Pay a little extra money for more storage, and encode at a decent bitrate. Every time you watch the movie, you'll remember the quality, and forget that you paid a few hundred dollars extra for the storage.

An finally, take a moment to think about the future of video. The era of building big video libraries at home is over. The future is streaming (for many, it's here now). In ten years, you will be able to get most movies and TV shows on demand, with a quality that exceeds anything you can create with these compromised bitrate encodes. Right now, in the background, I'm watching one of the backwater basketball games on ESPN3, their streaming service. I'm also switching to their main feed, broadcast on cable. The ESPN streaming has virtually identical spatial quality, with very few artifacts, although it IS degraded to 24p. For material that is already 24p (most movies), this compromise doesn't exist, and the result looks pretty darned good.

[creaothceann]

Once you have a really clean 24 fps progressive source, you can get pretty darn good results with some remarkably low bitrates. If you then use a really good encoder that can budget bits by doing multiple passes, you can dramatically improve that low bitrate result.

The other thing I learned, as DVD blanks got cheap, and then hard drives (and now memory sticks) became cheap is that, while you can get good encodes at low bitrates, high bitrate encodes sure look a lot better. Put another way, I learned that encoding at a low bitrate is not worth the effort. Instead, choose a decent bitrate and get a top-quality encode.

Right; in technical terms: use h.264's CRF (constant rate factor) instead of ABR (average bitrate) or multipass mode with a target bitrate. (Well, use the latter if you must fit the result onto a DVD or even a CD.) And forget that the CQP (constant quantizer) mode exists.