Maximizing cpu on x264CLI with core i7

[magnatique]

Hey Guys,

I`ve been trying to read the board all day for a solution but I can't quite put my finger on what I need.

I am trying to get an overall 90-100% cpu usage on my i7-920 command line encode using x264. launching multiple .bat files is an option, but right now all I can seem to be able to achieve is roughly 30% usage.

I tried running 4 batches at once, but everything went down to 12% cpu and 1-2fps encodes only after a bit.

I am using a 32bit version of x264 to work well with my avisynth, but I tested lordmulder`s benchmark tool and saw only a few % gain on a testfile... x64 wasn't maxing cpu.


Here is an example of my command line.

"Z:\_sharedapps\wavi\wavi.exe" "Z:\current_rhoz\mp4-ipod-etc\1280x720\file.avs" - | "C:\Program Files\megui\neroAacEnc.exe" -ignorelength -br 128000 -if - -of "v:\temp3\file.m4a"
"C:\Program Files\megui\tools\x264\x264.exe" --profile high --level 4.1 --preset fast --pass 1 --bitrate 1500 --stats "v:\temp3\file.stats" --thread-input --threads auto --deblock -1:-1 --b-adapt 2 --direct auto --ref 1 --vbv-bufsize 50000 --vbv-maxrate 50000 --no-mbtree --me dia --subme 2 --partitions none --trellis 0 --no-mixed-refs --output NUL "Z:\current_rhoz\mp4-ipod-etc\1280x720\file.avs"
"C:\Program Files\megui\tools\x264\x264.exe" --profile high --level 4.1 --preset fast --pass 2 --bitrate 1500 --stats "v:\temp3\file.stats" --thread-input --threads auto --deblock -1:-1 --b-adapt 2 --direct auto --ref 1 --vbv-bufsize 50000 --vbv-maxrate 50000 --no-mbtree --subme 2 --partitions all --trellis 0 --no-mixed-refs --aud --output "v:\temp3\file.264" "Z:\current_rhoz\mp4-ipod-etc\1280x720\file.avs"
del /Q "v:\encoded\file-720p.mp4"
"C:\Program Files\megui\tools\mp4box\mp4box.exe" -add "v:\temp3\file.264:fps=29.97" -add "v:\temp3\file.m4a#trackID=1" -tmp "v:\encoded" -new "v:\encoded\file-720p.mp4"
del /Q v:\temp3\*.*




if this is fine, then I am thinking perhaps I am limited not by X264 CLI but by avisynth/decoder?

here is a sample code of my avs.


Import( "\\fileserver\newz\_sharedapps\hdvars\lpt01-bak.avs") # sets proper network path
video=directshowsource(path +"content\file.mp4")
audio=directshowsource(path +"content\file-GOOD.aif",audio=true)
source=audiodub(video,audio)
source=trim(source,232,36869)
source=tomsmocomp(source,1,5,1)
source=lanczosresize(source,640,360)
source=colorYuV(source,levels="Tv->PC")
source=tweak(source, -6, 1.4, source=levels(source, 0, 1.1, 255, 0, 255)


output=MDegrain2i2(source,4,0)
output=limitedsharpenfaster(output,ss_x= 1.3,ss_y= 1.3, strength=150)
return output.converttoyv12()

function MDegrain2i2(clip source, int "overlap", int "dct")
{
overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct=default(dct,0) # use dct=1 for clip with light flicker
super = MSuper(source,pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=overlap, dct=dct)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=overlap, dct=dct)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=overlap, dct=dct)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=overlap, dct=dct)
source.MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
}




for decoding, I used klite mega codec pack setup with default stuff.

for this instance, the source file is a x264 mp4 file that was encoded using :

"C:\Program Files (x86)\megui\tools\x264\x264.exe" --profile high --level 4.1 --preset fast --tune fastdecode --crf 16.0 --thread-input --deblock -1:-1 --keyint 30 --min-keyint 2 --b-adapt 2 --direct auto --ipratio 1.1 --pbratio 1.1 --vbv-bufsize 30000 --vbv-maxrate 40000 --no-mbtree --merange 12 --subme 2 --partitions all --trellis 0 --mvrange 511 --nal-hrd --sar 1:1 --aud --slices 4 --output "Z:\current_rhoz\mp4-ipod-etc\640x360\file.mp4" "Z:\current_rhoz\mp4-ipod-etc\640x360\file.avs"



Please if possible, guide me in the proper direction.

If it is a prob from the decoder. let me know what to use.

If it is from avisynth, please guide me with some info on how to properly make the script with multiple threads and i7 friendly... or some more effective methods of removing grain and etc... than those I use in my script at the moment.

Also, while this script is resized at 640x360, it is also being encoded for 720p and 1080p ;)

[Blue_MiSfit]

Dude.

Your script is HEAVY :) LSF and MDegrain2 run quite slowly when single threaded (as default)

You need to either use MT or SetMTMode to multithread the script, or render it out to a lossless / near lossless mezzanine first with x264 (think crf 12) first! In fact, if you're going to run multiple encodes from a single filtered source, I'd suggest doing both methods - multithreading your script at 1080p and saving it at crf12, then using this as the source for a simple ffvideosource encode, scaled to your target file size.

Also, try something other than tomsmocomp for deinterlacing. It's pretty durn old - not to say it's not perfectly fine if you're happy with the results. Check out YADIF.

MiSfit

[magnatique]

I'll have a look at yadif thanks :)

I'm already making my sources as crf16 as I don't plan on going higher than bluray down the road with those sources, so I wanted to have an x264 version that I can just drop into a BD project... but you are right, I should color correct/apply degrain onto the source when doing that initial encode and then be done with it.

I'll give it a shot with MT/setMTMode. what is the current stable/bet version? I think I saw a bunch of versions on the boards and couldnt find an official thread. 2.8?

As for LSF, wouldn't I have to apply sharpen every time I resize down though for optimal results? or would there be a lighter version you would recommend that could do the job for a simple resize for different encode formats?

Degrain is mostly on 1080p HD sources , so I don't need to take a sandblaster on those sources ;) any other good filters you would recommend?

I know I ask on pretty much all plugins I use, but I figured I would revamp my methods if someone who knows better than I has anything to recommend hehe ;-)


But as for maximizing CPU, that script if I load it in other encoders such as rhozet carbon coder and encode say Flash/mov/wmv at once, I'd launch 5 concurent sources per profile (profile having 4xflash, 2xwmv 2xmov) it will max out the CPU at 100%... so technically the avs being slow doesn't come into play. As for MP4, is there known issues with running say 5-10 concurrent encodes? will it max the cpu, or they will not work too well when duplicating the work on 5-10 CMD's?

when I tried launching 4, I would barely get 10-12% CPU... would removing thread-input force avisynth to be sent to multiple threads and increase cpu usage? any tips there is appreciated.

[Blue_MiSfit]

If you encode at CRF16, you won't be able to drop it into a BD project! You'll have to generate BD specific streams with NAL-HRD, AUDs, short GOPs, and VBV. Unrestricted CRF16 will certainly fail muxing!

CRF16 is quite high quality, but not quite good enough for a mezz IMO. 1080p CRF12 usually comes out under 50gb for a full movie in my case, and sometimes a lot smaller. But, this may be overkill. Perhaps we can compromise on CRF14 ;)

The real test would be to make a mezz at CRF16 and then use this as the source for your other transcodes, and compare this to a CRF12 mezz. If the results are similar enough for you to be happy - then that's the end of that!

The version of MT Avisynth that I use is SEt's 2.5.8 build, in the AviSynth Development thread: http://forum.doom9.org/showthread.php?t=148117

Make sure to install avisynth.dll in your syswow64 folder (assuming 64 bit OS), and MT.dll in your plugins folder. Look at the MVTools docs for examples on using MT(...) to speed MDegrain2. The same process will work fine with LSF.

I think degraining / sharpening your mezz is the way to go. Sharpening after resize shouldn't produce better results. If anything, it would look worse IMO, since sharpening is usually best done at the highest possible resolution, then downscaled.

If you just need to clean up a little blocking and chroma noise, have a peek at fft3dgpu. It's awesome and fast.

I'm not sure why multiple x264 processes have issues versus Rhozet's queue. I've run 4 1080p encodes in parallel using x264 CLI without any issues, using up to 6GB of RAM.

~MiSfit

[magnatique]

as I'm trying to tweak how I run the script with MT() I am a bit at a loss...

this works :
levels(source, 0, 1.1, 255, 0, 255)
output=mt("MDegrain2i2(4,0)",8)

but I can't seem to find a way to make it work when storing it in variables/clips...

like :

output=levels(source, 0, 1.1, 255, 0, 255)
output=mt(output,"MDegrain2i2(output,4,0)",8)


it says mt function changed the height/width

[magnatique]

If you encode at CRF16, you won't be able to drop it into a BD project! You'll have to generate BD specific streams with NAL-HRD, AUDs, short GOPs, and VBV. Unrestricted CRF16 will certainly fail muxing!


~MiSfit



I'll try installing the MT you sent me to and test some more with that ;)

as for the BD compatible version, I had dug up a bunch of threads when I was tweaking my crf 16 and the size/quality difference was good enough for what I had in mind. I do add nal-hrd/slices 4 etc... in there and I followed some threads about being BD compatible. I have put my encode line in my first thread too, if you notice anything that would make it unappropriate, let me know.

Also, if you have a quick link to something/some tool I could use to test if it is Blu-ray compatible, that'd be great. I haven't searched yet, but I'll be doing that in a min after I am done with the reply. ... but if you have a link ready, I'll take it hehe ;-)
Thanks for your help btw ;)

[magnatique]

ok some heads up.

I am now pretty much using a good chunk of the cpu and it encodes MUCH faster.

the main gain was with dropping directshowsource for ffvideosource(). I went from roughly 8FPS to 37-40FPS just by doing that.

I'm playing around with MT() and setmpmode as well, see what can be achieved.

My 15 min vid is finishing up, I will review and make sure there are no bugs but it's promising.

without any degrain, I was getting 110-115FPS using ffvideosource over directshowsource


-----------------

On a sidenote, is there much quality loss with re-encoding an already encoded x264 with CRF 16 to another cRF 16 with the exact same settings? In taking your advice and applying the correction settings to the original encode, I was thinking of starting a batch re-encode of my actual mp4 crf16 sources and applying the degrain/sharpen/color corrects once and for all to that source... so could I just run it on the current mp4 source, or the quality loss is too big and it needs to be done off the original Vegas projects for example?

[poisondeathray]

the main gain was with dropping directshowsource for ffvideosource(). I went from roughly 8FPS to 37-40FPS just by doing that.

That's huge, but I think ffdshow and ffms2 use the same decoder based on libavcodec, so I don't think there should be that big of a difference

Can you clarify which decoder were you using for directshowsource (you said k-lite mega codec pack and I think it uses ffdshow by default)? If it was default ffdshow as you suggested earlier, did you set it to "ffmpeg-mt" ?

[LoRd_MuldeR]

Maybe he uses an outdated ffdshow. Or ffdshow was configured to use the "normal" libavcodec, instead of the "ffmpeg-MT" decoder. Or the number of decoding threads wasn't set appropriately. Or DirectShowSource didn't even select ffdshow as decoder. He doesn't mention that. Does he? Maybe I missed something.

Probably he could get even faster decoding speed with DirectShowSource() or DSS2() in combination with DivX' H.264 Decoder. At least that one is the fastest Freeware H.264 decoder on my system so far.

[magnatique]

well, I overwrote the ffdshow.ax file with the one from the thread misfit linked above.

as for directshowsource, it was default I believe. if I open ffdshow settings from klike codec pack, it is setc for AVC/h264 to libavcodec.

I think I am getting some pretty decent results with fft3dgpu and the fps I obtain is pretty good at 80 or so FPS Vs 40 with mdegrain2. I just don`t have enough knowledge of that tool to choose the proper settings. Would anyone have an input on what I should be using with a 1080p source? straight from camera source.

[Blue_MiSfit]

Oh right. I forgot about directshow :)


function MDegrain2i2(clip source, int "overlap", int "dct")
{
overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct=default(dct,0) # use dct=1 for clip with light flicker
MT("""
super = MSuper(source,pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=overlap, dct=dct)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=overlap, dct=dct)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=overlap, dct=dct)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=overlap, dct=dct)
source.MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
""",8,4)
}


Try that. And no MT in your MDegrain2i2 call. 8,4 means 8 threads and 4 pixels of overlap

~MiSfit

[magnatique]

Maybe he uses an outdated ffdshow. Or ffdshow was configured to use the "normal" libavcodec, instead of the "ffmpeg-MT" decoder. Or the number of decoding threads wasn't set appropriately. Or DirectShowSource didn't even select ffdshow as decoder. He doesn't mention that. Does he? Maybe I missed something.

Probably he could get even faster decoding speed with DirectShowSource() or DSS2() in combination with DivX' H.264 Decoder. At least that one is the fastest Freeware H.264 decoder on my system so far.


All of the above ? ;-)

to be honnest, I never thought before I tried it that the videosource was the bottleneck hehe.

version was 3065.

let me give it a shot with directshowsource and ffdshow-MT or what you suggested.

[LoRd_MuldeR]

as for directshowsource, it was default I believe.

Unfortunately it cannot be told easily which H.264 decoder is the "default" on your system. DirectShow picks the decoder with the highest "merit" value.

if I open ffdshow settings from klike codec pack, it is setc for AVC/h264 to libavcodec.

It should be set to "ffmpeg-mt", not "libavcodec". Also "Number of decoding threads" (on "Decoding options" page) should be set appropriately.

http://www.xvidvideo.ru/ffdshow-tryouts-project-x86-x64/ffdshow-tryouts-project-svn-3212-x86-x64.html

[magnatique]

on this last test :
directshow libavcodec : 30fps
directshow ffmpeg-mt: 40 fps
ffvideosource : 82fps

[LoRd_MuldeR]

on this last test :
directshow libavcodec : 30fps
directshow ffmpeg-mt: 40 fps
ffvideosource : 82fps

Still strange. Try updating your ffdshow (from the link above) and check the number of decoding threads.

(Since switching "libavcodec" -vs- "ffmpeg-mt" did have some effect at least, I assume that DirectShow is using ffdshow for decoding H.264)

[magnatique]

Oh right. I forgot about directshow :)


function MDegrain2i2(clip source, int "overlap", int "dct")
{
overlap=default(overlap,0) # overlap value (0 to 4 for blksize=8)
dct=default(dct,0) # use dct=1 for clip with light flicker
MT("""
super = MSuper(source,pel=2, sharp=1)
backward_vec2 = MAnalyse(super, isb = true, delta = 2, overlap=overlap, dct=dct)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=overlap, dct=dct)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=overlap, dct=dct)
forward_vec2 = MAnalyse(super, isb = false, delta = 2, overlap=overlap, dct=dct)
source.MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
""",8,4)
}


Try that. And no MT in your MDegrain2i2 call. 8,4 means 8 threads and 4 pixels of overlap

~MiSfit


mdegrain as I had set : 37FPS
fft3dgpu with some setting I grabbed randomly (some guidance on what to set for a 1080p source for basic cleanup would be appreciated as I have no idea hehe)
output=fft3dgpu(sigma=.5, sigma2=1.5, sigma3=2, sigma4=1, bt=3, bw=30, bh=30, ow=8, oh=8, wintype=2)
86FPS

mdegrain with your suggestion : [actually it returns an error, says invalid arguments to function MT and points to both my call to the function and line with """,8,4)]

[Blue_MiSfit]

Take it out of the function, and roll it directly into your script, like this (with IMO cleaner syntax)



#Not including audio and video load lines, not sure what you've settled on

audiodub(video,audio)
trim(232,36869)

tomsmocomp(1,5,1)
#Or YADIF(mode=1)

lanczosresize(640,360)
#Why resize before cleaning and sharpening? Faster, but worse quality.

#colorYuV(levels="Tv->PC")
#Why are you going to PC levels?? This should almost never be necessary.
#Furthermore, you should use something like SmoothLevels for better results.

#tweak(-6, 1.4)
#levels(0, 1.1, 255, 0, 255)
#This tweaking shouldn't be necessary IMO....

MT("""
super = MSuper(pel=2, sharp=1)
backward_vec2 = super.MAnalyse(isb = true, delta = 2, overlap=4)
backward_vec1 = super.MAnalyse(isb = true, delta = 1, overlap=4)
forward_vec1 = super.MAnalyse(isb = false, delta = 1, overlap=4)
forward_vec2 = super.MAnalyse(isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
""",8,4) # eight threads

MT("limitedsharpenfaster(ss_x= 1.3,ss_y= 1.3, strength=150)",8,4) #eight threads again

converttoyv12()
#Your source should damned well be YV12!!!


Try also changing the decoder to DivX H.264 or CoreAVC, buy installing either and disabling ffdshow's H.264 decoder, then using DSS2.

~MiSfit

[magnatique]

lanczosresize(640,360)
#Why resize before cleaning and sharpening? Faster, but worse quality.


noted!



#colorYuV(levels="Tv->PC")
#Why are you going to PC levels?? This should almost never be necessary.
#Furthermore, you should use something like SmoothLevels for better results.



do you have a suggestion for smoothlevels settings to use?

I do this since this output is meant to be for web streaming with flash players, and viewed on PC.




#tweak(-6, 1.4)
#levels(0, 1.1, 255, 0, 255)
#This tweaking shouldn't be necessary IMO....



The output for flash needs to have strong and bright colors, and sometimes the filmed source is not bright enough/ etc. the real goal is to have brighter colors though... I've used tweak for years, but if there is something else I don't know about that would be better, am all ears ;)




MT("""
super = MSuper(pel=2, sharp=1)
backward_vec2 = super.MAnalyse(isb = true, delta = 2, overlap=4)
backward_vec1 = super.MAnalyse(isb = true, delta = 1, overlap=4)
forward_vec1 = super.MAnalyse(isb = false, delta = 1, overlap=4)
forward_vec2 = super.MAnalyse(isb = false, delta = 2, overlap=4)
MDegrain2(super, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400)
""",8,4) # eight threads

MT("limitedsharpenfaster(ss_x= 1.3,ss_y= 1.3, strength=150)",8,4) #eight threads again


will give that a shot

if you have suggested settings with fft3dgpu, am all ears it ran really fast but I am not sure what to set for a simple / overall degrain of my original source.. I probably don't need all that mdegrain does.



Try also changing the decoder to DivX H.264 or CoreAVC, buy installing either and disabling ffdshow's H.264 decoder, then using DSS2.

~MiSfit

uninstalled/reinstalled klite codec with latest versions.

I disabled in ffdshow, installed divxh264decoder820.exe, getting 42 FPS vs 86 with ffvideosource()

[Blue_MiSfit]

hmmm sounds like your splitter might be bottlenecking things :(

This is why I don't like codec packs. Is there a config tool somewhere for adjusting filter merits?

I'd just sidestep the issue and stick with ffvideosource. I use it for most things nowadays except QuickTime MOVs, since there's no ProRes decoder in libavcodec...

~MiSfit

[magnatique]

Try also changing the decoder to DivX H.264 or CoreAVC, buy installing either and disabling ffdshow's H.264 decoder, then using DSS2.

~MiSfit

ahh I missed that part from LordMulder's post about dss2.

using dss2 I just got 60fps VS 104FPS with ffvideosource

[magnatique]

yup, this is what I'll do, I'll use ffvideosource ;)

I guess what I am mainly missing now is :
1- fft3dgpu's help on some basic settings, this was real fast but I am unsure how to set it for my sources.


2- any other tips on my reply to your suggested settings above for post #18 http://forum.doom9.org/showthread.php?p=1365961#post1365961


thanks to everyone for the help :)

OH and is there a quality loss on re-encoding again with the same settings my CFR 16 source to apply the degrain/color corrects ? IE ffvideosource my mp4, do the tweak/levels/degrain/sharpen save crf16 mp4, then use that newly re-encoded file with just a simple resize for all my multiple re-encodes I need?

and to make sure my mp4 is Blu-Ray compliant, is there a good easy way to test it?

[Blue_MiSfit]

1) FFT3DGPU

plane=4 == both chroma and luma processed
plane=3 == only chroma
plane=0 == only luma

sigma, sigma2, sigma3, and sigma4 all control different frequency bands. If you only specify sigma, then all others will be equal to this. I suggest a bell curve of sorts...

sigma=1, sigma2=2, sigma3=2.5, sigma4=1.5

Something like that, tweak as necessary.

If you must use tweak and levels, fine. They work well enough.

Smoothlevels is very simple. SmoothLevels("tv2pc") works fine, and does a nicer job than coloryuv does (scaling instead of just moving data). Look at both with histogram() at the end of your script to see the difference!

Making your mezzanine file at CRF16 will be fine. It's up to you to decide whether or not the resized encodes from this mezzanine file are of sufficient quality. As I said, I'd probably go lower, and use CRF12 for cases like this, but that may very well be overkill. Lossless certainly was for my purposes!

If you're targeting BluRay then you need to use short GOPs, nal-hrd, and VBV. I'm not sure if there's a free BluRay verification tool. If your file muxes in an authoring program, it's probably good enough :)

I'm not sure on the exact settings, but some searching should turn up safe bets.

~MiSfit

[ronnylov]

The reason to use x264 intermediate "mezzanine" files instead of lossless lagarith codec is because of faster decoding speed, or lower file size or something else?
I think using lossy compression as intermediate files may give perhaps a lower quality?

[magnatique]

The reason to use x264 intermediate "mezzanine" files instead of lossless lagarith codec is because of faster decoding speed, or lower file size or something else?
I think using lossy compression as intermediate files may give perhaps a lower quality?

In my case, both.

Misfit suggested CRF 12, I used 16.

I based my choice on a bunch of factors.

1- our HD files usually come from our producers in a m2t container, which are 25mbps , or we're looking into getting AVCHD which are usually AVC encoded at 17-22mbps. Ultimately, some will capture from tape to apple prores and they will be 100-200mbps.

With that in mind, we compared some prores VS m2t files we received from one producer and the quality was pretty much the same to us all visually. we compared avs stacked videos, not much difference... compared frame grabs, same.

2- Being AVCHD cams compress at 17-22mbps, I was comfortable with the idea of CRF 16 giving me a good size that comes to roughly 40some mbps.

3- In our editor, I put some stacked encodes from x264 and could find almost no diff between CRF 12 to 18. We did some more tests with frame grabs where I asked some of my guys to choose their highest quality frame or put them in order and nothing was consistent... for some the highest compressed media was the highest quality, some chose the middle, some the highest... but nothing much was different.

4- ditto on our test encodes, either from highest CRF or our CRF16, we couldn't spot which was best.

5- my goal was to keep a very high quality edited file that would suite all my future goals without needing to go back to the editing board while not taking 40 GB per 15 minute scenes. we're around 4GB per 15 MIN, and I tried encoding it to be Blu-Ray compliant without needing to re-encode. As misfit pointed out, I might not be, so I need to review this again hehe. But the settings I had chosen were from a thread about being BD compliant and were what I was given since I am not as knowledgeable as the people around here hehe


EDIT : didn't see you mention Lagarith. It's faster to decode in our editor , and also Lagarith will run me 40-60GB per file VS 4-5GB per file.

[Blue_MiSfit]

Mezzanine files are an interesting concept. The way I see it, there's usually no need for a perfect lossless copy of the source. Also, since lossless files are usually so large, it's usually good enough to do an extremely high quality intra-only x264 encode. This is VERY editable, and though it will take more CPU time to decode than Lagarith, it is also MUCH smaller and manageable. At a sufficiently low CRF, you wouldn't be able to tell any difference :)

Of course, if you're really trying to preserve every aspect of the quality (like in feature film production coming from telecine, or some high quality digital source like RED), you should use something that's even better, like ProRes, CineForm, or DNxHD (or a lossless codec), so you can preserve higher bit depth, less chroma subsampling etc... If you have compressed 4:2:0 sources, it's a whole different ballgame :)

~MiSfit

[ronnylov]

Thank you magnatique and Blue_MiSfit, very useful information (also the other things in this thread regarding how to optimize the speed)!

EDIT: If you specify crf 16 (or any other low value giving good quality) and other blu-ray parameters like level 4.1 and compliant vbv settings then the real rate factor may be higher in some parts of the video because of the vbv settings and level restrictions. But it can be a good way to maximize the quality of a blu-ray compliant encoding (a low crf setting limited by the vbv restrictions to maximum possible bitrate for blu-ray). You still may save some file size if other parts of the video gives good enough quality without maximize the bitrate of blu-ray on these parts.

[magnatique]

1) FFT3DGPU

plane=4 == both chroma and luma processed
plane=3 == only chroma
plane=0 == only luma

sigma, sigma2, sigma3, and sigma4 all control different frequency bands. If you only specify sigma, then all others will be equal to this. I suggest a bell curve of sorts...

sigma=1, sigma2=2, sigma3=2.5, sigma4=1.5

Something like that, tweak as necessary.

If you must use tweak and levels, fine. They work well enough.

Smoothlevels is very simple. SmoothLevels("tv2pc") works fine, and does a nicer job than coloryuv does (scaling instead of just moving data). Look at both with histogram() at the end of your script to see the difference!

~MiSfit


Thanks for all the input, getting back to it to polish things up :)

I really appreciate the help and info, some truly good knowledge being passed on Doom9. when I start reading posts it amazes me how far some of you are with these tools and how little I know compared to you guys, while at the same time knowing much more with video/editing/encoding than most other people I know in everyday life hehe ;-)


if you still have the patience, I was wondering if you might know... one part of my process with encoding these files, I have one guy who visually looks at the scene and boosts, with tweak, saturation/levels etc... to get some great flashy blues/reds/yellows etc... but being it;s a human input, sometimes the skin tones will get yellowish or too red etc... dpeending on how it was filmed at first.

When you mentionned histogram, I tried gathering more info on this as I've seen it mentionned a bunch of time but never really grapsed it fully. Anyhow, looking at it and reading a bunch of other threads that popped up, I'm just starting to wonder if there isn't a filter/plugin out there someone already developped that can analyse the video and correct most of these to get proper results? I am not processing some VHS or anything of the sort, just trying to polish these HD sources to get vibrant colors, attenuate some of the extremes when say someone has a spot in his face and brightness/contrast is messed etc... like an "overall fix" rather than multiple micro fixes.

[Blue_MiSfit]

Hmmm... color correction is not exactly my forte :)

There are lots of scripts and tools out there to adjust colors, but sadly I know very little about them.

Here's a good read:
http://forum.doom9.org/showthread.php?t=79898&highlight=ylevelsc

Other than that, not sure where else to point you... :)

Happy to be of help!!

~MiSfit

[magnatique]

OK Used smoothlevels now, much better results thanks :)

Using MDegrain2 now I achieve 39FPS
Using output=fft3dgpu(sigma=1,sigma2=2,sigma3=1.5,sigma4=1.5) 62 FPS

some tests I did, visually I am not very much able to distinguish which is best... with these results, I would opt for the faster fft3dgpu option...

But trying to play with histogram, I put it in luma mode which I now think is what you hinted about.

Original
http://www.magnatique.com/temp/OG.jpg

FFT3DGPU : I get these
http://www.magnatique.com/temp/fft3dgpu.jpg

it's less grainy, but it has these "mushroom style" lines, is that what is usually refered to a halo?

is there another filter I can run to smoothen this out, or is that something I shouldn't worry about and just move on with the faster solution?


Here is Degrain :
http://www.magnatique.com/temp/degrain.jpg


and looks like MT() doesn't run the mdegrain2 function right :/
http://www.magnatique.com/temp/degrainMT.jpg

[Blue_MiSfit]

it's less grainy, but it has these "mushroom style" lines, is that what is usually refered to a halo?

is there another filter I can run to smoothen this out, or is that something I shouldn't worry about and just move on with the faster solution?


This is banding. fft3dfilter / fft3dgpu tend to create / reveal it. I usually follow with gradfunkmirror(1.51) to dither out the result. But, this does so by adding noise... so it's a balancing act :)

Check your syntax, MVTools2 version, and MT / AviSynth.dll versions. Post your syntax for a sanity check ;)

Also, for looking at levels, just use Histogram(). It will display the levels off to the right. Compare ColorYUV with SmoothLevels with a histogram at the end of each ;)

MDegrain2 is amazingly good, but is quite a bit slower than FFT3DGPU, even with heavy multithreading. If you need really good results, use it. For moderate stuff, use FFT3DGPU, and for light stuff, I like DeGrainMedian.

Also, don't be _too_ concerned with the luma histogram. It's not the end of the world :) The most important thing is looking at the final encoded result (using a 2 pass encode at identical bitrates and settings of course). This is the only real way to examine the total effect of a given denoiser!

~MiSfit

[magnatique]

Check your syntax, MVTools2 version, and MT / AviSynth.dll versions. Post your syntax for a sanity check ;)


~MiSfit


yup, mvtools2 was the culprit I had an old version ;)

[magnatique]

Ok all quirks have been worked out, I got a satisfying 60FPS with what I was working on right now.

now reviewing other fast input solutions for .mov's ;-)

I was currently using qtinput, will see if something else is more recent/bench it.

[magnatique]

I'd just sidestep the issue and stick with ffvideosource. I use it for most things nowadays except QuickTime MOVs, since there's no ProRes decoder in libavcodec...

~MiSfit

what do you use for prores decoding?

[magnatique]

well I guess with QTinput I am having a hard time getting some decent speeds and am pretty much back to square one on those sources. I get a 35 fps max with it alone, I tried MTsource but with no good results the script would crash from time to time and not getting higher than 40 with no work on the script other than input. anyone has some ideas?

I was thinking of trying perhaps qt alternative, but everytime I go that route it ends up messing older avs that were with qtinput heh

[Blue_MiSfit]

MT doesn't work nicely with QTInput hehe...

However, QTInput seems to be THE ONLY way to get ProRes video into AviSynth.

Use mode=2 by default, and check audio / video durations. I've found that QTInput for some reason has garbage at the end of the audio track. So, I frequently use trim(1,0) to trim off any extra audio crap.

You can still use MT'd MDegrain2 with MOV sources. The ProRes decoder on Windows is single threaded and slow as shite. If you have spare CPU time, run heavier filtering (MDegrain2), slower x264 settings, or run multiple encodes in parallel. I do 3 at a time typically on an 8 core Penryn system. I can get ~17fps per encode without any filtering, using basically x264 defaults. At this point, the hard drive I/O is nearly maxed out. I can sustain over 60 MB/s read speed when doing this!

I do ProRes -> x264 ALL DAY LONG. It works, it's just a PITA sometimes...

Almost any other MOV you can do with FFMS2, just index it ahead of time. Photo JPEG (MJPG) / PCM is a common one for me, as are H.264 / AAC trailers. These run waaaay faster in FFMS2 (and correctly output YV12 instead of upsampled YUY2 like with QTInput)

~MiSfit

[magnatique]

a few more questions came to mind , some of them I ask because I can't test as I am not at the office;)

-With your normal non-prores MOV, do you use qtinput still?
- with QTinput, since it's single threaded, would opening a bunch of CLI encodes max out the cpu or the QT plugin would try to use the same thread for all instances hence not maximizing cpu?

- this might be a silly question... but if I need to encode from the same source 4 mp4 output formats (say IPOD/Iphone/720p/720p_higherbitrate meant for 1080p viewing), and I have 3 different sizes (320x180, 640x360, 1280x720), do I need to run the pass1 for all of these outputs?

right now I have 3 different avisynth scripts that are exactly the same but for the resize spec, I was wondering If I could save some time with doing :
audio encode
pass1 on 1280x720 source
pass2 ipod
pass2 iphone
pass2 720p
pass2 720p2
mux ipod
mux iphone
mux 720
mux 720p2

[Blue_MiSfit]

The only non ProRes MOVs I ever typically see contain MJPG or H.264. Both these can be opened nicely via ffms2, so I tend to use that route (though indexing a 150GB file is NO FUN AT ALL).

Opening multiple CLI encodes using QTInput implies a different process for each encode - so that means a thread for each encode. Also, if you use x264 through avs2yuv, you can break out AviSynth (and thus QuickTime) into its own process, and make x264 a 64 bit process. This makes high memory situations a little simpler. Not that it's terribly relevant to your scenario though :)

I wouldn't suggest using the stats file of a 720p encode as the source for a second pass encode at a different resolution! I'm not actually sure if it would work or not, but I would certainly avoid it! Also, I assume you're using different GOP structures for iPod / iPhone vs generic - since the former hardware platforms have piss poor decoders that can't do CABAC or B-Frames AFAIK - correct?

If you're pressed for time, either use faster settings, or just use 1 pass ABR with VBV, or (to take it futher) proper CBR.

~MiSfit