Max speed from mpeg2 to h264

[Gannjunior]

Hi,

I need more speed during encoding of many DVD to h264.
I tried different sw like megui, staxrip but I would like to improve dramatically fps.

I heard Nvidia CUDA GPU does miracles. For example in TMPGenc 4 and Premiere CS5. But what is not clear is if the "miracle" is only during the editing (to accelerate filter etc and so to have real time effect) or also in the final encoding.
That is what I need. I'm looking for something to improve only the compression from mpeg2 to h264.

Any suggestion?

Thanks
Ciao!

[nurbs]

CUDA can be used for decoding and deinterlacing with the DGNVTools. If your content is HD that can be beneficial, especially if you let it do the deinterlacing. For SD there is still the deinterlacing, but apart from that don't expect too much speed gains.
There are also CUDA encoders. They can be useful if one has a slow CPU and doesn't care about quality, but in your case, since your GPU is old and CPU is new, it's probably better to simply use a faster x264 preset. That will also cost some quality since you are turning the psy optimizations off if you use faster settings than the defaults, but the CUDA encoders aren't good to begin with so you should still get better quality unless you use the fastest ones.

[nm]

I need more speed during encoding of many DVD to h264.
I tried different sw like megui, staxrip but I would like to improve dramatically fps.

MPEG-2 decoding is fast, so your bottleneck is either in filtering or H.264 encoding. If you don't use heavy filters, speed can be gained by simply changing encoding parameters or by using more CPUs.

What x264 parameters have you tried and what kind of encoding speeds are you getting now?

[Groucho2004]

getting a faster CPU.

Look at his sig. It should be very fast.

[nm]

Look at his sig. It should be very fast.

Yes, I corrected the sentence already.

[nurbs]

Yeah, I got an 860 with a GHz slower clock and for SD resolutions it's faster than realtime even with --preset veryslow (for progressive content at --crf 20).

edit: Not totally true. I forgot that I limited the number of B-frames to 3 when I did that test.

[prOnorama]

Did a test on a progressive PAL DVD, AR ca. 2.40:1 (with cropping and resizing), on a AMD Phenom II X4 955 BE quadcore @ 3.2 Ghz:

x264 (--preset x) --crf 16

avs : 688x288p 0:0 @ 25/1 fps (cfr)

[i]--preset ultrafast
x264 : profile Constrained Baseline, level 2.1
encoded 5001 frames, 186.08 fps, 2576.02 kb/s
[i]--preset superfast
x264 : profile High, level 2.1
encoded 5001 frames, 182.58 fps, 2055.69 kb/s
[i]--preset veryfast
x264 : profile High, level 2.1
encoded 5001 frames, 181.03 fps, 859.70 kb/s
[i]--preset faster
x264 : profile High, level 2.1
encoded 5001 frames, 154.32 fps, 923.53 kb/s
[i]--preset fast
x264 : profile High, level 2.1
encoded 5001 frames, 106.55 fps, 1010.96 kb/s
[i]default
x264 : profile High, level 2.1
encoded 5001 frames, 91.45 fps, 981.13 kb/s
[i]--preset slow
x264 : profile High, level 2.1
encoded 5001 frames, 67.25 fps, 968.15 kb/s
[i]--preset slower
x264 : profile High, level 2.2
encoded 5001 frames, 28.82 fps, 935.38 kb/s
[i]--preset veryslow
x264 : profile High, level 3.1
encoded 5001 frames, 15.86 fps, 819.29 kb/s
[i]--preset placebo
x264 [info]: profile High, level 3.1
encoded 5001 frames, 7.95 fps, 822.16 kb/s

I didn't use multithreaded AviSynth, the bottleneck seems te be around 180 FPS here for normal AviSynth.

Kind of surprised the bitrates on --preset veryfast/faster are actually lower than the default preset, not what I expected :confused:

[sneaker_ger]

Kind of surprised the bitrates on --preset veryfast/faster are actually lower than the default preset, not what I expected :confused:

Don't draw the conclusion, that faster/veryfast create higher quality on this sample at the same bitrate than normal. CRF only means same quality if the settings are identical, which is obviously not the case.

[nurbs]

CRF only gives you roughly constant quality for the same settings, so faster settings don't necessarily have to result in higher bitrates. With the same settings and CRF turning off the psy optimizations reduces the bitrate by a lot. I guess if you'd turned of psy everything down from --preset fast would get a lower bitrate, because starting with faster RDO is off.

[stax76]

I heard Nvidia CUDA GPU does miracles.

With DGDecNV only with slow CPUs, with 4 or more cores gains are small, even with a HD source it might not be more then 1 fps.

[Blue_MiSfit]

Unless you have to deinterlace. Then it can be a HUGE improvement.

Derek

[Gannjunior]

Hi guys,

thanks for your answers


Off course I need to deinterlace. I usually use yadif and a little tweak of brightness and contrast.
And for the string compression: --preset fast --crf 23 --ref 3 --rc-lookahead 40 --deblock -1:-1 --subme 5 --trellis 0 --output "<target>" "<source>" ...as you can see the settings are quite "light"...

The problem is that I'm trying to help a friend he needs to encode many clips per week of about a couple of hours each one from mpeg2 to h264. And the problem is not h264 or encoding software...The problem is he has many clips to encode and maybe some hardware help is the only way to really boost encode...I believed CUDA could help also in final encoding..but it seems CUDA is only useful during editing a video to get real time effect...

Maybe the only way is to update from i920 to i980x?

Any suggestion is welcome

Thanks again

ciao!

[Blue_MiSfit]

Uhh.. of course you need to deinterlace? You do realize that most newer DVDs aren't actually interlaced at all, but are actually coded in 24p and contain RFF flags to apply a soft 3:2 pulldown during playback on devices that have to output interlaced analog sigals... right?

Stuff originated on film is either coded this way, or as hard 3:2 pulldown, in which case you don't deinterlace either - you IVTC (inverse telecine) via field matching and decimation, neither of which can run on a GPU.

If you mostly process content that is actually natively interlaced (sitcoms, porn, some animated content, live music etc) then yes GPU decoding will help you a lot.

To my eyes, CUDA deinterlacing looks at least as good as YADIF, if not better. It is also quite fast, and frees CPU cycles for encoding power. The latter is usually the biggest benefit, though with very fast deinterlacers like YADIF, it may or may not be a huge issue.

Try, and tell us your results :)

Derek

[nm]

Uhh.. of course you need to deinterlace? You do realize that most newer DVDs aren't actually interlaced at all, but are actually coded in 24p and contain RFF flags to apply a soft 3:2 pulldown during playback on devices that have to output interlaced analog sigals... right?

And since he's from Italy, most R2 DVD movies are straight 25p. Sometimes euro pulldown (2:2:2:2:2:2:2:2:2:2:2:3) may be used, but more often there are bad, field-blended conversions from NTSC sources. Those can sometimes be fixed with SRestore.

[Didée]

I would like to improve dramatically fps.
Weee ... "dramatically". :devil:

Basic question: What is your actual encoding speed?

Related: Show a typical Avisynth script you're using.

[Gannjunior]

Thanks again for you support Blue, mm and Didè!

nm,
unfortunately, football match clip I'm speaking about, are taken from skytv and these clip are always interlaced.

Here a little sample:
http://www.speedyshare.com/files/24838975/1.mpg

I will give you some numbers, based on the complete conversion of the whole match uploaded above (1h 43').

I did two test, encoding with staxrip, using in both two test the same h264 settings (see my preset in the attached file at the bottom, but it is something near this: --preset fast --crf 22 --ref 2 --rc-lookahead 40 --deblock 0:0 --subme 5 --trellis 0 ).

Difference between two test is a denoise: often it happens source is quite degradated, so, using CRF, I need more compressibility to contain size and to reduce the visive pixelment impact..


first test

MPEG2Source("F:\wisport\VIDEO_TS\VTS_01_1 temp files\VTS_01_1.d2v", CPU=6)
LoadCplugin("D:\Program Files (x86)\StaxRip\Applications\AviSynth plugins\Yadif\yadif.dll")
Yadif()
Crop(8,6,-20,-6)
BicubicResize(640,480,0,0.5)
tweak(bright=15,cont=0.85,sat=1.03)

final medium bitrate about 1350 kbps

about 45% cpu utilization during encoding

speed 105-110 fps, that is 24'-25' per clip

If I run another task of staxrip to saturate cpu, I can encode 2 clips in the same time, so it is how to hit the equivalent of about 200 fps.
------

second test

MPEG2Source("F:\wisport\VIDEO_TS\VTS_01_1 temp files\VTS_01_1.d2v", CPU=6)
LoadCplugin("D:\Program Files (x86)\StaxRip\Applications\AviSynth plugins\Yadif\yadif.dll")
Yadif()
Crop(8,6,-20,-6)
BicubicResize(640,480,0,0.5)
VagueDenoiser(threshold=3, chromaT=3)
tweak(bright=15,cont=0.85,sat=1.03)

final medium bitrate about 1250 kbps **

about 20-25% cpu utilization during encoding

speed 41-42 fps, that is about 1h per clip

If I run 3 more tasks of staxrip to saturate cpu I can encode almost 4 clips in the same time, so it is how to hit the equivalent of about 150 fps.

Hope it is more clear,

ciao!!


** I often need to use or to add other denoisers to get a more consistent impact on the compressibility (but off course encoding time grows...)

[nm]

Ok, since you need to deinterlace and sometimes denoise, DGDecNV (http://neuron2.net/dgdecnv/dgdecnv.html) should help with both speed and deinterlacing quality (compared to yadif).

I don't know how many parallel processes you can run with it though, if a single encode is still not fast enough to saturate your CPU. I also don't know if it exposes denoising and sharpening controls.

[Didée]

You are bottlenecked by the performance of the source filter. Obviously, you can not encode faster than you can decode the source.
Specifically, it's the postprocessing that sucks lots of performance. Without postprocessing, decoding is so much faster.

Alas, it seems multithreading is not a possibility to boost performance. I tried MTSource() for mpeg2source, performance was worse. For ffms2, the internal multithreading gives a big boost when not using pp ... but with postprocessing, there's not much difference. With pp, mpeg2source was faster than ffms2, anyway.
DGDecodeNV does not offer postprocessing at all. And without postprocessing, though it's quite fast, the competitors are even faster.


The numbers for decoding speed:

(PAL DVD source, i7-860, NV GT240)

postprocessing off H+V+dering
--------------------------------------
mpeg2source 460 fps 170 fps
DGDecodeNV 396 fps -------
ffmpegsource 1T 495 fps (125 fps) (singlethread + postprocessing = crashes VDub )
ffmpegsource 4T 777 fps 131 fps


Bottom line: your wished "dramatical" performance increase is definetly possible ... but only if you do *not* use postprocessing.

[nm]

You are bottlenecked by the performance of the source filter. Obviously, you can not encode faster than you can decode the source.
Specifically, it's the postprocessing that sucks lots of performance. Without postprocessing, decoding is so much faster.

I think he would also need to split Yadif and any denoisers he's using to multiple threads?

[Didée]

Yes, probably. I just wanted to point out the influence of source filters with internal deblocking/postprocessing. Not much point going into mutlithreading and searching for "faster filters", when the source filter happens to be the main bottleneck.

OTOH, when it's about converting a huge pile of DVDs anyway, then the easiest way is to run multiple encodes in parallel. When the queue contains dozens or hundreds of jobs, why jump through loops trying to make a single job run faster, when you simply can run several jobs in parallel. Of course you need sufficient ressources (RAM) for that ... but seeing Gannjunior has 12GB available, that shouldn't be a problem.;)

[Gannjunior]

nm, i think i will try neuron's DGDecodeNV.

Didè, thank for your analisy, interesting to know the speed impact of deblocking. But off course, with mpeg2 pp off, the presence of articats is, off course, more visible.

Just one thing: yesterday I did some attempt with only mpeg2source and no pp, but I could not do better than 160 fps (cpu at 95-100%)...

...you simply can run several jobs in parallel. Of course you need sufficient ressources (RAM) for that ... but seeing Gannjunior has 12GB available, that shouldn't be a problem...
:-) yes! as i told you in my last reply, it is exactly what I do: I have 4 staxrip folders and I run more tasks of it, until my cpu resource hits 100%.


thanks and ciao!

[Gannjunior]

Bought DGDecodeNV.

Tried SD mpeg2 to 576x432 h264 encoding.

Given some settings of avisynth and x264, I did a little test, changing only mpeg2 import settings. In test 1 dgindex and in test 2 dgdecodenv using its internal deinterlacing and resize (instead of yadif and bicubic resize of avisynth).

test 1 -> about 165 fps
test 2 -> about 295 fps

My GPU is Nvidia 470 GTX. The boost is really amazing.


I would like to advice dgdecodenv to my friends that uses Nvidia 9800 GT and GT 230: could I advice them, being sure they will have a similar % improvement?

Many thanks in advance

[Didée]

Yes. However, you're making bad use of your CPU. Those numbers tell that you're not using Multi-Threaded (MT) Avisynth. With such fast filtering and encoding settings, the CPU can't deliver data fast enough when using only one thread for Avisynth.

Pure rendering to screen:

mpeg2source("PAL_576i.d2v")
Yadif().bicubicresize(576,432)
FPS: 142.6

SetMTmode(5,4)
mpeg2source("PAL_576i.d2v")
SetMTmode(2)
Yadif().bicubicresize(576,432)
FPS: 341.2


See the difference?

The following benches all were using 4 Avisynth threads, except the one where indicated. "GPU" is DGDecNV with deinterlace=1 and GPU-resize.
Note: in all benches, CPU usage was ~98%. The one bench with singlethreaded Avisynth had only ~30% CPU usage.

# i7-860 @ 2.8 (stock)


--preset ultrafast --crf30

CPU: 315.75 fps
GPU: 349.01 fps ( + 10.5% )


--preset veryfast --crf 24

CPU: 132.82 fps (Avisynth: 1 thread)
CPU: 240.79 fps (Avisynth: 4 threads)
GPU: 297.61 fps ( + 23.6% )


--preset fast --crf 24

CPU: 127.69 fps
GPU: 150.12 fps ( + 17.5% )


--preset slow --crf 20

CPU: 62.72 fps
GPU: 66.33 fps ( + 5.7% )


You see, when making "good use" of the CPU (instead of "big waste"), then DGDecNV still gives a performance boost, but not nearly as much. It's mostly in range 5-20% faster. Your boost of almost 80% was only because you didn't use your CPU properly.

BTW, speed comes at a price: NV deinterlacing is one of the most poor ever seen (http://forum.doom9.org/showthread.php?p=1433847#post1433847) in Avisynth's world. ;) (I can think of only one that's worse: Alparysoft deinterlacer)

[Gannjunior]

Hi Didèe and happy new year!!

I had finally some time to do some tests because in the last two months I was really "under water" with my accountant job to close final yr balance!!! :eek:

Installing MT everything completely changed and with fast presets I break 400 fps!!

Thank you very much for your support :)