Megui and x264 - Need more speed!!!

[Fat80y]

Need some help from the gurus:

Mainly compressing TV caps, mostly HD - 1080i or 720p.

Low broadcast bit rates where I am, with the current settings below I cannot tell the difference between broadcast and compressed @ 3m on my 50" 1080p plasma.

What I am interested in is more speed. Currently get maybe 8-9 fps on 1080i up to 20 with 720p.

I am guessing it is a combination of deinterlacing, noise removal, and wrong (inefficient?) encode settings. Can anyone help maintain similar quality but faster encodes?

System is i7 @ 2.8ghz, 6gb ram, hyperthreading is on.

Megui x264 command line:

program --tune film --qp 30 --b-adapt 2 --ref 5 --merange 32 --me umh --partitions all --no-mixed-refs --no-fast-pskip --output "output" "input"

Typical AVS

global MeGUI_darx = 640
global MeGUI_dary = 351
LoadPlugin("C:\Program Files (x86)\megui\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("D:\working\current processing\tv show.d2v", cpu=4, info=3)
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)
Load_Stdcall_Plugin("C:\Program Files (x86)\megui\tools\yadif\yadif.dll")
Yadif(order=1)
#crop
#resize
LoadPlugin("C:\Program Files (x86)\megui\tools\avisynth_plugin\Convolution3DYV12.dll")
Convolution3D("movielq") # Heavy Noise

[Assassinator]

Megui x264 command line:

program --tune film --qp 30 --b-adapt 2 --ref 5 --merange 32 --me umh --partitions all --no-mixed-refs --no-fast-pskip --output "output" "input"

Just use one of x264's given presets. Those are the settings suggested by the x264 authors, and they know the costs and benefits of each setting best.

Try...
--preset fast
--preset faster
--preset veryfast


And use --crf instead of --qp. crf is a "smarter" ratecontrol system.

[Dark Shikari]

And use --crf instead of --qp. crf is a "smarter" ratecontrol system.
Moreso, qp is FOR DEBUGGING ONLY and anyone who uses it for real encoding (besides for setting lossless mode and a few other special cases) is a fool. CRF is the default for a reason; don't go out of your way to add stupidity where there isn't any to start with.

[Fat80y]

Thanks for the quick responses guys.

--qp was a huge snafu on my part - great catch!

I have just been using Megui to set the x264 parameters - and saved a custom profile and must have had constant quantisizer selected instead of constant quality!!! :stupid:

I'll try the basic presets now (didn't realise they were there!!!) I've been using that command line for a long time.... :eek:

:thanks:

[Adub]

Also, make sure your X264 is up to date, as more often then not there are speed improvements over the previous version.

[Fat80y]

Hmmm, I have done some testing, and believe I have a bottleneck somewhere.

Source= 1440x1080i mpg

All encodes are --crf 30 --ultrafast

1. As per avs above = 9fps

2. Deinterlace only = 43fps

3. No filters, crop deinterlace, nothing = 77 fps

On all the above encodes, CPU usage hovered around 12% for x264.exe and vfwx264.exe respectively. ie It was only using 2 cores out of 8 on my 920.

Switching off hyper threading made no difference.

Is this an avisynth bottleneck?

Further testing:

As 3. above, no filters etc at all, --crf 30 but now --slow and am getting 13 fps and CPU usage has risen to 85% combined for both processes.

I will test --slower in the next encode to see if I can actually max out my cpu.

What I hope / want to be able to do is encode @ 50fps utilising my entire cpu, and therefore getting the maximum quality from the time I can allocate to each encode.

Can anyone offer any insight?

[Mixer73]

If you want more speed and are running a Nvidia graphics card you should consider the DG NV Tools which use your graphics card as a frameserver, this may well give you more speed.

That said I'm not sure that a goal of 50fps encoding at high resolution is an attainable goal. I guess DS would know if anyone...

[Guest]

Convolution3D() is pretty expensive. Ditch it or or use a 2D filter instead.

You can ditch yadif too by deinterlacing on the GPU using DGDecNV, as mentioned by Mixer73 above.

[Fat80y]

Thanks for your replies Mixer73 and Neuron2, but you may have not realized my question has changed a little.

Removing all filters, deinterlacing etc from the avisynth script, x264 --crf 30 --ultrafast results in 77 fps encode speed on a 1080i source, but only 25% cpu utilization. So if I could somehow get x264 to use my other 6 cores, I should theoretically be able to encode at 4x77=308 fps?

So I am concerned that avisynth (frameserver) is not able to feed data to x264 fast enough. Does that make sense?

I will try your app neuron2 and report back.

[Fat80y]

neuron2, didn't realise you were charging for it. Not to worry, I have been using dgindex for years, so I definately have my $15 worth!

Will have to wait now for registration codes etc... :thanks:

[Guest]

Better to ask about threading and CPU utilization for x264 in one of the x264 threads, rather than one titled "Megui ..."

Just sent (what I assume is) your activation.

[Fat80y]

Just completed a few hours testing, and all I can say is wow!

Many thanks neuron2 for your excellent DGDecNV.

My GTX 295 is decoding @ ~ 90fps whilst deinterlacing, cropping and resizing. I got rid of convulution3d as it didn't make much difference to quality (and was about 4 fps speed hit), and am now encoding at ~ 32 fps (a 4.5x speed increase!!!) x264 still --medium --film --crf30.

The vfwx264.exe has cone from 12% (or 1 core) cpu usage to 1% indicating that frameserving speed was my bottleneck.

Its not the 50fps I was after, but it is a damn site better than 7 fps.

For those interested I usually encode 20 hours of a music channel every week end. This will dropped my encode time from 4 days to less than 16 hours. :thanks::thanks::thanks:

Now to justify a Nvidia GTX 480 gpu and a i7 980x cpu :devil:

Update: SD encodes at 105 fps!!!

[Guest]

Another happy customer. :p

[Mixer73]

Another happy customer. :p

Laugh, and I reckon the first time I ever suggested something on Doom9 that helped someone. Only took me 5 years!

[burfadel]

--crf 30 --ultrafast

Wow, the results would truly be terrible with those settings!

All these things outlined below are critical considerations, don't omit any of them!

- What revision of x264 are you using?
- Have you triend evisynth-MT? (and setting it correctly accordingly)
- Are you using the latest Yadif v1.7 which has lots of speed improvements or the older Yadif v0.9 that most people use except when using the latest Staxrip etc? Have you even considered Staxrip v1.1.6.4?

After taking into account the latest Avisynth v2.5.8-MT (its a mod version), Yadif v1.7, x264 (currently r1680), muxer (say for MKV, Mkvtoolnix v4.1.1) etc versions...

--tune film -b-adapt 2 --ref 5 --merange 32 --me umh --partitions all --no-mixed-refs --no-fast-pskip

Leave leave --merange and --partitions as the default (default when not setting quality settings). Setting it to '32' and 'all' respectively will hurt encoding speed and not give any noticeable benefit. Also don't set --no-mixed-refs and --no-fast-pskip, the latter which will reduce encoding speed and give no real benefit, especially when using such a high CRF.

I see you're encoding to 640x352, which is good. What resize filter is being used to adjust down from 1440x1080? You have bilinear, bicubic, lanczos, lanczos4 (same as lanczos with 4 taps specified), blackman, gauss, spline36, spline64 resizers. You should probably consider trying either blackmanresize (similar but better than lanczos) or spline36resize.

Use a much lower CRF, 30 is really high. Without sounding critical of your tv, it should look really crap compareed to the original at 30 pn a 50 inch screen. Are you sure its a 1080P tv, or is it just a tv capable of 1080P? There is a massive difference. A 1080P ready tv can accept 1080P but only display it at the resolution of the screen.

Whats the model number of the TV? how have you connected it to your computer to play it, or are you using a blu-ray player to play it?

I'm asking the model number of the TV to look it up and the true specs of it, because I can assure you with a 100 percent guarantee that there should be a very very noticeable difference between a 1440x1080 source, and a 640x352 encode at crf 30 (and especially at your originally set qp 30), even at 3m. There has to be some issue with the TV spec wise (or firmware) or no offence, your eyesight, if you can't tell the difference at 3m

A respectful output, including speed wise, is the programme and filter considerations I outlined about, have x264 use the default settings (maybe with tuning for film), and having it at CRF 22. Try the following:
--crf 22 --b-adapt 2 -ref 5 --aq-mode 2 --open-gop normal
which should give you fast encoding with decent output quality and file size wise.

When encoding standard def stuff (all that I bother with recording) on my q9400@3.2, I get still get over 25fps at worst with the settings below. FFmpegsource is r318, ffmpeg used is the latest daily, and filename for the source is replaced with the actual filename naturally, and spline64resize value is automatically adjusted in staxrip to approximate around an intended output of 225,000 pixels as set in the settings.
FFVideoSource(filename)SetMTMode(1,4)
Spline64Resize(640,352)
Gradfun2db(thr=1.55)
degrainmedian()

Settings for x264 (and yes, some seem a little excessive but I use them because I can!):
--crf 21 --keyint 400 --open-gop normal --ref 5 --rc-lookahead 80 --nr 400 --bframes 5 --b-adapt 2 --direct auto --subme 10 --trellis 2 --psy-rd 1:0.2 --me umh --aq-mode 2 --sync-lookahead 40

I do realise deinterlacing, resizing and decoding of 1440x1080 will be much slower, but in the case that its the decoding and deinterlacing thats the bottleneck, crippling x264 settings like crf 30 isn't the answer. Finding faster filters like you have with the decoder, yadif v1.7, avisynth-mt, utilising the latest x264 as well as anything else along the way is :)

Don't forget the inform us of your tv make/model, because like I said before, something is very off if you can't tell the difference even at 3m of 640x352 at crf 30 compared to 1440x1080 on what is supposedly a 1920x1080 progressive scan plasma! (doesn't matter if tv source is interlaced, good tv's will deinterlace it for you hence progressive scan, or 1080p).

[Dark Shikari]

Setting it to '32' and 'all' respectively will hurt encoding speed and not give any noticeable benefit.Settings for x264 (and yes, some seem a little excessive but I use them because I can!):So in short, wasteful settings are bad when other people use them, but good when you use them?

Just use the presets.

(Also, --nr 900 is high enough to completely utterly destroy some sources in my testing. I have no idea why you're using such a high value.)

[Fat80y]

Burfadel,

Thankyou for such a detailed response, however I believe you have made a few assumptions:

1. The script was an example only (which happened to be an SD source) I anamorphic encode everything (so as not to lose any detail) and resize to mod16 source resolution if necessary. I WAS using that script for all my encodes 1080i - 720p - 720i.

2. I don't encode bluray. Only recorded OTA. Hence deinterlacing is necessary.

3. My TV is a 50" Panasonic TH-P50V10A

http://panasonic.com.au/products/details.cfm?objectID=5102

I can assure you it is a 1080p panel. I watch everything in THX mode - everything seems more natural that way.

4. It is hooked up to my HTPC - q6600 @ stock with 4gb ram and a Nvidia GTX 295 playing through hdmi and latest MPC Home Cinema build. Standard post processing.

5. I live in Australia. Our DTV is crap. The sample clip I have been testing with is actually 1440x1080i 14mbit mpeg2 + ac3 2 channel sound. It looks as rubbish as it sounds. Full of blocks, halos, artifacts etc. It is of a car race, so plenty of panning, fast action on static back grounds etc. I settled on originally settled on --crf 30 because from 3m back on my couch it was indistinguishable to me from the original recording. Considering that was my best case viewing scenario, but I generally stream to my phone for viewing in bed the theoretical loss of quality is acceptable to me. I end up with a more flexible container (mkv) and files one tenth the size which makes for easier transmission over wireless. I mainly encode music video clips - which if you've ever watched one from the 60's 70's and 80's are in terrible shape to begin with...

6. According to my last eye test in April, I have 20/20 vision.


7. I was testing with --ultrafast because I correctly believed that x264 was being bottle necked by the frame server.

8. I hate tweaking avs files. Time is important to me. If Megui can't do it automatically with presets etc (I typically use one click encoding) I'm not interested. Hence the use of yadif and spline64.

9. I am currently reevaluating my choice of --crf 30. Currently am encoding the same test clip from --crf 16 in whole number increments to --crf 40 for another viewing test. I have stuck with --medium because that allows me to encode HD at real time or better speeds. I f I can justify it to myself, I will upgrade to i7 980x which may allow me to encode at --slow in real time.

9.5 I use the development server for Megui updates - so everything should be as current as possible. I have, however, noticed that Megui64 is being developed, but am not holding my breath...

10. Did I miss anything? Whew that is probably the longest post I have ever made on a internet forum. As you can see from my join date, I have been a long time lurker... :thanks: for everyone's contributions!

[Mixer73]

Fat80y: I'm also encoding crap Aussie OTA and I use CRF 24/25 and everyone always tells me it must be crap! However I think the quality is good enough and its for long term storage.

Can I ask whether you keep anythng you encode or do you just watch it once?

I guess I'm interested in the conjunction of the need for fastest encoding at the lowest quality level...

[Fat80y]

Fat80y: I'm also encoding crap Aussie OTA and I use CRF 24/25 and everyone always tells me it must be crap! However I think the quality is good enough and its for long term storage.

Can I ask whether you keep anythng you encode or do you just watch it once?

I guess I'm interested in the conjunction of the need for fastest encoding at the lowest quality level...

Hey Mixer73,

My partner and I are music nuts (specifically 80's and earlier) - you have probably guessed that I am recording rage. To be fair ABC HD is 720p but I was looking for a worst case scenario and found it on OneHD. I have 4 tuners in my HTPC and record anything that looks interesting - I keep everything I want to watch that I can't download elsewhere. (I like to let other people do the hard work for me!) Mainly documentaries. I have 8tb on my home server, but am looking to upgrade!

I can't wait for the day to come when I can download good quality drm and add free hd material at a reasonable price. Think itunes for movies / tv (with genius recommendations) without all the crap that goes with it.

I too used to be obsessed with quality - back in the day I used to encode @ 1fps 700mb dvd rips on my Duron 600mhz because theory said it was better. Then I actually did a few of my own viewing comparisons once I got the plasma this year and realized I was wasting my time. All those meticulous encodes still look like crap because the source is crap. The forum rules are right - there is no "best" - just "subjectively best for you!"

[burfadel]

So in short, wasteful settings are bad when other people use them, but good when you use them?

Just use the presets.

(Also, --nr 900 is high enough to completely utterly destroy some sources in my testing. I have no idea why you're using such a high value.)

Yes my mistake :) in my rush I took the settings from a muck around try different things encode I happened to be doing at the time! I've corrected it with the settings I usually use. That is, -- nr 400 and 5 ref, 5 b-frames.

I use those settings because quality vs filesize is my main concern, and speed second :) Since the question asked was purely about speed, I was intending to show my typical settings and processor to show that the speed shouldn't be too bad :)

I know the settings are fairly high, but I do get quite reasonable bitrate saving using them and the speed is very acceptable for me. To put in in perspective, preset slower is close to what I like, but I do prefer -subme 10 over -subme 9, without having p4x4 enabled. I also prefer 5 b-frames and 5 ref frames, variance AQ, otherwise its a good preset for my preferences (and also just starting to have open-gop enabled) :).

[Dark Shikari]

Yes my mistake :) in my rush I took the settings from a muck around try different things encode I happened to be doing at the time! I've corrected it with the settings I usually use. That is, -- nr 400 and 5 ref, 5 b-frames.

I use those settings because quality vs filesize is my main concern, and speed second :) Since the question asked was purely about speed, I was intending to show my typical settings and processor to show that the speed shouldn't be too bad :)I was referring to the rc-lookahead, which is even more useless than any of the things you deemed "useless".

[burfadel]

Burfadel,

1. The script was an example only (which happened to be an SD source) I anamorphic encode everything (so as not to lose any detail) and resize to mod16 source resolution if necessary. I WAS using that script for all my encodes 1080i - 720p - 720i.

2. I don't encode bluray. Only recorded OTA. Hence deinterlacing is necessary.

3. My TV is a 50" Panasonic TH-P50V10A

http://panasonic.com.au/products/details.cfm?objectID=5102

5. I live in Australia. Our DTV is crap. The sample clip I have been testing with is actually 1440x1080i 14mbit mpeg2 + ac3 2 channel sound. It looks as rubbish as it sounds. Full of blocks, halos, artifacts etc. It is of a car race, so plenty of panning, fast action on static back grounds etc.

8. I hate tweaking avs files. Time is important to me. If Megui can't do it automatically with presets etc (I typically use one click encoding) I'm not interested. Hence the use of yadif and spline64.

9. I am currently reevaluating my choice of --crf 30. Currently am encoding the same test clip from --crf 16 in whole number increments to --crf 40 for another viewing test. I have stuck with --medium because that allows me to encode HD at real time or better speeds. I f I can justify it to myself, I will upgrade to i7 980x which may allow me to encode at --slow in real time.

10. Did I miss anything? Whew that is probably the longest post I have ever made on a internet forum. As you can see from my join date, I have been a long time lurker... :thanks: for everyone's contributions!

Please excuse the cutting out of parts of your thread in the quote :)

I'm in Australia too, Adelaide to be precise!

The TV you have is an excellent choice as you already know! the tv does have excellent processing circuitary in it, which probably helps a lot with the perceived encode quality. HOWEVER I would strongly suggest, if you can, SELL IT NOW!

Check this new model out (effectively the replacement model for yours):
http://panasonic.com.au/products/details.cfm?objectID=5636

Benefits of this model:
- 5,000,000:1 contrast over your 2,000,000:1
- Its a 3D tv :) of course you need those stupid shutter glasses, but at least you can watch things like the next Underbelly in 3D (its apparently going to be recorded in that), as well as anything else that comes along
- 4 star enegy rating vs your 2 star
- 13th gen panel vs your 12th gen

The power you'd save from a 4 star vs a 2 star would make up the difference in selling and buying price (haggle), especially if you use it several hours a day! I'm guessing the PVR functions via a portable HDD doesn't interest you, but one consideration is the playback features of the TV. If you record on to an external drive, and have some video's on another, may save having the computer on all the time, and you then can use STR standby more effectively, again saving power :)

I agree DTV quality here isn't the best, although I think OneHD is one of the worst offenders. I thought they got rid of 1440x1080 several years ago and now are all 1920x1080 or 720P in the case of ABC?

If it is 1440x1080, I would assume you're in a regional centre or Canberra, lol (not sure if they're on 1920x1080 or not). Also the broadcasts are either 576i (standard def digital in Australia), and for the HD streams 576P (think SBS may still be 576P), 720P (Abc HD), and 1080i (7, Nine, Ten). 1440x1080 is a leftover from 2001 when they brought in digital TV here (yes it was that early in the capitals, even 2000?) since back then the hardware wasn't as capable. Australia was a pretty early adopter which unfortunately leaves us with Mpeg-2 as the broadcast standard. Considering each station has to have a SD version of the HD stream, and they want multichannelling (7Two, Go, ABC2, ABC3, ABC 24news, SBS 2, SBS world news, One), it does leave to bandwidth crippling a bit since they're all in the same Transport stream. For channel Ten, since they have OneHD, also have One in SD in addition to their normal channel, with another rumoured to start later in the year. That will suck! I have heard h.264 has been trialled recently though :)

Now, pleae say the motor race you were recording was something worthy like a round of V8 Supercars :D

[Fat80y]

Burfadel,

Appreciate the advice on the tv, but I waited 10 years to get this one (I was waiting for combination of decent size, 1080p, film mode for bluray and price point) - I think trading it now would be akin to chasing my tail in x264 tweaks! I've basically given up on saving power - with a pool, tropical fishtank, constant rain necessitating clothes dryer use, a missus who likes the heater on with the door open so it doesn't get stuffy.. The list goes on! Its quite liberating actually to say "to hell with it! - I like my luxuries and I'm willing to pay for it!" That being said, I have opted for 50% green power to encourage alternative energy research...

I'm in Newcastle actually, and my test encode is definitely 1440x1080i - it is from OneHD, and is the Goodwell Festival of Speed 10 minutes of classic mini racing!!!

[Adub]

I must admit, I usually recommend using something like crf 18-20 (I usually use around 19.5). However this is while encoding from DVD and Blu-ray sources. If you are happy with the quality at your current settings, then by all means enjoy the reduced space usage!

[Mixer73]

I've basically given up on saving power - with a pool, tropical fishtank, constant rain necessitating clothes dryer use, a missus who likes the heater on with the door open so it doesn't get stuffy.. The list goes on! Its quite liberating actually to say "to hell with it! - I like my luxuries and I'm willing to pay for it!" That being said, I have opted for 50% green power to encourage alternative energy research...

Keep the plasma, trade the wife.

[Fat80y]

I must admit, I usually recommend using something like crf 18-20 (I usually use around 19.5). However this is while encoding from DVD and Blu-ray sources. If you are happy with the quality at your current settings, then by all means enjoy the reduced space usage!

Yes bluray and dvd are usually much better source material than tv. I think the last DVD I did was at crf 22.

I must add that watching a crf 30 encode on my computer (which has a 1080p 24" LCD) you can notice the differences compared to the source. However my face is only 18" from the screen, not 3m, and of course the further away you are the less detail you can resolve. If I had to watch these on the computer, I would probably shoot for crf 25ish.

[Fat80y]

Update:

I can't for the life of me get this to work with megui's one click encoder. Is there a way?

Or what's the easiest way to batch convert a bunch of mpg's?