x264 "Macroblock Tree Ratecontrol" testing (committed) [Archive] - Page 10

me7

11th November 2009, 11:28

Weightp 2 adds two more refs, not one, so set refs to 4.

Does weightp 1 add 1 more ref frame?

The calculation is exactly the same as normal. The only difference is for broken devices that don't obey the spec.

Does this mean that weightp adds more refs that should be handled different then normal refs, but some broken decoders handle them like normal refs and therefore need a lower number of normal refs to stay within H264 level limitations?

shon3i

11th November 2009, 11:49

number of normal refs to stay within VBV limitations?I just have same dillema since we haved b-pyramid violation some time ago. I run encode and i will check if pass verification for level 4.1

Dark Shikari

11th November 2009, 12:30

The specification places no limits on reference frames, only on DPB usage. If you replicate reference frames via reordering, that doesn't use extra DPB space.

me7

11th November 2009, 13:56

The specification places no limits on reference frames, only on DPB usage. If you replicate reference frames via reordering, that doesn't use extra DPB space.

Can you explain what you meant by this post:
Weightp 2 adds two more refs, not one, so set refs to 4.

DarkZell666

11th November 2009, 18:35

Can you explain what you meant by this post:Weightp 2 adds two more refs, not one, so set refs to 4.

Total number of refs used = number of refs you asked for + number of refs added by weightp. (ie: 5+2=7 in your case).

G_M_C

11th November 2009, 19:02

nurbs

11th November 2009, 19:11

As I read it you only need to reduce refs by two if your decoder isn't spec compliant.

yuvi

11th November 2009, 19:27

To make it clear, the commit message referred to CoreAVC 1.x breakage, which was due to something completely different and breaks no matter the number of reference frames. I don't think any other decoders were known to have problems specific to weightp.

This iPhone/iPod breakage may or may not be related to my guess of non-spec limitations with references (could someone confirm whether ref <= 4 works with weightp 2 and ref > 5 breaks with weightp 1?) and is probably unique either to Apple or to the Samsung chips used.

Dark Shikari

11th November 2009, 20:20

Hmmm, does that make encoding for bluray less efficient on 1080p projects ?

The number of refs set in the commandline a big influence on encoding efficiency. On 1080p BD projects, you can only have max refs of 4. If i can only set refs=2 when using weightp=2 project, a lot of efficiency is lost as opposed to refs=4 without weigtp (on a 1080p project).

Or am i wrong in this conclusion ?Why is it that even when I say something half a dozen times nobody ever reads my posts?

The limitation is on DPB size, not number of reference frames. Of course you can set refs=4 with Blu-ray encoding. Whoever said you couldn't? Nobody.

G_M_C

11th November 2009, 22:10

Why is it that even when I say something half a dozen times nobody ever reads my posts?

The limitation is on DPB size, not number of reference frames. Of course you can set refs=4 with Blu-ray encoding. Whoever said you couldn't? Nobody.

Yes, that can be. But you also said that weightp adds two reference frames;

Weightp 2 adds two more refs, not one, so set refs to 4.

And I reacted to that. So actually DO read your posts, i might not fully understand your thoughtprocess behind your posts, but i DO read them. And i react to them.

And btw; My STD does NOT accept ref=4 and weightp=2 together on 1080p. Nor does ffdshow, MPC or CoreAVC. ArcSoft TMT3 works however.

Expiriments on 720p (max refs on BD beeing 9) gives;
refs=8 plus weightp=2 no go
refs=6 plus weightp=2 works fine.
All other setting in the CMD kept the same, only refs changed. Can give CMD if needed. Used latest Techouse build, posted in the patched thread.

So weightp does influence the settings I have to use for making BD's.

PS: Running a full-length 720p encode now with refs 6/weightp 2. Will mux to AVCHD/BD to see what happens on my STD.

rack04

11th November 2009, 22:16

And btw; My STD does NOT accept ref=4 and weightp=2 together on 1080p. Nor does ffdshow, MPC or CoreAVC. ArcSoft TMT3 works however.

Haven't tested on my standalone player but in my tests ffdshow works fine with those settings. Can you upload a sample that doesn't work?

G_M_C

11th November 2009, 22:35

Haven't tested on my standalone player but in my tests ffdshow works fine with those settings. Can you upload a sample that doesn't work?

Overwritten the sample with the 720p i'm doing now. I'll see now if ffdshow works with that one, hold on (I've allready seen that CoreAVC doesnt work with this one, but that was allready known to happen).

Nope the raw .264 stream doesnt want to play correctly with my version of ffdshow (using Albains latest beta to test if we can get HD-audio streaming running).

PPS: Is Techouse's build maybe at fault ?

This is my CMD btw
x264.exe --threads auto --thread-input --psnr --ssim "%IN_TITLE%.avs" --stats "%IN_TITLE%.stats" --output "%OUT_TITLE%.264" --pass 2 --bitrate 4900 --deblock -2:-2 --fps 25 --aud --slices 4 --nal-hrd --vbv-bufsize 30000 --vbv-maxrate 24000 --sar 1:1 --weightp 2 --bframes 3 --b-adapt 2 --ref 6 --no-fast-pskip --qpmax 40 --ipratio 1.3 --pbratio 1.2 --direct auto --subme 10 --trellis 2 --psy-rd 1.0:0.2 --partitions all --me umh --merange 24 --mvrange 511 --aq-strength 1.0 --aq-mode 1 --rc-lookahead 75

rack04

11th November 2009, 22:59

Here are a couple samples I just encoded for you to try.

http://www.mediafire.com/?mmj5dyqodoz

EDIT: I just saw your post. Why are you trying to play the raw h264?

G_M_C

11th November 2009, 23:04

Here are a couple samples I just encoded for you to try.

http://www.mediafire.com/?mmj5dyqodoz

EDIT: I just saw your post. Why are you trying to play the raw h264?

Monitor the progress, and since it's a test; See if the decoder " takes" it. No sence in spending time muxing if the decoder flukes anyway ...

Sharc

11th November 2009, 23:33

Test clip 720p / weightp=2 / refs=3 plays fine with mpc-hc H264/AVC (DXVA) and coreAVC, but SAP BDP-S360 rejects the avchd disc.

Added:
Problem was with the disc. I burned new samples and they all play fine with the SAP (1080p, 720p, 576).

shon3i

12th November 2009, 00:32

Good to know, anyway Scenarist BD reject the stream at importing stage, with unspecified error message.

x264 cmd i used to create stream:
--tune film --bitrate 8306 --pass 2 --stats .stats --keyint 24 --min-keyint 2 --slices 4 --vbv-maxrate 30000 --vbv-bufsize 30000 --sar 1:1 --level 4.1 --threads 12 --aud --nal-hrd --output fnf2.264 fnf2.avs

Dark Shikari

12th November 2009, 00:49

The Blu-ray specification does not have any problem with what x264 is currently doing, so Scenarist is broken (or NAL-HRD patch is, but I doubt that).

http://i33.tinypic.com/2ijkg2g.png

Also, what the hell is this discussion doing in this thread? It's completely off-topic.

shon3i

12th November 2009, 01:09

@Dark Shikari, my mistake, i tryed to import some other bad stream, not one which i produced for test. Scenarist passes stream, and muxing is just fine. Resulting BD plays fine on TMT2 and 3 and PowerDVD. Tomorow i will test on some BD Player (LG 370 and 390) and WD HDTV player. And i will post results.

btw i compared fades to other non p-weight, and i see nice improvment.

madeinlisboa

1st March 2010, 12:05

Hi,

do you think mbtree is useful with very grainy sources like 300 or it is just a waste of processor time? Thank you

Dark Shikari

1st March 2010, 18:36

Hi,

do you think mbtree is useful with very grainy sources like 300 or it is just a waste of processor time? Thank youIt should be useful, just less so than usual.

asarian

2nd July 2010, 01:34

Switching to 64-Bit OS is always a good thing, as it allows to use more than ~3 GB of physical RAM, it allows to run 64-Bit processes where needed and 32-Bit processes will still work fine.
Interesting. :)

I wonder how that works out in practice, though. Can current (32-bit) AviSynth filters still be used? (provided I install 32 bit AviSynth, of course, on my new Windows 7 64-bit install). And then still use 64-bit x264? I very vaguely remember a post of yours, a good while back, in which you used some neat piping trick to mix 32 and 64 bit processes.

I'm not at the limit yet (x264 only uses ca 1.4 GB yet); but I don't have a lot of room to manoeuvre in, either (like for increasing rc_lookahead and merange).

Thanks.

Redsandro

2nd July 2010, 01:48

As always, 32 bit filters are not compatible with 64 bit software. Thats's a problem for me, as I used avisynth for all my proxies in Adobe After Effects and Première. It was an awesome workflow while it lasted.

Now, the software has become 64 bit only, and I cannot use 64 bit avisynth because most useful filters are 32 bit only.

I wish there were 2 separate versions at the same time on my system, like .avs for 32 bit avisynth and .a64 for 64 bit avisynth. That would solve some of the problems. 64 bit simple proxies and 32 bit fancy filter stuff.

asarian

2nd July 2010, 02:26

As always, 32 bit filters are not compatible with 64 bit software. Thats's a problem for me, as I used avisynth for all my proxies in Adobe After Effects and Première. It was an awesome workflow while it lasted.

Now, the software has become 64 bit only, and I cannot use 64 bit avisynth because most useful filters are 32 bit only.
Yeah. Too bad at that. I do remember LoRd_MuldeR having made some sort of program, though, a while back, in which he piped the output of 32-bit AviSynth to 64-bit x264. Can't find it any more, though. :(

Redsandro

2nd July 2010, 02:35

Yes you can do piping in the command window with different programs. If you want to do right-click .avs or .avi to .mp4 encodes from Windows, using 32 bit avisynth and 64 bit x264, you can use my x264box script (http://www.redsandro.com/?app) where I use avs2yuv.exe. Or if you just want to know how it's done you can look at the source, but basically it's something like avs2yuv.exe [AVSFILE] -o - | x264-64.exe [PARAMETERS] - --demuxer y4m

For audio you can do something similar using bepipe.exe from the behappy package.

Redsandro

2nd July 2010, 03:02

Since x264 is obviously the bottleneck, I think there is a 1% speed difference at very best.

But if you encode a lot, encode audio too, want to mux and use the same settings for speed/quality in one click (okay two :P) you should try my script. It's VBscript, you can fully customize it.

I use it to create mp4 proxies for video editing, but I change the default settings to be faster for that, and slow for the final encode.

ImmortAlex

2nd July 2010, 03:39

That program is pipebuf.exe, and command line is something like

pipebuf.exe avs2yuv.exe video.avs -raw -o - : x264.exe ... -o video.mkv - 720x576 : 4

I still use it because of some bug in ffmpeg64 which breaks decoding with ffdshow through DirectShowSource() in AviSynth64. Maybe that bug was fixed, I don't know. That solution works well for me, and I am too lazy to test AviSynth64 again :)

Redsandro

2nd July 2010, 03:53

Oh yeah, buffered pipe, there was an article on this forum somewhere about the ideal buffer size.

IIRC the best speed improvement was a couple dozen of frames on a big encode, that would be less than 1%.
But with the name of the program known one could surely find back that topic and see the comparisons.

asarian

2nd July 2010, 10:28

Thanks very much, guys! :) invaluable information, as usual! Time to start some serious work on Windows 7 64-bit!

asarian

4th July 2010, 01:38

Or if you just want to know how it's done you can look at the source, but basically it's something like avs2yuv.exe [AVSFILE] -o - | x264-64.exe [PARAMETERS] - --demuxer y4m

That works brilliantly. :) Thanks!

Any chance I can get avs2yuv to show x264's estimated time display? I kinda miss that now.

rack04

4th July 2010, 02:03

That works brilliantly. :) Thanks!

Any chance I can get avs2yuv to show x264's estimated time display? I kinda miss that now.

You will have to add the number of frames, i.e. --frames XXXXX

pipebuf.exe avs2yuv.exe [AVSFILE] - | x264.exe [PARAMETERS] - --frames XXXXXX --stdin y4m - : 4

asarian

4th July 2010, 21:53

You will have to add the number of frames, i.e. --frames XXXXX

pipebuf.exe avs2yuv.exe [AVSFILE] - | x264.exe [PARAMETERS] - --frames XXXXXX --stdin y4m - : 4
Thanks. Too bad, though; means you have to determine the total amount of frames first, then add it as parameter. Sorta wrecks the batchfile processing I had made for it. :) Who knows, maybe I should just ask the avs2yuv author to simply restore the 'eta' function.

LoRd_MuldeR

4th July 2010, 21:58

Thanks. Too bad, though; means you have to determine the total amount of frames first, then add it as parameter. Sorta wrecks the batchfile processing I had made for it.:)

You don't have to. But without "--frames" set properly x264 won't be able to display the progress properly, as the total number of frames is unknown in the case of STDIN input.

So if you want proper progress display with STDIN input, you must pass the number of frames to x264 explicitly. You can use avs2yuv.exe for that purpose...

"avs2yuv.exe" -frames 1 "%INPPATH%" -o NUL

asarian

4th July 2010, 22:05

You don't have to. But without "--frames" set properly x264 won't be able to display the progress properly, as the total number of frames is unknown in the case of STDIN input.
Duh on me! LOL. Yes, of course, we're streaming from STDIN here: x264 can't possible know upfront.

So if you want proper progress display with STDIN input, you must pass the number of frames to x264 explicitly. You can use avs2yuv.exe for that purpose...

"avs2yuv.exe" -frames 1 "%INPPATH%" -o NUL
Your knowledge in all areas video never ceases to amaze me. :) Thank you very much!

asarian

4th July 2010, 22:29

Oh man, this proves even more useful than I thought!

"c:\x264\avs2yuv.exe" -frames 1 "M:\jobs\2010.avs" -o NUL
M:\jobs\2010.avs: 1920x796, 24000/1001 fps, 166750 frames

This way I can even extract total resolution from it too, which avs2yuv seems to calculate accurately, even with multiple crops:

FFVideoSource("M:\jobs\2010.mkv").ConvertToYV12()
Crop(0, 140, 0, -140)
MCTemporalDenoise(settings="high")
Crop(0, 2, 0, -2)

VERY useful, as I can now create a small function, inside the batch-file, to auto-determine the max allowed ref frames for the source! Yay avs2yuv! :)

asarian

5th July 2010, 03:24

Okay, based on LoRd_MuldeR's very useful 1-liner command for avs2yuv, I wrote, like I said, a little batch file script accordingly, which will extract the number of frames, to restore the x264 'eta' functionality with avs2yuv, and which will calculate the appropriate max number of reference frames allowed for the input source. Both will be parsed to the full avs2yuv/x264 command line.

Adapt path-names and such to your own personal needs.

---------------
@echo off
cls

echo Retrieving info for: "%1.avs" ...

"c:\x264\avs2yuv.exe" -frames 1 "M:\jobs\%1.avs" -o NUL 2>"c:\x264\res.txt"
set /P _line=<"c:\x264\res.txt"

set _restline=%_line:*.avs: =%
set _restlinea=%_restline:*x=%
call set _width=%%_restline:x%_restlinea%=%%

set _restlineb=%_restlinea:*, =%
call set _height=%%_restlinea:, %_restlineb%=%%

set _restlinec=%_restlineb:* fps, =%
set _restlined=%_restlinex: frames%

call set _frames=%%_restlinec:%_restlined%=%%

set /a _reframes = 12582912 / (((%_width% * %_height%) * 3) / 2)

echo .

echo Width: %_width%
echo Height: %_height%
echo Frames: %_frames%
echo Reframes: %_reframes%

echo .

"C:\x264\avs2yuv.exe" "M:\jobs\%1.avs" -o - | "C:\x264\x264.exe" - --demuxer y4m --frames %_frames% --crf %2 --sar 1:1 --level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 60000 --ref %_reframes% --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 10 --trellis 2 --partitions all --8x8dct --me tesa --threads auto --merange 32 --aq-strength %4 --aq-mode 2 --rc-lookahead 60 --qpmin %3 --thread-input --output "C:\video\%1.264"

nurbs

5th July 2010, 07:43

Same command line:
C:\x264\avs2yuv.exe" "M:\jobs\%1.avs" -o - | "C:\x264\x264.exe" - --demuxer y4m --frames %_frames% --crf %2 --sar 1:1 --level 4.1 --aud --preset placebo --vbv-bufsize 70000 --vbv-maxrate 60000 --bframes 8 --merange 32 --aq-strength %4 --aq-mode 2 --qpmin %3 --output "C:\video\%1.264"

Half of the stuff in your command line were defaults or included in placebo. As long as you don't specify the number of reference frames manually x264 will automatically reduce them if you use --level.
I don't think you need --aud unless you are doing Blu-Ray/AVCHD, which you aren't. If you don't specify --merange the above command line will use 24 which is still plenty. You should think about using the tunings (film/animation) so that psy-trellis is used and the psy and aq strenght is set according to content.

asarian

5th July 2010, 08:23

Same command line:
C:\x264\avs2yuv.exe" "M:\jobs\%1.avs" -o - | "C:\x264\x264.exe" - --demuxer y4m --frames %_frames% --crf %2 --sar 1:1 --level 4.1 --aud --preset placebo --vbv-bufsize 70000 --vbv-maxrate 60000 --bframes 8 --merange 32 --aq-strength %4 --aq-mode 2 --qpmin %3 --output "C:\video\%1.264"

Half of the stuff in your command line were defaults or included in placebo.
Yeah, I should really clean up that old, clunky (adapted) command line to the newer stuff. :)

As long as you don't specify the number of reference frames manually x264 will automatically reduce them if you use --level.
Well, the idea was actually to increase them to as high as is allowed for the current DPB. So, it will be 5 for 1920x800, and 9 for 1280x720, etc.

nurbs

5th July 2010, 08:59

--preset placebo uses 16 reference frames. With --level 4.1 they will be reduced. So, it will be 5 for 1920x800, and 9 for 1280x720, etc.

asarian

5th July 2010, 09:25

--preset placebo uses 16 reference frames. With --level 4.1 they will be reduced. So, it will be 5 for 1920x800, and 9 for 1280x720, etc.
Ah, I didn't realize that. :) Thanks. I thought '--level 4.1' would always just reduce it to 4 (standard Blu-Ray).

Redsandro

5th July 2010, 16:20

@echo off
cls

echo Retrieving info for: "%1.avs" ...

"c:\x264\avs2yuv.exe" -frames 1 "M:\jobs\%1.avs" -o NUL 2>"c:\x264\res.txt"
set /P _line=<"c:\x264\res.txt"

set _restline=%_line:*.avs: =%
set _restlinea=%_restline:*x=%
call set _width=%%_restline:x%_restlinea%=%%

set _restlineb=%_restlinea:*, =%
call set _height=%%_restlinea:, %_restlineb%=%%

set _restlinec=%_restlineb:* fps, =%
set _restlined=%_restlinex: frames%

call set _frames=%%_restlinec:%_restlined%=%%

set /a _reframes = 12582912 / (((%_width% * %_height%) * 3) / 2)

echo .

echo Width: %_width%
echo Height: %_height%
echo Frames: %_frames%
echo Reframes: %_reframes%

echo .

"C:\x264\avs2yuv.exe" "M:\jobs\%1.avs" -o - | "C:\x264\x264.exe" - --demuxer y4m --frames %_frames% --crf %2 --sar 1:1 --level 4.1 --aud --nal-hrd none --vbv-bufsize 70000 --vbv-maxrate 60000 --ref %_reframes% --b-pyramid normal --mixed-refs --bframes 8 --b-adapt 2 --weightb --no-fast-pskip --direct auto --subme 10 --trellis 2 --partitions all --8x8dct --me tesa --threads auto --merange 32 --aq-strength %4 --aq-mode 2 --rc-lookahead 60 --qpmin %3 --thread-input --output "C:\video\%1.264"

Nice batch! I didn't know you could do that line splitting in Command.