x265 HEVC Encoder

[LigH]

What exactly is the changed condition to enable 2-pass CRF? If I read the patches correctly: Use a VBV maximum bitrate in the 1st pass, so you can either let x265 calculate the optimal RF with VBV constraints to achieve a target size (in VBR mode), or use a specified RF with VBV constraints regardless of a target size (in VBV-CRF mode)?
__

New build with changed 2-pass VBV-CRF conditions:

x265 1.9+15-425b583f25db (GCC 5.3.0)

[Ma]

Originally Posted by pradeeprama

I see that you're trying to resort --pools +,+ to effectively mean --pools 24,24 (or whatever the # threads per socket is) on Windows.

Yes, for '--pools +,+' and '--pools *' and no pools options it simulates '--pools 24,24' option. I think it is simple and more effective than current code and better than backout commit 10983.

I hope that you find better fix.

[qyot27]

I apologize for not following up on this much earlier, but this is a basic rundown of the issues when cross-compiling x265-yuuki currently:

input-filters.diff doesn't cleanly apply because the amount of fuzz got too large.

CMake has to be run twice to even detect that -DENABLE_LAVF has been turned on and get the locations for the libs. This is due to the position of the ENABLE_LAVF block in CMakeLists.txt
coming before the ENABLE_CLI block, but requiring the CLI is enabled. If the ENABLE_LAVF block is moved to a position after the CLI is turned on, or if the CLI requirement is removed
from the ENABLE_LAVF block, then it can detect the libs on the first try (although it seemed to cause other issues during building or linking).

There are problems linking against FFmpeg's private libs, especially when all of them are built static. zlib, bzip2, lzma, and iconv are all autodetected and used if present when FFmpeg is configured,
and these cause errors on link with the lavf module because the libs haven't been properly added to the linker list by cmake. To say nothing of when more expansive configurations with more
external libs are present, or for other options that are enabled by default (like what's brought in if the user doesn't explicitly use --disable-network). If the user explicitly disables these four libraries
and the network support when building FFmpeg, then x265 can link to them just fine, but otherwise it requires the user to manually append the list of linker libs to the failed link command, and this
then breaks the ability of ninja to do a proper install, even though the .exe is okay.

FFmpeg API usage needs to be updated. PIX_FMT_* needs to be changed to AV_PIX_FMT_* to allow compilation to proceed successfully, and even though it's just a warning, uses of deprecated
functions should be updated to non-deprecated ones (which x264 recently fixed for the lavf module there):

[78/88] Building CXX object CMakeFiles/cli.dir/input/lavf.cpp.obj
/home/qyot27/x265-build/x265-yuuki/source/input/lavf.cpp: In member function 'bool x265::LavfInput::readPicture(x265_picture&, x265::InputFileInfo*)':
/home/qyot27/x265-build/x265-yuuki/source/input/lavf.cpp:108:13: warning: 'void av_free_packet(AVPacket*)' is deprecated [-Wdeprecated-declarations]
             av_free_packet(&pkt);
             ^
In file included from /usr/i686-w64-mingw32/include/libavformat/avformat.h:318:0,
                 from /home/qyot27/x265-build/x265-yuuki/source/input/lavf.h:12,
                 from /home/qyot27/x265-build/x265-yuuki/source/input/lavf.cpp:29:
/usr/i686-w64-mingw32/include/libavcodec/avcodec.h:4040:6: note: declared here
 void av_free_packet(AVPacket *pkt);
      ^
/home/qyot27/x265-build/x265-yuuki/source/input/lavf.cpp:108:13: warning: 'void av_free_packet(AVPacket*)' is deprecated [-Wdeprecated-declarations]
             av_free_packet(&pkt);
             ^
In file included from /usr/i686-w64-mingw32/include/libavformat/avformat.h:318:0,
                 from /home/qyot27/x265-build/x265-yuuki/source/input/lavf.h:12,
                 from /home/qyot27/x265-build/x265-yuuki/source/input/lavf.cpp:29:
/usr/i686-w64-mingw32/include/libavcodec/avcodec.h:4040:6: note: declared here
 void av_free_packet(AVPacket *pkt);
      ^
/home/qyot27/x265-build/x265-yuuki/source/input/lavf.cpp:108:32: warning: 'void av_free_packet(AVPacket*)' is deprecated [-Wdeprecated-declarations]
             av_free_packet(&pkt);
                                ^
In file included from /usr/i686-w64-mingw32/include/libavformat/avformat.h:318:0,
                 from /home/qyot27/x265-build/x265-yuuki/source/input/lavf.h:12,
                 from /home/qyot27/x265-build/x265-yuuki/source/input/lavf.cpp:29:
/usr/i686-w64-mingw32/include/libavcodec/avcodec.h:4040:6: note: declared here
 void av_free_packet(AVPacket *pkt);
      ^
[88/88] Linking CXX executable x265.exe
[8bit:$]

The ENABLE_LAVF block in input.cpp should really probably be changed to accept any file format lavf can. AviSynth was the main reason to argue it before, but really, only supporting *.mp4 and
*.mkv as input is too limiting (other containers and secondary extensions like *.m2ts, *.ts, *.mpg, *.ogv, *.webm, *.vob and several others are also found commonly enough to be nice to open).
The easiest solution would be to simply make the block into an else and the return statement, so anything that's not y4m or a pipe gets handled by lavf (and presumably lavf would error out in the
case that it can't open the file, although this probably needs more error-proofing in input/lavf.cpp unless it already does emit a clean error and not a crash).


The zimg filter also fails to build because of an error in zimgfilter.h:

[86/88] Building CXX object CMakeFiles/cli.dir/filters/filters.cpp.obj
FAILED: /usr/bin/i686-w64-mingw32-g++   -DENABLE_LAVF -DENABLE_LSMASH -DENABLE_MKV -DENABLE_ZIMG -DEXPORT_C_API=1 -DHAVE_INT_TYPES_H=1 -DHIGH_BIT_DEPTH=0 -DX265_ARCH_X86=1 -DX265_DEPTH=8 -DX265_NS=x265 -D_WIN32_WINNT=_WIN32_WINNT_WINXP -D_WIN32_WINNT_WIN7=0x0601 -D__STDC_LIMIT_MACROS=1 -mfpmath=sse -march=pentium3 -msse -mtune=pentium3 -O3 -DNDEBUG -I/home/qyot27/x265-build/x265-yuuki/source/. -I/home/qyot27/x265-build/x265-yuuki/source/common -I/home/qyot27/x265-build/x265-yuuki/source/encoder -I.    -Wall -Wextra -Wshadow -std=gnu++98 -march=i686 -Wno-array-bounds -ffast-math -mstackrealign -fno-exceptions -MMD -MT CMakeFiles/cli.dir/filters/filters.cpp.obj -MF CMakeFiles/cli.dir/filters/filters.cpp.obj.d -o CMakeFiles/cli.dir/filters/filters.cpp.obj -c /home/qyot27/x265-build/x265-yuuki/source/filters/filters.cpp
In file included from /home/qyot27/x265-build/x265-yuuki/source/filters/filters.cpp:27:0:
/home/qyot27/x265-build/x265-yuuki/source/filters/zimgfilter.h:56:5: error: 'zimg_resize_context' does not name a type
     zimg_resize_context* resizeCtx[3];
     ^
/home/qyot27/x265-build/x265-yuuki/source/filters/zimgfilter.h:57:5: error: 'zimg_depth_context' does not name a type
     zimg_depth_context* depthCtx;
     ^
[86/88] Building CXX object CMakeFiles/cli.dir/x265.cpp.obj
ninja: build stopped: subcommand failed.

This is with a git build of zimg, so 2.0.4 or something. With the older 1.1.1 release (which I tested only because I thought maybe it might have been expecting that), there's even more compiler
noise thrown and the above errors still apply.


For reference, this is the set of instructions I'm following:

FFmpeg:
git clone git://source.ffmpeg.org/ffmpeg.git && \
cd ffmpeg && \
    ./configure --prefix=/usr/i686-w64-mingw32 --cross-prefix=i686-w64-mingw32- \
    --enable-gpl --enable-version3 --disable-w32threads --enable-avresample \
    --disable-encoders --disable-muxers --disable-doc --disable-debug \
    --disable-devices --disable-avdevice --enable-avisynth --cpu=pentium3 \
    --extra-cflags="-mfpmath=sse -march=pentium3 -msse -mtune=pentium3" \
    --target-os=mingw32 --arch=x86 && \
make -j$(nproc) && \
    sudo checkinstall --pkgname=ffmpeg-minimal-mingw --pkgversion="$(git \
    rev-list --count HEAD)-g$(git rev-parse --short HEAD)-$(date --rfc-3339=date | \
    sed 's/-//g')-git" --backup=no --deldoc=yes --delspec=yes --deldesc=yes \
    --strip=yes --fstrans=no --default

Add --disable-network --disable-zlib --disable-bz2 --disable-lzma to keep FFmpeg from
calling in private libs, otherwise, see pkg-config note at the end.

mkdir -p ~/x265-build/12bit ~/x265-build/10bit ~/x265-build/8bit && \
cd ~/x265-build && \
hg qclone https://bitbucket.org/msg7086/x265-yuuki && \
cd x265-yuuki && \
hg qpush -a && \

Condense the ifdef ENABLE_LAVF block in source/input/input.cpp to:
#ifdef ENABLE_LAVF
    else
        return new LavfInput(info);
#endif

Correct uses of PIX_FMT_* to AV_PIX_FMT_*
sed -i 's/case PIX_FMT/case AV_PIX_FMT/g' source/input/lavf.cpp

cd ../12bit && \
    cmake ../x265-yuuki/source -G "Ninja" -DCMAKE_INSTALL_PREFIX="$HOME/x265_build/x265-12bit" \
    -DCMAKE_TOOLCHAIN_FILE="/usr/i686-w64-mingw32/toolchain-i686-w64-mingw32.cmake" \
    -DCMAKE_CXX_FLAGS="-mfpmath=sse -march=pentium3 -msse -mtune=pentium3" \
    -DHIGH_BIT_DEPTH:bool=on -DMAIN12:bool=on -DENABLE_SHARED:bool=off \
    -DENABLE_ASSEMBLY:bool=off -DEXPORT_C_API:bool=off -DENABLE_CLI:bool=off \
    [-DWINXP_SUPPORT:bool=on] && \
ninja && \
    sudo checkinstall --pkgname=x265-main12-mingw --pkgversion="$(grep X265_VERSION \
    build.ninja | sed 's/X265_VERSION=/\t/' | cut -f2 | sed 's/ /\t/g' | cut -f1)-$(date \
    --rfc-3339=date | sed 's/-//g')-hg" --backup=no --deldoc=yes --delspec=yes \
    --deldesc=yes --strip=yes --fstrans=no --default cp libx265.a /usr/i686-w64-mingw32/lib/libx265_main12.a && \
mv *.deb ~/mingw_debs

# Build 10-bit:
cd ../10bit && \
    cmake ../x265-yuuki/source -G "Ninja" -DCMAKE_INSTALL_PREFIX="$HOME/x265_build/x265-10bit" \
    -DCMAKE_TOOLCHAIN_FILE="/usr/i686-w64-mingw32/toolchain-i686-w64-mingw32.cmake" \
    -DCMAKE_CXX_FLAGS="-mfpmath=sse -march=pentium3 -msse -mtune=pentium3" \
    -DHIGH_BIT_DEPTH:bool=on -DENABLE_SHARED:bool=off -DENABLE_ASSEMBLY:bool=off \
    -DEXPORT_C_API:bool=off -DENABLE_CLI:bool=off [-DWINXP_SUPPORT:bool=on] && \
ninja && \
    sudo checkinstall --pkgname=x265-main10-mingw --pkgversion="$(grep X265_VERSION \
    build.ninja | sed 's/X265_VERSION=/\t/' | cut -f2 | sed 's/ /\t/g' | cut -f1)-$(date \
    --rfc-3339=date | sed 's/-//g')-hg" --backup=no --deldoc=yes --delspec=yes \
    --deldesc=yes --strip=yes --fstrans=no --default cp libx265.a /usr/i686-w64-mingw32/lib/libx265_main10.a && \
mv *.deb ~/mingw_debs

# Only the .a files from 12-bit and 10-bit are installed to the system,
# to reduce the chances of conflicts.

# Build 8-bit:
cd ../8bit && \
    cmake ../x265-yuuki/source -G "Ninja" -DCMAKE_INSTALL_PREFIX="/usr/i686-w64-mingw32" \
    -DCMAKE_PREFIX_PATH="/usr/i686-w64-mingw32" \
    -DCMAKE_TOOLCHAIN_FILE="/usr/i686-w64-mingw32/toolchain-i686-w64-mingw32.cmake" \
    -DCMAKE_CXX_FLAGS="-mfpmath=sse -march=pentium3 -msse -mtune=pentium3" \
    -DENABLE_SHARED:bool=off -DEXTRA_LINK_FLAGS=-L. -DLINKED_10BIT:bool=on -DLINKED_12BIT:bool=on \
    -DEXTRA_LIB="/usr/i686-w64-mingw32/lib/libx265_main10.a;/usr/i686-w64-mingw32/lib/libx265_main12.a" \
    -DENABLE_LSMASH:bool=on -DENABLE_MKV:bool=on -DENABLE_LAVF:bool=on [-DWINXP_SUPPORT:bool=on] && \

    cmake ../x265-yuuki/source -G "Ninja" -DCMAKE_INSTALL_PREFIX="/usr/i686-w64-mingw32" \
    -DCMAKE_PREFIX_PATH="/usr/i686-w64-mingw32" \
    -DCMAKE_TOOLCHAIN_FILE="/usr/i686-w64-mingw32/toolchain-i686-w64-mingw32.cmake" \
    -DCMAKE_CXX_FLAGS="-mfpmath=sse -march=pentium3 -msse -mtune=pentium3" \
    -DENABLE_SHARED:bool=off -DEXTRA_LINK_FLAGS=-L. -DLINKED_10BIT:bool=on -DLINKED_12BIT:bool=on \
    -DEXTRA_LIB="/usr/i686-w64-mingw32/lib/libx265_main10.a;/usr/i686-w64-mingw32/lib/libx265_main12.a" \
    -DENABLE_LSMASH:bool=on -DENABLE_MKV:bool=on -DENABLE_LAVF:bool=on [-DWINXP_SUPPORT:bool=on] && \
ninja && \
sed -i 's/lx265/lx265 -lx265_main10 -lx265_main12/' x265.pc && \
    sudo checkinstall --pkgname=x265-mingw --pkgversion="$(grep X265_VERSION \
    build.ninja | sed 's/X265_VERSION=/\t/' | cut -f2 | sed 's/ /\t/g' | cut -f1)-$(date \
    --rfc-3339=date | sed 's/-//g')-hg" --backup=no --deldoc=yes --delspec=yes \
    --deldesc=yes --strip=yes --fstrans=no --default ninja install && \
mv *.deb ~/mingw_debs

CMake step needs to be run twice so FFmpeg is detected, but it only gets detected
if it's installed to the system mingw-w64 area.  Then, if the zlib/bz2/lzma/iconv libs
and network support are still enabled, it fails to link.  This can be fixed by appending the output of:
PKG_CONFIG_PATH=/usr/i686-w64-mingw32/lib/pkgconfig pkg-config --cflags --libs --static libavcodec libavformat libswscale

to the failed link command*.  After this, the install step doesn't work, but x265.exe does.

*which, if following the FFmpeg instructs at the beginning, should be something like:
-I/usr/i686-w64-mingw32/include -lavcodec -lws2_32 -liconv -lm -llzma -lbz2 -lz -lpsapi -ladvapi32 -lshell32 -lswresample -lavutil -lavformat -pthread -lswscale -L/usr/i686-w64-mingw32/lib

sudo apt-get purge ffmpeg-minimal-mingw
(this needs to be purged to prevent it interfering with future ffmpeg or mpv builds)

[MeteorRain]

Originally Posted by qyot27

I apologize for not following up on this much earlier, but this is a basic rundown of the issues when cross-compiling x265-yuuki currently:

Thank you for all the efforts.

Please note that I have moved my repo to Github due to some personal reason.

And if you can provide some fixes, feel free to send a PR for me.

The zimg referenced is a pretty old version, mainly due to when I built this patch, it was still in 1.x . If you somehow port it to 2.x, please send a PR. I don't have much time to look into it for the moment.

My builds on Linux (and was successful) are compiled against:
zimg 1.1.1
ffmpeg 2.8.5-1+b1 (debian 9 sys pkg)
l-smash rev1384

[MeteorRain]

Originally Posted by qyot27

I apologize for not following up on this much earlier, but this is a basic rundown of the issues when cross-compiling x265-yuuki currently:

input-filters.diff doesn't cleanly apply because the amount of fuzz got too large.

Using HG is always a hassle for me. Moving to git and I'm feeling much better when dealing with patches.

Originally Posted by qyot27

CMake has to be run twice to even detect that -DENABLE_LAVF has been turned on and get the locations for the libs.

Thanks, this has been fixed and pushed.

Originally Posted by qyot27

There are problems linking against FFmpeg's private libs, especially when all of them are built static. zlib, bzip2, lzma, and iconv are all autodetected and used if present when FFmpeg is configured,
and these cause errors on link with the lavf module because the libs haven't been properly added to the linker list by cmake. To say nothing of when more expansive configurations with more
external libs are present, or for other options that are enabled by default (like what's brought in if the user doesn't explicitly use --disable-network). If the user explicitly disables these four libraries
and the network support when building FFmpeg, then x265 can link to them just fine, but otherwise it requires the user to manually append the list of linker libs to the failed link command, and this
then breaks the ability of ninja to do a proper install, even though the .exe is okay.

I guess, with the latest patch, cmake should be able to detect your pkgconfig files and automatically include these dependencies into consideration. If not, we'll see if someone will send a PR to fix it. I don't have much experience on cmake, sorry.

Originally Posted by qyot27

FFmpeg API usage needs to be updated. PIX_FMT_* needs to be changed to AV_PIX_FMT_* to allow compilation to proceed successfully, and even though it's just a warning, uses of deprecated
functions should be updated to non-deprecated ones (which x264 recently fixed for the lavf module there):

These have been fixed with latest patch. BTW I couldn't find the x264 fixes so I modified it according to your description.

Originally Posted by qyot27

The ENABLE_LAVF block in input.cpp should really probably be changed to accept any file format lavf can.

The LAVF part is ported poorly. Besides, I planned to have VFR support but didn't actually finished that. So I cannot guarantee that it's working well with various of types. I'd probably extend the list a bit, but I'd like to keep it minimal until someone has actually reviewed the code.

Originally Posted by qyot27

The zimg filter also fails to build because of an error in zimgfilter.h:

This is with a git build of zimg, so 2.0.4 or something. With the older 1.1.1 release (which I tested only because I thought maybe it might have been expecting that), there's even more compiler
noise thrown and the above errors still apply.

This is on my todo-list, but is also open for PR.

[qyot27]

Originally Posted by MeteorRain

Using HG is always a hassle for me. Moving to git and I'm feeling much better when dealing with patches.

I agree, I have practically zero working knowledge of Mercurial (enough to clone and look at the log, and I managed to figure out qrefresh/qpush when dealing with the patchsets). I actually was more or less ready to send a pull request on Github, but it looks like I'll need to review which changes can go through now.

I guess, with the latest patch, cmake should be able to detect your pkgconfig files and automatically include these dependencies into consideration. If not, we'll see if someone will send a PR to fix it. I don't have much experience on cmake, sorry.

The solution I'd ultimately came up with was to spin the case of a static FFmpeg off into its own option so that the existing ENABLE_LAVF continues to work as it always has with shared libs.

I'll test to see if the fix I'd implemented is irrelevant now.

These have been fixed with latest patch. BTW I couldn't find the x264 fixes so I modified it according to your description.

Huh, I thought I'd seen x264 stop complaining about using deprecated functions (av_free_packet, etc.) when building the lavf input. The pix_fmt IDs were separate from that.

[qyot27]

I've opened a pull request with two patches to address these points:

Originally Posted by MeteorRain

I guess, with the latest patch, cmake should be able to detect your pkgconfig files and automatically include these dependencies into consideration. If not, we'll see if someone will send a PR to fix it. I don't have much experience on cmake, sorry.

It's not so much that the pkgconfig files weren't detected, it was that it wasn't adding the content of Libs.private to the linker list. It was populating the right *_STATIC_* variables in CMakeCache.txt, though, so I was able to get around the problem by calling the variables directly.

The LAVF part is ported poorly. Besides, I planned to have VFR support but didn't actually finished that. So I cannot guarantee that it's working well with various of types. I'd probably extend the list a bit, but I'd like to keep it minimal until someone has actually reviewed the code.

Even though I'd simply prefer a wildcard, I went ahead and extended the list manually with the most common (or significant) extensions I could think of. The only one I forgot to include was *.webm, but considering those are just a subset of Matroska, it's not like the user can't just rename them *.mkv.

[MeteorRain]

Originally Posted by qyot27

I've opened a pull request with two patches to address these points:

Thanks, and that has been [del]merged[/del] rebased.

[n808]

Trying to encode a standard test sequence (BasketballDrill) using 2-pass and the latest 1.9 build ( as well as earlier builds), the 2nd pass creates a bitrate just a fraction of the requested..

My parameters are the same as I am using for x264:

x265 BasketballDrill_832x480_50.yuv --input-res 832x480 --frames 100 --preset slow --fps 50 -o bits --keyint 100 --bitrate 1000 --pass 1
x265 BasketballDrill_832x480_50.yuv --input-res 832x480 --frames 100 --preset slow --fps 50 -o bits --keyint 100 --bitrate 1000 --pass 2

Stats for the 2 passes:
1:encoded 100 frames in 2.94s (33.99 fps), 891.82 kb/s, Avg QP:36.8
2:encoded 100 frames in 3.46s (28.86 fps), 109.70 kb/s, Avg QP:51.00

There must be something basic I am missing..?

[benwaggoner]

Originally Posted by n808

Trying to encode a standard test sequence (BasketballDrill) using 2-pass and the latest 1.9 build ( as well as earlier builds), the 2nd pass creates a bitrate just a fraction of the requested..

My parameters are the same as I am using for x264:

x265 BasketballDrill_832x480_50.yuv --input-res 832x480 --frames 100 --preset slow --fps 50 -o bits --keyint 100 --bitrate 1000 --pass 1
x265 BasketballDrill_832x480_50.yuv --input-res 832x480 --frames 100 --preset slow --fps 50 -o bits --keyint 100 --bitrate 1000 --pass 2

Stats for the 2 passes:
1:encoded 100 frames in 2.94s (33.99 fps), 891.82 kb/s, Avg QP:36.8
2:encoded 100 frames in 3.46s (28.86 fps), 109.70 kb/s, Avg QP:51.00

There must be something basic I am missing..?

Encoding just 100 frames is just 2 seconds, way less than a GOP, and just 4x the slow's lookahead of 25 frames. How does the actual output look? You can output to bits_1p.hevc and bits_2p.hevc to compare.

I would expect the 2nd pass rate control to be a whole lot better than the 1st for such a short clip. If there's even a second of black leader, that'd throw everything off.

[n808]

It's a commonly used test sequence for standards development, including HEVC, so there's no black leader..

x264 encodes the same 100 frames with the exact same parameters like this:
1:encoded 100 frames, 237.53 fps, 885.64 kb/s
2:encoded 100 frames, 73.05 fps, 1045.64 kb/s

Something is not working right. x265 seems to clamp to QP to max 51. I may check if libx265 in ffmpeg works any better.

[LigH]

It would probably be useful to see the complete console output and maybe even a CSV log (level 1 or 2) of this second pass. There must be a reason why the target bitrate is apparently not relevant for x265 in the 2nd pass.
_

BTW:

For a more efficient command line, you may like to convert raw YUV to Y4M; this may be possible with y4mtools, MJPGTools (Linux), AviSynth with RawSource26 and avs2pipemod, or ffmpeg (-f yuv4mpegpipe).

[n808]

I re-created the same results with a compressed version (x264 crf 10) available here. if anyone else wants to try.

I converted this to Y4M using ffmpeg. Then ran the same two commands (adjusted to remove size and fps). The output in the 2nd pass is at the bottom. It's likely that "Error: 2pass curve failed to converge" may be causing the problem, but I don't know how to set parameters to avoid it. If I try higher bitrates, e.g 2000kbps, the error above is gone, but it still misses the mark completely (1795 kbps 1st pass / 789 kbps 2nd pass). Similarly, 2nd pass for for 4000 and 8000 kbps is much much worse than 1st pass, with a lower bitrate that requested.

P.S. I am using raw YUV due to some tools in my toolchain do not yet support Y4M.

====
y4m [info]: 832x480 fps 50/1 i420p8 sar 1:1 frames 0 - 99 of 100
raw [info]: output file: bit
x265 [info]: HEVC encoder version 1.9+32-c2228fb8151d
x265 [info]: build info [Windows][GCC 5.3.0][64 bit] 8bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main profile, Level-3.1 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: frame threads / pool features : 2 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [warning]: Error: 2pass curve failed to converge
x265 [warning]: target: 1000.00 kbit/s, expected: 1140.97 kbit/s, avg QP: 68.7127
x265 [warning]: try increasing target bitrate
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut : 10 / 100 / 40
x265 [info]: Lookahead / bframes / badapt : 25 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb : 1 / 1 / 0
x265 [info]: References / ref-limit cu / depth : 4 / 1 / 1
x265 [info]: AQ: mode / str / qg-size / cu-tree : 1 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress : ABR-1000 kbps / 0.60
x265 [info]: tools: rect limit-modes rd=4 psy-rd=2.00 rdoq=2 psy-rdoq=1.00
x265 [info]: tools: signhide tmvp strong-intra-smoothing deblock sao stats-read
[1.0%] 1/100 frames, 5.75 fps, 738.00 kb/s, eta 0:00:17
[10.0%] 10/100 frames, 22.52 fps, 166.76 kb/s, eta 0:00:03
[19.0%] 19/100 frames, 26.32 fps, 137.94 kb/s, eta 0:00:03
[27.0%] 27/100 frames, 27.69 fps, 129.30 kb/s, eta 0:00:02
[35.0%] 35/100 frames, 27.91 fps, 125.79 kb/s, eta 0:00:02
[43.0%] 43/100 frames, 28.33 fps, 125.13 kb/s, eta 0:00:02
[51.0%] 51/100 frames, 28.70 fps, 124.24 kb/s, eta 0:00:01
[59.0%] 59/100 frames, 28.99 fps, 122.76 kb/s, eta 0:00:01
[67.0%] 67/100 frames, 29.13 fps, 122.07 kb/s, eta 0:00:01
[75.0%] 75/100 frames, 29.41 fps, 121.13 kb/s, eta 0:00:00
[84.0%] 84/100 frames, 29.72 fps, 118.84 kb/s, eta 0:00:00
[92.0%] 92/100 frames, 29.89 fps, 117.43 kb/s, eta 0:00:00

x265 [info]: frame I: 1, Avg QP:51.00 kb/s: 308.80
x265 [info]: frame P: 24, Avg QP:51.00 kb/s: 183.78
x265 [info]: frame B: 75, Avg QP:51.00 kb/s: 83.48
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 8.0% 0.0% 4.0% 60.0% 28.0%

encoded 100 frames in 3.29s (30.40 fps), 109.80 kb/s, Avg QP:51.00

[LigH]

x265 [warning]: Error: 2pass curve failed to converge
x265 [warning]: target: 1000.00 kbit/s, expected: 1140.97 kbit/s, avg QP: 68.7127
x265 [warning]: try increasing target bitrate

True, a strong hint, but directions are not yet clear to me. Apparently x265 believes that QP 51 is sufficient to reach the target bitrate, which is nonsense and results in a bitrate way below par. I will try to recreate your test, but don't have the same hardware (sometimes bugs are CPU instruction dependend, who knows...).

BTW: Is this BasketballDrill_832x480_50.yuv downladable somewhere? It's not in Derf's collection, unfortunately.
__

Trying similar parameters with a smaller file on an AMD Phenom-II X4, everything works well.

I believe a common reason for 2-pass issues was that a statistics file created in a 1st pas was not read correctly or misinterpreted... For now I have no better idea.
__

I can confirm this behaviour, using a CRF10 AVC-in-MKV source of BasketballDrill (uploaded a copy to my archive).

Furthermore, for different other sources, 2-pass VBR misses the target bitrate, the result has about 75-80%.

I will notify the x265 Developers mailing list about this case.
__

I was able to narrow it down between a few archived builds:

This issue was introduced between versions 1.8+167-e951ab673b1c and 1.8+201-769081eb5f4c.

[LigH]

Hum, funny stuff.

I received BasketballDrill_832x480_50.yuv again from staff of the Uni Hannover. It has originally 501 frames. Running them, the bitrate calculation converges.

So I have to wonder if the reason behind this issue is indeed that you encode only 100 frames with a keyframe distance of 100 frames.

Indeed: Adding "--frames 100 --keyint 100" to the command line provokes the miscalculation.

[n808]

@LigH: Thanks for the help! However, I still cannot get it to work properly, and I am little confused by a couple of your answers.

When I use all 501 frames, it somewhat improves, but 2nd pass is still much worse than 1st pass:
1: encoded 501 frames in 14.57s (34.38 fps), 1029.45 kb/s, Avg QP:35.94
2: encoded 501 frames in 31.30s (16.00 fps), 711.98 kb/s, Avg QP:38.16

When I encode the shortened 100 frames version (Y4M) with neither --frames, nor --keyint, the resulting bitrate is no better than with those parameters. So it seems 2 pass does not work on short clips at all. I tried other clips (100 frames) and the problem is the same.

>> Furthermore, for different other sources, 2-pass VBR misses the target bitrate, the result has about 75-80%.

OK. That's what I am seeing as well, for longer sequences and other bitrates. See 1st paragraph. It seems 1-pass is the only option for now.

>> Trying similar parameters with a smaller file on an AMD Phenom-II X4, everything works well.

Is this also the case for the command line parameters and clip in question (posted above)? How about the 75-80% 2nd pass issue in general?

(Yeah, these test sequences are hard to track down without the right connections...)

[LigH]

Remember, I found that this issue was introduced during a specific period of development. Now we have to give the developers some time to discover which change exactly caused it, so they can later fix it... But I hope I could help good enough to find the reason quickly now.

I believe the extreme result of being unable to converge always happens when you have only one GOP; when (what I guess may be the reason) the statistics of the first GOP are accidentally skipped, then everything is skipped because that was only one. The default keyframe interval is already more than 100 frames, so encoding only 100 frames will provoke it already without explicitly limiting the keyframe interval to only 100 frames, which could have been e.g. 250 or 300 frames without this parameter.

[n808]

Originally Posted by LigH

Remember, I found that this issue was introduced during a specific period of development. Now we have to give the developers some time to discover which change exactly caused it, so they can later fix it...

Sure - that's expected of course. I was only trying to clarify what was still confusing to me, so thanks for explaining.

Here's what I think is the conclusion:

• There is a bug where 2-pass will fail completely: encoding less than one GOP
• There is a bug where 2-pass works very poorly, reaching only 75-80% of requested bitrate. I am not sure if it's possible to get 2-pass rate control to work in current releases.

[Ma]

x265-1.8+167:

i:\speed\2p>ffmpeg -i b.mkv -v warning -f yuv4mpegpipe - | x265-18+167 --y4m - -p slow --bitrate 1000 --keyint 100 -o w2.hevc --
pass 2
y4m  [info]: 832x480 fps 50/1 i420p8 sar 1:1 unknown frame count
raw  [info]: output file: w2.hevc
x265 [info]: HEVC encoder version 1.8+167-e951ab673b1c
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main 10 profile, Level-3.1 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: frame threads / pool features       : 2 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut       : 10 / 100 / 40
x265 [info]: Lookahead / bframes / badapt        : 25 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / 0 / 0
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 1 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : ABR-1000 kbps / 0.60
x265 [info]: tools: rect rd=4 psy-rd=0.30 rdoq=2 psy-rdoq=1.00 signhide tmvp
x265 [info]: tools: strong-intra-smoothing deblock sao stats-read
x265 [info]: frame I:      1, Avg QP:27.14  kb/s: 9306.40
x265 [info]: frame P:     24, Avg QP:30.98  kb/s: 2060.87
x265 [info]: frame B:     75, Avg QP:36.53  kb/s: 467.95
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 8.0% 0.0% 4.0% 60.0% 28.0%

encoded 100 frames in 11.99s (8.34 fps), 938.64 kb/s, Avg QP:35.10

x265-1.8+168:

i:\speed\2p>ffmpeg -i b.mkv -v warning -f yuv4mpegpipe - | x265-18+168 --y4m - -p slow --bitrate 1000 --keyint 100 -o w2.hevc --
pass 2
y4m  [info]: 832x480 fps 50/1 i420p8 sar 1:1 unknown frame count
raw  [info]: output file: w2.hevc
x265 [info]: HEVC encoder version 1.8+168-73cc1cd4c7fc
x265 [info]: build info [Windows][MSVC 1900][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main 10 profile, Level-3.1 (Main tier)
x265 [info]: Thread pool created using 4 threads
x265 [info]: frame threads / pool features       : 2 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [warning]: Error: 2pass curve failed to converge
x265 [warning]: target: 1000.00 kbit/s, expected: 1133.83 kbit/s, avg QP: 68.7129
x265 [warning]: try increasing target bitrate
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : star / 57 / 3 / 3
x265 [info]: Keyframe min / max / scenecut       : 10 / 100 / 40
x265 [info]: Lookahead / bframes / badapt        : 25 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / 0 / 0
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 1 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : ABR-1000 kbps / 0.60
x265 [info]: tools: rect rd=4 psy-rd=0.30 rdoq=2 psy-rdoq=1.00 signhide tmvp
x265 [info]: tools: strong-intra-smoothing deblock sao stats-read
x265 [info]: frame I:      1, Avg QP:51.00  kb/s: 358.00
x265 [info]: frame P:     24, Avg QP:51.00  kb/s: 194.60
x265 [info]: frame B:     75, Avg QP:51.00  kb/s: 88.01
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 8.0% 0.0% 4.0% 60.0% 28.0%

encoded 100 frames in 5.98s (16.71 fps), 116.29 kb/s, Avg QP:51.00

So, the prime suspect is 73cc1cd

[LigH]

No surprise, Sherlock! Of course, the most important change in the rate control.