x265 HEVC Encoder

[LigH]

Probably also an issue to investigate...

When was your last fresh clone? I'll try one right now.
__

Version info added in all required places:

#include <winresrc.h>

VS_VERSION_INFO VERSIONINFO
  FILEVERSION    0,9,0,25
  PRODUCTVERSION 0,9,0,25
# ...
    BLOCK "StringFileInfo"
    BEGIN
      BLOCK "04090000"
      BEGIN
        VALUE "FileDescription",  "HEVC video encoder"
        VALUE "FileVersion",      "0.9+25-bd987db26d5d"
        VALUE "InternalName",     "x265"
        VALUE "LegalCopyright",   "Multicoreware: GPLv2 or commercial"
        VALUE "OriginalFilename", "libx265.dll"
        VALUE "ProductName",      "x265"
        VALUE "ProductVersion",   "0.9+25-bd987db26d5d"
      END
    END
    BLOCK "VarFileInfo"
    BEGIN
      VALUE "Translation", 0x409, 1200
    END
END

[foxyshadis]

The ASM crashes used to happen from time to time when an SSE4 instructions was included in the SSE2 branch. Hopefully Min Chen can take a look at it and see if something like that's the case here.

[LigH]

Originally Posted by D3C0D3R

EDIT:Or better should send this directly into developers mailing lists?

You may have missed my later edit in my reply; so I sent a brief report on your behalf today.

[raine]

I just have a few quick questions regarding x265, I hope it's OK to ask them here.

Given the recent developments (hevc and opus), I'm planning to batch convert a huge film archive (6TB), mostly not-even-HD content encoded as H.264, using medium or slow preset via ffmpeg, with the hopes of shrinking the archive. I don't know how important psy-rd, trellis, intra-refresh etc. are.

• Is it mature enough? (for an ordinary eye, for a home archive. I'm not Netflix or Hollywood so my requirements aren't that high )
• Are there important improvements waiting in the queue? Should I start already or is it worth waiting?
• What parameters do you suggest for encoding? I'm planning to go with -preset slow only, without using -x265-params, but if it turns out to be too slow, I can switch to medium with -x265-params me=star. (as far as I know, there are no -tune film or psy-rd based tunings yet).

I made a small experiment with a 1280x720 H264 video from my archive, using medium preset with crf=22. The resulting file was less than half of the original file with no visible (to me, a keen eye or PSNR/etc may differ) reduction in quality. So it seems it's definitely worth trouble. Even with a recent Haswell i7 processor, the video encoding (I used -c:a copy) was a bit slow (~14 FPS), though this is expected. I hope it will get faster in time though.

I will experiment with slow and slower and make a detailed post next time.

[LigH]

Try to estimate the energy costs caused by the required encoding time and decide if it is worth to pay your energy supplier for weeks or months of converting.

The faster you choose the preset, the less efficient a further compression will be. And it will certainly be lossy as well. As far as I remember, there is still some structure flattening. But that is the cost of efficient redundancy removal: Sometimes it is too optimistic in deciding what may be unimportant.

[raine]

Originally Posted by LigH

Try to estimate the energy costs caused by the required encoding time and decide if it is worth to pay your energy supplier for weeks or months of converting.

The faster you choose the preset, the less efficient a further compression will be. And it will certainly be lossy as well. As far as I remember, there is still some structure flattening. But that is the cost of efficient redundancy removal: Sometimes it is too optimistic in deciding what may be unimportant.

Thanks for the quick response! The energy cost is OK, I have a low-TDP Haswell, I didn't even heard any fan noise during encoding. I also expect the loss, it's lossy -> lossy after all; as long as I don't realize it, it's OK. Maybe this translates to some lower CRF; I used 22.

[LigH]

@ D3C0D3R:

Can you provide a call stack of the crash to the developer mailing list? That would help finding the close range of the reason; the developers don't have the same hardware to check it.

But don't ask me how to do that... Let's hope that Windows will display a verbose crash dialog where you can extract details from. Crash address + your specific EXE may be a minimal anchor.

Well, maybe it can also be provoked with a better CPU by masking SSE4 out.

[D3C0D3R]

Originally Posted by LigH

You may have missed my later edit in my reply; so I sent a brief report on your behalf today.

Thank you! I've have look at x265-devel yesterday, but they tried to fix something unrelated, because --n0-asm works.

I will try to retrive stacktrace, or some debug info.
Its obvious that bug-report instruction here http://www.videolan.org/developers/x265.html ./configure --enable-debug simply copy-pasted from x264 and should be updated.

I suppose CMAKE_BUILD_TYPE, CMAKE_CXX_FLAGS must be set to Debug.

[D3C0D3R]

Ok. I compile x265 with CMAKE_BUILD_TYPE = Debug.
And run this:

gdb --args x265_debug --crf 20  --input ffmpeg_g.exe --input-depth 8 --input-res 320x240 --fps 20 --preset=medium --asm=0x014007F --ssim --log=4 --output 1_1.265

Then type "run" in gdb console:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 2916.0xf28]
0x00533950 in x265_patial_butterfly_inverse_internal_pass2_ssse3 ()
(gdb) bt
#0  0x00533950 in x265_patial_butterfly_inverse_internal_pass2_ssse3 ()
#1  0x00533b6e in x265_idct8_ssse3 ()
#2  0x0441f339 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb) disass $pc-32,$pc+32
Dump of assembler code from 0x533930 to 0x533970:
   0x00533930 <x265_patial_butterfly_inverse_internal_pass1_ssse3+368>: jb     0x533916 <x265_patial_butterfly_inverse_internal_pass1_ssse3+342>
   0x00533932 <x265_patial_butterfly_inverse_internal_pass1_ssse3+370>: pop    %es
   0x00533933 <x265_patial_butterfly_inverse_internal_pass1_ssse3+371>: psubd  %xmm5,%xmm1
   0x00533937 <x265_patial_butterfly_inverse_internal_pass1_ssse3+375>: psrad  $0x7,%xmm1
   0x0053393c <x265_patial_butterfly_inverse_internal_pass1_ssse3+380>: packssdw %xmm1,%xmm4
   0x00533940 <x265_patial_butterfly_inverse_internal_pass1_ssse3+384>: movq   %xmm4,0x30(%edi)
   0x00533945 <x265_patial_butterfly_inverse_internal_pass1_ssse3+389>: movhps %xmm4,0x40(%edi)
   0x00533949 <x265_patial_butterfly_inverse_internal_pass1_ssse3+393>: ret
   0x0053394a <x265_patial_butterfly_inverse_internal_pass1_ssse3+394>: nopw   0x0(%eax,%eax,1)
=> 0x00533950 <x265_patial_butterfly_inverse_internal_pass2_ssse3+0>:   movdqa (%edi),%xmm0
   0x00533954 <x265_patial_butterfly_inverse_internal_pass2_ssse3+4>:   movdqa %xmm0,%xmm4
   0x00533958 <x265_patial_butterfly_inverse_internal_pass2_ssse3+8>:   pshufb 0x6a7950,%xmm4
   0x00533961 <x265_patial_butterfly_inverse_internal_pass2_ssse3+17>:  pmaddwd 0x6a7960,%xmm4
   0x00533969 <x265_patial_butterfly_inverse_internal_pass2_ssse3+25>:  phsubd %xmm4,%xmm5
   0x0053396e <x265_patial_butterfly_inverse_internal_pass2_ssse3+30>:  pshufd $0x4e,%xmm4,%xmm4
End of assembler dump.

eax            0xc55b90 12934032
ecx            0x163d900        23320832
edx            0x10     16
ebx            0x30     48
esp            0x10bf8c8        0x10bf8c8
ebp            0x623990 0x623990 <_ZSt16__convert_from_vRKPiPciPKcz+6437264>
esi            0x623970 6437232
edi            0x10bf8cc        17561804
eip            0x4e63f0 0x4e63f0 <x265_patial_butterfly_inverse_internal_pass2_ssse3>
eflags         0x10216  [ PF AF IF RF ]
cs             0x1b     27
ss             0x23     35
ds             0x23     35
es             0x23     35
fs             0x3b     59
gs             0x0      0

xmm0           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x31, 0xf5, 0xff, 0xff,
    0x4c, 0x8, 0x0, 0x0, 0xb1, 0x3, 0x0, 0x0, 0xd7, 0x6, 0x0, 0x0}, v8_int16 = {0xf531, 0xffff, 0x84c, 0x0,
    0x3b1, 0x0, 0x6d7, 0x0}, v4_int32 = {0xfffff531, 0x84c, 0x3b1, 0x6d7}, v2_int64 = {0x84cfffff531,
    0x6d7000003b1}, uint128 = 0x000006d7000003b10000084cfffff531}
xmm1           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x8000000000000000}, v16_int8 = {0xd1, 0x4,
    0x0, 0x0, 0xea, 0x0, 0x0, 0x0, 0x22, 0x4, 0x0, 0x0, 0xe5, 0xf6, 0xff, 0xff}, v8_int16 = {0x4d1, 0x0, 0xea,
    0x0, 0x422, 0x0, 0xf6e5, 0xffff}, v4_int32 = {0x4d1, 0xea, 0x422, 0xfffff6e5}, v2_int64 = {0xea000004d1,
    0xfffff6e500000422}, uint128 = 0xfffff6e500000422000000ea000004d1}
xmm2           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000},
  v16_int8 = {0x80, 0x4, 0x80, 0x4, 0x80, 0xfb, 0x40, 0xf9, 0xc0, 0xfd, 0x0, 0x0, 0x0, 0x0, 0x80, 0xfb},
  v8_int16 = {0x480, 0x480, 0xfb80, 0xf940, 0xfdc0, 0x0, 0x0, 0xfb80}, v4_int32 = {0x4800480, 0xf940fb80,
    0xfdc0, 0xfb800000}, v2_int64 = {0xf940fb8004800480, 0xfb8000000000fdc0},
  uint128 = 0xfb8000000000fdc0f940fb8004800480}
xmm3           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x0}, v16_int8 = {0xa7, 0xf8,
    0xff, 0xff, 0xaf, 0xff, 0xff, 0xff, 0x74, 0xfe, 0xff, 0xff, 0x0, 0x0, 0x0, 0x0}, v8_int16 = {0xf8a7, 0xffff,
    0xffaf, 0xffff, 0xfe74, 0xffff, 0x0, 0x0}, v4_int32 = {0xfffff8a7, 0xffffffaf, 0xfffffe74, 0x0}, v2_int64 = {
    0xffffffaffffff8a7, 0xfffffe74}, uint128 = 0x00000000fffffe74ffffffaffffff8a7}
xmm4           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000},
  v16_int8 = {0xdb, 0x6, 0x41, 0x4, 0x39, 0xfc, 0x1, 0xf9, 0xd1, 0x4, 0xea, 0x0, 0x22, 0x4, 0xe5, 0xf6},
  v8_int16 = {0x6db, 0x441, 0xfc39, 0xf901, 0x4d1, 0xea, 0x422, 0xf6e5}, v4_int32 = {0x44106db, 0xf901fc39,
    0xea04d1, 0xf6e50422}, v2_int64 = {0xf901fc39044106db, 0xf6e5042200ea04d1},
  uint128 = 0xf6e5042200ea04d1f901fc39044106db}
xmm5           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x80, 0x82, 0x0, 0x0, 0xc0,
    0xd5, 0x0, 0x0, 0xc0, 0x5, 0xfe, 0xff, 0x0, 0x87, 0x0, 0x0}, v8_int16 = {0x8280, 0x0, 0xd5c0, 0x0, 0x5c0,
    0xfffe, 0x8700, 0x0}, v4_int32 = {0x8280, 0xd5c0, 0xfffe05c0, 0x8700}, v2_int64 = {0xd5c000008280,
    0x8700fffe05c0}, uint128 = 0x00008700fffe05c00000d5c000008280}
xmm6           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0, 0x8, 0x0, 0x0, 0x0,
    0x8, 0x0, 0x0, 0x0, 0x8, 0x0, 0x0, 0x0, 0x8, 0x0, 0x0}, v8_int16 = {0x800, 0x0, 0x800, 0x0, 0x800, 0x0,
    0x800, 0x0}, v4_int32 = {0x800, 0x800, 0x800, 0x800}, v2_int64 = {0x80000000800, 0x80000000800},
  uint128 = 0x00000800000008000000080000000800}
xmm7           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000},
  v16_int8 = {0x5e, 0xfe, 0x62, 0xf5, 0x27, 0xfc, 0x7e, 0xff, 0xf7, 0xf5, 0xf3, 0x9, 0x0, 0xf6, 0x9d, 0xfe},
  v8_int16 = {0xfe5e, 0xf562, 0xfc27, 0xff7e, 0xf5f7, 0x9f3, 0xf600, 0xfe9d}, v4_int32 = {0xf562fe5e,
    0xff7efc27, 0x9f3f5f7, 0xfe9df600}, v2_int64 = {0xff7efc27f562fe5e, 0xfe9df60009f3f5f7},
  uint128 = 0xfe9df60009f3f5f7ff7efc27f562fe5e}

[LigH]

I hope this function name already helps narrowing the search. If not, the developers may need your ffmpeg_g.exe together with this result.

[D3C0D3R]

Originally Posted by LigH

the developers may need your ffmpeg_g.exe together with this result.

Post updated. Failure is _completly_ independent from yuv-input.

I look for some public-avaible builds. one which i found ran under 32-bit xp, from x265.ru (8-bit 0.9+29). And it fails exact way, at exact movdqa, only functions names unavailble in debugger, because exe without debug-info.

Sorry, now for me convenient to reply here instead mailing lists.

[LigH]

They are already on a track. IIUC, here is an instruction that may lead to a crash on SSSE3 only CPUs when the stack is not aligned. Gosh, how much do you need to know about it to discover that?!

[smegolas]

Originally Posted by raine

I just have a few quick questions regarding x265, I hope it's OK to ask them here.

Given the recent developments (hevc and opus), I'm planning to batch convert a huge film archive (6TB), mostly not-even-HD content encoded as H.264, using medium or slow preset via ffmpeg, with the hopes of shrinking the archive. I don't know how important psy-rd, trellis, intra-refresh etc. are.

• Is it mature enough? (for an ordinary eye, for a home archive. I'm not Netflix or Hollywood so my requirements aren't that high )
• Are there important improvements waiting in the queue? Should I start already or is it worth waiting?
• What parameters do you suggest for encoding? I'm planning to go with -preset slow only, without using -x265-params, but if it turns out to be too slow, I can switch to medium with -x265-params me=star. (as far as I know, there are no -tune film or psy-rd based tunings yet).

I made a small experiment with a 1280x720 H264 video from my archive, using medium preset with crf=22. The resulting file was less than half of the original file with no visible (to me, a keen eye or PSNR/etc may differ) reduction in quality. So it seems it's definitely worth trouble. Even with a recent Haswell i7 processor, the video encoding (I used -c:a copy) was a bit slow (~14 FPS), though this is expected. I hope it will get faster in time though.

I will experiment with slow and slower and make a detailed post next time.

No, this would be not a good idea.

x265 is not mature enough yet. It is still in beta. Everybody complains about blurring with live action content. Witness how much x264 improved since the first year. Will be same with x265.

There are no hardware decoders so we don't know what 'correct' settings are to ensure widespread compatiblity. (eg HEVC equivalent of H.264 HP@L4.1)

6TB isn't really a huge collection. Pretty soon it will fit on one disk. If you have run out of space just buy more storage, it would be cheaper than your electricity bill for encoding that many movies.

We were at the NAB show in Las Vegas last week, giving demonstrations to many of the companies attending, showing x265 encodes of popular video sequences side by side with x264 encodes (the gold standard for quality today). Attendees were blown away by the quality of x265. We demonstrated 2 streams played back in sync, showing the middle 50% of two 4K clips on a 4K monitor. Here is a photo of our demo...

Sample frame grabs from Crowd_Run4K50 @ 4 Mbps are here... http://x265.org/CrowdRun4K_4Mbps.7z
You can repeat these tests yourself, using the SVT clips from http://media.xiph.org/video/derf/ - Crowd Run, Old Town Cross, Ducks Take Off, In To Tree.... take your pick.

H.265 provides for much higher video coding accuracy. The thing you will notice is that x265 encodes don't have all of the temporal artifacts of H.264. x265 is typically winning comparisons against x264 even at half the bit rate. There were cases where x265 was preferred even at 1/4 the bit rate
(Crowd Run 4K - x265 veryslow 4 Mbps vs x264 veryslow @ 16 Mbps), due to the lack of temporal artifacts and macroblocking in x265.

[Blue_MiSfit]

I wish I had stopped by your booth

This is an impressive demonstration, but I think x265 is still too slow for any real use - especially for 4k. I haven't looked at alternatives yet (the glacial broadcast industry is still a long ways away from HEVC), but is there hope for significant speedup or are we already to the point where further ASM optimizations could give only 10-20% more?

To amend the above, I have looked at Elemental Live. Their HEVC encoder is supposedly real-time for 4kp30 using a single server, and 4kp60 using two servers strapped together. This is of course using scads of very powerful and expensive GPUs, but would the same thing theoretically be possible with today's technology using CPU-only encoders like x265? Let's pretend we have a beastly quad-socket fire breathing 48 core monster at our disposal..

Thanks,
Derek

Originally Posted by Blue_MiSfit

I wish I had stopped by your booth

This is an impressive demonstration, but I think x265 is still too slow for any real use - especially for 4k. I haven't looked at alternatives yet (the glacial broadcast industry is still a long ways away from HEVC), but is there hope for significant speedup or are we already to the point where further ASM optimizations could give only 10-20% more?

To amend the above, I have looked at Elemental Live. Their HEVC encoder is supposedly real-time for 4kp30 using a single server, and 4kp60 using two servers strapped together. This is of course using scads of very powerful and expensive GPUs, but would the same thing theoretically be possible with today's technology using CPU-only encoders like x265? Let's pretend we have a beastly quad-socket fire breathing 48 core monster at our disposal..

Thanks,
Derek

We had a private suite. We'll look at getting a booth next year.

It's true that you don't get these significant improvements in video compression efficiency for free. HEVC requires substantially higher compute resources for a given performance level. x265 can scale performance nicely on multiple socket machines, and we will continue to optimize for our studio / broadcast customers that use these systems.

A number of current and pending x265 licensees are building real-time encoders. One was demonstrating real-time 4K encoding based on x265, using multiple processors (dividing the incoming stream into segments). It looked great!

[Procrastinating]

Is the 1.0 release intended to be the one which is "fit for public consumption", or will it be progression as usual?

[Kurtnoise]

https://bitbucket.org/multicoreware/x265/wiki/RoadMap
https://bitbucket.org/multicoreware/x265/wiki/TODO

[raine]

Originally Posted by smegolas

No, this would be not a good idea.

x265 is not mature enough yet. It is still in beta. Everybody complains about blurring with live action content. Witness how much x264 improved since the first year. Will be same with x265.

There are no hardware decoders so we don't know what 'correct' settings are to ensure widespread compatiblity. (eg HEVC equivalent of H.264 HP@L4.1)

6TB isn't really a huge collection. Pretty soon it will fit on one disk. If you have run out of space just buy more storage, it would be cheaper than your electricity bill for encoding that many movies.

Thanks so much! I think I will wait until the stable 1.0 release then (assuming all items in Quality/Efficiency TODOs are cleared).
However, I haven't seen any blurring so far (using x265 0.9, ffmpeg 2.2.1) with crf through 22 to 26. Is this an issue with the older versions?

As I mentioned above, I have a low-TDP Haswell CPU. The power supply itself is 80W (comparable to an old-fashioned bulb), so there is really no issue regarding the electricity usage. When I say Haswell, people seem to think of a big ATX machine with 800W power supply

And BTW, on average, I found that x265 version is 40-60% of the original file. For HD content, the encoding rate is around 10fps whereas for low-definition videos (360p or even less) it is around 40fps. I couldn't see any visual loss using crf 22-24, and it is very difficult for me to tell them apart with crf=26.

In my case, buying more storage is not a problem, but having external harddisks floating around is; I'm trying to minimize the number of harddisks, while still having space to add more movies. And yes, I'm planning to buy a 6TB harddisk

[mandarinka]

I don't think recompressing video collections was ever a good idea.
You will kill a great deal of the content, and what is really the gain? /are you planning to do this each five years when a new scheme or improved encoders are available?/

BTW, about that "1.0". It isn't going to be a particularly significant, complete, "stable" or how you want to call it milestone. It looks like it will be more of a normal, scheduled release like the ones before it.

See on the mailing list.