Taking submissions for a small encoder comparison

Dark Shikari · 7th May 2010, 08:34

With the MSU encoder comparison coming out, we're going to have a good comparison (hopefully) of objective quality (SSIM) between various encoders. Unfortunately, not only does their comparison contain basically only H.264 encoders (plus Theora and Xvid), it also doesn't focus on visual comparisons, so I figured I would make my own small one to get a feel for the "current state of the art". This is a visual comparison only; no measurements of SSIM or PSNR will even be made.

Source: parkjoy (1080p50)
Target file size: 17000KB (can be a bit over or under if the encoder won't reach the exact value, obviously). Using constant quality mode is fine as long as you can get it close to the target size. If the encoder has no 2-pass ABR mode, use constant quality mode (as I did with Dirac).
Keyframe interval: 250

Stuff I already have:

x264 (PSNR-optimized and psy-optimized)
VP7
Theora (Thusnelda and Ptalabvorm)
Dirac (through Schroedinger)
Xvid
ffmpeg mpeg-4
ffmpeg mpeg-1
Bink
SVQ1
SIF1
CudaH264Enc
Ateme (v1.5 and v2.0)
Elecard
Samsung+BBC H.265 proposal
Mainconcept 8.5
Microsoft VC-1 SDK

Stuff I'm interested in:

GPU encoders (e.g. Badaboom)
Any other comedy options you can think of that can hit the target bitrate
VP8 (if you have the encoder for this, please contact me directly and we can arrange for just the decoded output to be uploaded)
Other H.265 proposals?
Reference encoders (JM, JSVM, etc)
Anything else?

To submit a test, just upload the video file along with the settings you used to create it. All encoders should use their "best" settings possible, whatever they happen to be.

In a few days, after I get a few more submissions, I'll post a blind visual test where people can give scores to each codec.

Rationale for the choice of test clip:

1. It's a lossless source. Using pre-compressed sources results in all kinds of stupid arguments about which encoder benefits the most from the compression artifacts already present in the source. It's also free to distribute.

2. It contains a wide variety of high-detail, highly colorful content that should be a serious challenge for any encoder.

3. It shouldn't bias too heavily towards any one encoder like many of the other standard test clips will:
a. It's relatively high motion, so it won't bias heavily against encoders without B-frames or qpel (as, say, mobcal does).
b. It's not so high motion that it would cripple video formats that don't support motion vectors longer than 16 pixels (e.g. Theora).
c. It's not something that benefits an unreasonably large amount from some of x264's algorithms (which is why I picked this and not parkrun).

Overall it should serve as a good, relatively fair test. Ideally I'd want to test more clips, but I don't have that much time and doing psy comparisons is a lot of effort.

About the source:

The source is from the SVT Multiformat test set (thanks Sveriges Television!). It was filmed using an ARRI ArriFlex 765 camera on 65mm film using 1/100th second exposure time, hence the staggering level of detail all the way up to 2160p.

morqueozwald · 8th May 2010, 01:26

Theora Ptalarbvorm: This was the most recent svn version that affects output, the front end was ffmpeg2theora also the most recent version.
The command line: ffmpeg2theora --optimize -V 13600 --two-pass -K 250 park_joy_1080p.y4m
http://www.mediafire.com/file/qjmnjnzrznn/ParkJoy.ogv

This falls under the "comedy" option.
Quicktime H264 bitrate set to 14200 to hit target size, key frame 250, two pass and no restriction on profile. I first encoded the file with x264 lossless so Quicktime could read it.
http://www.mediafire.com/file/ikynzymzmto/ParkJoy2.mp4

Dark Shikari · 8th May 2010, 01:38

Quote:

Originally Posted by morqueozwald

Theora Ptalarbvorm: This was the most recent svn version that affects output, the front end was ffmpeg2theora also the most recent version.
The command line: ffmpeg2theora --optimize -V 13600 --two-pass -K 250 park_joy_1080p.y4m
http://www.mediafire.com/file/qjmnjnzrznn/ParkJoy.ogv

This falls under the "comedy" option.
Quicktime H264 bitrate set to 14200 to hit target size, key frame 250, two pass and no restriction on profile. I first encoded the file with x264 lossless so Quicktime could read it.
http://www.mediafire.com/file/ikynzymzmto/ParkJoy2.mp4

The Quicktime video appears to have completely screwed up brightness...

Ptalabvorm added.

morqueozwald · 8th May 2010, 20:50

Quote:

Originally Posted by Dark Shikari

The Quicktime video appears to have completely screwed up brightness...

I never noticed this before but this is apparently a very common problem. Finder's information on the file says "Color Profile: HD (1-1-1)" whereas x264 produced files don't say anything (w/r to color) in the info. Could this be responsible?

Dark Shikari · 8th May 2010, 21:37

@shon3i

Added.

shon3i · 9th May 2010, 01:22

Mainconcept (SDK 8.5 Brodcast encoder)

Settings used:
TwoPass encoding
Profile@Level: High 5.1
Entropy Coding: CABAC
Performance: 15-Best Quality
Psychovisual: Luma & Contrast & Complexity, strength: 100-100-100
Deblock: 0:0
B-pyramid
Ref frames: 16
GOP Length: 250

http://www.mediafire.com/?zvzgn0omion

Tomorow i will encode with Ateme v2 engine encoder. Maybe someone can post Elecard with Converter Studio Pro, because have more advanced options

filetree · 19th June 2010, 03:41

Quote:

Originally Posted by Dark Shikari

The Quicktime video appears to have completely screwed up brightness...

Ptalabvorm added.

It's actually a gamma thing. There's a decent workaround (that requires QTPro) but it's still []. It sucks that there are a lot of good broadcast/live hardware that will only allow video in .mov format.
Unless I'm mistaken x264 cannot be held in an mov container.

What about Sorenson3 codec? As old as it is, it is sadly still being used a lot of places.

nakTT · 19th June 2010, 19:06

Hi Dark,

Can we expect to see SIF1 v1.10 in the comparison?

hellfred · 19th June 2010, 22:29

I have created an up to date theora ptalarbvorm clip:
park_joy_V13600_softtarget.ogv

Code:

command used:
ffmpeg2theora-ptalarbvorm.exe --no-skeleton --speedlevel 0 --soft-target -K 250 -V 13600 --two-pass --noaudio park_joy_1080p.y4m
Binary from:
http://lists.xiph.org/pipermail/theo...ne/004164.html

Changes to the theora encoder, which might have had an impact on the video quality since the first theora ptalarbvorm clip provided at page #1:

Code:

18.05.2010: https://trac.xiph.org/changeset/1723...ra-ptalarbvorm
04.06.2010: https://trac.xiph.org/changeset/1727...ra-ptalarbvorm

shark37 · 8th September 2010, 20:51

Much better snow encoding, at least without ghosting artefacts.

park_joy_1080p-snow-qp_11.avi
Size: 17,556,882
FFmpeg version SVN-r24986

Code:

ffmpeg -i park_joy_1080p.y4m -vcodec snow -strict -2 -qscale 11 -cmp satd -mbcmp satd -subcmp satd -me_method iter -refs 4 -y foo.avi

Credits for command line go to akupenguin.

AnonCrow · 8th May 2010, 03:21

CudaH264Enc through MediaCoder 0.7.3.4640, ABR and VBR (constant quantizer) encodes, Mediacoder preset files included :

http://www.mediafire.com/?sharekey=b...f0d6082f1c2cd0

... although, the file size produced by the VBR preset varies by a couple KB between encodes.
( brief help on cudah264enc cli options: http://forum.mediacoderhq.com/viewto...&t=6896#p21699 )

Comedy options:

A M-JPEG encode should always included, either as a comedy option or to make the MPEG-1 encode feel better about itself.
Possibly also MNG, APNG and M-JPEG 2000 encodes to complement the M-JPEG one.

Dark Shikari · 8th May 2010, 04:11

CUDAH264Enc added.

shon3i · 8th May 2010, 20:38

Ateme (eavc engine 1.5)

Settings used:
TwoPass encoding
Profile@Level: High 5.1
Entropy Coding: CABAC
Quality: Full
Psychovisual: Best
Adaptive Deblock: 0
WeightP
RefP: 5
RefB: 5
Max GOP Size: 250
OpenGOP

http://www.mediafire.com/?yuykyyzjhzm

Elecard (Converter Studio 3.2)

Settings used:
TwoPass encoding
Profile@Level: High 5.1
Entropy Coding: CABAC
Quality: Highest
Psychovisual: Complexity & Luma, strength: 4
Deblock: 0:0
B-pyramid
Ref frames: 16
GOP Length: 250

http://www.mediafire.com/?y4miontkzmn

Biggiesized · 9th May 2010, 06:13

Any particular reason to use the clip at 1080p50? Why not do a Blu-ray compatible shoot-out (that is, 720p50)? That way you can throw in Blu-ray encoders if users have access to them.

Dark Shikari · 9th May 2010, 06:35

Quote:

Originally Posted by Biggiesized

Any particular reason to use the clip at 1080p50? Why not do a Blu-ray compatible shoot-out (that is, 720p50)? That way you can throw in Blu-ray encoders if users have access to them.

You can easily use a Blu-ray encoder by setting the framerate to 25 and halving the bitrate

Biggiesized · 9th May 2010, 07:28

Can x264 do 1080i MBAFF encoding for Blu-ray?

Dark Shikari · 9th May 2010, 07:29

Quote:

Originally Posted by Biggiesized

Can x264 do 1080i MBAFF encoding for Blu-ray?

Yes, it can.

Schrade · 9th May 2010, 08:02

If anyone has access to an Elemental Technologies GPU encoder, I'd love to see that as one of the samples

Blue_MiSfit · 9th May 2010, 19:44

You forgot DivX!

I'll do a couple encodes, even though I never ever use DivX

[edit]
Scratch that... I can't get the VFW encoder to accept the input. It complains "the source image format is not acceptable". I think it may be a profile issue.

Maybe an HC powered MPEG-2 encode?
[/edit]

~MiSfit

Dark Shikari · 9th May 2010, 20:13

Quote:

Originally Posted by Blue_MiSfit

You forgot DivX!

I'll do a couple encodes, even though I never ever use DivX

Their encoder crashed for me, so I couldn't get it to work. DivX ASP, that is.

DivX H.264 might be interesting.

7th May 2010, 08:34	#1 \| Link
Dark Shikari x264 developer Join Date: Sep 2005 Posts: 8,666	Taking submissions for a small encoder comparison With the MSU encoder comparison coming out, we're going to have a good comparison (hopefully) of objective quality (SSIM) between various encoders. Unfortunately, not only does their comparison contain basically only H.264 encoders (plus Theora and Xvid), it also doesn't focus on visual comparisons, so I figured I would make my own small one to get a feel for the "current state of the art". This is a visual comparison only; no measurements of SSIM or PSNR will even be made. Source: parkjoy (1080p50) Target file size: 17000KB (can be a bit over or under if the encoder won't reach the exact value, obviously). Using constant quality mode is fine as long as you can get it close to the target size. If the encoder has no 2-pass ABR mode, use constant quality mode (as I did with Dirac). Keyframe interval: 250 Stuff I already have: x264 (PSNR-optimized and psy-optimized) VP7 Theora (Thusnelda and Ptalabvorm) Dirac (through Schroedinger) Xvid ffmpeg mpeg-4 ffmpeg mpeg-1 Bink SVQ1 SIF1 CudaH264Enc Ateme (v1.5 and v2.0) Elecard Samsung+BBC H.265 proposal Mainconcept 8.5 Microsoft VC-1 SDK Stuff I'm interested in: GPU encoders (e.g. Badaboom) Any other comedy options you can think of that can hit the target bitrate VP8 (if you have the encoder for this, please contact me directly and we can arrange for just the decoded output to be uploaded) Other H.265 proposals? Reference encoders (JM, JSVM, etc) Anything else? To submit a test, just upload the video file along with the settings you used to create it. All encoders should use their "best" settings possible, whatever they happen to be. In a few days, after I get a few more submissions, I'll post a blind visual test where people can give scores to each codec. Rationale for the choice of test clip: 1. It's a lossless source. Using pre-compressed sources results in all kinds of stupid arguments about which encoder benefits the most from the compression artifacts already present in the source. It's also free to distribute. 2. It contains a wide variety of high-detail, highly colorful content that should be a serious challenge for any encoder. 3. It shouldn't bias too heavily towards any one encoder like many of the other standard test clips will: a. It's relatively high motion, so it won't bias heavily against encoders without B-frames or qpel (as, say, mobcal does). b. It's not so high motion that it would cripple video formats that don't support motion vectors longer than 16 pixels (e.g. Theora). c. It's not something that benefits an unreasonably large amount from some of x264's algorithms (which is why I picked this and not parkrun). Overall it should serve as a good, relatively fair test. Ideally I'd want to test more clips, but I don't have that much time and doing psy comparisons is a lot of effort. About the source: The source is from the SVT Multiformat test set (thanks Sveriges Television!). It was filmed using an ARRI ArriFlex 765 camera on 65mm film using 1/100th second exposure time, hence the staggering level of detail all the way up to 2160p. __________________ Follow x264 development progress \| akupenguin quotes \| x264 git status ffmpeg and x264-related consulting/coding contracts \| Doom10 Last edited by Dark Shikari; 9th May 2010 at 19:24.

8th May 2010, 21:37	#5 \| Link
Dark Shikari x264 developer Join Date: Sep 2005 Posts: 8,666	@shon3i Added. __________________ Follow x264 development progress \| akupenguin quotes \| x264 git status ffmpeg and x264-related consulting/coding contracts \| Doom10

9th May 2010, 01:22	#6 \| Link
shon3i BluRay Maniac Join Date: Dec 2005 Posts: 2,419	Mainconcept (SDK 8.5 Brodcast encoder) Settings used: TwoPass encoding Profile@Level: High 5.1 Entropy Coding: CABAC Performance: 15-Best Quality Psychovisual: Luma & Contrast & Complexity, strength: 100-100-100 Deblock: 0:0 B-pyramid Ref frames: 16 GOP Length: 250 http://www.mediafire.com/?zvzgn0omion Tomorow i will encode with Ateme v2 engine encoder. Maybe someone can post Elecard with Converter Studio Pro, because have more advanced options Last edited by shon3i; 9th May 2010 at 02:34. Reason: new link

19th June 2010, 22:29	#9 \| Link
hellfred FFmpeg fan Join Date: Dec 2003 Location: Germany Posts: 427	updated theora clip I have created an up to date theora ptalarbvorm clip: park_joy_V13600_softtarget.ogv Code: command used: ffmpeg2theora-ptalarbvorm.exe --no-skeleton --speedlevel 0 --soft-target -K 250 -V 13600 --two-pass --noaudio park_joy_1080p.y4m Binary from: http://lists.xiph.org/pipermail/theo...ne/004164.html Changes to the theora encoder, which might have had an impact on the video quality since the first theora ptalarbvorm clip provided at page #1: Code: 18.05.2010: https://trac.xiph.org/changeset/1723...ra-ptalarbvorm 04.06.2010: https://trac.xiph.org/changeset/1727...ra-ptalarbvorm Last edited by hellfred; 19th June 2010 at 22:31.

8th September 2010, 20:51	#10 \| Link
shark37 Registered User Join Date: May 2003 Posts: 34	Much better snow encoding, at least without ghosting artefacts. park_joy_1080p-snow-qp_11.avi Size: 17,556,882 FFmpeg version SVN-r24986 Code: ffmpeg -i park_joy_1080p.y4m -vcodec snow -strict -2 -qscale 11 -cmp satd -mbcmp satd -subcmp satd -me_method iter -refs 4 -y foo.avi Credits for command line go to akupenguin.

8th May 2010, 01:26	#2 \| Link
morqueozwald Registered User Join Date: Jan 2010 Posts: 3	Theora Ptalarbvorm: This was the most recent svn version that affects output, the front end was ffmpeg2theora also the most recent version. The command line: ffmpeg2theora --optimize -V 13600 --two-pass -K 250 park_joy_1080p.y4m http://www.mediafire.com/file/qjmnjnzrznn/ParkJoy.ogv This falls under the "comedy" option. Quicktime H264 bitrate set to 14200 to hit target size, key frame 250, two pass and no restriction on profile. I first encoded the file with x264 lossless so Quicktime could read it. http://www.mediafire.com/file/ikynzymzmto/ParkJoy2.mp4

19th June 2010, 19:06	#8 \| Link
nakTT Registered User Join Date: Dec 2008 Posts: 415	Hi Dark, Can we expect to see SIF1 v1.10 in the comparison?

8th May 2010, 03:21	#11 \| Link
AnonCrow Registered User Join Date: Aug 2009 Location: 61.45° , 23.86° Posts: 120	CudaH264Enc through MediaCoder 0.7.3.4640, ABR and VBR (constant quantizer) encodes, Mediacoder preset files included : http://www.mediafire.com/?sharekey=b...f0d6082f1c2cd0 ... although, the file size produced by the VBR preset varies by a couple KB between encodes. ( brief help on cudah264enc cli options: http://forum.mediacoderhq.com/viewto...&t=6896#p21699 ) Comedy options: A M-JPEG encode should always included, either as a comedy option or to make the MPEG-1 encode feel better about itself. Possibly also MNG, APNG and M-JPEG 2000 encodes to complement the M-JPEG one.

8th May 2010, 04:11	#12 \| Link
Dark Shikari x264 developer Join Date: Sep 2005 Posts: 8,666	CUDAH264Enc added. __________________ Follow x264 development progress \| akupenguin quotes \| x264 git status ffmpeg and x264-related consulting/coding contracts \| Doom10

8th May 2010, 20:38	#13 \| Link
shon3i BluRay Maniac Join Date: Dec 2005 Posts: 2,419	Ateme (eavc engine 1.5) Settings used: TwoPass encoding Profile@Level: High 5.1 Entropy Coding: CABAC Quality: Full Psychovisual: Best Adaptive Deblock: 0 WeightP RefP: 5 RefB: 5 Max GOP Size: 250 OpenGOP http://www.mediafire.com/?yuykyyzjhzm Elecard (Converter Studio 3.2) Settings used: TwoPass encoding Profile@Level: High 5.1 Entropy Coding: CABAC Quality: Highest Psychovisual: Complexity & Luma, strength: 4 Deblock: 0:0 B-pyramid Ref frames: 16 GOP Length: 250 http://www.mediafire.com/?y4miontkzmn

9th May 2010, 06:13	#14 \| Link
Biggiesized Registered User Join Date: Apr 2007 Location: Pittsburgh, PA Posts: 225	Any particular reason to use the clip at 1080p50? Why not do a Blu-ray compatible shoot-out (that is, 720p50)? That way you can throw in Blu-ray encoders if users have access to them.

9th May 2010, 07:28	#16 \| Link
Biggiesized Registered User Join Date: Apr 2007 Location: Pittsburgh, PA Posts: 225	Can x264 do 1080i MBAFF encoding for Blu-ray?

9th May 2010, 08:02	#18 \| Link
Schrade Registered User Join Date: Mar 2004 Posts: 339	If anyone has access to an Elemental Technologies GPU encoder, I'd love to see that as one of the samples

9th May 2010, 19:44	#19 \| Link
Blue_MiSfit Derek Prestegard IRL Join Date: Nov 2003 Location: Los Angeles Posts: 5,989	You forgot DivX! I'll do a couple encodes, even though I never ever use DivX [edit] Scratch that... I can't get the VFW encoder to accept the input. It complains "the source image format is not acceptable". I think it may be a profile issue. Maybe an HC powered MPEG-2 encode? [/edit] ~MiSfit __________________ These are all my personal statements, not those of my employer :) Last edited by Blue_MiSfit; 9th May 2010 at 19:58.