Thor: a new codec from Cisco (reference implementation available)

[MoSal]

Cisco presented Thor, a new royalty-free video codec, at the IETF93 meetings in Prague. A high-level description of the codec was submitted as a draft (https://tools.ietf.org/html/draft-fuldseth-netvc-thor-00) earlier this month.

An open-source reference implementation is now publicly available at github (https://github.com/cisco/thor).

The draft's abstract describes the codec as:

Thor is designed to achieve high compression efficiency with
moderate complexity, using the well-known hybrid video coding
approach of motion-compensated prediction and transform coding.


Other than working with the IETF, which implies the willingness to write a usable standard document. I don't think Thor will have any advantages over VP9/VP10. However, some techniques that Thor uses might prove usable to other codecs under development. Daala developers are already experimenting (https://www.ietf.org/proceedings/93/slides/slides-93-netvc-5.pdf) with a couple of those techniques.

[MoSal]

The IETF93 NETVC meeting at which Thor was presented:
http://recordings.conf.meetecho.com/Playout/watch.jsp?recording=IETF93_NETVC_II&chapter=chapter_1

The slides:
https://www.ietf.org/proceedings/93/slides/slides-93-netvc-4.pdf

[wiak]

compiled a windows build today
http://nwgat.ninja/thor-video-codec/

[dapperdan]

Interesting, at first glance it appears to be an attempt to tempt existing IP holders in H.264/H.265 to defect to royalty-free since the existing processes have stymied that for the last decade or so.

Bold move.

[pieter3d]

That IETF document is really sparse on details. It appears to be pretty simple, not really any new features. If I had to guess it is probably in between H.264 and HEVC/VP9 in terms of compression efficiency.

[wiak]

That IETF document is really sparse on details. It appears to be pretty simple, not really any new features. If I had to guess it is probably in between H.264 and HEVC/VP9 in terms of compression efficiency.
yep, did you read the daala hackathlon pdf? hehe

[pieter3d]

I looked at the code - it appears to be largely a dumbed-down version of HEVC. There are even constants and macros in the code with the name HEVC in it.

[wiak]

I looked at the code - it appears to be largely a dumbed-down version of HEVC. There are even constants and macros in the code with the name HEVC in it.
alot of codecs these days share techniques
yet some, like daala or diarc are alot different, the lapped transform is a pretty interesting technology

[Tommy Carrot]

Thanks for the build, wiak.

I ran a few tests with the encoder. It is horrendously slow in the high efficiency settings, and it has a few issues (cuts off the last few frames if b-frames are enabled), but the quality is pretty good, much better than the current daala encoder, it's actually not that far from x265. Considering the immature state of the encoder (non-adaptive b-frames, no scenecut detection, no psy-rdo), it's quite promising.

[wiak]

Thanks for the build, wiak.

I ran a few tests with the encoder. It is horrendously slow in the high efficiency settings, and it has a few issues (cuts off the last few frames if b-frames are enabled), but the quality is pretty good, much better than the current daala encoder, it's actually not that far from x265. Considering the immature state of the encoder (non-adaptive b-frames, no scenecut detection, no psy-rdo), it's quite promising.
ooh, well the current daala encoder is basicly a experimental patch work
and the current thor code was posted to githu a week ago

its good that there are more codecs at IETF, the Thor MC seem to fix issues with the current Daala MC

[dapperdan]

I looked at the code - it appears to be largely a dumbed-down version of HEVC. There are even constants and macros in the code with the name HEVC in it.


Cisco contributed to H.264/5 and have chosen to freely licence that IPR in Thor and any outcome of the NetVC project.

They've also had teams of lawyers deciding what bits of H.264/5 are freely reusable.

So their strategy of sticking close to H.264/5 makes a lot of sense. If they can tempt any of the other IP holders to join in, then it'll make even more sense.

The blossoming of lots of interesting new codecs will happen after the royalty-bearing patent stranglehold on the industry us broken.

[Bloax]

The blossoming of lots of interesting new codecs will happen after the royalty-bearing patent stranglehold on the industry us broken.
What do you mean patents strangle progress? :rolleyes:

Maybe you're in with those filthy pirates that say that copyright hinders creativity!! :eek:

*cough*

Based off Tommy Carrot's little writeup this sounds like a very exciting entry to the field, though.
Paint me interested.

[MoSal]

Thanks for the build, wiak.

I ran a few tests with the encoder. It is horrendously slow in the high efficiency settings, and it has a few issues (cuts off the last few frames if b-frames are enabled), but the quality is pretty good, much better than the current daala encoder, it's actually not that far from x265. Considering the immature state of the encoder (non-adaptive b-frames, no scenecut detection, no psy-rdo), it's quite promising.

Unless someone starts making some crazy claims, I don't think it's very useful at this point to compare Daala with other (traditional) codecs quality-wise.

There are many basic features that are not implemented yet, or still need a lot of work. 64x64 blocks, multiple-references, mixed-predicion, bi-prediction, better intra-prediction, better MC, better MPs, and finally, that promised efficient deringing filter. All this work is on the roadmap. And the plan was to finish most of it by the end of this year. But I think that's a little bit too ambitious of a goal.

[Tommy Carrot]

Unless someone starts making some crazy claims, I don't think it's very useful at this point to compare Daala with other (traditional) codecs quality-wise.
You are right of course, i was just surprised that this codec, despite being almost as early stage in the development as daala, produces such a good quality. It is possible though (i haven't looked at the source code) that many parts were borrowed from the HM encoder, leaving out the features that they considered patent-risks. That would explain the good performance, and the similar characteristics to h.265.

There are many basic features that are not implemented yet, or still need a lot of work. 64x64 blocks, multiple-references, mixed-predicion, bi-prediction, better intra-prediction, better MC, better MPs, and finally, that promised efficient deringing filter. All this work is on the roadmap. And the plan was to finish most of it by the end of this year. But I think that's a little bit too ambitious of a goal.

Daala has improved immensely in the last year or so, but it's still not comparable to the mature codecs. They started to fall behind the roadmap in the last few months though, so yes, i'd not expect to get the main features finished for at least a year from now or so.

[foxyshadis]

For the 32x32 and 64x64 transform sizes, only the 16x16 low frequency coefficients are quantized and transmitted.

Interesting idea that could be implemented in HEVC today (explicitly setting them to 0 instead of just tossing them). I wonder if this would have any effect on quality, since those really are mostly insignificant. If Thor intends to set them to random low-level noise on decoding, though, that's impossible for HEVC to copy.

[Gravitator]

Interesting idea that could be implemented in HEVC today (explicitly setting them to 0 instead of just tossing them). I wonder if this would have any effect on quality, since those really are mostly insignificant. If Thor intends to set them to random low-level noise on decoding, though, that's impossible for HEVC to copy.
Random noise? Why not transfer the approximate noise filter / decoder...

[dapperdan]

I guess it's just a coincidence that the name fits the Xiph naming scheme of strong female characters from pop culture?

[wiak]

this "MC" is a interesting case of what works works for others tooo
http://i.imgur.com/X6Nzlcf.png
http://i.imgur.com/D45fMKi.png

this forum reaily needs a img scaler (like gmail has)

[foxyshadis]

Random noise? Why not transfer the approximate noise filter / decoder...

Because there's no point, you don't need to bother. In high frequencies, low level random noise is indistinguishable from grain or shaped noise, and since grain doesn't much change across a single frame, all you need to transmit is one frame-level variance value to fill in all the missing coefficients. It also helps cover up other sources of noise, including compression noise. It's so consistent you can mostly get away with one value for the whole movie, but that's really stepping away from reproduction into stylized.

Shaped noise (and explicit coefficients) matters more for the lower frequencies, for synthesizing film grain and texture, which is usually more trouble than it's worth except in extremely low-bandwidth video. FGM suffers from being complicated enough to be all things to all users, so no one wants it, when a simpler noise generator might have succeeded. I was pretty disappointed when HEVC completely punted on even trying to improve it.

Alternately, instead of messing with the idct you could just sweep random noise across each decoded frame in the same way; it looks nearly as good, but more 'digital' which turns some people off.

Obviously, this doesn't apply to CGI and film stock with low noise and sharp detail. Also obviously, I don't subscribe to the cult of PSNR and perfect reproduction, not when it leaves better subjective quality on the table.

[colours]

I went through the presentation slides and this bit jumped out at me: Thor uses a nonseparable low-pass filter to compute a (0.5,0.5) offset (for luma only). Looking at the source code points to the kernel being [[0,1,1,0], [1,2,2,1], [1,2,2,1], [0,1,1,0]]/16. No negative coefficients at all?

I'm kinda curious about how much this strange choice of a nonseparable interpolation filter for the (0.5,0.5) offset improves on just using the separable 6-tap interpolation filter which is also used for all the other fifteen offsets.

[Parabola]

Hi colours,

This non-separable filter is a Gaussian-like low pass filter. I don't think it is intended to be better than the separable 6-tap, rather it is designed to give the encoder more flexibility.

There are 16 sub pixel offsets per integer MV position. They break down as follows:

(0,0) - the integer position, no filtering - motion compensation is a block copy
[6 positions] - e.g. (0.25, 0) or (0, 0.5) - one direction filtered with short, band-pass-ish filter - prediction may suffer softening, ringing or aliasing
[8 positions] - e.g. (0.25, 0.25) or (0.75, 0.5) - both direction filtered with short, band-pass-ish filter - prediction may suffer softening, ringing or aliasing
(0.5,0.5) - the mid position - motion compensation is a low pass filter - prediction will be soft

By selecting the mid position, the encoder is low-pass filtering the prediction. If motion hits (0.5, 0.5) but low-pass filtering is not beneficial then the encoder would likely use the neighbouring separable vector having lowest cost, e.g. (0.25, 0.5). I doubt this would hit RD badly.

Of course an alternate way of achieving this would be have a separate flag, or index to specify the motion compensation filter to be applied. Pure speculation on my part but perhaps this approach has prior art that the Thor team wishes to avoid. Or perhaps combining the MC filter mode with MVD signalling is cheaper since Thor does not use arithmetic coding.

[Kurtnoise]

fyi, Daala's Entropy Coder has been pulled (https://github.com/cisco/thor/pull/8) to Thor and vice-versa (https://review.xiph.org/874/)...:) Nice work btw.

From Monty Montgomery (http://xiphmont.livejournal.com/67534.html)...

[hajj_3]

Thor has been officially announced: http://blogs.cisco.com/collaboration/world-meet-thor-a-project-to-hammer-out-a-royalty-free-video-codec

[pandy]

Thor has been officially announced: http://blogs.cisco.com/collaboration/world-meet-thor-a-project-to-hammer-out-a-royalty-free-video-codec

Yupi, another video codec... :confused:

[mandarinka]

Not necessarily, at this point is is just a draft parallel to Daala. IETF will probably standarddize just one codec through their NetVC programme. It will likely take years anyway.

[LigH]

German technology sites and blogs of different topics start to discuss Thor already. Fefe's "conspiracy" blog (or rather a blog of political and social interests) points (http://blog.fefe.de/?ts=ab34af5e) at the fact that Cisco used to be happy while being a member of the H.264 patent pool, being on the earners' side ... but because H.265 has two independent patent pools, and licenses are not even capped, Cisco got afraid of becoming a main payer this time.

Capitalism is a great concept as long as you make profits.

[pandy]

Not necessarily, at this point is is just a draft parallel to Daala. IETF will probably standarddize just one codec through their NetVC programme. It will likely take years anyway.

Well... i believe Thor will be not Daala and Daala wil be not a Thor.
From my perspective Cisco is interested in niche codec capable to perform real time video streaming with reasonable resources usage even with less quality. Daala on other side pursue quality with reasonable resource usage - it almost same but not the same.
Probably both will share some ideas but at some point Daala should be capable to do more where Thor remain mostly niche codec.

German technology sites and blogs of different topics start to discuss Thor already. Fefe's "conspiracy" blog (or rather a blog of political and social interests) points (http://blog.fefe.de/?ts=ab34af5e) at the fact that Cisco used to be happy while being a member of the H.264 patent pool, being on the earners' side ... but because H.265 has two independent patent pools, and licenses are not even capped, Cisco got afraid of becoming a main payer this time.

Capitalism is a great concept as long as you make profits.

Accordingly to my knowledge Cisco is no longer under strict H.265 pressure as they selling division (where H.265 seem to be most used) to Technicolor.

[LigH]

A codec for a limited purpose may be easier to optimize than a general purpose codec. Not sure which interest Cisco may have mainly, but e.g. "video telephony" with face presentation may be narrow enough to optimize strongly for it.

[mandarinka]

I don't think there is that much space to differentiate with the format itself. Most of the needs can be addressed by encoder/decoder configuration/architecture, so it IMHO makes more sense to use a general-purpose format that will benefit from network effects and such.

[benwaggoner]

I don't think there is that much space to differentiate with the format itself. Most of the needs can be addressed by encoder/decoder configuration/architecture, so it IMHO makes more sense to use a general-purpose format that will benefit from network effects and such.
Yes, generally a flexible format can be used more narrowly for specific tasks.

However, codecs for extremely low latency don't need a lot of features useful for file-based encoding or streaming. For example, B-frames aren't used in VC since even a single one adds several frames of latency end-to-end. Robust error concealment and correction is also a lot more important than for other scenarios.

[pandy]

A codec for a limited purpose may be easier to optimize than a general purpose codec. Not sure which interest Cisco may have mainly, but e.g. "video telephony" with face presentation may be narrow enough to optimize strongly for it.

Build 3D face model, apply picture of face as texture, do motion capture from video - is there anything more efficient than this?
:D

[mandarinka]

Build 3D face model, apply picture of face as texture, do motion capture from video - is there anything more efficient than this?
:D

Then you subtract that prediction from source and you realize that you have residual that is ten times worse than what you had in 2002 with H.263 ;)

[pandy]

Then you subtract that prediction from source and you realize that you have residual that is ten times worse than what you had in 2002 with H.263 ;)

Maybe and... how this is important from conference perspective?

[kuchikirukia]

Maybe and... how this is important from conference perspective?

If H.263 would be give better quality while being less resource intensive, why wouldn't you just use H.263?

[pandy]

If H.263 would be give better quality while being less resource intensive, why wouldn't you just use H.263?

I doubt if there is something more efficient (bitrate) than mocap 3d recreation... similarity to audio LPC (functional).

[LigH]

Back to the purpose of this thread ... we have a new codec. Who was able to test it?

I'm just trying, and it is a little annoying; at first the lack of documentation from the CLI ('-?' doesn't show a brief help, '-h' doesn't either, just like omitting any parameter); then the assumption from the available config examples that only raw YUV is supported, not even Y4M, and possibly no pipe either, lack of documentation, you know (so I get afraid I may have to waste disk space to create a copy of my Y4M test samples without header). Luckily, thorenc supports the use of config files as input (https://github.com/cisco/thor):
Thorenc -cf config.txt

So reading through the sources a bit ... oh, Y4M is supported in the encoder ... as reconstructed output. And ... oh, as input too, so I may be able to omit these parameters from the config file?

Here are some defaults from enc/strings.c:
add_param_to_list(&list, "-cf", NULL, ARG_FILENAME, NULL);
add_param_to_list(&list, "-if", NULL, ARG_FILENAME, &params->infilestr);
add_param_to_list(&list, "-ph", "0", ARG_INTEGER, &params->file_headerlen);
add_param_to_list(&list, "-fh", "0", ARG_INTEGER, &params->frame_headerlen);
add_param_to_list(&list, "-of", NULL, ARG_FILENAME, &params->outfilestr);
add_param_to_list(&list, "-rf", NULL, ARG_FILENAME, &params->reconfilestr);
add_param_to_list(&list, "-stat", NULL, ARG_FILENAME, &params->statfilestr);
add_param_to_list(&list, "-n", "600", ARG_INTEGER, &params->num_frames);
add_param_to_list(&list, "-skip", "0", ARG_INTEGER, &params->skip);
add_param_to_list(&list, "-width", "1920", ARG_INTEGER, &params->width);
add_param_to_list(&list, "-height", "1080", ARG_INTEGER, &params->height);
add_param_to_list(&list, "-qp", "32", ARG_INTEGER, &params->qp);
add_param_to_list(&list, "-f", "60", ARG_FLOAT, &params->frame_rate);
add_param_to_list(&list, "-lambda_coeffI", "1.0", ARG_FLOAT, &params->lambda_coeffI);
add_param_to_list(&list, "-lambda_coeffP", "1.0", ARG_FLOAT, &params->lambda_coeffP);
add_param_to_list(&list, "-lambda_coeffB", "1.0", ARG_FLOAT, &params->lambda_coeffB);
add_param_to_list(&list, "-early_skip_thr", "0.0", ARG_FLOAT, &params->early_skip_thr);
add_param_to_list(&list, "-enable_tb_split", "0", ARG_INTEGER, &params->enable_tb_split);
add_param_to_list(&list, "-enable_pb_split", "0", ARG_INTEGER, &params->enable_pb_split);
add_param_to_list(&list, "-max_num_ref", "1", ARG_INTEGER, &params->max_num_ref);
add_param_to_list(&list, "-HQperiod", "1", ARG_INTEGER, &params->HQperiod);
add_param_to_list(&list, "-num_reorder_pics", "0", ARG_INTEGER, &params->num_reorder_pics);
add_param_to_list(&list, "-dqpP", "0", ARG_INTEGER, &params->dqpP);
add_param_to_list(&list, "-dqpB", "0", ARG_INTEGER, &params->dqpB);
add_param_to_list(&list, "-mqpP", "1.0", ARG_FLOAT, &params->mqpP);
add_param_to_list(&list, "-mqpB", "1.0", ARG_FLOAT, &params->mqpB);
add_param_to_list(&list, "-dqpI", "0", ARG_INTEGER, &params->dqpI);
add_param_to_list(&list, "-intra_period", "0", ARG_INTEGER, &params->intra_period);
add_param_to_list(&list, "-intra_rdo", "0", ARG_INTEGER, &params->intra_rdo);
add_param_to_list(&list, "-rdoq", "0", ARG_INTEGER, &params->rdoq);
add_param_to_list(&list, "-max_delta_qp", "0", ARG_INTEGER, &params->max_delta_qp);
add_param_to_list(&list, "-encoder_speed", "0", ARG_INTEGER, &params->encoder_speed);
add_param_to_list(&list, "-deblocking", "1", ARG_INTEGER, &params->deblocking);
add_param_to_list(&list, "-clpf", "1", ARG_INTEGER, &params->clpf);
add_param_to_list(&list, "-snrcalc", "1", ARG_INTEGER, &params->snrcalc);
add_param_to_list(&list, "-use_block_contexts", "0", ARG_INTEGER, &params->use_block_contexts);
add_param_to_list(&list, "-enable_bipred", "0", ARG_INTEGER, &params->enable_bipred);


Let's start with a test file I have available as raw YUV 4:2:0 anyway:
Thorenc.exe -if Johnny_1280x720_60.yuv -width 1280 -height 720 -f 60 -n 600 -of Johnny_1280x720_60.bit -rf Johnny_1280x720_60.recon.y4m
It does something ... at 2 seconds per frame on an AMD Phenom-II X4. But the LC result (https://www.mediafire.com/download/0olmw5twiofmrr1/Johnny_1280x720_60.bit) looks already quite convenient with the rather low-complexity defaults (not even B frames or multi refs).

Preparing a config for higher efficiency, testing again...

-if Johnny_1280x720_60.yuv ; Input yuv sequence
-of Johnny_1280x720_60_HE.bit ; Output bitstream
-rf Johnny_1280x720_60_HE.recon.y4m ; Reconstructed yuv sequence
-stat Johnny_1280x720_60_HE.stat.txt ; Statistics to file
-width 1280 ; Witdh of luminance
-height 720 ; Height of luminance
-n 600 ; Number of frames to encode
-f 60 ; Frame rate in Hz
-qp 32 ; Quantization parameter
-HQperiod 12 ; Period of high quality frames
-mqpP 1.2 ; QP multiplier for low quality P frames
-dqpI -2 ; QP offset for intra frames
-lambda_coeffI 1.2 ; Multiplier for lambda - I frames
-lambda_coeffP 1.2 ; Multiplier for lambda - P frames

;
;High complexity operating point
;
-intra_rdo 1 ; Use RDO for choosing intra mode
-enable_tb_split 1 ; Enable splitting of a block in 4 transform blocks
-enable_pb_split 1 ; Enable splitting of an inter block in 4 prediction blocks
-early_skip_thr 0.3 ; Early skip threshold
-max_num_ref 4 ; Number of reference frames
-use_block_contexts 1 ; Use block contexts
-enable_bipred 1 ; Enable biprediction
-encoder_speed 0 ; Encoder complexity parameter (0: Slow, 1: Moderate: 2: Fast)

Encoding speed drops to more than 10 seconds per frame. — After 3.5 hours for 600 frames: Less than 3 frames per minute. Well, the x265 encoder for HEVC is extremely optimized, in comparison. But apart from that ... the HE result (https://www.mediafire.com/download/ic1ook94uwj4zuc/Johnny_1280x720_60_HE.bit) looks rather convenient, too (just blurred, yet structured, hardly any ringing), at a much smaller size even.

The stats file tells little, just few statistical values, it's not like a 2-pass bitrate distribution stats file:
NFR kbps PSNRY PSNRU PSNRV
600 161.215 38.010 44.001 44.578

I doubt I will easily do "same size" comparisons very soon, due to the very low encoding speed, but maybe in some future... a MediaFire shareable folder is reserved for samples.

[Havokdan]

Maybe offtopic: http://aomedia.org

WHAT IS THE ALLIANCE?
The Alliance for Open Media is founded by leading Internet companies focused on developing next-generation media formats, codecs and technologies in the public interest. The new Alliance is committing its collective technology and expertise to meet growing Internet demand for top-quality video, audio, imagery and streaming across devices of all kinds and for users worldwide.

The initial project will pursue a new, open royalty-free video codec specification and open-source implementation based on the contributions of members, along with binding specifications for media format, content encryption and adaptive streaming, thereby creating opportunities for next-generation media experiences.

Day one founding members are Amazon, Cisco, Google, Intel Corporation, Microsoft, Mozilla and Netflix.

[BadFrame]

Maybe offtopic: http://aomedia.org

This needs it's own topic I think, it's quite a bomb in the video codec world.

[dapperdan]

I gathered a few of the announcement posts about the Alliance and started a thread here:

http://forum.doom9.org/showthread.php?t=172550

[mandarinka]

Presentation about Thor from VDD 2015: https://www.youtube.com/watch?v=g6m_N3QlqOI

I didn't yet have time to watch it whole, but it seems that Thor is basically something that Cisco originally proposed for HEVC, but with lower complexity (which means the lack of CABAC-grade entropy coding is intentional, meh!), and now it is basically being recycled. Which also means that Thor as a format is not going to be competitive with HEVC, even if it had a competitively-tuned encoder available which it AFAIK doesn't have anyway.

I hope the influence of these low-complexity aims won't cripple NetVC's overall compression strength, as Thor and Cisco is another input into the NetVC program apart from Daala.

[Nil Einne]

Maybe and... how this is important from conference perspective?

Probably quite important? My understanding is a lot of the stuff Cisco sells for video conferencing isn't intended to be Skype with low latency. Latency is important of course, but for something like Cisco TelePresence for example, Cisco even designs the rooms so they are similar, so it's basically like the person or people are really there. I'm not sure if they're interested in Thor, or other medium complexity codecs for that or it's more targeted at lower end solutions, but the point remains just because it's video telephony doesn't mean the quality (as well as low latency and reliability and of course cost) isn't important.

Anyway speaking more generally for Thor, whatever work can help the royalty free codec space is IMO good. While I'm not the sort of person desperate for royalty free to take over (although I'm not a big fan of the US's software patent regime either), if it does take over and it's always possible royalty payment disagreements mean it will, I would prefer we have the best possible solution. And unless there is a major change in the software patent regime, or in how patent holders act, that's going to mean taking what ever we can get. It seems Thor could have some minor or even more significant benefits to the NetVC, although mandarinka has said, hopefully this won't be in a low complexity (and therefore likely lower quality/bitrate) fashion.