Temporary lossless format for 10-bit video, with good random access?

[wonkey_monkey]

I'm trying to work with some HDR video files which I only need small snippets from. I've gathered the snippets together with a single .avs file and now I want to encode the output to a file that I can read reasonably quickly.

FFV1 seemed like a good/only choice, as it's a lossless intraframe codec with 10-bit support. So I used ffmpeg to encode an .avi file (I think I tried .mp4 first but some error came up).

I can open this file with ffmpegsource2 or BSSource (I shouldn't need the latter's frame accuracy guarantee, since it's an AVI file with an infraframe codec). It's a little slow - 12fps or so - but the bigger problem is that it's very slow to seek backwards and for random access.

Does anyone know why this might be the case - it's an intra-frame codec so shouldn't reading a random frame be just as fast as reading the next frame? - and what a better option might be?

[FranceBB]

There's always the good old HuffYUV which is all intra, supports up to 16bit planar, and should be reasonably fast to seek. That's what I use all the time.

PQ HuffYUV:

ffmpeg.exe -hide_banner -i "AVS Script.avs" -vcodec ffvhuff -color_range 1 -color_primaries bt2020 -color_trc smpte2084 -colorspace bt2020nc -c:a pcm_s32le -ar 48000 -y "output.mov"

pause

HLG HuffYUV:

ffmpeg.exe -hide_banner -i "AVS Script.avs" -vcodec ffvhuff -color_range 1 -color_primaries bt2020 -color_trc arib-std-b67 -colorspace bt2020nc -c:a pcm_s32le -ar 48000 -y "output.mov"

pause

UTVideo also used to be good and I've used it in the past, although I decided to stick with HuffYUV.

There are other alternatives too, of course, like x264 and x265 using the lossless mode, however I always found them harder / heavier to decode.

Otherwise, if you just want a lossless uncompressed 10bit output there's v210 but keep in mind that a 720x576 25i TFF SD file is like 220 Mbit/s if I remember correctly, so for UHD it would be huge and at that point you're gonna be slowed down by the network (if it's an SMB Share):

PQ v210:

ffmpeg.exe -hide_banner -i "AVS Script.avs" -vcodec v210 -color_range 1 -color_primaries bt2020 -color_trc smpte2084 -colorspace bt2020nc -c:a pcm_s32le -ar 48000 -y "output.mov"

pause

HLG v210:

ffmpeg.exe -hide_banner -i "AVS Script.avs" -vcodec v210 -color_range 1 -color_primaries bt2020 -color_trc arib-std-b67 -colorspace bt2020nc -c:a pcm_s32le -ar 48000 -y "output.mov"

pause

[SeeMoreDigital]

Out of interest... How well does HEVC lossless perform when set to 10 or even 12-bit mode?

In 10-bit mode you can even play the encodes on a smart TV that supports HEVC...

[tormento]

Well, you can play with nvenc lossless too if you have a nvidia card.

Pros: it's really fast
Cons: it's not as efficient as other cpu based codecs

[wonkey_monkey]

Well HuffyUV at least doesn't seem to suffer, at least not as badly, from the same slow random/reverse access as FFV1, though I'm still not sure why FFV1 should do so. Looks like it'll be about twice the size of FFV1 though.

I didn't think UTVideo supported >8 bit.

v210 is slightly slower backwards than forwards. 16Gb for 30 seconds, and I need 5m45s I do have the space, for now.

HuffyUV is giving me 70fps (forwards, anyway) vs 20fps for v210 though. Guess it's a disc bottleneck, even though it's a USB3 SSD. Edit: oops, no, it's the internal SATA III SSD.

Is there a particular reason to favour a .mov container?

[FranceBB]

Originally Posted by SeeMoreDigital

Out of interest... How well does HEVC lossless perform when set to 10 or even 12-bit mode?

Well, I didn't do UHD progressive stuff with x265 in lossless mode, but I did SD interlaced and it performed pretty poorly, which is not surprising given that nearly no optimization went into it as the use-case was almost non existent. https://forum.doom9.org/showthread.php?t=185461


A snippet from my own post from 2024:

Originally Posted by FranceBB

Duration: 2h and 7 minutes
Video Codec: v210 720x576 25i TFF 4:2:2 BT601 10bit planar
Audio Codec: 1 track 4ch PCM 24bit 48000Hz
Container: avi

Lossless Encoding candidates:

- UTVideo 720x576 25i TFF 4:2:2 BT601 10bit planar
- H.264 720x576 25i TFF 4:2:2 BT601 10bit planar
- H.265 720x576 25i TFF 4:2:2 BT601 10bit planar
- FFV1 720x576 25i TFF 4:2:2 BT601 10bit planar
- HuffYUV 720x576 25i TFF 4:2:2 BT601 10bit planar

Output (from bigger to smaller):

0) v210 332 Mbit/s (315 GB)
1) HuffYUV 99 Mbit/s (97 GB)
2) H.265 72 Mbit/s (71 GB)
3) H.264 64 Mbit/s (65 GB)
4) UTVideo 54 Mbit/s (56 GB)
5) FFV1 37 Mbit/s (39 GB)

Yes, you read that right, H.265 performs worse than H.264 in lossless mode with interlaced contents, but both were better than HuffYUV.

Originally Posted by wonkey_monkey

I didn't think UTVideo supported >8 bit.

Oh, it definitely does as long as it's YUV 4:2:2, not sure about other sampling, though.

Originally Posted by wonkey_monkey

HuffyUV is giving me 70fps (forwards, anyway) vs 20fps for v210 though. Guess it's a disc bottleneck, even though it's a USB3 SSD.

You're lucky it's a locally attached SSD, imagine doing it via an SMB Share xD

Anyway, I don't really use v210 for anything other than SD.

Originally Posted by wonkey_monkey

Is there a particular reason to favour a .mov container?

Nah, you can use anything you want. Even .mkv would work fine when it's a matter of opening it with other open source stuff, but if you have to send it somewhere ideally .mov is probably the least painful option that still allows people to decode it. You could very well create an .avi, but the problem with an .avi container is that it won't preserve metadata, I'm not talking about MaxCLL/MaxFALL, I'm talking about even basic stuff like colormatrix, transfer, primaries and range which is a bit of a problem with HDR.

[StainlessS]

There is also MagicYUV [MagicYUV_v1.2_win-rev2.exe free, later paid {Trialware $10}],
https://www.videohelp.com/software/M...n=Lf32QdtLXHSD

[Emulgator]

For mezzanines I am daily throwing around 4k50/59.94p apch .mov encoded by FFMPEG (10bit 422HQ)
Can be handled quite universely, MPC-BE, MPC-HC, Subtitle Edit, VD, VD2, Topaz, Vegas Pro 13/14/15/16.
I used to use MagicYUV Ultimate often as well, but apch .mov was more widely accepted.

[pbristow]

I just had to to look up what apch meant... Apparently it's "Adobe Prores (something something)", and the first half-dozen search hits are all from people having difficulties with playing it. This link, though, gives a hint as to why that might be:
https://discussions.apple.com/thread...69?sortBy=rank

So... It's basically Adobe's custom variation of an otherwise standard container format (.mov), which could contain absolutely anything... But as long as whatever data stream you stuff into it is universally readable, it shouldn't cause problems... Except that someone at Microsoft seems not to have got that message, and is making their software spit out "apch format not supported" messages, when the problem is more likely to be whatever codec was used for the stream inside. =:o\

Thanks for the educational diversion! =:o}

[SeeMoreDigital]

Originally Posted by FranceBB

Yes, you read that right, H.265 performs worse than H.264 in lossless mode with interlaced contents, but both were better than HuffYUV.
.

I see... But you only tested interlaced SD content

[Z2697]

FFV1 is an inter-frame codec/encoder

Note: I'm just assuming from the fact that it has keyframe and non-key-frame... and no frame level multithread.
FFmpeg does claim it to be an intra-frame codec.
https://github.com/FFmpeg/FFV1?tab=r...e#introduction

[Z2697]

Originally Posted by FranceBB

There are other alternatives too, of course, like x264 and x265 using the lossless mode, however I always found them harder / heavier to decode.

CABAC can be hard on computation when there's a lot of data going through it.
With HEVC you don't have a choice, but using CAVLC with AVC makes it way faster, not without the decreased compression ratio of course.

(With lower bitrates the cost of CABAC is usually small enough and masked by costs of other components)

And what's the setting of that size comparison? You might given FFV1 unfair favor if you compare it with intra-only AVC.

[johnmeyer]

You might want to consider "visually lossless" codecs. They eliminate the performance (and also size) problems. Way back when, Cineform was the codec most people used. I have gotten a little out of date, but Cineform still works for my uses and I think you may be able to still download it for free from the GoPro site (it used to cost close to $1,000). A quick search suggests that people now often use ProRes with one of its high profile settings.

I seldom use a truly lossless codec because I don't see any advantages over a visually lossless codec. I say that having spent hours doing A/B tests between the original and the lossy compressed versions, using the high quality settings. I could never detect any differences.