Good encoder for master recording

[Kivenkantaja]

Hi all,

I've captured data from an analog camrecorder, using the Huffyuv lossless codec for video, and WAV for audio.
These files are obviously big (10Go for less than 15min).

Can you give me advice on a good way to compress more these files ?
First, I was doing lossless compression using "x264 --crf 0" and FLAC fo audio.
But it's still big, and x264 requires chroma subsampling from the 4:2:2 of Huffyuv. So this is kind of lossy rencoding.

Finally, I guess that compressing with a lossy codec, but with good quality settings, would be enough. But which one ? (something which provide invisible loss for the human eye and not too much complexity for further reeconding)

Thanks in advance !

[um3k]

You could try using HCEnc to encode 4:2:2 MPEG at a high bitrate and max quality settings.

[mp3dom]

You can also choose to encode to MPEG2 422P@ML.
It's a 4:2:2 Mpeg2, generally I-frame only (but you can choose to do I-P, discarding B frames). You can then decode it with DGIndex without too much trouble. The bitrates goes up to 50Mbps so 25-30 min. would be about 8 Gb.
The I-frame only structure is a standard format for IMX and some professional NLE for post-production.
I'm not sure, but FFmpeg should be able to encode this format.

[TheSkiller]

You can also try a M-JPEG codec like the Morgan Multimedia MJPEG or the PicVideo MJPEG Codec, and there's also a MJPEG Codec built into the ffdshow "Codec". They are all capable of 4:2:2 (YUY2) encoding. I personally use the Morgan one, you can set the quality from 1 (unwatchable) to 100 (lossless). Using 95 or so will give you a visually lossless encode and is much smaller than the HuffYUV file.

Using MPEG2 I-frame only is pretty much the same as M-JPEG, but if you use P-frames in the MPEG2 encode the file size of the MPEG2 encode would be smaller than M-JPEG.

Edit: Converting from HuffYUV to a M-JPEG codec is very fast, it's so fast that you should be using 2 physical hard drives for this task (one HDD reads the HuffYUV source the other writes the M-JPEG compressed AVI), because otherwise the HDD would cause a bottleneck having to read and write BIG amounts of data at the same time.

[Blue_MiSfit]

Yeah, HC would be my preference. 4:2:2 support is a big deal, plus the whole free thing... :)

If you aren't going to do any more effects work on the content, why are you worried about going from 4:2:2 to 4:2:0? Harmless! Just do it, and instead of using --qp 0 (lossless) x264, use --crf 14 or something, maybe with a short GOP size, and --tune fastdecode if it's HD.

I can't adequately express how much better x264 would actually be for this task if you don't absolutely positively need to keep the 4:2:2.

Derek

[pandy]

Still You can use x264 for lossless compression only resize (Point Resize) twice vertical size of the video.

[lordsmurf]

I would suggest the Matrox MPEG-2 codecs.
There's some discussion (and download links) here: http://www.digitalFAQ.com/forum/showthread.php/dv-color-space-2270.html?p=11936#post11936
That gives you 4:2:2 and good archival quality, smaller than HuffYUV, using one of the best MPEG-2 codecs made.

I would not encode down to H.264.

[Ghitulescu]

How do you intend to play them?

[Kivenkantaja]

First, thank you for all your posts !

I've realized that I did not highlight well my purpose with these videos...

So, here is my plan :

I load these HuffYUV videos with an AviSynth script and make treatment ; then I encode into a format that fit for my personal use (which is x264+AAC).
The .avs does, for example : trim, crop, resize, deinterlace, add title, fade in, fade out, denoise... by the way, I'm using Convolution3D which requires 4:2:0...

So, now, I would like to keep these "master records" in case I need to re-encode them into another format, or make a compilation or what ever. But, keeping these "HuffYUV files" take a lot of disk space...
That's why I would like to find a good compromise between low file size and keeping the raw files (without treatment), with compression which does not affect too much the quality for future processing on these videos.

I tried to follow the 422P@ML standard, which is a good start. But regarding your comments, I think that I can go deeper in the compression. Firstly by downsampling chroma to 4:2:0. Maybe, encoding into a DVD standard (I mean MPEG-2 with DVD bitrate, maybe MP@ML ?) format is just enough !

The image size is 720x576, the frequency is 25img/s. And... if other data is useful, or if I'm not clear, tell me :)

[mp3dom]

Convolution3D is available also as a YUY2 filter (4:2:2). If you want to downsample to 4:2:0 you could try Lagarith or Ut Video codec. Both are lossless and both works also in YV12 colorspace. You should have a smaller file size than HuffYUV and 422P@ML.
Encoding to DVD is a bad idea if you want to edit in the future the footage (unless you only pretend to trim or add some small effects, in that case there are tools that can edit directly on the MPEG2 re-encoding only the parts that you're editing and passing the rest 'as-is')

[TheSkiller]

Yeah, I wouldn't use DVD MPEG2 for this purpose either. But if you want stronger compression then what about DV. It's fixed to 25Mbit/s so this will give you 13GB per hour. That's a good trade off between quality and storage space and keeps the flexibility you had with the original HuffYUV file.

[Ghitulescu]

The DV has some problems with the colour. However I'll keep saying that the DV is still the best format for casual users.
I would use the HuffYUV or similar.
What I use is the DVD encoding I did "before". I simply keep the DVDs.

[mariush]

lagarith would probably give you higher compression, with smaller decoding speed.

[dansrfe]

Nothing beats lagarith at lossless ;)

[2Bdecided]

The DV has some problems with the colour.His source is an analogue camcorder. It has about 30 lines of chroma resolution across the screen. The 180 lines of chroma resolution you get from NTSC DV are not going to make it any worse.

However, extreme noise on captures can visibly upset DV encoding (though of course MPEG-2, unless I-frame only at 25Mbps which no one does, is far worse).

Cheers,
David.

[TheSkiller]

The image size is 720x576, the frequency is 25img/s.It's a PAL capture so with DV he would end up with interlaced 4:2:0 (YV12).

[2Bdecided]

It's a PAL capture so with DV he would end up with interlaced 4:2:0 (YV12).True - though I've yet to find a beneficial use for interlaced YUY2 in this context. The final output is bound to be YV12 (interlaced or progressive), and any processing / denoising / deinterlacing works fine in YV12 (indeed, some functions don't work in YUY2).

Don't forget that PAL VCRs are already averaging the chroma in adjacent-but-one lines (in fields, not frames) to remove colour cross talk. YV12 isn't losing any real information.

The conversion from DV YV12 to MPEG-2 YV12 causes a very very small but just visible degradation with high quality digital sources - I'm not convinced it can ever be visible with VHS.

Cheers,
David.

[Kivenkantaja]

Indeed it's an interlaced PAL capture.

I'm not finding any beneficial use too for keeping YUY2, but I wanted confirmation from people who knows more than me :)

The conversion from DV YV12 to MPEG-2 YV12 causes a very very small but just visible degradation with high quality digital sources - I'm not convinced it can ever be visible with VHS.
In fact, I just did not want to loose "stupidly" crucial information. (Maybe I would have lost too much important data by converting into MPEG-2 DVD standard).

So, I've launched this command line :
ffmpeg -i video.avi -target pal-dv video.dv

Encoding is quite fast, file size is good (what I expected), and I can't find degradation of the quality (invisible for me) : it's perfect !

So, thank you very much for you support :)

I just have one step remaining. I would like to see the quality after the treatment and the encoding with x264 I used to do. The problem is that I can't load the "video.dv" with AviSynth. I think that I need a decoder but I guessed that ffdshow would be able to decode it ... I'll dig on this in few days and post the result.

Thanks again :)

[poisondeathray]

The problem is that I can't load the "video.dv" with AviSynth. I think that I need a decoder but I guessed that ffdshow would be able to decode it ... I'll dig on this in few days and post the result.



You should be able to load "video.dv" into avisynth with ffmpegsource2 , or re-wrap it into .mov container with mpegstreamclip or ffmpeg, then use QTInput()

[TheSkiller]

Don't forget that PAL VCRs are already averaging the chroma in adjacent-but-one lines (in fields, not frames) to remove colour cross talk.Yes, but this only happens as long as the VCR is in it's standard mode, a good S-VHS VCR allows the user to set a so called EDIT-mode (also called Tape-Dub) in which all of the usual low pass / average filters (chroma vertically and luma horizontally + user selectable strength of outline sharpening "sharpness") are disabled. The picture is much more pure in this mode but there is also a bit more noise. But still, YV12 is totally fine here.

[Kivenkantaja]

You should be able to load "video.dv" into avisynth with ffmpegsource2 , or re-wrap it into .mov container with mpegstreamclip or ffmpeg, then use QTInput()

Indeed it's working thanks :)

But I can't read it with ffdshow through Media Player Classic-HC.

When I'm looking into the configuration of the video decoder of ffdshow, I have the "DV" and "Other DV" lines which are decoded with libavcodec. But MPC is not able to read it... and would like to know why...

[kolak]

Nothing beats lagarith at lossless ;)

Canopus Losselss codec will be probably bigger, but way faster.
Very good codec.

Andrew

[poisondeathray]

Indeed it's working thanks :)

But I can't read it with ffdshow through Media Player Classic-HC.

When I'm looking into the configuration of the video decoder of ffdshow, I have the "DV" and "Other DV" lines which are decoded with libavcodec. But MPC is not able to read it... and would like to know why...

Do you mean the native file , not through avisynth ?

It also depends on what directshow decoders and splitters you have installed . For example, I can read .dv files natively in MPC . MPCHC might not have a .dv splitter. I use mainconcept's dv splitter and decoder

Another option you could try is to encode to PAL DV-AVI through ffmpeg (instead of .dv format)

[Kivenkantaja]

I've finally find a solution (in particular with the help of the FFmpeg mailing list)

The problem was that <code>-target pal-dv</code> does not provide a good FourCC ("dv " instead of "dvsd"), even with the parameter <code>-vtag dvsd</code>.

So, I use this command line :
<code>-vcodec dvvideo -pix_fmt yuv420p -r 25</code>
"-vcodec dvvideo " can be replaced by "-f dv" I guess.

Thanks again for your attention and your help !