Home movies archiving

[drob]

My dads camera saves mpeg2, until now he was editing the footage, then creating a DVD media for family distribution, and saving an uncompressed avi copy for archiving purposes.

Reading throw the forum i understood that since the footage was lossy o begin g with, there is no point in keeping it uncompressed, would transcoding it to h264 at a high bitrate, say 10000kbs be good for a transparent archiving purposes? would lossy h264 be better? on a side note is lossy h264 hardware player compatible?

[CarlEdman]

I am not quite sure I understand. If the camera captures mpeg2, the resulting file will be compressed. Does you father then go out of his way to uncompress the mpeg2 and store it in a truly lossless format? Why would he do that when he can just store the compressed mpeg2s?

As for transcoding video captured by power and real-time constrained devices (and hence probably compressed in a highly inefficient manner) to h264, it can be a big win. In my experience, a size reduction of a factor of 10 or 20 without any loss of quality is often perfectly achievable.

Without more information on the resolution, frame-rate, type of material, and type of compression of the source material, it is impossible to say what your target bit-rate should be (though 10,000 kbps is almost certainly much, much too high).

In fact, for archival purposes you probably should not be targetting a bit-rate, but pick a constant rate factor. Something between 20 and 25 works for many people and applications like yours, but only you and your eyes can pick the right trade-off at the end.

Finally, h264 compressed materials will no longer play on standard DVD players. It will play on almost any computer with suitable software and, depending on the parameters picked, many game consoles (such as PS3s or Xbox360s), portable devices, and other hardware decoders.

[nm]

I am not quite sure I understand. If the camera captures mpeg2, the resulting file will be compressed. Does you father then go out of his way to uncompress the mpeg2 and store it in a truly lossless format? Why would he do that when he can just store the compressed mpeg2s?

You forgot editing. If he does a lot of manual work to compose the final video from the MPEG-2 clips by adjusting colors and adding transition effects, the result needs to be re-encoded to some format. Lossless is probably overkill, but I'd archive the original captures from the camera as-is, and encode the final edited work with x264 at CRF 16. Quality-wise, that encode should be more than good enough for later use, i.e. re-encoding for some particular playback device.

CRF encodes can be made compatible with most standalone HD players by following Blu-ray spec limits (VBV and coding parameters). Lossless H.264 is not compatible. Xbox360 and AppleTV are somewhat crippled in their H.264 playback capabilities, so I'd forget about them if you don't own one.

[drob]

Yes the editing phase was dropped, hes process is importing into pinnacle studio, editing and so on, then create a dvd of final edit, and archive an uncompressed avi.

Here is the mediainfo of a small capture, do note he always uses 16:9 ration, as opposed to 4:3 here, the camera is set to ultra fine, specified by the manual as 8500 kbps VBR.


ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings, BVOP : Yes
Format settings, Matrix : Default
Duration : 8s 640ms
Bit rate mode : Variable
Bit rate : 5 831 Kbps
Nominal bit rate : 8 500 Kbps
Width : 720 pixels
Height : 576 pixels
Display aspect ratio : 4:3
Frame rate : 25.000 fps
Standard : PAL
Resolution : 8 bits
Colorimetry : 4:2:0
Scan type : Interlaced
Scan order : Top Field First
Bits/(Pixel*Frame) : 0.562
Stream size : 6.01 MiB (92%)

[Lam3rD]

Encoding it with crf 16 x264 is a great option - it will be around 30-120% of the size of the DVD and better looking, sure there might be a slight difference between it and the uncompressed, but is a great balance between filesize and loss. If you are too picky you can try with lower crf, lower --qpmax (like 20-24, or even 16 if you are way too picky) and lower --qpmin (6-8), but your files will get bigger.

[drob]

Since my dad is no spring chicken, i am going to need some very simple + preset GUI for him, something that he would simply put in the AVI and get an MP4 container out, so obviously MeGUI is not an option, out of all the GUI option which would you recommend?

[Lam3rD]

StaxRip is a good (near) 1 click solution

[drob]

I am now testing both StaxRip @ ASXGui, both seems very nice. should i use film tune? also for future compatibility should i specify level 4.1 instead of unrestricted?

I started with StaxRip template of film HQ and changed the quality to 16 and the level to 4.1, any other fine tuning i should make considering this input, and lastly should i crop, considering this is a 4:3?

Encoding settings from Mediainfo:
Encoding settings : cabac=1 / ref=3 / deblock=1:-1:-1 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.0:0.2 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-3 / threads=3 / sliced_threads=0 / nr=0 / decimate=1 / mbaff=0 / constrained_intra=0 / bframes=3 / b_pyramid=0 / b_adapt=1 / b_bias=0 / direct=1 / wpredb=1 / wpredp=2 / keyint=250 / keyint_min=25 / scenecut=40 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Encoded date : UTC 2010-02-16 20:08:33
Tagged date : UTC 2010-02-16 20:08:33

Source:
ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings, BVOP : Yes
Format settings, Matrix : Default
Duration : 8s 40ms
Bit rate mode : Variable
Bit rate : 8 900 Kbps
Width : 720 pixels
Height : 576 pixels
Display aspect ratio : 4:3
Frame rate : 25.000 fps
Standard : PAL
Resolution : 8 bits
Colorimetry : 4:2:0
Scan type : Interlaced
Scan order : Top Field First
Bits/(Pixel*Frame) : 0.858
Stream size : 8.57 MiB (91%)

[CarlEdman]

I am now testing both StaxRip @ ASXGui, both seems very nice. should i use film tune? also for future compatibility should i specify level 4.1 instead of unrestricted

Tune film works pretty well for most source material taken of real environments.

As for level, at your resolution and frame rate, 3.1 would probably be optimal. Higher levels are unlikely to help much, if any, and could be rejected by some low-power devices.

I started with StaxRip template of film HQ and changed the quality to 16 and the level to 4.1, any other fine tuning i should make considering this input, and lastly should i crop, considering this is a 4:3?

Not having seen your source material, I suspect that quality 16 may be overkill. Try encoding at 18, 20, 22, 24, etc. and check if you can see any difference in either motion or still frames. On the other hand, even crf 16 may be so much of an improvement over uncompressed storage that you don't really care about any additional space savings.

As for cropping, it should be done if and only if there are black borders in the source material. With anamorphic encoding, there may not be such borders, even if the source material is 4:3 and 720:576!=4:3.

[RunningSkittle]

drob: read the x264 long help page if you want to learn more about presets and tunings or read here: http://mewiki.project357.com/wiki/X264_Settings

If your content does not explicitly match the tune options, leave it blank!

by the way, with the new preset and tuning system, this is what a modern command looks like:
$x264 infile.avs --preset slow --level 4.1 --crf 16 --tune film -o outfile.mp4

Yes, thats all there is to it ;) so maybe consider making a small batch file for him if all of his encodes will be identical!

[nm]

Encoding settings from Mediainfo:
... / mbaff=0 / ...

Source:
...
Scan type : Interlaced
Scan order : Top Field First
...

That source should be encoded as interlaced.

[drob]

Your all so helpful and i am learning new stuff, i am use to mpeg2 encoding so H264 is all new to me.

I saw no black borders, but StaxRip automatically added crop to 704 on the template so i wasn't sure.

Shouldn't i deinterlace for future proof and more efficient encoding? again StaxRip added interlace automatically, source is definitely interlaced.

[nm]

Shouldn't i deinterlace for future proof and more efficient encoding?
Encoding as interlaced is usually more efficient than bobbing to 50p first, even with current x264 and it's limited intelaced encoding mode. Furthermore, if you are later making DVDs or 576i Blu-rays from the encode, you'd need to interlace again, wasting the deinterlacing effort and possibly some quality. Deinterlacers will likely get better in the future, so I'd archive interlaced footage as interlaced.

[Ghitulescu]

Uncompressed AVI in D1 size? That's about 60-70 GB per hour. Where did he stored this HUGE amount of data?

Being given an MPEG-2 stream, the logical way of archiving would have been as DVD. But since it was edited into something else, you can pick up either MPEG-2 again (useful for DVDs and future SDTV BDRs) or H.264 (BDR). These 2 codecs will give you the maximum freedom in picking up a player.

[drob]

You know when it gets scary? when all the numbers and switches start to make sense, i did lots of reading and am starting to create my own template in StaxRip (which is a really great beginner GUI) using all your input and tips.
The only thing i am still not sure of is encoding as interlaced, all guides i saw for x264 say deinterlace before, and going for progressive encoding sounds more prudent considering the major move to HD displays. why would reencoding to DVD require going interlace again? aren't DVD's producing progressive frames if input progressive material an outputting to progressive display (aka all flat HD screens)?

Lastly t what profile do the speed lost become unperceivable, should i go higher then slow?

[nm]

The only thing i am still not sure of is encoding as interlaced, all guides i saw for x264 say deinterlace before, and going for progressive encoding sounds more prudent considering the major move to HD displays. why would reencoding to DVD require going interlace again? aren't DVD's producing progressive frames if input progressive material an outputting to progressive display (aka all flat HD screens)?
DVD and Blu-ray only support 50i (=25i) and 25p PAL video. 576p50 is not allowed. Blu-ray supports 720p50, so you could upscale if bitrate doesn't matter. For DVD, you need to either deinterlace to 576p25 and lose temporal information, or encode as interlaced.

When your videos are archived as interlaced, it's easy to deinterlace later when re-encoding for some device that supports progressive video better. If you deinterlace now with method X, you might later notice that it sucks in some cases and the only way to go back is to reauthor the video from original clips (provided that you still have them).

If you decide to deinterlace, currently the best options for bobbing are slow AviSynth filters (MVBob, MCBob, TempGaussMC, ...) and leveraging Nvidia's GPU filters through DGNVTools. Choosing the right filter and setting it up properly is much more important than x264 settings, so you should spend some time on it.

Lastly t what profile do the speed lost become unperceivable, should i go higher then slow?
You mean at which preset quality/bitrate is no longer improved significantly? "Slow" is pretty good choice. I'd only use "slower" or "veryslow" when encoding for heavily bitrate-constrained distribution.

[drob]

So i went ahead and encoded interlaced as advised, however it seems Pinnacle Studio which is what my dad uses, has a problem with reading the interlaced encode, it would play fine up until the middle of the clip, then the picture would freeze, sound would continue to play until the end of the clip. If i feed it deinterlaced material there is no problem.
So unless i can find a pinnacle solution i have to deinterlace, StaxRip uses yadif, which as i understand is a very good filter, so hopefully it wouldn't cause to much damage, viability wise it looks the same as the original.

[nm]

So i wen ahead and encoded interlaced as advises, however it seems Pinnacle Studio which is what my dad uses, has a problem with reading the interlaced encode, it would play fine up until the middle of the clip, then the picture would freeze, sound would continue to play until the end of the clip. If i feed it deinterlaced material there is no problem.
Were the container and stream framerates set correctly to 25 fps?

So unless i can find a pinnacle solution i have to deinterlace, StaxRip uses yadif, which as i understand is a very good filter, so hopefully it wouldn't cause to much damage, viability wise it looks the same as the original.
Yadif is a decent realtime software deinterlacer, but its not good enough for archival-quality work in my opinion. It smoothes/smears fine details a lot and leaves edges with angles less than 20° jagged. Yadif+mcdeint retains detail and texture better, but it's still not as good as the best filters in AviSynth or the graphics cards (and they aren't perfect either).

[drob]

I believe the framerate is set, StaxRip uses mp4box, and it does have a framerate switch, to make sure i downloaded yamb and filed it myself, to no evil. btw premier elements reads the file just fine, and it also plays in windows media player ok, just pinnacle studio has problems with it.

As for yadif, StaxRip uses avisynth and yadif, it is a port form mplayer. so since avisynth is there, what other deinterlacers would you recommend?

Available filters built into StaxRip:
yadif
FieldDeinterlace
TomsMoComp
SelectEven
SeparateFields

In order not to get too off topic with the filtering issue, i had split the thread into the Avisynth forum.
http://forum.doom9.org/showthread.php?p=1375986#post1375986

[benwaggoner]

Using deblocking in the MPEG-2 decoder may also help a bit if there is any blocking in the source. 8.9 Mbps for SD is a good bitrate for MPEG-2, but a lot of those cameras have pretty cheap encoder ASICs.

If your source is 25i, you want to archive at either 25i or 50p in order to keep all the temporal data. Deinterlacing without bob throws out half the temporal samples, and will make high motion a lot more jerky. And it'll have to go back to either 25i or 25p for the DVD anyway.

[drob]

OK considering interlace in - interlace out, here is my final command line, any input it welcomed:

--preset placebo --tune film --crf 16 --qpmin 8 --sar 16:11 --qpmax 20 --interlaced

[Blue_MiSfit]

Skip --preset placebo, and use something reasonable like veryslow, slower, or slow (in that order)

Placebo is just that - a placebo :) Especially when using such a low CRF number, there is no reason to push x264 to 100%.

~MiSfit

[drob]

Hey i just do as daddy told me, he wants top quality regardless of speed lol
btw i managed to solve the Studio MP4 interlaced issue, converting to m2ts solves it.

[nm]

Hey i just do as daddy told me, he wants top quality regardless of speed lol
You get about the same quality with --preset placebo --crf 16 as with --preset slow --crf 16. "Placebo" is over 5 times slower and saves some space (less than 10 % compared to "slow"). Unless you can use the heat produced by your computer to reduce heating costs of your house, the additional storage space would be cheaper than the encoding time (electricity cost). So, at winter it may make some sense, at summer it doesn't.

[drob]

Oh you made my day :), it is winter here at the moment so it might come in handy :thanks:

So its not worth going beyond slow? not even slower???

[Boulder]

Sorry to pop in, but is there any information when x264 might adopt TFF/BFF flagging? I've got loads of interlaced material to encode, including quite a lot of home movies in DV.

[nm]

So its not worth going beyond slow? not even slower???

I'd use medium or slow for this kind of task.

Sorry to pop in, but is there any information when x264 might adopt TFF/BFF flagging?

When the NAL-HRD patch (http://forum.doom9.org/showthread.php?t=152127) is accepted to the main tree. That will probably happen real soon now, but you can also apply the patch yourself or get one of the patched builds.

[drob]

Thank you all, with the combination of these settings and the FFT3DFilter my dad is really happy with the results, compression is about 6% of original and quality is better then original, you cant get better then that. i learned a lot from you good people :thanks: