Transcoding ATSC TS to Blu-Ray Compatible With ffmpeg

[Quantumstate]

I've researched this problem for weeks, and am no closer to a solution as I am not a coder. (sorry) Trying to transcode a
US mpeg2 transport stream, using x264 to ==> h.264. Need to transcode SD & HD non-interlaced, and SD & HD
interlaced, and want to just copy over the audio. Destination is hopefully mkv.

I've succeeded in transcoding the video using the x264 command (by rote), but it is a very slow process taking hours
for a 30 minute clip. And I can not figure out how to extract and copy over the audio, and it is so complex
I gave up. So I am now attempting with ffmpeg.

redacted

redacted

Tested this, but get tons of:
# ffmpeg -f mpeg2video -i infile.mpg -pass 1 -an ~/dl/test.avi
FFmpeg version SVN-r18242, Copyright (c) 2000-2009 Fabrice Bellard, et al.
configuration: --prefix=/usr --extra-cflags=-Wall -g -fPIC -DPIC --cc=ccache cc --libdir=${prefix}/lib \
--shlibdir=${prefix}/lib --bindir=${prefix}/bin --incdir=${prefix}/include/ffmpeg --enable-shared \
--enable-libmp3lame --enable-gpl --enable-libfaad --mandir=${prefix}/share/man --enable-libvorbis \
--enable-pthreads --enable-libfaac --enable-libxvid --enable-postproc --enable-libamr-nb --enable-libamr-wb \
--enable-x11grab --enable-libgsm --enable-libx264 --enable-libtheora --enable-libdc1394 --enable-nonfree \
--disable-stripping --enable-avfilter --enable-libdirac --disable-decoder=libdirac --enable-libschroedinger \
--disable-encoder=libschroedinger --enable-libspeex --enable-avfilter-lavf --disable-mmx --disable-altivec \
--disable-armv5te --disable-armv6 --disable-vis
libavutil 50. 2. 0 / 50. 2. 0
libavcodec 52.22. 3 / 52.22. 3
libavformat 52.32. 0 / 52.32. 0
libavdevice 52. 1. 0 / 52. 1. 0
libavfilter 0. 5. 0 / 0. 5. 0
libswscale 0. 7. 1 / 0. 7. 1
libpostproc 51. 2. 0 / 51. 2. 0
built on Mar 30 2009 08:55:31, gcc: 4.3.3

Seems stream 0 codec frame rate differs from container frame rate: 59.94 (60000/1001) -> 29.97 (30000/1001)
Input #0, mpegts, from 'infile.mpg':
Duration: 00:29:56.57, start: 56560.600411, bitrate: 2984 kb/s
Program 1
Stream #0.0[0x51]: Video: mpeg2video, yuv420p, 704x480 [PAR 10:11 DAR 4:3], 15000 kb/s, 29.97 tbr, 90k tbn, 59.94 tbc
Stream #0.1[0x54]: Audio: ac3, 48000 Hz, stereo, s16, 192 kb/s
File '~/dl/test.mkv' already exists. Overwrite ? [y/N] y
[libx264 @ 0x251e140]using SAR=10/11
[libx264 @ 0x251e140]using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cache64
[libx264 @ 0x251e140]profile High, level 4.1
Output #0, mkv, to '~/dl/test.mkv':
Stream #0.0: Video: libx264, yuv420p, 704x480 [PAR 10:11 DAR 4:3], q=2-31, pass 1, 1 kb/s, 29.97 tbn, 29.97 tbc
Stream mapping:
Stream #0.0 -> #0.0
Press [q] to stop encoding
[libx264 @ 0x251e140]VBV underflow (-26543 bits)
[libx264 @ 0x251e140]VBV underflow (-15191 bits)
[libx264 @ 0x251e140]VBV underflow (-14047 bits)
[libx264 @ 0x251e140]VBV underflow (-20319 bits)
[libx264 @ 0x251e140]VBV underflow (-15991 bits)
[libx264 @ 0x251e140]VBV underflow (-17183 bits)
[libx264 @ 0x251e140]VBV underflow (-13631 bits)
[libx264 @ 0x251e140]VBV underflow (-17631 bits)
[libx264 @ 0x251e140]VBV underflow (-14631 bits)
[libx264 @ 0x251e140]VBV underflow (-18631 bits)
[libx264 @ 0x251e140]VBV underflow (-15079 bits)0.87 bitrate= 296.3kbits/s
[libx264 @ 0x251e140]VBV underflow (-18575 bits)
[libx264 @ 0x251e140]VBV underflow (-14215 bits)
[libx264 @ 0x251e140]VBV underflow (-13519 bits)
[libx264 @ 0x251e140]VBV underflow (-18895 bits)
[libx264 @ 0x251e140]VBV underflow (-13855 bits)
[libx264 @ 0x251e140]VBV underflow (-19591 bits)
[libx264 @ 0x251e140]VBV underflow (-14671 bits)
[libx264 @ 0x251e140]VBV underflow (-20351 bits)
[libx264 @ 0x251e140]VBV underflow (-14327 bits)
[libx264 @ 0x251e140]VBV underflow (-18743 bits)
[libx264 @ 0x251e140]VBV underflow (-14503 bits)1.23 bitrate= 356.8kbits/s
[libx264 @ 0x251e140]VBV underflow (-14663 bits)
[libx264 @ 0x251e140]VBV underflow (-19415 bits)
... etc, etc

Am I on the right track?

- What is a VBV underflow? I can't find anything out there explaining it or the correction.
- When transcoding it's only using one core. I'm setting to 3 threads... is there an auto setting?
- It won't let me set f=matroska in the preset file. Is this because it is specifically an output option?

I'm new at this.

[Quantumstate]

OK, now I get it.

I have finally managed an approach which synthesizes best practices from many sources; I did a transcode of two mpeg2 files with the output at the same bitrate as the input; and the resulting h.264 files were just about the same size as the input mpeg2 files!

No wonder I never got any help with the questions I've posted here at various times over the past month or so. The dirty little shameful secret is that x264 is no better than ATSC mpeg2! And no one wants to admit it. This explains why everyone is hiding and there is no authoritative reference for best practices.

Maybe there'll be those who desperately criticize me as 'not knowing how to do it right', but they are disingenuous as they didn't lift a finger to help and are probably lying anyway.

Fine. No more time wasted on this pursuit, thankfully.

[SquallMX]

OK, now I get it.

I have finally managed an approach which synthesizes best practices from many sources; I did a transcode of two mpeg2 files with the output at the same bitrate as the input; and the resulting h.264 files were just about the same size as the input mpeg2 files!

No wonder I never got any help with the questions I've posted here at various times over the past month or so. The dirty little shameful secret is that x264 is no better than ATSC mpeg2! And no one wants to admit it. This explains why everyone is hiding and there is no authoritative reference for best practices.

Maybe there'll be those who desperately criticize me as 'not knowing how to do it right', but they are disingenuous as they didn't lift a finger to help and are probably lying anyway.

Fine. No more time wasted on this pursuit, thankfully.

:stupid:, if you use the same bitrate obviously the file size for the x264 stream is going to be the same!!!, try using about 50% of the original bitrate, the quality should be almost as good as the original.

[Blue_MiSfit]

The dirty little shameful secret is that x264 is no better than ATSC mpeg2! And no one wants to admit it. This explains why everyone is hiding and there is no authoritative reference for best practices.


LOL!!! You're either trolling or understand nothing about video compression.

Don't expect any help with comments like that!!

Just for H.264's sake, I'll help.

1) Filesize = bitrate*length of movie. Therefore, if you tell x264 to hit the same bitrate as the input, it will do just that! Thankfully, H.264 is HUGELY more efficient than MPEG-2, and x264 is a particularly good implementation of H.264, so you can quite safely reduce the bitrate quite a bit. What's your input file's bitrate? I'm guessing 19.4mbps if it's an ATSC HD stream? Try setting x264 to 10mbps for starters. You can probably go much lower without much loss of quality

2) You can either use 2 pass VBR mode or CRF mode. The former is useful when trying to hit a specific filesize / bitrate. The latter is usefeul when trying to achieve constant quality without regard to how big the output is. You simply pick a number that produces good results according to your eyes, and let the encoder do its thing. For my eyes, CRF looks great between 18 and 22, and I standardize on 21 for most of my BluRay backups. This usually ends up being visually transparent at between 1/3 and 2/3 the size of the source. CRF is fast, because it only requires one pass. 2 pass VBR (as the name implies) requires 2 passes, and is therefore half the speed of CRF mode.

The decision is yours.

3) I'm not familiar with using x264 via ffmpeg, but you shouldn't be using VBV unless you have a hardware device that requires it, or are streaming.

4) Don't give up for lack of understanding. x264 is the most efficient end-product lossy video codec out there. I can quite confidently assure you of that, and most everyone on this board will back me up on that. You think we all use x264 because it sucks? Give me a break... :p

Don't believe me? Take a look here:
http://mirror05.x264.nl/Dark/website/compare.html

This was actually done with a pretty old version of x264, and doesn't include the latest magic bits :)

~MiSfit

[Quantumstate]

This is the first helpful input I've gotten. Thanks you two.

So I'd inferred (since this info is not written down anywhere I could find) that the compression algos for mpeg2 and h.264 are radically different, and so at the same bitrate h.264 should do much better. So why does making ffmpeg decompress the transport stream to raw video, and recompressing it with x264 at the same bitrate, not make a smaller file? Why must you cut the bitrate in order to reduce the filesize? I could do the same with mpeg2.

I don't care about transcode speed or making a particular filesize; only that it be alot smaller while keeping the quality. But I am not a videophile, I am a real estate developer, so you're saying that reducing bitrate to 1000kb retains acceptable quality? Is this for SD or HD? I've been working with an SD 720p ATSC file which is reported to be
Seems stream 0 codec frame rate differs from container frame rate: 59.94 (60000/1001) -> 29.97 (30000/1001)
Input #0, mpegts, from '1093_20090301073000.mpg':
Duration: 00:29:56.57, start: 56560.600411, bitrate: 2984 kb/s
Program 1
Stream #0.0[0x51]: Video: mpeg2video, yuv420p, 704x480 [PAR 10:11 DAR 4:3], 15000 kb/s, 29.97 tbr, 90k tbn, 59.94 tbc
Stream #0.1[0x54]: Audio: ac3, 48000 Hz, stereo, s16, 192 kb/s
[libx264 @ 0x1053130]using SAR=10/11
[libx264 @ 0x1053130]using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.1 Cache64
[libx264 @ 0x1053130]profile High, level 4.1
Output #0, matroska, to '/dev/null':
Stream #0.0: Video: libx264, yuv420p, 704x480 [PAR 10:11 DAR 4:3], q=2-31, pass 1, 2984 kb/s, 1k tbn, 29.97 tbc
Stream mapping:
Stream #0.0 -> #0.0
So it says the video stream is 15000 kb/s, but empirical testing indicates it's actually the 2984 kb/s. I don't understand the distinction. And should the framerate be 30 or 60? Would this be true of both progressive and interlaced? (Output will be to a 1280x720 DLP projector) What container is BluRay in? Surely not matroska. Would I need to convert to 24fps for true BluRay compatibility?

I assume an HD ATSC stream would have a much higher bitrate. Is there any standard for setting compression bitrate to retain say, 95% of video quality? Like 33% of source bitrate?

I am doing two-pass, and don't understand why the stats file is not getting written. I get ffmpeg2pass-0.log in the same directory I run the ffmpeg command in, but it stays 0 bytes. The first pass is useless if it's not making stats. Here's what I'm doing:
ffmpeg -f mpeg2video -i infile.mpg -vpre ~/.mythtv/SD720p.ffpreset -pass 1 -f matroska -y -an /dev/null
ffmpeg -f mpeg2video -i infile.mpg -vpre ~/.mythtv/SD720p.ffpreset -pass 2 -f matroska -y -acodec copy outfile.mkv

b=2984000
b_qfactor=1.1
b_strategy=2
bf=3
deblockalpha=-1
deblockbeta=-1
directpred=auto
flags=+loop
flags2=+dct8x8
flags2=+mixed_refs
flags2=+wpred
g=250
i_qfactor=1.1
keyint_min=25
level=41
me_method=umh
subq=7
partitions=+partp8x8
partitions=+partb8x8
partitions=+parti4x4
partitions=+parti8x8
qcomp=0.5
refs=6
threads=3
trellis=2
vcodec=libx264
And it only ever uses one core, despite the threads setting. I don't understand this.

[Amato]

So why does making ffmpeg decompress the transport stream to raw video, and recompressing it with x264 at the same bitrate, not make a smaller file?

Because the bitrate (and the length of the video) determines the file size. You're setting the file size to be the same and wondering why the file size is the same.

Is there any standard for setting compression bitrate to retain say, 95% of video quality?

How do you define "95% of the video quality"?

[Guest]

So why does making ffmpeg decompress the transport stream to raw video, and recompressing it with x264 at the same bitrate, not make a smaller file? Because it is the same bitrate! It's been explained to you: file size = bitrate * duration of movie. So if the duration is the same and the bitrate is the same, then the file size is the same.

Why must you cut the bitrate in order to reduce the filesize? See above.

I could do the same with mpeg2. Yes, but to target a given level of quality, you can get away with a smaller bitrate (and hence file size) if you use H.264 instead of MPEG2.

I've been working with an SD 720p ATSC file That's a contradiction because 720p is HD.

[Quantumstate]

Because the bitrate (and the length of the video) determines the file size. You're setting the file size to be the same and wondering why the file size is the same.
Are we not compressing with mpeg2 and with h.264? And isn't h.264 supposed to be so much better? So why is it not, at a given bitrate?

How do you define "95% of the video quality"?
I don't know how to say it any clearer.

Because it is the same bitrate! It's been explained to you: file size = bitrate * duration of movie. So if the duration is the same and the bitrate is the same, then the file size is the same.
I don't think you've thought about this.

I am beginning to infer that the 15000 kb/s bitrate above is actually of the raw uncompressed video stream, and that the 2984 kb/s is after. So compressing 15000 with mpeg2 at some arbitrary bitrate like 2984 kb/s gives no worse filesize than with h.264? H.264 is supposed to be far more efficient. What, is h.264's benefit only at bitrates below some standard? What standard? There will be compression vs degradation curves, and I don't think this research has ever been done.

That's a contradiction because 720p is HD.
This program sure isn't HD, I can tell you that. (This Old House)

[Guest]

Are we not compressing with mpeg2 and with h.264? And isn't h.264 supposed to be so much better? So why is it not, at a given bitrate? AVC is better for a given bitrate as long as the bitrate is below *saturation*, in that the quality will be better. At some point as the bitrate is raised, an encoder cannot find anything useful to do with more bits and so the quality maxes out. If you choose a really high bitrate, then you won't see much quality difference at that bitrate because both have saturated. If you choose a bitrate well below saturation, AVC will give much better quality. Since high compression is usually the goal, we operate in the domain well below saturation. AVC will also reach saturation before MPEG2, which is another way of seeing that it is better.

I don't know how to say it any clearer. But that's due to your ignorance, not ours. There are many ways to define "quality".

I don't think you've thought about this. LOL.

What, is h.264's benefit only at bitrates below some standard? What standard? Yes, indeed. It will show up at bitrates below the saturation point.

There will be compression vs degradation curves, and I don't think this research has ever been done. Of course such metrics have been gathered.

This program sure isn't HD, I can tell you that. (This Old House) It may be upsampled from SD content, but 720p is by definition HD. If you start using your own terminology, you're going to have trouble communicating with experts.

[Amato]

Are we not compressing with mpeg2 and with h.264? And isn't h.264 supposed to be so much better? So why is it not, at a given bitrate?


Are you talking about better quality? If you encode H.264 from MPEG2 video at the same bitrate, since the MPEG2 video probably has a very high bitrate, the H.264 video will basically look the same as the MPEG-2 video. Are you expecting it to look better than the source?

Or are you talking about a better filesize? If you want a lower filesize, pick one. Choosing the same filesize as the source for your encoded file and then asking "shouldn't the filesize of my encoded file be lower" is pointless :)

[Quantumstate]

But that's due to your ignorance, not ours. There are many ways to define "quality".
Pffffff
Thanks for the useful response...

Of course such metrics have been gathered.
But you do not feel that they should be shared...

It may be upsampled from SD content, but 720p is by definition HD. If you start using your own terminology, you're going to have trouble communicating with experts.
Another 'useful' reply...

BTW, the definition of an expert is one who is able to communicate knowledge in a structured way... not one who can do stuff but has it all hoity-toity fuzzy nebulous and wants to be a High Priest. I am asking you clear structured questions, and you have not been able to respond, so indeed we are not communicating.

I am pretty good at Linux and for 11 years have helped n00bs significantly. I don't just answer questions, but provide config files and step-by-steps (http://forums.debian.net/viewtopic.php?t=36173&highlight=conky), although I have never enjoyed that luxury myself. WRT this h.264 thing I have had to slog and fight for every scrap of information, and it pisses me off when airs are put on. No time for poseurs. If you know it, share it in the spirit of the open-source software we all enjoy for free. If not, don't bother to posture. We all see right through it.

[Guest]

Here's one:

http://cat.inist.fr/?aModele=afficheN&cpsidt=18524758

You can easily find many more with Google.

You better lose the attitude, or you can get yourself in trouble.

[Blue_MiSfit]

@Quantumstate:

You're quite confused. Forget everything you know, and start from scratch.

Please consider that neuron2 is quite an expert, and is well versed in the topics you're confused about, and your attitude is understandably frustrating to him, and probably others here :) We're willing to help, just relax!

Let's figure out a few things first. I'll state what we know.

1) According to FFMPEG, Your input is an MPEG Transport Stream containing 720x480 (480p) MPEG-2 video, not 720p.
2) Your goal is to compress this to a smaller file / lower bitrate than the source. You're apparently okay with using the MKV container. Good choice!

I don't use FFMPEG, so I'm not sure what the bitrate numbers mean. One says 15,000kbps (15mbps), and the other says 2984kbps. Please download DGIndex, and run the file through it so we can figure out the true bitrate. Simply drop the MPG into DGIndex and push F5. Let this process finish, and tell us what DGIndex reports the average bitrate to be.

Then we can proceed further with determining a workflow to output your desired result - a smaller file (lower bitrate), with transparent quality to your source.

Also, it might be helpful for us to see a stream sample. Please download DGSplit, load your file into it, and extract 25-50mb chunks. Wait until there are a few chunks done, then upload one of these chunks to a free file hosting service. I like Mediafire, but megaupload is fine too (lots of others, just no Rapidshare please)

~MiSfit

[Quantumstate]

Oh no, I'll tell you this is one big part of the misunderstanding: I thought x264 was Linux only. I thought I was amongst my own kind.

Thanks Blue_MiSfit, but I haven't used Microsoft products for over a decade. Is there any other way to get the info you need?

And yes I now see that it is 704x480. I am in the middle of floating a new investment fund, and have alot going on. If I ever get close to making this work I'd like to look into resizing and cropping a bit.

[Blue_MiSfit]

While we wait, I'll attempt to clear up your terminology a bit:


I am beginning to infer that the 15000 kb/s bitrate above is actually of the raw uncompressed video stream, and that the 2984 kb/s is after. So compressing 15000 with mpeg2 at some arbitrary bitrate like 2984 kb/s gives no worse filesize than with h.264?

Uncompressed 4:2:0 digital video is quite large. A lot more than 15mbps. I'm not sure what the exact number is, but it's close to 150mbps. Feel free to correct me, anyone :) I think the 15,000kbps number you're seeing is simply a flag in the transport stream, put in there likely for compatibility's sake by the broadcaster.

Now, here's how re-encoding works. The MPEG-2 video (which I suspect is about 3mbps) is decoded by the MPEG-2 decoder into uncompressed YV12 (the particular flavor of 4:2:0 that we're concerned with). This uncompressed video is then fed into x264, which in turn re-compresses it to whatever bitrate you choose.

If you choose the same bitrate as the MPEG-2 input (again, I'm guessing it's about 3mbps), you will get an output that's exactly (or very near) the same size. Now, since we're using H.264 (and x264 in particular), you can surely get away with a much lower bitrate while maintaining visual transparency. In other words, your 3mbps source can likely be compressed to 2mbps or lower without much perceptible loss in quality.

I'm guessing that at 3mbps, your source probably doesn't look too good, as that's a very low bitrate for 720x480 MPEG-2. In other words, there may be lots of compression artifacts in the source - i.e.things like macroblocking, low detail, and a percieved lack of sharpness. I won't go any further on that topic until we see a sample.


[How do you define "95% of the video quality"?]
I don't know how to say it any clearer.


Well, unfortunately 95% of the video quality means nothing in terms of bitrate :) Thankfully as I explained earlier, x264 has a rate-control method called CRF (constant rate factor). This is similar to what you want. You simply give x264 a number (good values are usually between 18 and 22). The higher the number, the lower the final bitrate, and the lower the perceived quality. To me, "95%" quality defines what I go for when I do a BluRay rip. In my case, that usually means CRF21. Typically, this means that a ~20-30GB BluRay video stream can be reduced to 10-15GB without any difference in quality. I have pretty good eyes, and do this stuff for a living, so I consider that (a 50% reduction in bitrate) to be quite impressive!

But, you can see the problem with saying "95% quality"? It's a relatively meaningless number, from an encoding perspective :)


What, is h.264's benefit only at bitrates below some standard? What standard?

As neuron2 said, all codecs (and ultimately all video standards) will eventually hit a saturation point. Ultimately, it all depends on your source. Let's consider an example.

Say I have a beautiful 1080p Apple ProRes422 file at ~150mbps. It's visually lossless to the HDCAM-SR master, but is a real heavyweight at 150GB (including 7.1ch 24 bit uncompressed audio). If I want to compress this to MPEG-2 or H.264, I have to choose a bitrate.

Now, at 80mbps, both MPEG-2 and H.264 should damned well be transparent :)!!! As we reduce bitrate down towards 40mbps, they should still both be excellent, but we may start to see flaws in the MPEG-2 that aren't present in H.264. As we ramp down even further to 25 or even 15mbps, the MPEG-2 will start to show flaws. Film grain will loose its natural quality, and begin to look over-compressed. Details may begin to fade, etc. But at these bitrates, a good H.264 encoder (like x264) can maintain near perfect transparency of most sources!

Unfortunately, there's no one-size-fits-all number. In other words, I can't honestly say that "x264 is 50% better than MPEG-2 all the time" because that may not be the case with every title! Plus it always depends which implementation you're using (i.e. Apple H.264 or x264, or for MPEG-2 - Digital Rapids MPEG-2 versus CinemaCraft MPEG-2 (CCE)), and again it's always source dependent. Lest we not forget, everyone sees video differently!

~MiSfit

[Blue_MiSfit]

x264 is platform independent.

I'd suggest installing WINE. Lots of extremely useful tools for open-source video encoding are windows-only at the moment, like neuron2's stuff, and AviSynth.

If we have to continue with what you have.. fine :) How big is your file, and what's the duration of the video?

~MiSfit

[Dark Shikari]

Oh hey guys (http://i40.tinypic.com/24yxz0g.png) ;)

[Quantumstate]

I'm guessing that at 3mbps, your source probably doesn't look too good, as that's a very low bitrate for 720x480 MPEG-2. In other words, there may be lots of compression artifacts in the source - i.e.things like macroblocking, low detail, and a percieved lack of sharpness.
Exactly. This particular This Old House looks horrible, but I chose it as it's a smaller file. (638.6MB) Most of my work will be on multi-gig files, interlaced * prog, SD & HD.

OK, CRF21 it is. But now to translate that to ffmpeg. And would that apply to HD -and- SD? Interlaced and prog?


Say I have a beautiful 1080p Apple ProRes422 file at ~150mbps. It's visually lossless to the HDCAM-SR master, but is a real heavyweight at 150GB (including 7.1ch 24 bit uncompressed audio). If I want to compress this to MPEG-2 or H.264, I have to choose a bitrate.
Point taken. But I haven't even been able to ascertain rules-of-thumb for choosing a bitrate. I'll bet there's an underlying principle.

OK, I've managed to get DGIndex running, and the avg bitrate for this clip is 2.753Mb/s.

Have a Lodge meeting now, so have to go.

[Blue_MiSfit]

CRF21 looks good to me, hopefully it will for you. And yes, CRF21 is a great place to start for HD or SD.

Interlaced, I'd suggest deinterlacing. x264 supports interlaced video, but IIRC it's not hugely efficient.

Again, I'm not an avid FFMPEG user, but here's what your goal is:

"I want to convert a 3mbps 480p MPEG-2 video to CRF21 x264" :)

~MiSfit

[Blue_MiSfit]

Alright... stay tuned. Because I'm feeling inquisitive, I'm going to figure out an ffmpeg command that will do everything you need ;)
~MiSfit

[Guest]

Alright... stay tuned. Because I'm feeling inquisitive, I'm going to figure out an ffmpeg command that will do everything you need ;)
~MiSfit
What assumptions are you going to make? He could have 3:2, interlaced, field blends, HD/SD, etc., and he hasn't posted a sample.

Don't bite off more than you can swallow. :)

[Blue_MiSfit]

:D That's a very good point, neuron2!

I was going to assume 24p, and his ffmpeg logs indicate 720x480 (though at 29.97fps... darn!)

ffmpeg has at least rudimentary inverse telecine and deinterlacing capabilities.. I would hope?

I mean, the "best" solution would be to get him to use your tools + AviSynth, but that's a bit outside my present abilities, given the whole Linux thing ;)

~MiSfit

[Mr VacBob]

ffmpeg doesn't have any IVTC, but it does have a bad non-adaptive deinterlacer. pullup and yadif could be ported from mplayer whenever libavfilter is finally done, I guess.

[nm]

I mean, the "best" solution would be to get him to use your tools + AviSynth, but that's a bit outside my present abilities, given the whole Linux thing ;)
Quantumstate is after an automatic solution that allows him to encode a variety of SD and HD captures with a few scripts: http://forum.doom9.org/showthread.php?t=145830

One option, that I suggested earlier, would be to create two scripts: one for 720p60 and other for encoding 30i (1080i30 and SD) as interlaced. That would simplify things a lot and avoid corrupting the video with bad filtering. Deinterlacing/IVTC on playback would give as good results as he now gets with the MPEG-2 sources. If he can use VDPAU, deinterlacing quality should be pretty nice -- better than anything else currently available on Linux except for some heavy AviSynth filters, perhaps.

Since there would be no need for filtering, this solution would be achievable with ffmpeg and x264 alone, no other tools required.

[Quantumstate]

I've just noticed that DGIndex says my sample clip is interlaced. None of my Linux tools give me this information, and I must rely on the frame size to make that determination. For example I have a Doc Martin clip (HD) which is 1280x720, so I can only conclude that that's progressive. Although I haven't specified any deinterlacing for my sample clip, it's done it anyway somehow; this surprises me.

My goal is to transcode certain of my ATSC shows (int & prog, SD & HD) to BluRay compatible format with little or no loss of quality. Although all my storage is hard-drive based, I believe that the BluRay format has the best potential to be a long-lived archival format. Container doesn't matter as transcoding would be trivial, and it seems matroska has good potential for longevity.

nm thanks for your input in that thread, but it's over my head. Indeed multiple scripts are fine, although at this point I'd be grateful for just a working command for each of the four eventualities. Indeed I have VDPAU and it works well with my 9300 IGP, but I am skeptical of interlaced. It has a bad rep for video quality and I'd like to avoid it if possible. Here's a great 'Rosetta Stone' (http://sites.google.com/site/linuxencoding/x264-ffmpeg-mapping) for mapping x264 options to ffmpeg, and it says:
--interlaced (x264)
none(FFmpeg)
Enables interlaced encoding. x264's interlaced encoding is not as efficient as its progressive encoding; consider deinterlacing for maximum effectiveness.
I don't mind additional filtering, as long as I can determine what's needed, and it doesn't involve Winduhs. Above are the best ffmpeg commands I've been able to come up with, and to the preset we add crf=21. I've packed all the options I can into the preset, and those remaining must be put on the command-line or else they error out, I guess because they are source- or destination-dependent. I wonder whether CRF is BluRay compatible?

[nm]

I've just noticed that DGIndex says my sample clip is interlaced. None of my Linux tools give me this information, and I must rely on the frame size to make that determination. For example I have a Doc Martin clip (HD) which is 1280x720, so I can only conclude that that's progressive. Although I haven't specified any deinterlacing for my sample clip, it's done it anyway somehow; this surprises me.
Post the sample clip so we can take a look and perhaps suggest alternative ways to handle things.

Indeed I have VDPAU and it works well with my 9300 IGP, but I am skeptical of interlaced. It has a bad rep for video quality and I'd like to avoid it if possible. Here's (http://sites.google.com/site/linuxencoding/x264-ffmpeg-mapping) a great 'Rosetta Stone' for mapping x264 options to ffmpeg, and it says:
--interlaced (x264)
none(FFmpeg)
Enables interlaced encoding. x264's interlaced encoding is not as efficient as its progressive encoding; consider deinterlacing for maximum effectiveness.
Encoding an interlaced (not telecined) 30i source as interlaced is more efficient than deinterlacing it to 60p and encoding as progressive. Deinterlacing to 30p allows more efficient encoding, but you would lose motion fluidity. Also, 1080p60 is not compatible with Blu-ray specs, only 1080i30/i25 and 1080p24/p23.976 are. 1080p30 is not strictly compatible, but it would probably work with many hardware players and all software players.

Telecined sources make things a bit more complicated. First you need to detect the type of the source: is it interlaced or soft/hard-telecined, or perhaps a hybrid of interlaced and telecined scenes. The only reliable way to do this is by going through the video frame-by-frame and looking for changes in combing with your own eyes. Now, if the source is interlaced and you decide to inverse telecine to 24p, quality is lost and motion will be jerky. If the source is telecined and you deinterlace to 30p or 60p, quality is again lost and motion is jerky (although less noticeably at 60p).

So, choosing between deinterlacing and inverse telecine requires manual guidance and wrong choice leads to badly encoded video. If you still decide to go for it, you also need filters from AviSynth, MEncoder or HandBrake (which includes some filters ported from MEncoder).

Post samples from a few different shows that you want to encode. It makes helping a lot easier.

[Quantumstate]

Thanks nm, but I've just spent two hours looking for a transport stream editing tool, finally finding TSMuxer (http://smlabs.net/tsmuxer_en.html), but unfortunately the binary's libraries are 32bit and I'm 64. And in the source code there's no configure or Makefile, so I suspect it is Winduhs only.

It will clearly take me at least a couple more hours to find some kind of transport stream editing app and get the editing done, and I must get some legal work done for my actual job. (gopchop is PS only, not TS)

I could do this in a minute or two with MythTV if I were willing to chop up a good show. But I'll need a TS editor anyway.

[Blue_MiSfit]

Indeed.

One of the most difficult things (until you become very experienced doing it) is figuring out exactly how to process different types of interlacing. The nice thing about the Windows world - there's lots of tools to help you do that, like DGIndex, AviSynth (and a great editor for it - AvsP), and so on. Most of these tools work well in WINE however.

If you have VDPAU, then you can safely code natively interlaced content (59.94 distinct pictures per second in 29.97fps) and let your GPU bob-deinterlace to 60p on playback. nVidia GPUs are very good at this, and it's much simpler that spending lots of CPU time doing a proper bob-deinterlace prior to encoding. Keep in mind, some of x264's magic options don't work with interlaced content. I forget which..

You can quite easily just encode BluRay compatible H.264 + AC3 and keep it in an MKV for now, then remux to BluRay file structure later if you need to burn it. Pretty good plan.

Do keep in mind BluRay's restrictions on bitrates. You'll need to do 2 pass encoding with VBV for any HD content, but SD is probably safe with CRF, since the bitrates will be low enough that you'll have little chance of violating the VBV of the hardware decoder. Don't quote me on that, as I've only ever burned 1080p AVCHD.


I've just noticed that DGIndex says my sample clip is interlaced. None of my Linux tools give me this information, and I must rely on the frame size to make that determination

You can't :(. 1280x720 is an exception because it's always progressive - BUT in many cases can be 24p content with lots of duplicate frames to make it 60p. In such a case, you need to decimate it down to 24p, which can be a bit tricky!

If you're dealing with such a wide range of sources, want to do a good job, and don't mind spending a bit of time to learn the basics of one of the most useful tools I've ever come across - you should really consider using AviSynth!

Also, there's lots more folks on here that are well versed with AviSynth, as its modular approach gives it quite a bit of flexibility that ffmpeg lacks.

~MiSfit

[Quantumstate]

If you have VDPAU, then you can safely code natively interlaced content (59.94 distinct pictures per second in 29.97fps) and let your GPU bob-deinterlace to 60p on playback. nVidia GPUs are very good at this, and it's much simpler that spending lots of CPU time doing a proper bob-deinterlace prior to encoding. Keep in mind, some of x264's magic options don't work with interlaced content. I forget which..
Don't want to really rely on always having VDPAU. This is for archival purposes and I'd rather not tie to a given architecture or format other than BluRay.


Do keep in mind BluRay's restrictions on bitrates. You'll need to do 2 pass encoding with VBV for any HD content, but SD is probably safe with CRF, since the bitrates will be low enough that you'll have little chance of violating the VBV of the hardware decoder. Don't quote me on that, as I've only ever burned 1080p AVCHD.
Oh dear. I don't know what CRF or VBV are, but it sounds like for best all-round flexibility I should stay with VBV. (However that's done)


If you're dealing with such a wide range of sources, want to do a good job, and don't mind spending a bit of time to learn the basics of one of the most useful tools I've ever come across - you should really consider using AviSynth!
Agh.

[Guest]

I've just noticed that DGIndex says my sample clip is interlaced. Believe it or not, that doesn't necessarily mean that the content is interlaced! Progressive content can be encoded as interlaced. You have to inspect the video by eye to determine its actual nature.

[Blue_MiSfit]

Don't want to really rely on always having VDPAU. This is for archival purposes and I'd rather not tie to a given architecture or format other than BluRay.


Understandable. Thankfully interlaced H.264 is perfectly compatible with BluRay, or software decoding on both Linux and Windows. It just so happens that VDPAU and DXVA (the windows equivalent) usually do a very good job of it, in real-time, using the GPU.


Oh dear. I don't know what CRF or VBV are, but it sounds like for best all-round flexibility I should stay with VBV. (However that's done)

CRF is the 1 pass mode we were chatting about. VBV usually implies traditional 2 pass encoding. In simple terms, VBV modifies the rate control to impose restrictions on the maximum instantaneous bitrate. This is important when you're concerned with hardware decoding, since such decoders only have x amount of buffer, and you can't exceed this. Thankfully, x264's 2 pass VBV is pretty darned good. CBR is still a little wonky in my opinion, but that's another issue alltogether :)

You can continue with ffmpeg if you wish, but for handling a wide range of pure interlaced / telecined / progressive content, there really is no substitute :) As neuron2 says, the encoding mode does not dictate the content. In fact, many times folks leave their MPEG-2 encoders in interlaced mode, even when coding pure progressive content.

There are plenty of tools out there that automate this entire process. Unfortunately, many (that I'm familiar with anyway) work on "Winduhs" as you call it :p

~MiSfit

[Quantumstate]

OK. I've understood the limitations of ffmpeg, although it does do alot of things automatically, and I can make it half-ass work.

I attempted using raw x264 commando, er commandline, but I can not figure out how to get audio over with timesync into matroska.

It freaks me out that there are different interlace methods, and that we can't definitively determine whether a file is actually interlaced without visually looking at it. I can't believe there's not a bit set in the mpeg header for interl/prog; that situation is ****-up. What were they thinking?!

Anyway I've tried raw x264 but am not smart enough. I've tried mencoder but that is like another planet. And I gather here that ffmpeg is inadequate. One thing's for sure, I am an open-source type and it's that or nothing. No beans for the gargling rapacious monopolies. And AVSynth over wine is possible, but disagreeable. This process seems to be a bottomless pit and my time is very limited as I am right in the middle of floating a real estate vulture fund. Maybe it's hopeless and I should just keep the mpeg2 files. I'd only gain 50% with h264, risk out-of-sync audio, and spending days/weeks beating out all the freakin' microscopic intricacies for little overall gain.

[Guest]

It freaks me out that there are different interlace methods, and that we can't definitively determine whether a file is actually interlaced without visually looking at it. I can't believe there's not a bit set in the mpeg header for interl/prog; that situation is ****-up. What were they thinking?! If you did some research instead of shooting from the hip in a totally uninformed way, you would know that such a bit does exist and that it is so often misused that it is not reliable. Somebody, horrors, has to inspect the video to know if the content is interlaced or progressive and set the bit. Given that lots of encoding is done automatically without intervention by humans, is it so surprising that encoders just default to interlaced? Encoding progressive content as interlaced is much less serious than the converse, so if no intervention is possible it's safest just to do everything as interlaced.

And if you had a real understanding of what interlacing is, you would understand why there are so many different ways to undo it, each satisfying the various tradeoffs involved in a unique way.

[multimediaman]

just use mplayer, mencoder and mkvmerge combo to avoid fmmpeg switch nightmare :

mencoder -ovc x264 -x264encopts crf=20:frameref=3:bframes=5:subq=7:psy-rd=1.0:8x8dct:b_pyramid:weight_b:threads=auto:me=umh -nosound -of rawvideo -o output.264 inputfile
mplayer inputfile -dumpaudio -dumpfile output.something
mkvmerge output.264 output.something 0:"insert fps here"fps -o output.mkv

EDIT

There is something else to add/change like vbv stuff etc -to x264encopts parametres. I am not familiar with BluRay. And matroska is out of question.