Encoding BD M2TS to x264vfw/AVI

[JimmyBarnes]

Putting aside whether what I am doing is good or not (I know many in this forum despise x264vfw and AVI)..

When I encode The Day The Earth Stood Still (2008) as indicated above with CRF=20.0 and Zero Latency enabled, it gives a much larger file 3.29GB compared to when ZL is disabled (1.81GB)

Would this also happen with x264 CLI?

One effect of having ZL enabled is that a given frame number in both M2TS and the transcode seem to have the same content, whereas if ZL is not enabled, there is a significant difference between the frame numbers in the original and the rip.

Is that due to using AVI as a container or would it also be the case in MKV?

[Blue_MiSfit]

1) Zero latency is a terrible idea in this situation :) It sacrifices a lot of compression capability to provide as little latency as possible - which only matters for real-time, latency sensitive applications like videoconferencing. So, that explains the much larger file sizes you're seeing with it enabled :)

2) Yes it would happen with CLI. It's the same x264 ;)

3) When you say "a given frame number in both M2TS and the transcode seem to have the same content" - how are you determining this? It depends very much on how the video is decoded. Are you doing interleaves in avisynth? If so, you should use something frame accurate. Unfortunately my usual go-to (ffms2) isn't perfect when it comes to Transport Streams. The manual states: "M2TS, TS: Seeking seems to be off a few frames here and there".

Conveniently, my usual workflow for processing BluRay content is to first use eac3to on the M2TS files(s) (actually on the root of the disc and let it handle the branching etc). Using eac3to I can remux the video stream into an MKV, which is totally frame accurate with FFMS2. At this point you're at least standing on solid ground, and can begin evaluating the source versus the transcodes. As a bonus, eac3to can extract or transcode your audio streams, and dump, subtitles, all in one read pass over the BluRay disc.

Keep in mind that H.264 in AVI is generally frowned upon, and even though FFMS2 should theoretically be frame accurate with it, you may run into unforseen circumstances. The old problem of packed bitstream in Xvid comes to mind... Therefore, I'd have to suggest you stay away from using x264vfw unless you have a very specific application - i.e. live capture with VirtualDub or another VFW app. In any case it's probably a good idea to remux your output AVI into MKV or something anyway :devil:! The other option is to use one of the experimental vfw builds that have their own muxers built in. In this case, you configure a 264 / MP4/ MKV output directly within the x264vfw configuration. The AVI file that your transcoder / capture app generates is just a placeholder. This is quite handy :)

I hope that helps. somewhat. :D

Derek

[Caroliano]

Try with no b-frames (zero-latency, among other bad-for-compression things, do this). This alone should solve the problem of number of frames and their position. B-frames are not suported in AVI/VFW spec, and the hack to make them work deletes some frames and in the process mess up with their numbering. That is why it is not advised to put H.264 in AVI. Why would you need to do that nowadays?

Description of the B-frames hackery for Mpeg4ASP: http://forum.doom9.org/showthread.php?t=80430 (for H.264 is ~ the same, but worse, and packed bitstream don't work)

[JimmyBarnes]

1) Zero latency is a terrible idea in this situation :) It sacrifices a lot of compression capability to provide as little latency as possible - which only matters for real-time, latency sensitive applications like videoconferencing. So, that explains the much larger file sizes you're seeing with it enabled :)

2) Yes it would happen with CLI. It's the same x264 ;)

Only reason I tried it was that when I encode the whole M2TS, it finishes with end-credits still showing, even tho the M2TS has some 2 seconds of blank frames at the end. Appending extra blank frames at the end of the transcode didn't fix the problem.

Using Zero Latency just for the end-credits made them finish normally (i.e. on blank frames), then the end-credits could be appended to the main movie (without ZL) to give a compact result.

3) When you say "a given frame number in both M2TS and the transcode seem to have the same content" - how are you determining this?

Just by eye-balling frame content in the M2TS and the AVI for the same frame.

Conveniently, my usual workflow for processing BluRay content is to first use eac3to on the M2TS files(s) (actually on the root of the disc and let it handle the branching etc). Using eac3to I can remux the video stream into an MKV, which is totally frame accurate with FFMS2.

I'm using TMPGEnc4XP to transcode the M2TS directly to x264/AVI. Of the 6 BD rips I've done, this is the first to give this particular problem.

I use eac3to to obtain subs and obtain and transcode audio streams.

Keep in mind that H.264 in AVI is generally frowned upon

Well aware of that, but its what I feel comfortable with and it meets my needs. Of course its possible to add an AVI to mkvmergeGUI and produce an MKV.

Pity that some guru could not modify VDub or VDubMod so that it fully supports MKV.


I hope that helps. somewhat. :D

Indeed it does. Many thanks for a courteous and helpful reply

JB

[JimmyBarnes]

Try with no b-frames (zero-latency, among other bad-for-compression things, do this). This alone should solve the problem of number of frames and their position.

Wouldn't that result in much less compresssion?

B-frames are not suported in AVI/VFW spec, and the hack to make them work deletes some frames and in the process mess up with their numbering.

Not something that's visible in the hundreds of AVI/Xvid rips I've done over the years, with b-frames enabled, and watched carefully throughout.

That is why it is not advised to put H.264 in AVI. Why would you need to do that nowadays?

Description of the B-frames hackery for Mpeg4ASP: http://forum.doom9.org/showthread.php?t=80430 (for H.264 is ~ the same, but worse, and packed bitstream don't work)

Never used Packed bitstream even with Xvid.

[nm]

Only reason I tried it was that when I encode the whole M2TS, it finishes with end-credits still showing, even tho the M2TS has some 2 seconds of blank frames at the end.
Is A/V sync ok in the output AVI over the entire movie?

Just by eye-balling frame content in the M2TS and the AVI for the same frame.
The corresponding frame number in AVI will be off by something up to the maximum number of B-frames, as Caroliano explained. If you're not using B-frames (like when using zero-latency), there is no padding and the frames will be at the same positions.


I'm using TMPGEnc4XP to transcode the M2TS directly to x264/AVI. Of the 6 BD rips I've done, this is the first to give this particular problem.

I use eac3to to obtain subs and obtain and transcode audio streams.
Since you're handling audio and subtitles separately with a command-line tool, why don't you do the same for video with x264 CLI? Then mux the streams to MKV and be done with it. Or if you prefer a GUI for the whole deal, try RipBot264, StaxRip or MeGUI, for example.

Pity that some guru could not modify VDub or VDubMod so that it fully supports MKV.
That would mean rewriting most of the application completely since it would need to use something else than VFW internally. Avidemux is a similar app built on top of open source multimedia libraries, but I don't know how its Blu-ray support has been progressing.




Try with no b-frames (zero-latency, among other bad-for-compression things, do this). This alone should solve the problem of number of frames and their position.
Wouldn't that result in much less compresssion?

Yes, but not as much less as with zero-latency.

[JimmyBarnes]

Is A/V sync ok in the output AVI over the entire movie?

Have yet to watch the TDTESS x264 throughout, tho no probs when sampling at 10+ spots.

The other 5 BD rips I have watched carefully throughout and observed no AV out-of-sync (to which I am very sensitive). These rips show the same loss of frame correspondence, so in this sense are like TDTESS.

The main reason I concerned myself with this was to try to minimise the size of end-credits (like with Xvid rips), tho with total x264 video sizes 2-4 GB, size of end-credits is usually much less of an issue.

The easiest method is to rip the whole movie at a given CRF, end-credits included, and for most of them I did this, but TDTESS was peculiar in that the last frame still had end-credits showing.

Since you're handling audio and subtitles separately with a command-line tool, why don't you do the same for video with x264 CLI? Then mux the streams to MKV and be done with it. Or if you prefer a GUI for the whole deal, try RipBot264, StaxRip or MeGUI, for example.

Had a rocky start to BD ripping: Started with RipBot but had this strange prob where it disabled my FLV playback, something which I cannot do without. Tried MeGUI but found it too hard to understand.

With x264vfw, they have hidden the Advanced options GUI tabs, but the default settings appear to give good rips, with a few tweaks like "Preset: Slow" "Tuning: Film" "Profile: High"
i.e. it's very easy to use and gives good results with a minimum of fuss.

Note that TMPGEnc4XP does not offer MKV as a container (otherwise I would try it). It has a great interface which makes it very easy to crop, resize, cut etc.

Yes, but not as much less as with zero-latency.

What is the command to disable b-frames?

Thanx for your feedback

[nm]

What is the command to disable b-frames?

--bframes 0 in x264 CLI. I have no idea about x264vfw.

[MasterNobody]

JimmyBarnes
Didn't you see the warning when tried encode without Zero Latency?
x264vfw [warning]: Few frames probably would be lost. Ways to fix this:
x264vfw [warning]: - if you use VirtualDub or its fork than you can enable 'VirtualDub Hack' option
x264vfw [warning]: - you can enable 'File' output mode
x264vfw [warning]: - you can enable 'Zero Latency' option
As I understand you use TMPGEnc4XP so this leave only option 2 and 3. But because 'Zero Latency' have sacrifice in compression efficiency I would recommend you to use option 2. You still can output to AVI with File output mode (it is one of supported muxers for file output) if you need it.

[Blue_MiSfit]

Once you write an AVS script, doing the actual encode is usually quite simple.


x264 input.avs --crf 20 --tune film --output output.mp4


Something like that is usually all you really need. Why bother with a GUI when you can just shift+right click in the folder that contains your AVS, and punch "Open command window here", type in a few bits, and go.

Still, I guess I understand the desire to stay away from the CLI. I'm sorry to hear you had issues with Ripbot etc. MeGUI can be overwhelming to a newbie, I agree! Take a look at Lord_Mulder's x264 GUI. It's incredibly simple!!!

Derek

[JimmyBarnes]

JimmyBarnes
Didn't you see the warning when tried encode without Zero Latency?

The warning only appears when using VDubMod and x264vfw.

No warning at all appears when using TMPGEnc4XP and x264vfw.

[JimmyBarnes]

Anyway watched TDTESS thru carefully last night and no lip-sync or other AV out-of-sync probs.

I should mention that for the 6 BD to X264 rips I've done, I found I had to use Audio Skew Correction +2400 ms when muxing in AC3 (and the same delay for subs) to get sync with video.

Comparing the first non-blank frame in TDTESS in the source M2TS and the x264 rip, it becomes evident why: It occurs 58 frames earlier in the M2TS than the rip, so clearly the rip's frames are delayed by this amount.

58 frames at 23.976 fps = 2419 ms

When I do the transcode to x264 in TMPGEnc4XP I do it without audio, in case you were wondering.

[JimmyBarnes]

Once you write an AVS script, doing the actual encode is usually quite simple.


x264 input.avs --crf 20 --tune film --output output.mp4


Something like that is usually all you really need. Why bother with a GUI when you can just shift+right click in the folder that contains your AVS, and punch "Open command window here", type in a few bits, and go.

I could handle that.

What about the other things like Preset: Slow, Profile: High, Level: Auto?

Are there CLs for various type of movies, like ones optimized for extreme fast motion? Or does CRF handle all that?

How do you determine the best CRF to use for a given movie?

TIA

[Blue_MiSfit]

add --preset slow or --preset fast as necessary. --profile high or --profile baseline etc.

You never know which CRF to use, you have to experiment a bit :)

x264.exe --fullhelp will give you a lot of info :)

Derek

[Sharktooth]

you may also have a look here: http://mewiki.project357.com/wiki/X264_Settings

[JimmyBarnes]

you may also have a look here: http://mewiki.project357.com/wiki/X264_Settings

Thanx, but which of these settings do you alter from default on a regular basis?

[JimmyBarnes]

You never know which CRF to use, you have to experiment a bit :)


I'm told CRF18 gives "transparent" results = indistinguishable from the original whereas CRF23 seems to be a de facto default setting which will give good results for most movies (maybe some sort of minimum standard?).

Would you agree with the above opinions, or qualify in some way?

[LoRd_MuldeR]

Thanx, but which of these settings do you alter from default on a regular basis?

You shouldn't have to explicitly alter any options "on a regular basis" ;)

Instead just pick the preset (something between "ultrafast" and "veryslow") that gives the best "speed -vs- quality" trade-off for your personal needs. And that's it!

And even if you overwrite options manually, the presets should give you a good indication on which options are reasonable and which are not...

(For example everything used in "placebo" is overkill and thus makes no sense to use. Everything used in presets "fast" and below tweaks for speed rather than quality)


I'm told CRF18 gives "transparent" results = indistinguishable from the original whereas CRF23 seems to be a de facto default setting which will give good results for most movies (maybe some sort of minimum standard?).

Would you agree with the above opinions, or qualify in some way?

In general I'd agree.

But since "perceived quality" depends so much on the individual source, on the playback equipment and on the personal preferences, you must test/decide this yourself!

[JimmyBarnes]

You shouldn't have to explicitly alter any options "on a regular basis" ;)

Instead just pick the preset (something between "ultrafast" and "veryslow") that gives the best "speed -vs- quality" trade-off for your personal needs. And that's it!

Reminds me of the results I have gotten with x264vfw where I have made minimal changes from defaults and still get very good results. A lot has to do, I believe, with the quality of the original BDs, I hope they don't get slack on it as time progresses, I have seen some pretty dreadful DVDs.

Do you mind telling me the best site to download your x264 GUI? And will it work OK with WinXP Pro sp3 32-bit?

thanks

[kypec]

:search:C'mon man, it's not that hard to find the correct thread (http://forum.doom9.org/showthread.php?t=144140) with all necessary info therein.
I don't see a reason why it shouldnt' work under your WinXP.

[LoRd_MuldeR]

Reminds me of the results I have gotten with x264vfw where I have made minimal changes from defaults and still get very good results. A lot has to do, I believe, with the quality of the original BDs, I hope they don't get slack on it as time progresses, I have seen some pretty dreadful DVDs.

As a matter of fact, re-encoding will never improve the overall quality. At best the additional "generation" loss can be minimized.

So the final quality is limited by source's quality. You won't "magically" get great encodes from a lousy source.

(There are however cases where the source has a specific "problem" or "artifact" that can be fixed with some Avisynth magic)


Do you mind telling me the best site to download your x264 GUI? And will it work OK with WinXP Pro sp3 32-bit?

http://tinyurl.com/22jueyq

And yes, it works on 32-Bit Windows, but without the x64 features, of course.

[Blue_MiSfit]

Another vote for Lord_Mulder's x264 GUI. It's fantastic :)