Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-4 Encoder GUIs

Reply
 
Thread Tools Search this Thread Display Modes
Old 12th March 2013, 03:23   #7021  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
ryszardzonk,
I've had a similar problem quite a few times when trying to re-encode a small section of video at a later date (say to encode subtitles in a section which needed them but I didn't realise originally) but generally the error message when trying to rejoin the sections at least appears to make sense as it'll look something like this: (lengths: 41 and 42). However I have had instances where the private data is reported as being different while the lengths are reported as being the same and the two files still won't append properly. It's happened with older versions of x264 and I've tried different versions of MKVMergeGUI with the same result. I've not been able to solve it though.
Only one time did I think I'd found the cause.... I'd set the colour characteristics in the command line when encoding one section but not the other, and that seemed to be enough to do it, but obviously that's not what's happening in your case.

Anyway, as a workaround you could probably just encode both scripts as a single encode which should eliminate the problem. You'd create a script which imports both your single scripts while joining them together, so you'd end up with a script which looks like this:

Import ("E:\script 1.avs") ++ Import("E:\script 2.avs")

How you'd handle the audio probably depends on what sort of editing you'd added to the original script, but the above should solve the video part of the problem.

I've got some encodes currently sitting in the job queue where in order to apply different cropping to a section of video, I've split the encoding job into three scripts to later join the output files with MKVMergeGUI. I've done this quite a few times and never had a problem joining them when they've been encoded one after the other. A few weeks ago I encoded a video that way using one computer to encode two sections of it, while a second encoded the other and they still joined without a problem.
Later today, after the encodes I have in the job queue finish, I'll report back regarding whether I could join the output files successfully. If I can, I'll try some quick encodes again using the same scripts, but with your x264 settings to see if for some reason that makes a difference. It'd be nice to get to the bottom of this. Currently I'm just using the x264 default settings and the slow or slower speed preset.

Last edited by hello_hello; 12th March 2013 at 03:34.
hello_hello is offline   Reply With Quote
Old 12th March 2013, 03:33   #7022  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
PS. Here's where I asked about the problem in the MKVToolnix thread. While it was nice to receive an explanation regarding what a codec's private data actually is, it didn't really help me solve it. http://forum.doom9.org/showthread.php?p=1610745#post1610745

Mosu: "CodecPrivate data is an element that stores data that the codec in question needs in order to decode the file properly. Different codecs have very different requirements for their private data (it's often called "codec initialization data" as wel). For example, Vorbis needs its codebook which tells the codec how to expand the encoded/compressed stuff back to the uncompressed stuff that can be played back.

Same with AVC/h.264. The CodecPrivate data contains (amongst other things) the sequence parameter sets and picture parameter sets. They contain important information like pixel resolution, codec features used etc.

So it is generally technially impossible to decode video from an encoding A with the private data from encoding B (and mkvmerge can do nothing about it). However, there are situations in which that will work: if the codec private data only differns in unimportant fields (e.g. the frame rate is also stored in the sequence parameter sets, and if they and only they differ then there should not be any problem). That's why mkvmerge doesn't prevent you from doing it
."

I guess the next step, if it's possible to view the "codec private data", would be to look at it for each encode to determine where it's different, but if there's a way to do so I don't know how.

Last edited by hello_hello; 12th March 2013 at 03:43.
hello_hello is offline   Reply With Quote
Old 12th March 2013, 07:25   #7023  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
ryszardzonk,
I've narrowed the problem down a little.....
Encoding 2 files using 2 pass encoding (I used the same bitrate as your last log file) produces the "codec private data" error when you try to append the output files, and so far that seems to be regardless of any other x264 settings you may be using. If you use CRF encoding and try to append the output files, there's no error. As I pretty much always use CRF encoding these days......

Maybe now I've discovered cause, someone more knowledgeable than myself might be able to work out the "why" part, or maybe offer a way to prevent the problem from happening when using 2 pass encoding.

I'd take a guess and suggest appending a 2 pass encode to a CRF encode will once again produce the error, but I haven't tested it. It's a very hot day where I am and I want to give the PC a rest because it's not helping with the room temperature. I'd be thinking though, when I've tried to re-encode a section of old video and then append the new version to the original, I've used CRF encoding to re-encode a section of it while the original probably used 2 passes.... back before I'd discovered the joy of CRF encoding.
hello_hello is offline   Reply With Quote
Old 12th March 2013, 08:54   #7024  |  Link
Kurtnoise
Swallowed in the Sea
 
Kurtnoise's Avatar
 
Join Date: Oct 2002
Location: Aix-en-Provence, France
Posts: 5,191
didnt test yet on my side but this doesnt look like an error...more a warning (mkvtoolnix-wise).

Your input are mp4 or raw h264 ?

Could you post also your logfile, especially the muxing part ?

Last edited by Kurtnoise; 12th March 2013 at 08:57.
Kurtnoise is offline   Reply With Quote
Old 12th March 2013, 10:12   #7025  |  Link
ryszardzonk
Registered User
 
Join Date: Jan 2013
Posts: 37
@hello_hello

To start with your very first post I did make mistakes in the past as well and back than MKV was always right as for example I forgot to add audio to one of the source files so it obviously did not match. Sometimes the errors where becouse of more subtle differences as different settings in AVS scripts like with ot without color correction, but that was easily eliminated with creation of "Avisynth profiles". Encoder settings presets where also a big help. This time around it is something completly different so I am somewhat glad it is reproducible so hopefully the solution will be found.
I'll try using different x264 versions now to see if they are to blame. I trust here that if I just replace recent version of x264 with an old one the new MeGUI version would not complain to any missing or moved setting or anything of that sort

I have tried the workaround You mention in the past which did the trick, but I have 2 issues with it. One is that it adds another layer of work which may not seem much but I record number of of videos daily and taking every extra step is not what tigers like best. Another is the issue with audio which is quite error prone to out of sync errors for the additional videos.

As far as the determining differences in codec private data stuff it seems out of my league and I would rather not comment on something I did not even egxamine in the past at all

I was not aware of the CRF method so I did a little bit of reading about it which actually sounds quite promising as it is definetly better judge of scene complexity than I am but it seems it will not be miracle quality boost "CRF mode gives almost exactly equivalent quality to 2pass at the same bitrate".
I have gained some experience as of what bitrate to give to any given video and ussually aim to use preset which gives codec enought bitrate (usually falls between 0,11 - 0,13 bits per pixel and about 15%-20% less for anime) depending on its scene changing needs so that it itself does not really a must have but definetly worth trying out. Hopefully crf would take less time than 2pass encode even with its result/time unpredictability.

It seems however that if the issue with 2pass encoding will not be resolved I would have to move to the crf method regardless.
Thanks for finding out that it is 2 pass encode behaviour that have changed!

Last edited by ryszardzonk; 12th March 2013 at 10:37. Reason: link fixing
ryszardzonk is offline   Reply With Quote
Old 12th March 2013, 10:34   #7026  |  Link
ryszardzonk
Registered User
 
Join Date: Jan 2013
Posts: 37
Quote:
Originally Posted by Kurtnoise View Post
didnt test yet on my side but this doesnt look like an error...more a warning (mkvtoolnix-wise).

Your input are mp4 or raw h264 ?

Could you post also your logfile, especially the muxing part ?
I am using MP4 format to encode videos. I never used RAWAVC

Well mkvtoolnix would be the last I would blame for the change as version 5.6 was used with previous megui stable where it all just worked and reverting to 5.6 to join newly created files results in the same error. Still mkvtoolnix somehow knows there is the difference is the video in the private data as program is able to recognize it hence the warning/error message. Maybe it could inform of what is actually different

warning message
Quote:
Warning: The track number 0 from the file 'D:\mp4\rt dok 03 02 - 2.mp4' can probably not be appended correctly to the track number 0 from the file 'D:\mp4\rt dok 03 02 - 1.mp4': The codec's private data does not match (lengths: 42 and 42). Please make sure that the resulting file plays correctly the whole time. The author of this program will probably not give support for playback issues with the resulting file.
muxing message for videos files only just like they appear after encode [no audio muxed to them]
Quote:
mkvmerge v6.1.0 ('Old Devil') built on Mar 2 2013 14:32:37
'D:\mp4\rt dok 03 02 - 1.mp4': Using the demultiplexer for the format 'QuickTime/MP4'.
'D:\mp4\rt dok 03 02 - 2.mp4': Using the demultiplexer for the format 'QuickTime/MP4'.
'D:\mp4\rt dok 03 02 - 1.mp4' track 0: Using the output module for the format 'AVC/h.264'.
'D:\mp4\rt dok 03 02 - 2.mp4' track 0: Using the output module for the format 'AVC/h.264'.
The file 'D:\mp4\rt dok 03 02 - 1.mkv' has been opened for writing.
'D:\mp4\rt dok 03 02 - 1.mp4' track 0: Extracted the aspect ratio information from the MPEG-4 layer 10 (AVC) video data and set the display dimensions to 544/304.
Appending track 0 from file no. 1 ('D:\mp4\rt dok 03 02 - 2.mp4') to track 0 from file no. 0 ('D:\mp4\rt dok 03 02 - 1.mp4').
The cue entries (the index) are being written...
Muxing took 4 seconds.
ryszardzonk is offline   Reply With Quote
Old 12th March 2013, 11:16   #7027  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
The encodes seem fine. The warning comes from MKVToolnix when you try to join them. I'd say it's an encoder issue, and nothing to do with MeGUI as such, but I'm guessing.

I just ran another two short encodes and tried to join them together. Identical scripts both times, except I used Trim() to encode different sections of it. When using 2 pass encoding MKVMergeGUI complains about the codec's private data not being the same. If I repeat the process using CRF encoding, there's no problem appending the output files.

I tried to post both log files but the forum complained about my post being too long. I tried to upload the samples as attachments but apparently the file size limit for this video conversion forum is 200KB. Only at doom9....

Here's a link instead, assuming there's no ridiculous forum limitation on those as well. I changed the name of the output files because last time I uploaded video files without doing so they were deleted for being unauthorised. The log files are included.
hello_hello is offline   Reply With Quote
Old 12th March 2013, 11:24   #7028  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
ryszardzonk,
I didn't see your post until after submitting my last one. It's interesting to see you say the behaviour for 2 pass encoding, in relation to appending files, has changed at some stage. I didn't know as I almost never encode that way.

But yeah.... when I saw your encoder settings, that's where the penny dropped for me regarding where to start looking for the cause of the problem. I was going to add a little CRF vs 2 pass encoding rant to my post, but thought it might be a little out of place (I know it upsets some people when discussions take place in a discussion forum), but I guess maybe you've seen the light already.
No, it doesn't provide a quality boost. You've read about it encoding the same way as 2 pass encoding if the resulting bitrate is the same (all else being equal) but the difference is CRF lets you pick the quality you want while the file size/bitrate varies accordingly, whereas 2 pass lets you choose the bitrate/file size, but the quality is the unknown. And CRF is faster too....
CRF18 is around where the x264 encoder is roughly considered to be "transparent", but anything up to CRF22 will probably still look pretty good. How high you can go before the quality drop starts to annoy you is a personal thing. I pretty much encode everything using CRF18 myself.

Last edited by hello_hello; 12th March 2013 at 11:36.
hello_hello is offline   Reply With Quote
Old 12th March 2013, 13:18   #7029  |  Link
ryszardzonk
Registered User
 
Join Date: Jan 2013
Posts: 37
@hello_hello
I did few CFR encode tests and I confirm that creating the video with using that setting works as there is no problem with joining the created files together. Sweet

I am fairly positive I may speak of behavior change as I create movies for several years and no such problem appeared previously. IMHO if all settings remain constant and the program/s version change and previous setting no longer work than definitely one may speak of behavior change and in that particular case probably even bug/error.

I tend to agree that it seems like the thing to do is just to confirm which x264 version is to blame. Hopefully nothing older that 2200.

Could be that CFR method is just superior (seems like you say constant good quality, faster encodes, but I would definitely would have to make more test encodes prior to really establish my own opinion) that almost no one uses 2pass encoding anymore and problem was spotted only by those few like me that use ancient method of encoding that is about decade old

PS Rant is last thing I fought about when you mentioned CFR, but more like a hint as getting best quality possible with the use of certain codec settings.

PS Back to x264 version tests

Last edited by ryszardzonk; 12th March 2013 at 18:40. Reason: typo: lat --> last
ryszardzonk is offline   Reply With Quote
Old 12th March 2013, 17:21   #7030  |  Link
ryszardzonk
Registered User
 
Join Date: Jan 2013
Posts: 37
I have run tests with using builds available x264.nl for playback issues with 2 files being joined together while the files have been created with Automated 2pass profile.

It turns out that somewhere between rev 2216 and 2230 the change was made that altered codec private data just enough for the files not being able to be joined together

Tested revisions:
2200: OK
2216: OK
2230: broken
2245: broken
2273: broken

Test therefore reveled this is not a Megui nor MKVtoolnix issue
ryszardzonk is offline   Reply With Quote
Old 12th March 2013, 17:54   #7031  |  Link
hello_hello
Registered User
 
Join Date: Mar 2011
Posts: 4,823
Quote:
Originally Posted by ryszardzonk View Post
@hello_hello
PS Rant is lat thing I fought about when you mentioned CFR, but more like a hint as getting best quality possible with the use of certain codec settings.
I usually do rant about it.
But yeah given hard drive space isn't an issue any more and squishing a file down to fit on a disc is less important than it once was, I think the majority of people would just use CRF encoding these days.
If I had to pick a word, it'd be about getting "consistent" quality rather than "best" quality, although because the bitrate varies you're not wasting bits on quality you can't see when encoding one file and having too few bits for the next.

Quote:
Originally Posted by ryszardzonk View Post
It turns out that somewhere between rev 2216 and 2230 the change was made that altered codec private data just enough for the files not being able to be joined together.
Problem solved.... well "cause found"... at least
I wonder if the x264 developers know it's happening? I'm not sure where you'd report such a "bug".
hello_hello is offline   Reply With Quote
Old 13th March 2013, 08:46   #7032  |  Link
Poutnik
Useless idea generator
 
Poutnik's Avatar
 
Join Date: Apr 2004
Location: Europe, Czech Republic, Brno
Posts: 332
I have recently tried to mux SRT subtitles to AVI, making MKV, using adaptive and MKV muxers.
I have noticed it did not allow me to use AVI as the Audio source.
I thought it may take audio stream as default, but it did not.

But if I have launched standalone mmg GUI from MKVmerge tool of MeGUI,
it took audio stream automatically.

Do I have to demux AVI, or provide AVS audio script to do that in MeGUI ?
Or, are MeGUI muxers aimed to elementary stream muxing only ? ( Using last MeGUI stable )

Edit>I have checked Always mux MKV encoding via MKVmerge.
__________________
Vista64 Premium SP2 / C2D E4700 2.6GHz/ 6GB RAM/ Intel GMA 3100 / DTV Leadtek DONGLE GOLD USB2 /
focused to DVB-T MPEG2 PS capture -> ProjectX -> M2V/MP2 -> MeGUI/AVS -> MP4[AVC/AAC]

Last edited by Poutnik; 13th March 2013 at 08:49.
Poutnik is offline   Reply With Quote
Old 13th March 2013, 09:13   #7033  |  Link
goldensun87
Registered User
 
Join Date: Oct 2008
Posts: 42
Hello everyone, I just ripped my BD of Wreck It Ralph, and I wanna encode it. But, I see that the main movie is split into many m2ts files. So, will I be able to encode the entire movie if I rename the files like, "filename_1", "filename_2, ... and load filename_1 into the file indexer? Or do I have to do something else?
goldensun87 is offline   Reply With Quote
Old 13th March 2013, 09:49   #7034  |  Link
AMED
Registered User
 
AMED's Avatar
 
Join Date: Sep 2004
Location: Auckland, New Zealand
Posts: 466
@goldensun87

Use the HD stream extractor and it will merge all the m2ts in to one mkv video file and any audio languages you choose.
__________________
A Man Eating Duck
AMED is offline   Reply With Quote
Old 13th March 2013, 12:19   #7035  |  Link
goldensun87
Registered User
 
Join Date: Oct 2008
Posts: 42
Quote:
Originally Posted by AMED View Post
@goldensun87

Use the HD stream extractor and it will merge all the m2ts in to one mkv video file and any audio languages you choose.
Thank you . I extracted as raw h264, then remuxed into m2ts. Now, I will attempt to encode down to 720p. Hopefully it will do so without a hitch.

Edit: Just out of curiosity, would my idea have worked?

Last edited by goldensun87; 13th March 2013 at 12:22.
goldensun87 is offline   Reply With Quote
Old 13th March 2013, 19:57   #7036  |  Link
AMED
Registered User
 
AMED's Avatar
 
Join Date: Sep 2004
Location: Auckland, New Zealand
Posts: 466
It should work but it would be easier (possibly safer) to just mux the video in to MKV. I think FFMS can have some issues with seeking when using M2TS files (is this still true?).

My procedure is;

1. Rip full disc in MakeMKV.
2. Use the HD stream extractor to get the streams I want, video = mkv, audio = AC3/DTS
3. Index with FFMS
4. Use AVISynth creator to crop, resize and apply MvDegrain1
5. Autoencode just the video portion with CRF 19 and --grain and mux in the audio, chapter and subtitles.
__________________
A Man Eating Duck
AMED is offline   Reply With Quote
Old 14th March 2013, 09:29   #7037  |  Link
Octo-puss
Registered User
 
Join Date: Jan 2008
Posts: 571
Why does it take MeGui like three seconds to close? Before you ask - no I am not running out of time, just curious what's going on in the background so intense it takes so long to shut down, when much more complex programs close instantly :P
Octo-puss is offline   Reply With Quote
Old 14th March 2013, 12:37   #7038  |  Link
Kurtnoise
Swallowed in the Sea
 
Kurtnoise's Avatar
 
Join Date: Oct 2002
Location: Aix-en-Provence, France
Posts: 5,191
this depends on your machine but megui saves all profiles, settings, jobs and delete files if needed when you close it...
Kurtnoise is offline   Reply With Quote
Old 14th March 2013, 14:38   #7039  |  Link
Octo-puss
Registered User
 
Join Date: Jan 2008
Posts: 571
Well, my machine is pretty fast, and MeGui does this even if I simply launch it and immediatelly close. There's nothing to save or process...
Octo-puss is offline   Reply With Quote
Old 14th March 2013, 17:40   #7040  |  Link
Kurtnoise
Swallowed in the Sea
 
Kurtnoise's Avatar
 
Join Date: Oct 2002
Location: Aix-en-Provence, France
Posts: 5,191
Quote:
Originally Posted by ryszardzonk View Post
Please advice of way to fix that.
Using raw output file (i.e h264 instead of mp4 or mkv), concatenation seems to be ok. No more warning like this...

Quote:
Originally Posted by Octo-puss View Post
Well, my machine is pretty fast, and MeGui does this even if I simply launch it and immediatelly close. There's nothing to save or process...
Each time you launch MeGUI and close it afterwards, all settings/jobs/profiles are saved...
Kurtnoise is offline   Reply With Quote
Reply

Tags
megui

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 09:58.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.