MeGui and Progressive 59.94 fps

[Zag]

You'll have to forgive my noob questions but I have been reading for a couple of weeks and although I have been able to get megui up and encoding everything I have worked with so far was already at 23.96fps.

I have some HD movies captured at 1280x720 @59.94 progressive fps. These were originally theater releases shot at 24 fps so should I IVTC to 23.96 or should I leave it alone? If so how do you do it in megui? I have seen other posts talking about using using SelectEven() or fdecimate() or SelectEvery(5,2,4). I assume I manually add one or more of these statements in the avs script or is there a setting in MeGui?

I know a lot depends on your source but is there a rule of thumb on when you should or should not use Telecine or IVTC.

[berrinam]

I know a lot depends on your source but is there a rule of thumb on when you should or should not use Telecine or IVTC.
Standard IVTCs only work if your source is of the frame pattern 3 progressive followed by 2 combed frames. MeGUI will do that detection as well, and will tell you if it is. If it is, then great! You could manually add an IVTC in yourself, or you could tell MeGUI to 'Analyse' the source to suggest its filters.

If it isn't, and you are sure that it isn't entirely interlaced, then I suggest that this thread be moved to the AviSynth Usage forum.

[foxyshadis]

If they're progressive and have no (or at least minimal) combing artifacts, like it's a bobbed telecine, you might just use TDecimate(cycle=10,cycleR=6) to bring it back down to 24 fps as well as possible. Now I don't know if you could recover the original 30i telecine, which would probably give a smoother 24p, but that would be a lot more work. Try Tdec first.

[Zag]

I always run the 'Analyze' tool first but it usually comes back with 'Source declared progressive' and it will not suggest any filters, it just says nothing to do.

[Doom9]

well.. your source is probably framedoubled but still progressive, so the detection picks it up as such. But perhaps berrinam will extend the source detection to detect such content and handle it properly by applying the proper decimation.

[berrinam]

well.. your source is probably framedoubled but still progressive, so the detection picks it up as such. But perhaps berrinam will extend the source detection to detect such content and handle it properly by applying the proper decimation.
Sure, if (a) someone tells me where to get such samples, or (b) if someone describes them sufficiently, and helps to test it when I've furthered the algorithm.

[Zag]

I can give you a sample, just tell me how long of a sample you need. Would 60 seconds do or do you need a longer sample. Also I use videoredo to snip it out is that ok. A place to upload it would be nice too.

[berrinam]

I can give you a sample, just tell me how long of a sample you need. Would 60 seconds do or do you need a longer sample. Also I use videoredo to snip it out is that ok. A place to upload it would be nice too.
60 seconds would probably be enough for me to work out how to recognise it, but it would also be nice to have a few minutes to test on. If it isn't possible for you to do that, I'll just simulate it by repeating it several times over.

I have no place to upload. Why don't you try one of the free file hosting services reviewed on http://www.free-webhosts.com/free-file-hosting.php

As long as I can input the video into AviSynth, the sample is fine.

[Zag]

Ok..I am working on it...I'll post back when I am done...I already have the 60 seconds clip (90 MB), I'll add another 3 minute clip.

Looks like this is going to take a while. My uploads speeds kind of suck (384 Mb/s) and the free file hosting places have MB limits. How would you feel if I setup an FTP site and you get them straigh from me. It's going to take a little while due to my slow upload speeds but if you don't mind I won't mind. The 3 minute clip is an additional 268MB. Let me know if this is ok and I'll PM you the relevant information.

Check your PM.

[berrinam]

As you can probably tell, I have already got your file now, and I've done some testing with it. The initial results are looking quite good. Anyway, it would be good if you could also run my test on your computer, with the entire movie as a sample (the thing is that I don't know if my results are good just because it's a particularly easy part of the film or not). Can you try your film out with the following method (as well as anyone else with this sort of sample):

Firstly, create an AviSynth script which looks like the following:

file="your_results_file.csv"
mpeg2source("yoursource.d2v")
selectrangeevery(every=500,length=50)

global sep="-"
global source = last
c = source
c = WriteFile(c, file, "diff")
c = FrameEvaluate(c,"global diff = 0.50*YDifferenceFromPrevious(source) + 0.25*UDifferenceFromPrevious(source) + 0.25*VDifferenceFromPrevious(source)")
crop(c,0,0,16,16)
You can then pass through this file exactly once, so that the analysis results are written to "your_results_file.csv". The easiest way that I do this is by using MeGUI's 'Queue Analysis pass' button. All you do is load the avs file, press that button, then launch the new job created in the queue.

After that, you will have a csv file containing your results. Open it, and you should see lots of results in column A. Put a motion threshold in B1 (I think that 1 is the best, so try that first and tweak later if you want). Then, paste this Visual Basic macro into the Excel macros. You can do this by going Alt+F11 or Tools->Macros->Visual Basic Editor. In the new window, just paste the following code, then press F5 to run it. When you go back to your spreadsheet, you should see some numbers in cells C1-I1. These numbers signify as follows:

C1: Total number of sections (groups of 5 frames) analysed
D1: Number of sections with 0 motion frames
E1: Number of sections with 1 motion frame
F1: Number of sections with 2 motion frames
...
I1: Number of sections with 5 motion frames

Now, in normal sources, I1 should be by far the largest, followed by all the others somewhere in between. But in your source, F1 should be by far the largest, as there are only two frames in every five that differ from the preceding ones.

Visual Basic Macro:
Sub countsections()
Dim i As Integer
i = 1
Dim asections As Integer
Dim bsections As Integer
asections = 0
bsections = 0
Dim count1 As Integer
Dim count2 As Integer
Dim j As Integer
Dim totalsections As Integer
Dim sectionscounts(0 To 5) As Integer

While Sheet1.Cells(i + 5, 1) <> ""
count1 = 0
count2 = 0
For j = 1 To 5
If Sheet1.Cells(i, 1) - Sheet1.Cells(1, 2) > 0 Then
count1 = count1 + 1
Else
count2 = count2 + 1
End If
i = i + 1
Next j
totalsections = totalsections + 1
sectionscounts(count1) = sectionscounts(count1) + 1
Wend
Sheet1.Cells(1, 3) = totalsections
For i = 0 To 5
Sheet1.Cells(1, i + 4) = sectionscounts(i)
Next i
End Sub

So, if you could do what I just said on your sample (use a threshold of 1 -- I'm pretty sure that's close enough to the best) and then report back with the numbers in columns C1 through I1, that would be good. Ideally, F1 should be at least 2 times, if not 3 times the value of all the other columns.

Also, if your results are good, can you try playing with the SelectRangeEvery line in the avisynth script I posted? A few other suggestions (which should make it run faster while hopefully not losing much accuracy) are:
selectrangeevery(every=500,length=5)
selectrangeevery(every=1000,length=10)
selectrangeevery(every=500,length=20)
selectrangeevery(every=50,length=5)

Thanks a lot for your input.

[Zag]

I tried doing as you said. I ran the avs file and got the results (43601 values in column A). I placed a value of 1 in B1 then I pasted your VB macro but when I run it I get the following error:

Run-Time Error '6'
Overflow

If you wish you can download the .csv file from the same location.


update:

I am not a programmer but based on the error I was receiving I changed the Dim statements from Intiger to Long. Now the macro runs and I get the following results:


selectrangeevery(every=500,length=50)
C1 D1 E1 F1 G1 H1 I1
8720 250 269 6273 1820 94 14


selectrangeevery(every=500,length=5)
872 0 16 382 430 41 3


selectrangeevery(every=1000,length=10)
872 11 27 519 292 21 2


selectrangeevery(every=500,length=20)
3488 85 106 2342 888 61 6


selectrangeevery(every=50,length=5)
8713 17 239 3622 4377 433 25

[berrinam]

Ok, I seem to be getting bad results with length=5, and I think I know why. Can you try this slightly modified avs script?
file="your_results_file.csv"
mpeg2source("yoursource.d2v")

global sep="-"
global source = last
c = source
c = WriteFile(c, file, "diff")
c = FrameEvaluate(c,"global diff = 0.50*YDifferenceFromPrevious(source) + 0.25*UDifferenceFromPrevious(source) + 0.25*VDifferenceFromPrevious(source)")
selectrangeevery(every=500,length=5)
crop(c,0,0,16,16)
With SelectRangeEvery at the end, it should not be causing problems when it jumps.

Thanks for your testing -- it is very valuable.

[Zag]

Ok. I am running your new script but the amount of time it will take as increased substantially. Your other script only took a couple of minutes while this one is going to take about an hour.

Update:
Something is not right. The script ran but the file won't completely load in exel (it exceeds the exel limit of 65536 rows). I tried to run the VB macro on the 65536 rows that did load but I get the following error:

Run-Time error '1004'
Application-defined or object-defined error


Not sure what else I can try.

[berrinam]

Ok. I am running your new script but the amount of time it will take as increased substantially. Your other script only took a couple of minutes while this one is going to take about an hour.
How many frames are there? Are you sure you got the script right, because it runs at about the same speed here, and has the same number of frames. Specifically, are you sure you have SelectRangeEvery in the second last line?

[Zag]

There are 174254 total frames (I am not sure if this is the frames you are referring to) and I just copied and pasted you script in a notepad text file, changed the 'mpeg2source' file name to the appropriate file name and saves as .avs. Loaded the .avs file in megui and hit the 'Queued analysis pass' and started the queued file, just like before.

[berrinam]

Aah, found the problem. My mistake. Move the SelectRangeEvery call right to the end.

[berrinam]

It turns out that your sample is quite interesting. I ran source detection on it after making changes that I thought would find the decimation, and it detected it as hybrid film/ntsc. It turns out that, since each group of 5 had 3 duplicates, there was no group of 5 frames with motion throughout, and as a result, all of the sections were ignored. That caused the bad detection, and it brought another case to my attention which I hadn't thought of before.

Perhaps someone with more experience can help me with this. For reasons of simplicity and accuracy, source detection ignores any group of 5 frames in which there is no motion between each consecutive frame. While there are normally enough sections analysed for this not to matter, in some cases, like the case discussed in this thread, this breaks the analysis completely, as source detection has ignored all sections and has nothing to go on. My question is whether you can think of any situations in which there will always be some frames with no motion in each group of 5. Also, do you have any suggestions for these situations?

[Zag]

I don't know if you still want the results of the last script (I put the selectrange statement at the end) but here it is:

871 24 20 673 147 6 1


I also went ahead and encrypted the movie using the HQ-Slow profile but added the SelectEvery(5,2,4) statement which reduced the fps to 23.975. I am very happy with the results. The movie plays very smoothly and audio is in sync. Much better results than my previous attempt when it left the fps at 59.94.

[berrinam]

Here's a testing version of MeGUI with updated source detection. It works on the sample you gave me, so can you see if it works on your sample?

http://www.savefile.com/files.php?fid=4852333

[Zag]

I ran the new megui. First it gave me an error about not being able to load the default profile and said something about being protected. In any case I pressed ok and I had it analyze the movie. It came back with 'Declared as Repitition-upconverted. Decimation is Required' it then selected 'Tritical Decimate'

[berrinam]

As far as I know, that is the correct filter, and it is the intended response, so it looks like all works. I'll commit it to the official repository then.

[Zag]

Just to follow up, I used your new version on a new movie that I just captured from the same source. Your new detection worked so I went ahead and encoded it. The results were very good (the fps dropped to 23.976) so it looks like it's working just fine. The only unsual thing about is that it won't play in TCPMP (it says unrecognized format) but it plays fine in MCP and Zoomplayer using CoreAVC. Thanks for all your help.

On a separate issue I seem to be having a problem with audio encoding to NAAC (used avisynth encoding with variable bitrate and LC profile). The resulting .mp4 file won't work in a .mp4 container when muxed. When I play the audio file on it's own it plays just fine but when muxed all I get is a very loud screeching moment and then silence. On one movie this happens right at the beginning and on another it happens about half way through the movie. When I mux the original ac3 file in a .mkv container, it all works file.