DGPulldown 1.0.9 Final supports in-place operation [Archive]

Chainmax

1st April 2006, 14:35

In this thread (http://forum.doom9.org/showthread.php?p=807759&posted=1#post807759), I'm trying to make a DVD out of a 14.999fps source. The recommendation thus far seems to be to go PAL by speeding up to 16.667fps and then use DGPulldown to reach 25fps. Now, I seem to recall a post somewhere in here that said that any pulldown excepting the classic 23.976--->29.97 one would introduce some jittering (line moving artifacting, right?), is that true? If so, is there any way around it?

neuron2

2nd April 2006, 00:16

If you'd said "judder" I'd know what you are talking about. Even 23.976 -> 29.97 introduces judder.

But "line moving artifact"? You're baffling me.

Chainmax

3rd April 2006, 14:01

It's probably judder then, sorry for my ignorance :o.

neuron2

3rd April 2006, 17:10

The judder is an unavoidable side effect of the field duplication. It is for you to judge whether it is tolerable for a given rate conversion.

Chainmax

3rd April 2006, 18:22

I see, but would 16.667-->25 introduce more judder than 23.976-->29.97?

Mug Funky

4th April 2006, 08:37

not really. the judder is usually due to the frames being on the screen for different amounts of time (2 fields v 3 fields). going from 16.67... to 25 will be an exact 3:3 pulldown, meaning every film frame is on screen for the same amount of time.

though it'll still appear slightly jerky seeing as the frame rate is lower after all, and when you track a moving object with your eye (on an interlaced screen), you'll see 3 ghost images as your eye moves (hope that makes sense. the effect is quite distinct. try track a medium-fast pan with your eye and see how things look).

Deep_Eyes

9th April 2006, 04:42

would this method benifit low-bitrate anime encoding?

foxyshadis

9th April 2006, 05:43

Only if mastering to DVD, but there you have lots of bitrate, usually. For pcs and handhelds, decimate the framerate to 8 or 12fps if you have to but never purposely introduce judder or interlacing. (Interlacing will look bad on them anyway, or take extra processing to remove messily.)

DJKC

2nd May 2006, 16:31

Was test this out today using the in place mode. Seemed to run okay and said it completed, but can't use the resulting file in IFOedit or Scenarist. IFOedit gave a message saying "Too many dropped frames" and stopped. Scenarist stopped at roughly the same point with this in the log:
Error J:\working\VTS02\Scenaid\NewD2V\VTS__02_P01.I-TFF.16~9_1.mpv
Error : Number of fields (48) in GOP(# 8531) or between sequence_header_code should be equal or smaller than 36.
Error : Frame rate (3 = reserved) is wrong.
Warning : Number of SequenceEndCode is 0.

Wondering if there's anything obvious I might have done wrong? The video was flagged as interlaced on the original DVD but had no artifacts that I could find so I processed it as suggested in the third option in this guide http://www.doom9.org/mpg/big3-BatchCCEWS.htm + resized it to 720x480. Then I ran dgpulldown on it and got the file I'm having problems with now. I've got it running again processing it as if the source was interlaced to see if it makes a difference, but wondering if there are any suggestions people might have about what else could be wrong.

neuron2

2nd May 2006, 16:46

Was test this out today using the in place mode. What happens if you do everything the same but don't use in-place operation? Also, please tell me all the DGPulldown settings you used. How did you create the base M2V file? It sounds like you haven't encoded it for DVD compliance.

DJKC

2nd May 2006, 17:22

Unfortunately I didn't make a backup of the file before trying it, so I can't say. I'm in the middle of encoding it again.

Rip+encode was done with DIF4U+scenaid+batchccews using CCE SP 2.70. Options in batchccews were Progressive, TFF, and Close All GOPS. DGpulldown options were 25->29.97, Set Time Codes, and Modify file in place. AVS script was:
import("C:\Program Files\DoItFast4U\new.avs\addaudio.avs")
LoadPlugin("C:\Program Files\ScenAid\DGDecode.dll")
Mpeg2Source("J:\working\VTS02\Scenaid\NewD2V\VTS__02_P01.I-TFF.16~9_1.d2v",idct=0)
AddAudio()
LanczosResize(720,480)
ConvertToYUY2()

Thanks for your response.

edit:
You were right, found a stupid error elsewhere in the encode process that was hosing things up an giving me a non-compliant final stream. Sorry for posting here before I finished all my reencode tests.

Mr_Odwin

13th June 2006, 22:56

I've recently had some problems with DGPulldown1.07 and the inplace option. I use QuEnc 0.70 and the encodes appear to be fine before the pulldown operation and creating a new file as part of the pulldown results in a good file too. However, when using the inplace option the resulting file plays okay for the majority of the time but then starts to stutter, plays too quickly and shows distorted flashes while the video plays.
I've been using 23.976->25 and the original file has been about 3.5GB in size. I've not investigated it fully but when I reduced the resolution of the vid to 352x288 and lowered the bitrate to 600 so that the filesize was considerably smaller (0.5GB-ish) there was no problem. I am also using Avisynth 2.57 Alpha 3 if that matters.
I watched the gui for a while (sad I know) and the second number didn't seem to change as much towards the end of the file where the problem tends to arise.

neuron2

13th June 2006, 23:02

Perform pulldown on your source M2V twice (keep a backup): do it once without in-place operation and once with in-place operation. Then perform a binary comparison of the two resulting files. Are they the same?

It's not clear whether you are reporting just an issue with in-place operation or a problem with the pulldown itself. This test will clarify that and then we can take it from there.

Mr_Odwin

13th June 2006, 23:04

How do I perform a binary comparison?

neuron2

14th June 2006, 00:04

Google "binary file compare"

Are you saying it only happens when you use in-place?

Mr_Odwin

14th June 2006, 20:49

It only happens when I use in-place. I was mistaken above in that it actually occurs with a 23.976->29.97 pulldown. (I've not checked the 23.976->25)

My AVS file:

AVISource("C:\Documents and Settings\JAMES PIKE\My Documents\My Applications\X3.avi", false)
ChangeFPS("ntsc_film")
LanczosResize(720,480)
ConvertToYV12()

My commandline to obtain an m2v:

"C:\Documents and Settings\JAMES PIKE\My Documents\My Applications\FAVC\QuEnc\Quenc.exe" -i "C:\Documents and Settings\JAMES PIKE\My Documents\My Applications\FAVC\Working Folder 0\Title0.avs" -o "C:\Documents and Settings\JAMES PIKE\My Documents\My Applications\FAVC\Working Folder 0\Title0.m2v" -b 8000 -maxbitrate 8000 -dc 8 -1 -nohq -vbr -noscene notrell -nocgop -nointerlaced -noextreme -gopsize 12 -maxbframes 2 -nocmatrix -aspectratio 16:9 -mpeg2mux noaudio -silent -auto -close

I then perform pulldown with the GUI, or command line. Using default options apart from choosing in-place or not. The results of the binary file comparison begin with:

C:\Documents and Settings\James\My Documents\FAVC\Working Folder 0>cmp title0.m2v title0.m2v.pulldown.m2v -L[0]
BinaryFileCompare-32bit v2.30 Copyright (c) 1998-2000,2004 Vladimir Tarasov
WARNING! The files have different size (3676978814:3676978813)
Comparing title0.m2v and title0.m2v.pulldown.m2v ... 164936 differences.
Working time: 748.33 seconds.

Here is the report file created:
http://homepages.nildram.co.uk/~pike/report.cmp

It only seems to happen on large m2v files, with respect to filesize that is. Here's what it looks like:

http://img227.imageshack.us/img227/9602/vlcsnap3014402mi.png

http://img106.imageshack.us/img106/1128/vlcsnap3025236dv.png

neuron2

14th June 2006, 21:02

Just to be clear...

I want you to pull it down twice, once in-place and once not in-place. Then compare those two. Don't compare to the original M2V. Is that what you did?

If so, we have a couple options. Best is if you can mail me the original M2V stream on a DVD data disk. If you can do that, I'll PM you my address. Otherwise, try to duplicate this with an M2V less than 500MBytes and you can then upload that to my server. Please advise.

Meanwhile I'll try to duplicate it with a large M2V of my own.

Mr_Odwin

14th June 2006, 21:08

I'm sorry to say that I did understand what you asked me to do. They have both been pulled down. Title0.m2v is the one that was done in-place and a quick look at mediainfo or gspot confirms this.
I've also done this on my work computer and can confirm that it's not just some fault with my home computer hardware as the problem was replicable.

neuron2

14th June 2006, 21:10

Sorry, my edit crossed your post.

Mr_Odwin

14th June 2006, 22:03

I have an easier solution, well if you're up for a bit of encoding time.
http://homepages.nildram.co.uk/~pike/DGCheck.zip

In the zip file there are three files: a .bat, an .avi and an .avs. If you just modify these to suit your particular file paths then it produces an m2v file that I've just checked and have problems with when doing a 23.976->29.97 in-place pulldown.
Note the great elegance of the avs file. :p
The encoding shouldn't take very long.

neuron2

15th June 2006, 06:58

I encoded your script with HC and it made a 130MByte m2v file. The comparison showed no differences.

If your file is that small, please upload it somewhere and provide the link.

neuron2

15th June 2006, 07:46

OK, I made a 3.8GByte M2V from the DVD "Rainbow Bridge", and I was able to duplicate the issue. It appears to break after the famous 2GB file limit. I will investigate. Thank you for pointing it out.

neuron2

15th June 2006, 08:17

Found the problem. When a contributor made the changes for in-place operation, he used fseek() instead of _lseeki64(). I'll fix it.

Mr_Odwin

15th June 2006, 10:06

Great stuff. Don't know if I've said it already but I really like your tool and I only spotted this because I use it so frequently.:o

neuron2

15th June 2006, 10:30

Update: I have it working but there's no equivalent of setvbuf() for low-level I/O, which means it has become dreadfully slow. I could add application buffering as done in DGIndex, but I'm not highly motivated about it. I added in-place operation only because someone contributed the (broken) code. How important is in-place operation to you?

Mr_Odwin

15th June 2006, 10:58

Not hugely. It's nice to conserve disk space, but it's easy enough to work around.

neuron2

16th June 2006, 22:08

OK. Just to update you, I have completed the recoding to solve the 2GB limit and I added buffered low-level I/O so in-place operation is now very fast. I want to regression test it tonight and then I'll give you a beta.

Also, I discovered that the TFF/BFF handling was broken. You can see that by pulling down a stream and then pulling down that pulled down one. The two will have different field orders as shown by DGIndex. They should be the same. The TFF/BFF detection was not taking into account display versus coding order and rather than get into all that complication, I decided to just make it user configurable via the GUI and CLI.

neuron2

17th June 2006, 02:24

OK, Mr_Odwin, here ya go!

Here is in-place operation that works for large files (>2GB). It also has configurable TFF/BFF and displays more meaningful stats. Note that in-place operation is now about 30% faster than non-in-place operation. Finally, the elapsed time for processing is displayed.

The changes were extensive, so please test everything thoroughly. Your feedback will be gratefully received.

http://neuron2.net/dgpulldown/dgpulldown108b3.zip

Mr_Odwin

17th June 2006, 21:44

Ace! I'll use it and if I manage to break it again I'll let you know. Thanks again for a great tool.

neuron2

18th June 2006, 09:45

Bump for new version: beta 3, see above.

edguit@r

3rd July 2006, 00:04

guys,

I have an mpv file that resulted from a 23.976->29.970 pulldown made with dgpulldown 1.0.3. I want to undo this process e retrieve the original file. How do I do that? Any help?

I need to do this because I made a mistake: the correct pulldown should be 25.000->29.970 and not 23.976->29.970.

neuron2

3rd July 2006, 00:37

You can't recreate the original with this tool, but it doesn't matter. Just run it again on the already processed file with the new setting. It will replace all the flags with the correct pattern.

Audionut

16th September 2006, 16:49

Feature request.

Ability to select output file. Mainly so I can read from 1 HDD and output to another.

Mr_Odwin

16th September 2006, 18:50

Feature request.

Ability to select output file. Mainly so I can read from 1 HDD and output to another.

You can do that if you use the command line.

Chainmax

16th September 2006, 19:37

Thanks for this latest release, neuron2 http://smilies.vidahost.com/otn/wink/thumb.gif.

neuron2

17th September 2006, 00:03

Ability to select output file. Mainly so I can read from 1 HDD and output to another. Good idea. I will add that feature. Thank you for the suggestion.

Audionut

17th September 2006, 06:11

Good idea. I will add that feature. Thank you for the suggestion.

Thanks.

I'm guessing that the speed is limited to HDD speeds.
So it should roughly double the speed of processing.

Thanks again.

neuron2

19th September 2006, 03:02

So it should roughly double the speed of processing. Here you go, Audionut. Please advise if it performs as you expect.

http://neuron2.net/dgpulldown/dgpulldown.html

Maybe I should fix the NTSC racism as well. :)
I.e., presets exist only for -> 29.97. We could add 23.976 -> 25 and 24 -> 25. Is anything else useful?

Audionut

19th September 2006, 05:15

With a 65500 frame video, output to same hard drive = 124secs and output to a different drive = 110secs.
Not excatly double speed. But.

Excellent. Thankyou.

[posters opinion]
Not a high priority, perhaps a browse button for the output for a final release.
Just to give it that polished finish, so to speak.

presets exist only for -> 29.97. We could add 23.976 -> 25 and 24 -> 25.
I've never used it but wouldn't the custom button suit that purpose well?
IMO if you were to add extra presets, you would be best served to add all possible presets, to avoid any future requests.
Which, would start to make the GUI cluttered.[/end posters opinion]

Thanks again.

SeeMoreDigital

19th September 2006, 12:00

neuron2

19th September 2006, 14:08

If you have an NTSC 29.970fps IVTC MPEG-2 source containing progressive/interlaced frames, would it be possible to dump the interlaced frames in order to obtain a pure progressive MPEG-2 stream running at 23.970fps?Are you asking for an option to strip the RFF flags? That could be a useful thing as it is analogous to doing Ignore Pulldown Flags in DGIndex.

SeeMoreDigital

19th September 2006, 17:21

Are you asking for an option to strip the RFF flags? That could be a useful thing as it is analogous to doing Ignore Pulldown Flags in DGIndex.Yes please...

I was recently sent some (19Mbps) 1920x1080 MPEG-2 samples to test with my Z500, which all run at 23.970fps (pure progressive). They looked and played "better-than-good" ;)

neuron2

20th September 2006, 02:02

I added the browse output file button. I also discovered that DGPulldown can already remove all pulldown (see the revised help file):

http://neuron2.net/dgpulldown/dgpulldown.html

Removing pulldown is AV-sync-safe only if the pulldown is consistent throughout the stream, e.g., 3:2 pulldown throughout.

Paulcat

28th September 2006, 16:52

If you are taking requests...how about a "shut down computer on completion" checkbox. I was running a 25 to 29.97 pulldown on a file yesterday (a 4.1 GB m2v file) and doing other stuff around the house at the same time. The pulldown took about 10 minutes and that was all I was planning to do at that point, so it would be nice to have the pc shut off without me having to go back to it. Yeah, it's pure laziness, but WTF.

I did note that using the drag'n'drop option, the destination file name defaulted to ".pulldown.m2v" not "sourcepath/filename.m2v.pulldown.m2v" and ended up in the DGPulldown directory. Is it possible to have the default destination path the same as the source path?

Everything else works just peachy like. Great work.

Paul

neuron2

28th September 2006, 22:56

I've fixed the stupid drag'n'drop bug. Thanks for pointing it out. I'll upload the fixed version when I get home tonight.

I doubt I'll add a shutdown option. :(

neuron2

29th September 2006, 02:04

Drag-and-drop bug fixed:

http://neuron2.net/dgpulldown/dgpulldown109.zip

Pookie

29th September 2006, 08:24

Paulcat - you can do an autoshutdown if you set up a Batch file to do your pulldowns. Download Pstools from

http://download.sysinternals.com/Files/PsTools.zip
The file you want is called psShutdown.exe . Put it in the same folder that you have DGpulldown.exe

Just save the lines below as Pulldown.bat

dgpulldown.exe input.m2v -srcfps 25.0 -destfps 29.97 -o output.m2v

psShutdown -k

You'll have to edit the dgpulldown line to suit your source file name and frame rates (unless you know about %1 variables)
The psShutdown line turns the power off (-k) in 20 seconds after the pulldown completes.

canuckerfan

1st October 2006, 02:17

Quick question. I'm working with a progressive source, and dgpulldown makes me choose between TFF and BFF. Shouldn't there be a non-interlaced option. Or am I misinterpreting things?

Malow

1st October 2006, 02:55

dbloom

1st October 2006, 21:47

neuron2, its POSSIBLE to use flags to create a hybrid mpeg2 from a file + timecodes?

ive created a topic time go, asking this...

i convert dvd 2 matroska vfr, then, i can extract a cfr-avi, convert to mpeg2, then, add a "variable pulldown" based on timecode file (when extracting from mkv to avi) to create a mpeg2 like original?

sorry for bad english...

I implemented this a year or two ago in my own program: http://davidbloom.home.mchsi.com/readme/readme.html#. It uses pulldown codes from the TMPGEnc IVTC editor.

I have not encountered any compatibility problems with the hybrid frame rate output files on any DVD players or any MPEG-2 files. It works great for music videos...

The only problem is, it is not compatible with those timecode files, and it is limited to ~65K output fields. It's really just a proof of concept...

neuron2

2nd October 2006, 00:46

Or am I misinterpreting things? Pulldown necessarily makes combed frames from progressive material. They have to have their fields played in the right order.

Paulcat

2nd October 2006, 17:27

Thanks for the fix.

The autoshutdown is not really an issue like I said, unless you allow for multiple pulldowns in your GUI in future builds..

Paul

Malow

3rd October 2006, 05:25

I implemented this a year or two ago in my own program: http://davidbloom.home.mchsi.com/readme/readme.html#. It uses pulldown codes from the TMPGEnc IVTC editor.

I have not encountered any compatibility problems with the hybrid frame rate output files on any DVD players or any MPEG-2 files. It works great for music videos...

The only problem is, it is not compatible with those timecode files, and it is limited to ~65K output fields. It's really just a proof of concept...

tnks for the reply... well, at least is possible.. ;)

using this method for mkv, is my actual dream for a "full" backup... ;)