Using DeDup

doctorkhv · 13th September 2022, 08:01

Using DeDup.
After removing duplicates, it turns out a terrible out of sync video and audio with any assumefps(), but I wanted 25
What to change in the script?

Code:

Import("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\functions\AudioFunctions.avs")
Import("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\functions\VideoFunctions.avs")
LoadPlugin("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\FFMS2.dll")
Import("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\FFMS2.avsi")
LoadPlugin("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\NicAudio.dll")
LoadPlugin("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\SplineResize.dll")

video = FFmpegSource2("S:\Classic_cinema\Night.m2ts", rffmode=0, threads=1, cachefile="D:\Temp\00000.m2ts.ffindex").AssumeFPS(23.976)
audio = NicAC3Source("D:\Temp\3899_3.ac3")
AudioDub(video, audio)

ConvertToYV12()

###[FILTERING]###
XviD4PSPPluginsPath = "C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\"
LoadPlugin(XviD4PSPPluginsPath + "dedup.dll")


converttoyv12()
#dupmc(log="E:\01DUP\Night.txt")

dedup(threshold=3.0, maxcopies=3, maxdrops=3, log="E:\01DUP\Night.txt")
assumefps(25) # or 18 or 20 or ntsc_film or film or pal_film or ntsc_video or leave 29.959

hello_hello · 13th September 2022, 11:35

You can't remove duplicates with Dedup and end up with a constant frame rate because the duplicates won't be spread out evenly.

Dedup needs to create a timecodes file, so for the second pass it'd be something like:
dedup(threshold=3.0, maxcopies=3, maxdrops=3, log="E:\01DUP\Night.txt", times="E:\01DUP\Night_Timecodes.txt")
I assume you're running a full DupMC analysis pass to create the log file?

The timecodes file can be added after the video is encoded (while muxing) with MKVToolNix for MKVs. I'm not sure what the best tool is to do the same for MP4. Alternatively if you're encoding with x264 you can add the timecodes file to the x264 command line and encode in VFR mode.
--tcfile-in "E:\01DUP\Night_Timecodes.txt"

I've no idea whether hardware players are likely to support VFR xvid.

If you need a constant frame rate there's plugins such a Dup that replace duplicates with exact copies.
http://avisynth.nl/index.php/Externa...rame_Detectors

Alternatively you can follow Dedup in the script with TimecodeFPS to convert to any constant frame rate by replacing the duplicates and/or dropping frames if need be. It'll use the Dedup timecodes to convert to a CFR.

TimeCodeFPS("E:\01DUP\Night_Timecodes.txt", FPSNum=24000, FPSDen=1001)

I suspect though, you mightn't be decoding the source correctly, which will cause A/V sync problems too. Is the m2ts file 23.976fps to begin with?
If it's DVD video and it's not being decoded at exactly 23.976fps or 29.97fps, it probably means the video is VFR (contains repeat field flags) and AssumeFPS can't convert VFR to a CFR because FFMS2 just outputs the average frame rate and AssumeFPS has no way of knowing where the duplicates should be. For DVD video try FFmpegSource2(rffmode=1) so it'll obey the repeat flags and should output 29.97fps. For true VFR sources you might need to tell FFMS to output a constant frame rate, but be careful as it'll drop frames if need be, so ideally the frame rate should be high enough that it only needs to add duplicates. FFmpegSource2(FPSNum=30000, FPSDen=1001).

Or if the source is VFR you can tell FFMS2 to create a timecodes file that Dedup can use to create a new timecodes file after removing duplicates that can be used when muxing, or for TimeCodesFPS to convert to a CFR.

FFmpegSource2(timecodes="E:\Night_FFMS_Times.txt")
Dedup(log="E:\Night.txt", timesIn="E:\Night_FFMS_Times.txt", times="E:\Night_Dedup_Times.txt")
TimeCodeFPS("E:\Night_Dedup_Times.txt", FPSNum=24000, FPSDen=1001)

And some plugins create timecodes files with the following first line.
# timecode format v2
And some are created with the first line as
# timestamp format v2
Some tools expect one or the other, so you might need to manually change it if the timecodes aren't recognised.

doctorkhv · 13th September 2022, 22:40

Many thanks for the detailed answer!!!

There are 500 scenes in the film and there is one duplicate frame between scenes.

Seems like it worked out well?

doctorkhv · 19th September 2022, 00:18

Seems like it worked out well? https://youtu.be/HStLAepzBQU

hello_hello · 19th September 2022, 08:42

I don't think it's right.

I've only looked at the first minute or so but at the very beginning where the car first comes into the shot it's motion looks jerky, and again when it's going out of shot where it even seems to go backwards and forwards a couple of times, and when the car goes past the flag and the camera is panning the motion of the background objects doesn't seem smooth to me.

I've no idea what might have happened to it when it was re-encoded by YouTube, so maybe upload a small section of the source video somewhere for others to look at. A few minutes where there's movement should do. If anything it looks like it might be a standard telecined film that needs to be decoded at 29.97fps and then have reverse telecine applied for 23.976, but it's impossible to tell much from a YouTube sample. If you want to upload a sample there's lots of places.

https://workupload.com/
https://www.udrop.com/
https://ufile.io/
https://fileupload.win/

doctorkhv · 19th September 2022, 09:18

I agree with you.

The first scene isn't exactly smooth, but there are over a thousand scenes in this movie.

coolgit · 19th September 2022, 12:59

The film seems to overly denoised and overly temporal smoothed. The characters face look like plastic dolls. Frame 2303 somewhat defective according to Vdub.

doctorkhv · 19th September 2022, 22:47

Quote:

Originally Posted by coolgit

The film seems to overly denoised and overly temporal smoothed. The characters face look like plastic dolls. Frame 2303 somewhat defective according to Vdub.

Thank you for finding such a movie scene, with interpolation at 50 fps, this happens often in fast scenes.

To make the faces not look like plastic dolls, I can just add coarse grain, but this will increase the bitrate and cause other changes when converting to YouTube.

doctorkhv · 19th September 2022, 23:13

Quote:

Originally Posted by hello_hello

... If anything it looks like it might be a standard telecined film that needs to be decoded at 29.97fps and then have reverse telecine applied for 23.976, but it's impossible to tell much from a YouTube sample. If you want to upload a sample there's lots of places.

https://workupload.com/
https://www.udrop.com/
https://ufile.io/
https://fileupload.win/

I can't select an episode/segment of the original movie.

PHP Code:


			
General

ID                             : 1 (0x1)

Complete name                  : S:\Classic_cinema\Night of the Living Dead (1968)\BDMV\STREAM\00000.m2ts

Format                         : BDAV

Format/Info                    : Blu-ray Video

File size                      : 16.7 GiB

Duration                       : 1 h 35 min

Overall bit rate mode          : Variable

Overall bit rate               : 24.9 Mb/s

Maximum Overall bit rate       : 35.5 Mb/s



Video

ID                             : 4113 (0x1011)

Menu ID                        : 1 (0x1)

Format                         : AVC

Format/Info                    : Advanced Video Codec

Format profile                 : High@L4.1

Format settings                : CABAC / 4 Ref Frames

Format settings, CABAC         : Yes

Format settings, ReFrames      : 4 frames

Codec ID                       : 27

Duration                       : 1 h 35 min

Bit rate mode                  : Variable

Maximum bit rate               : 34.0 Mb/s

Width                          : 1 920 pixels

Height                         : 1 080 pixels

Display aspect ratio           : 16:9

Frame rate                     : 23.976 (24000/1001) FPS

Color space                    : YUV

Chroma subsampling             : 4:2:0

Bit depth                      : 8 bits

Scan type                      : Progressive

Color range                    : Limited

Color primaries                : BT.709

Transfer characteristics       : BT.709

Matrix coefficients            : BT.709





Audio #1

ID                             : 4354 (0x1102)

Menu ID                        : 1 (0x1)

Format                         : DTS XLL

Format/Info                    : Digital Theater Systems

Commercial name                : DTS-HD Master Audio

Muxing mode                    : Stream extension

Codec ID                       : 134

Duration                       : 1 h 35 min

Bit rate mode                  : Variable

Channel(s)                     : 2 channels

Channel layout                 : L R

Sampling rate                  : 48.0 kHz

Frame rate                     : 93.750 FPS (512 SPF)

Bit depth                      : 16 bits

Compression mode               : Lossless

Language                       : English



Audio #2

ID                             : 4355 (0x1103)

Menu ID                        : 1 (0x1)

Format                         : AC-3

Format/Info                    : Audio Coding 3

Commercial name                : Dolby Digital

Codec ID                       : 129

Duration                       : 1 h 35 min

Bit rate mode                  : Constant

Bit rate                       : 448 kb/s

Channel(s)                     : 2 channels

Channel layout                 : L R

Sampling rate                  : 48.0 kHz

Frame rate                     : 31.250 FPS (1536 SPF)

Bit depth                      : 16 bits

Compression mode               : Lossy

Stream size                    : 307 MiB (2%)

Language                       : English

Service kind                   : Complete Main

hello_hello · 20th September 2022, 03:07

If it's already 23.976fps I'm not sure why you'd need to do anything to it, but you can split the video into sections with MKVToolNix or TSMuxer.

doctorkhv · 20th September 2022, 04:19

Quote:

Originally Posted by hello_hello

If it's already 23.976fps I'm not sure why you'd need to do anything to it, but you can split the video into sections with MKVToolNix or TSMuxer.

To split a movie into sections without re-encoding, I use a Boilsoft Video Splitter

rgr · 20th March 2025, 12:50

Does DupMC still work for anyone?
I get at every frame:

"Error reading source frame 1215: Avisynth read error: CAVIStreamSynth: System exception - Access Violation at Ox00007FFAC5D284E3"

Script:

Code:

ffms2("projekt.mp4")
DupMC(log="dupmc-log.txt")

StvG · 20th March 2025, 17:54

Quote:

Originally Posted by rgr

Does DupMC still work for anyone?
I get at every frame:

"Error reading source frame 1215: Avisynth read error: CAVIStreamSynth: System exception - Access Violation at Ox00007FFAC5D284E3"

Script:

Code:

ffms2("projekt.mp4")
DupMC(log="dupmc-log.txt")

Yes, I just test it. Make sure you're using the version from 2021 (DeDup-0.17-20210227.7z)

StainlessS · 20th March 2025, 20:18

Quote:

ffms2("projekt.mp4")
DupMC(log="dupmc-log.txt")

Quote:

"Error reading source frame 1215: Avisynth read error: CAVIStreamSynth: System exception - Access Violation at Ox00007FFAC5D284E3"

Try LSmashVideoSource("projekt.mp4")

rgr · 21st March 2025, 17:01

LSMASH "failed to read an input file".
LWLibavVideoSource - same effect as with ffms2.

StainlessS · 22nd March 2025, 01:13

Presumably it fails even without DupMC line.

Is it possible that it is not an h264 stream inside the mp4 container, check via MediaInfo.
And, does the clip play ok in some media player app.

Maybe post MediaInfo log.

EDIT: Posted in another thread (perhaps pertinent to the problem).

Quote:

Originally Posted by rgr

Well, I have AMD (Ryzen) + Radeon Vega...

StvG · 22nd March 2025, 05:26

Yes, it looks likes it's related to the AMD CPU.
Dedup x64 is build with Intel C++ Compiler because of the ASM code (the original ASM is "converted" to ASM x64).
Either some ASM error is triggered or the AMD CPU doesn't recognize instruction(s).

13th September 2022, 11:35	#2 \| Link
hello_hello Registered User Join Date: Mar 2011 Posts: 5,017	You can't remove duplicates with Dedup and end up with a constant frame rate because the duplicates won't be spread out evenly. Dedup needs to create a timecodes file, so for the second pass it'd be something like: dedup(threshold=3.0, maxcopies=3, maxdrops=3, log="E:\01DUP\Night.txt", times="E:\01DUP\Night_Timecodes.txt") I assume you're running a full DupMC analysis pass to create the log file? The timecodes file can be added after the video is encoded (while muxing) with MKVToolNix for MKVs. I'm not sure what the best tool is to do the same for MP4. Alternatively if you're encoding with x264 you can add the timecodes file to the x264 command line and encode in VFR mode. --tcfile-in "E:\01DUP\Night_Timecodes.txt" I've no idea whether hardware players are likely to support VFR xvid. If you need a constant frame rate there's plugins such a Dup that replace duplicates with exact copies. http://avisynth.nl/index.php/Externa...rame_Detectors Alternatively you can follow Dedup in the script with TimecodeFPS to convert to any constant frame rate by replacing the duplicates and/or dropping frames if need be. It'll use the Dedup timecodes to convert to a CFR. TimeCodeFPS("E:\01DUP\Night_Timecodes.txt", FPSNum=24000, FPSDen=1001) I suspect though, you mightn't be decoding the source correctly, which will cause A/V sync problems too. Is the m2ts file 23.976fps to begin with? If it's DVD video and it's not being decoded at exactly 23.976fps or 29.97fps, it probably means the video is VFR (contains repeat field flags) and AssumeFPS can't convert VFR to a CFR because FFMS2 just outputs the average frame rate and AssumeFPS has no way of knowing where the duplicates should be. For DVD video try FFmpegSource2(rffmode=1) so it'll obey the repeat flags and should output 29.97fps. For true VFR sources you might need to tell FFMS to output a constant frame rate, but be careful as it'll drop frames if need be, so ideally the frame rate should be high enough that it only needs to add duplicates. FFmpegSource2(FPSNum=30000, FPSDen=1001). Or if the source is VFR you can tell FFMS2 to create a timecodes file that Dedup can use to create a new timecodes file after removing duplicates that can be used when muxing, or for TimeCodesFPS to convert to a CFR. FFmpegSource2(timecodes="E:\Night_FFMS_Times.txt") Dedup(log="E:\Night.txt", timesIn="E:\Night_FFMS_Times.txt", times="E:\Night_Dedup_Times.txt") TimeCodeFPS("E:\Night_Dedup_Times.txt", FPSNum=24000, FPSDen=1001) And some plugins create timecodes files with the following first line. # timecode format v2 And some are created with the first line as # timestamp format v2 Some tools expect one or the other, so you might need to manually change it if the timecodes aren't recognised. Last edited by hello_hello; 13th September 2022 at 18:51.

13th September 2022, 22:40	#3 \| Link
doctorkhv Registered User Join Date: Mar 2019 Posts: 48	Many thanks for the detailed answer!!! There are 500 scenes in the film and there is one duplicate frame between scenes. Seems like it worked out well? Last edited by doctorkhv; 19th September 2022 at 00:18.

20th March 2025, 12:50	#12 \| Link
rgr Registered User Join Date: Jun 2022 Posts: 176	Does DupMC still work for anyone? I get at every frame: "Error reading source frame 1215: Avisynth read error: CAVIStreamSynth: System exception - Access Violation at Ox00007FFAC5D284E3" Script: Code: ffms2("projekt.mp4") DupMC(log="dupmc-log.txt")

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

19th September 2022, 00:18	#4 \| Link
doctorkhv Registered User Join Date: Mar 2019 Posts: 48	Seems like it worked out well? https://youtu.be/HStLAepzBQU

19th September 2022, 08:42	#5 \| Link
hello_hello Registered User Join Date: Mar 2011 Posts: 5,017	I don't think it's right. I've only looked at the first minute or so but at the very beginning where the car first comes into the shot it's motion looks jerky, and again when it's going out of shot where it even seems to go backwards and forwards a couple of times, and when the car goes past the flag and the camera is panning the motion of the background objects doesn't seem smooth to me. I've no idea what might have happened to it when it was re-encoded by YouTube, so maybe upload a small section of the source video somewhere for others to look at. A few minutes where there's movement should do. If anything it looks like it might be a standard telecined film that needs to be decoded at 29.97fps and then have reverse telecine applied for 23.976, but it's impossible to tell much from a YouTube sample. If you want to upload a sample there's lots of places. https://workupload.com/ https://www.udrop.com/ https://ufile.io/ https://fileupload.win/

19th September 2022, 09:18	#6 \| Link
doctorkhv Registered User Join Date: Mar 2019 Posts: 48	I agree with you. The first scene isn't exactly smooth, but there are over a thousand scenes in this movie.

19th September 2022, 12:59	#7 \| Link
coolgit Registered User Join Date: Apr 2019 Posts: 246	The film seems to overly denoised and overly temporal smoothed. The characters face look like plastic dolls. Frame 2303 somewhat defective according to Vdub.

20th September 2022, 03:07	#10 \| Link
hello_hello Registered User Join Date: Mar 2011 Posts: 5,017	If it's already 23.976fps I'm not sure why you'd need to do anything to it, but you can split the video into sections with MKVToolNix or TSMuxer.

21st March 2025, 17:01	#15 \| Link
rgr Registered User Join Date: Jun 2022 Posts: 176	LSMASH "failed to read an input file". LWLibavVideoSource - same effect as with ffms2.

22nd March 2025, 05:26	#17 \| Link
StvG Registered User Join Date: Jul 2018 Posts: 569	Yes, it looks likes it's related to the AMD CPU. Dedup x64 is build with Intel C++ Compiler because of the ASM code (the original ASM is "converted" to ASM x64). Either some ASM error is triggered or the AMD CPU doesn't recognize instruction(s).