Using DeDup

[doctorkhv]

Using DeDup.
After removing duplicates, it turns out a terrible out of sync video and audio with any assumefps(), but I wanted 25
What to change in the script?

Import("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\functions\AudioFunctions.avs")
Import("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\functions\VideoFunctions.avs")
LoadPlugin("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\FFMS2.dll")
Import("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\FFMS2.avsi")
LoadPlugin("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\NicAudio.dll")
LoadPlugin("C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\SplineResize.dll")

video = FFmpegSource2("S:\Classic_cinema\Night.m2ts", rffmode=0, threads=1, cachefile="D:\Temp\00000.m2ts.ffindex").AssumeFPS(23.976)
audio = NicAC3Source("D:\Temp\3899_3.ac3")
AudioDub(video, audio)

ConvertToYV12()

###[FILTERING]###
XviD4PSPPluginsPath = "C:\Program Files (x86)\XviD4PSP 5\dlls\AviSynth\plugins\"
LoadPlugin(XviD4PSPPluginsPath + "dedup.dll")


converttoyv12()
#dupmc(log="E:\01DUP\Night.txt")

dedup(threshold=3.0, maxcopies=3, maxdrops=3, log="E:\01DUP\Night.txt")
assumefps(25) # or 18 or 20 or ntsc_film or film or pal_film or ntsc_video or leave 29.959

[hello_hello]

You can't remove duplicates with Dedup and end up with a constant frame rate because the duplicates won't be spread out evenly.

Dedup needs to create a timecodes file, so for the second pass it'd be something like:
dedup(threshold=3.0, maxcopies=3, maxdrops=3, log="E:\01DUP\Night.txt", times="E:\01DUP\Night_Timecodes.txt")
I assume you're running a full DupMC analysis pass to create the log file?

The timecodes file can be added after the video is encoded (while muxing) with MKVToolNix for MKVs. I'm not sure what the best tool is to do the same for MP4. Alternatively if you're encoding with x264 you can add the timecodes file to the x264 command line and encode in VFR mode.
--tcfile-in "E:\01DUP\Night_Timecodes.txt"

I've no idea whether hardware players are likely to support VFR xvid.

If you need a constant frame rate there's plugins such a Dup that replace duplicates with exact copies.
http://avisynth.nl/index.php/External_filters#Duplicate_Frame_Detectors

Alternatively you can follow Dedup in the script with TimecodeFPS (http://avisynth.nl/index.php/TimecodeFPS) to convert to any constant frame rate by replacing the duplicates and/or dropping frames if need be. It'll use the Dedup timecodes to convert to a CFR.

TimeCodeFPS("E:\01DUP\Night_Timecodes.txt", FPSNum=24000, FPSDen=1001)

I suspect though, you mightn't be decoding the source correctly, which will cause A/V sync problems too. Is the m2ts file 23.976fps to begin with?
If it's DVD video and it's not being decoded at exactly 23.976fps or 29.97fps, it probably means the video is VFR (contains repeat field flags) and AssumeFPS can't convert VFR to a CFR because FFMS2 just outputs the average frame rate and AssumeFPS has no way of knowing where the duplicates should be. For DVD video try FFmpegSource2(rffmode=1) so it'll obey the repeat flags and should output 29.97fps. For true VFR sources you might need to tell FFMS to output a constant frame rate, but be careful as it'll drop frames if need be, so ideally the frame rate should be high enough that it only needs to add duplicates. FFmpegSource2(FPSNum=30000, FPSDen=1001).

Or if the source is VFR you can tell FFMS2 to create a timecodes file that Dedup can use to create a new timecodes file after removing duplicates that can be used when muxing, or for TimeCodesFPS to convert to a CFR.

FFmpegSource2(timecodes="E:\Night_FFMS_Times.txt")
Dedup(log="E:\Night.txt", timesIn="E:\Night_FFMS_Times.txt", times="E:\Night_Dedup_Times.txt")
TimeCodeFPS("E:\Night_Dedup_Times.txt", FPSNum=24000, FPSDen=1001)

And some plugins create timecodes files with the following first line.
# timecode format v2
And some are created with the first line as
# timestamp format v2
Some tools expect one or the other, so you might need to manually change it if the timecodes aren't recognised.

[doctorkhv]

Many thanks for the detailed answer!!!

There are 500 scenes in the film and there is one duplicate frame between scenes.

Seems like it worked out well?

[doctorkhv]

Seems like it worked out well? https://youtu.be/HStLAepzBQU

[hello_hello]

I don't think it's right.

I've only looked at the first minute or so but at the very beginning where the car first comes into the shot it's motion looks jerky, and again when it's going out of shot where it even seems to go backwards and forwards a couple of times, and when the car goes past the flag and the camera is panning the motion of the background objects doesn't seem smooth to me.

I've no idea what might have happened to it when it was re-encoded by YouTube, so maybe upload a small section of the source video somewhere for others to look at. A few minutes where there's movement should do. If anything it looks like it might be a standard telecined film that needs to be decoded at 29.97fps and then have reverse telecine applied for 23.976, but it's impossible to tell much from a YouTube sample. If you want to upload a sample there's lots of places.

https://workupload.com/
https://www.udrop.com/
https://ufile.io/
https://fileupload.win/

[doctorkhv]

I agree with you.

The first scene isn't exactly smooth, but there are over a thousand scenes in this movie.

[coolgit]

The film seems to overly denoised and overly temporal smoothed. The characters face look like plastic dolls. Frame 2303 somewhat defective according to Vdub.

[doctorkhv]

The film seems to overly denoised and overly temporal smoothed. The characters face look like plastic dolls. Frame 2303 somewhat defective according to Vdub.

Thank you for finding such a movie scene, with interpolation at 50 fps, this happens often in fast scenes.

To make the faces not look like plastic dolls, I can just add coarse grain, but this will increase the bitrate and cause other changes when converting to YouTube.

[doctorkhv]

... If anything it looks like it might be a standard telecined film that needs to be decoded at 29.97fps and then have reverse telecine applied for 23.976, but it's impossible to tell much from a YouTube sample. If you want to upload a sample there's lots of places.

https://workupload.com/
https://www.udrop.com/
https://ufile.io/
https://fileupload.win/

I can't select an episode/segment of the original movie.

General
ID : 1 (0x1)
Complete name : S:\Classic_cinema\Night of the Living Dead (1968)\BDMV\STREAM\00000.m2ts
Format : BDAV
Format/Info : Blu-ray Video
File size : 16.7 GiB
Duration : 1 h 35 min
Overall bit rate mode : Variable
Overall bit rate : 24.9 Mb/s
Maximum Overall bit rate : 35.5 Mb/s

Video
ID : 4113 (0x1011)
Menu ID : 1 (0x1)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.1
Format settings : CABAC / 4 Ref Frames
Format settings, CABAC : Yes
Format settings, ReFrames : 4 frames
Codec ID : 27
Duration : 1 h 35 min
Bit rate mode : Variable
Maximum bit rate : 34.0 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709


Audio #1
ID : 4354 (0x1102)
Menu ID : 1 (0x1)
Format : DTS XLL
Format/Info : Digital Theater Systems
Commercial name : DTS-HD Master Audio
Muxing mode : Stream extension
Codec ID : 134
Duration : 1 h 35 min
Bit rate mode : Variable
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 93.750 FPS (512 SPF)
Bit depth : 16 bits
Compression mode : Lossless
Language : English

Audio #2
ID : 4355 (0x1103)
Menu ID : 1 (0x1)
Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
Codec ID : 129
Duration : 1 h 35 min
Bit rate mode : Constant
Bit rate : 448 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 31.250 FPS (1536 SPF)
Bit depth : 16 bits
Compression mode : Lossy
Stream size : 307 MiB (2%)
Language : English
Service kind : Complete Main

[hello_hello]

If it's already 23.976fps I'm not sure why you'd need to do anything to it, but you can split the video into sections with MKVToolNix or TSMuxer.

[doctorkhv]

If it's already 23.976fps I'm not sure why you'd need to do anything to it, but you can split the video into sections with MKVToolNix or TSMuxer.

To split a movie into sections without re-encoding, I use a Boilsoft Video Splitter

[rgr]

Does DupMC still work for anyone?
I get at every frame:

"Error reading source frame 1215: Avisynth read error: CAVIStreamSynth: System exception - Access Violation at Ox00007FFAC5D284E3"

Script:
ffms2("projekt.mp4")
DupMC(log="dupmc-log.txt")

[StvG]

Does DupMC still work for anyone?
I get at every frame:

"Error reading source frame 1215: Avisynth read error: CAVIStreamSynth: System exception - Access Violation at Ox00007FFAC5D284E3"

Script:
ffms2("projekt.mp4")
DupMC(log="dupmc-log.txt")

Yes, I just test it. Make sure you're using the version from 2021 (DeDup-0.17-20210227.7z)

[StainlessS]

ffms2("projekt.mp4")
DupMC(log="dupmc-log.txt")
"Error reading source frame 1215: Avisynth read error: CAVIStreamSynth: System exception - Access Violation at Ox00007FFAC5D284E3"
Try LSmashVideoSource("projekt.mp4")

[rgr]

LSMASH "failed to read an input file".
LWLibavVideoSource - same effect as with ffms2.

[StainlessS]

Presumably it fails even without DupMC line.

Is it possible that it is not an h264 stream inside the mp4 container, check via MediaInfo.
And, does the clip play ok in some media player app.

Maybe post MediaInfo log.

EDIT: Posted in another thread (perhaps pertinent to the problem).
Well, I have AMD (Ryzen) + Radeon Vega...

[StvG]

Yes, it looks likes it's related to the AMD CPU.
Dedup x64 (http://avisynth.nl/index.php/AviSynth%2B_x64_plugins) is build with Intel C++ Compiler because of the ASM code (the original ASM is "converted" to ASM x64).
Either some ASM error is triggered or the AMD CPU doesn't recognize instruction(s).