Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 20th April 2021, 16:32   #1  |  Link
abyss616
Registered User
 
Join Date: Dec 2008
Posts: 98
Encoding 1080i clips with x264 incredibly slow

For years I've been capturing video from my DirecTV box using a Hauppauge box at 720p60. I recently decided that I would try to capture at 1080i30 since I recently upgraded my computer which should be able to handle such material.

This is the script I normally use for 720p60:
Code:
v=LWLibavVideoSource("J:\Clip.TS")
a=LWLibavAudioSource("J:\Clip.TS").Normalize()
AudioDub(v,a)
ColorMatrix()
UnDot()
ConvertToYV12()
Trim(373,108160)
When I encode using x264 @ 2500 in MeGUI, I get ~120 fps on the first pass and ~60 on the second. The CPU runs at 99% the entire time.

When I used the 1080i clips in that same script, the result was a very jumpy video, which I somewhat expected. After some research, I decided upon this script:

Code:
v=LWLibavVideoSource("J:\Clip.TS",fpsnum=30000, fpsden=1001)
a=LWLibavAudioSource("J:Clip.TS").Normalize()
AudioDub(v,a)
ColorMatrix()
UnDot()
ConvertToYV12()
Trim(700,218080)
AssumeTFF()
AssumeFrameBased()
SeparateFields()
Spline36Resize(1920,1080)
SelectEven()
That gave me a nice picture and smooth output. The only glitch I would see would be when editing in VirtualDub, when going frame-by-frame, I would see a random frame pop up every now and again, usually after jumping ahead several thousand frames at once.

However, when encoding in MeGUI using the same x264 settings, I was getting ~10 fps on the first pass! This a major dropoff and a dealbreaker (the CPU is running at 99% here too). Is this to be expected? I realize I'm asking Avisynth to do more, but didn't think it would slow down this much. Is there anything I can do to speed this up?
abyss616 is offline   Reply With Quote
Old 20th April 2021, 17:27   #2  |  Link
videoh
Useful n00b
 
Join Date: Jul 2014
Posts: 1,541
Run AVSMeter on your script and you'll see if the script is your bottleneck. You've got some silly stuff in there BTW, but others will surely elaborate on that.
videoh is offline   Reply With Quote
Old 20th April 2021, 18:57   #3  |  Link
abyss616
Registered User
 
Join Date: Dec 2008
Posts: 98


AVSMeter screencap attached.

The ColorMatrix, UnDot and ConvertToYV12 are holdovers that I picked up awhile back. I haven't revisited them to see if they are still valid.
Attached Images
 

Last edited by abyss616; 20th April 2021 at 19:05.
abyss616 is offline   Reply With Quote
Old 20th April 2021, 22:30   #4  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Metropolitan City of Milan, Italy
Posts: 1,765
You're saying you record in 1080i which basically means that you're already on 1920x1080 29.970 interlaced, so why are you doing this:

Code:
AssumeTFF()
AssumeFrameBased()
SeparateFields()
Spline36Resize(1920,1080)
SelectEven()
I mean, why are you separating fields, resizing the fields and selecting the even ones after resizing?

Quote:
Originally Posted by abyss616 View Post
When I used the 1080i clips in that same script, the result was a very jumpy video, which I somewhat expected.
You can't use filters unless they're field-based...


This is what you can do to encode in 1080i NTSC using/refactoring your own script:

Code:
#Indexing
v=LWLibavVideoSource("J:\Clip.TS",fpsnum=30000, fpsden=1001)
a=LWLibavAudioSource("J:Clip.TS").Normalize()
AudioDub(v,a)

#trim
Trim(700,218080)

#Let's go to 4:2:0 planar 8bit
Converttoyv12(matrix="Rec.601", interlaced=true)

#From BT601 to BT709
ColorMatrix(mode="Rec.601->Rec.709", interlaced=true, threads=0, thrdmthd=0)

#Bobbing to 60p to remove dots
Bob()
UnDot()

#Re-interlace to 30i
assumeTFF()
separatefields()
selectevery(4,0,3)
weave()
At this point, you can encode with x264 using the --tff flag which will trigger MBAFF mode.
If the final result is choppy, it means that the fields were somehow not read properly.
Sometimes you need to specify AssumeTFF or AssumeBFF to get them correctly. Anyway, let me know...

Last edited by FranceBB; 20th April 2021 at 22:33.
FranceBB is offline   Reply With Quote
Old 21st April 2021, 02:08   #5  |  Link
abyss616
Registered User
 
Join Date: Dec 2008
Posts: 98
First: thank you for responding. I appreciate the help.
Second: why did I do what I did? Because I don't know what I'm doing! This is my first time working with 1080i material - I looked up a few threads on deinterlacing and that's what was recommended. ::shrug::

Using your edits gives me this in AVSMeter:


... and the resulting video is chaos - tries to give me a seizure, with or without the --tff switch.

MediaInfo says my source media is:
Scan type: Interlaced
Scan type, store method: Separated fields
Scan order: Top Field First
abyss616 is offline   Reply With Quote
Old 21st April 2021, 13:11   #6  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Metropolitan City of Milan, Italy
Posts: 1,765
Quote:
Originally Posted by abyss616 View Post

... and the resulting video is chaos - tries to give me a seizure, with or without the --tff switch.
Well, there's only one way to find out: upload few seconds of the source file so that I can take a look at it.
If it's really interlaced and the field parity is correct, then I guess there's something wrong in the metadata of the source file and the way they're passed to Avisynth by LWLibav.
For instance, if the source file says progressive but it's actually interlaced you would need an AssumeTFF() or AssumeBFF() before everything in Avisynth, otherwise bad things are gonna happen. But in the meantime, try with this:


Code:
#Indexing
FFMpegSource2("J:\Clip.TS", atrack=-1)
AssumeTFF()

#Loudness Correction to -24LUFS
Normalize(0.24)

#trim
Trim(700,218080)

#Let's go to 4:2:0 planar 8bit
Converttoyv12(matrix="Rec.601", interlaced=true)

#From BT601 to BT709
ColorMatrix(mode="Rec.601->Rec.709", interlaced=true, threads=0, thrdmthd=0)

#Bobbing to 60p to remove dots
Bob()
UnDot()

#Re-interlace to 30i
assumeTFF()
separatefields()
selectevery(4,0,3)
weave()
and encode with --tff

if it's still a mess, then try with:

Code:
#Indexing
FFMpegSource2("J:\Clip.TS", atrack=-1)
AssumeBFF()

#Loudness Correction to -24LUFS
Normalize(0.24)

#trim
Trim(700,218080)

#Let's go to 4:2:0 planar 8bit
Converttoyv12(matrix="Rec.601", interlaced=true)

#From BT601 to BT709
ColorMatrix(mode="Rec.601->Rec.709", interlaced=true, threads=0, thrdmthd=0)

#Bobbing to 60p to remove dots
Bob()
UnDot()

#Re-interlace to 30i
assumeTFF()
separatefields()
selectevery(4,0,3)
weave()
and encode with --tff.

If once again is a mess, then I gotta take a look at the the sample...
There are many things that can go wrong with interlaced stuff, but there's no way for us to find out without a sample, unfortunately...

Quote:
Originally Posted by abyss616 View Post
MediaInfo says my source media is:
Scan type: Interlaced
Scan type, store method: Separated fields
Scan order: Top Field First
Well, that looks like a normal TFF, so it would be weird for the indexer not to have passed the field parity correctly unless, of course, the metadata are lying.

Quote:
Originally Posted by abyss616 View Post
why did I do what I did? Because I don't know what I'm doing! This is my first time working with 1080i material
Don't worry. The question for you then is: do you want to preserve interlacing or would you rather like to have the final encode as progressive?
What I tried to do with the script is to deliver an interlaced result, but you could as well go for a progressive output.
Honestly it's up to you.
In my case, working in broadcast (although in a PAL country) I have to go to interlaced, but if you're a home user, you can go to progressive as well.

A few things to keep in mind:

- if you keep the encode as interlaced, you're gonna need less space as interlaced encodes take less space

- it's gonna be up to the decoder to deinterlace or bob-deinterlace for display according to the metadata. Pretty much any decent decoder can easily do that, like any players on Windows (MPV, MPC-HC, PotPlayer etc) or TVs etc.



Anyway, I'm gonna be here waiting for a sample.
Oh and in case you wanna go to progressive, we're gonna use something better than bob() of course xD


Side note about the "Loudness Correction": you have put a "Normalize()" there in your script without anything, so I assumed you wanted to use it to perform some kind of correction, so I put the values there, but if it's not what you wanted to do, then feel free to remove it.

Last edited by FranceBB; 21st April 2021 at 13:15.
FranceBB is offline   Reply With Quote
Old 21st April 2021, 15:24   #7  |  Link
abyss616
Registered User
 
Join Date: Dec 2008
Posts: 98
1) I've changed so many things around it's hard for me to keep track what I've tried, what I haven't, what worked, what didn't, and what the results look like.

2) It seems that demuxing the original TS file into separate audio/video streams with LWLibavVideoSource makes all the difference in the world in terms of encoding speed. The original script you posted encodes at ~8 fps using TS file as video input and ~76 fps using a demuxed h.264 video stream in an MKV container for video input.

3) What does it tell you that I took out everything from the script after ColorMatrix (so, Bob, UnDot, AssumeTFF, etc...) and just used the --tff switch in x264, and the result doesn't look all that bad? No combing effects, no jumping or anything. It's hard to get a great comparison when you're having to go back and forth between the original and output, but I don't think I can tell much of a difference.
abyss616 is offline   Reply With Quote
Old 21st April 2021, 17:09   #8  |  Link
videoh
Useful n00b
 
Join Date: Jul 2014
Posts: 1,541
Take FranceBB's advice and post a link to an unprocessed source clip with good motion. 50MB should be enough. You can use DGSplit on the source to cut it. Otherwise we are all just guessing.
videoh is offline   Reply With Quote
Old 21st April 2021, 18:07   #9  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Metropolitan City of Milan, Italy
Posts: 1,765
Quote:
Originally Posted by videoh View Post
Take FranceBB's advice and post a link to an unprocessed source clip with good motion. 50MB should be enough. You can use DGSplit on the source to cut it. Otherwise we are all just guessing.
Yep. @abyss616 post a sample.
I can't speculate over what could have gone wrong...

Quote:
Originally Posted by abyss616 View Post
3) What does it tell you that I took out everything from the script after ColorMatrix and just used the --tff switch in x264, and the result doesn't look all that bad?
That you didn't mess up the fields 'cause Colormatrix is interlacing-aware?

But honestly, man, post a sample...
If you think that it's private and you wanna keep it confidential, can't you trim a part where there's only a landscape or nature or a part in which there's nobody and just objects? Anything will do...
FranceBB is offline   Reply With Quote
Old 21st April 2021, 18:49   #10  |  Link
abyss616
Registered User
 
Join Date: Dec 2008
Posts: 98
@FranceBB - sent you a PM.

Sorry - I somehow missed the part where you said you were waiting on me to post a sample.
abyss616 is offline   Reply With Quote
Old 21st April 2021, 19:54   #11  |  Link
videoh
Useful n00b
 
Join Date: Jul 2014
Posts: 1,541
Make it available for everyone!
videoh is offline   Reply With Quote
Old 21st April 2021, 23:08   #12  |  Link
abyss616
Registered User
 
Join Date: Dec 2008
Posts: 98
Quote:
Originally Posted by videoh View Post
Make it available for everyone!
What's a free host service I can use? I'm not putting my Google account out for the public.
abyss616 is offline   Reply With Quote
Old 22nd April 2021, 00:38   #13  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 9,631
Mediafire is popular.
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 22nd April 2021, 01:01   #14  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,346
also https://www.solidfiles.com/
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 22nd April 2021, 02:03   #15  |  Link
abyss616
Registered User
 
Join Date: Dec 2008
Posts: 98
https://www.mediafire.com/file/fn81z...Moment.TS/file
abyss616 is offline   Reply With Quote
Old 22nd April 2021, 11:27   #16  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Metropolitan City of Milan, Italy
Posts: 1,765
Quote:
Originally Posted by abyss616 View Post
@FranceBB - sent you a PM.
Gotcha!

Oh! But this is the NCAA! We air this! hahaha
When I first came to Sky Sports in 2017 they were like "and then there's the NCAA" and I was like "what the heck is that?" and they were like "Oh, college stuff" xD

Anyway, I can totally say that it's not your fault: the file is a recording from a CBS live feed, so it's truly interlaced and can be easily bobbed to 59.940fps progressive.
Now the problem is that both FFMpegSource2 and LWLibavVideoSource are broken due to the container which confuses them (they've never been working great with .ts as container) which is why you saw improvements once you remuxed.
Anyway, this is a simple H.264 truly interlaced top field first 29.970i and guess what indexed it correctly? The latest and greatest ffms2? Nope! LSMASH.dll? Nope... The 12 years old indexer DGAVCDecode made by Donald Graft (by the way, he's here, it's "videoh" and he's not a noob, the "noob" thing in his description is a joke, he knows more than you and me combined xD).




So, once again, with kudos to Donald Graft for getting it right when others got it wrong, here is your script, using the 12 years old DGAVCDecode for video and the 15 years old NicAudio for audio (boys, we're getting old):

Code:
#Indexing
video=DGAVCDecode_AVCSource(dga="Z:\2021 One Shining Moment.dga", i420=yes, deblock=yes)
audio=NicAC3Source("Z:\2021 One Shining Moment PID 1100 2_0ch 48KHz 384Kbps DELAY -68ms.ac3")
AudioDub(video, audio)

#Trimming ads
trim(161, 5578)

#Bob-deinterlacing
AssumeTFF()
QTGMC(Preset="Medium")

#Clipping
Limiter(min_luma=16, max_luma=235, min_chroma=16, max_chroma=240)

#Loudness Correction
Normalize(0.22, show=false)


Note: the last two parts (Clipping + Loudness) are optional as you don't have to air anything and you can skip those if you want. Then encode with x264 like you've always done for progressive materials with no flags.

Oh well since we're here, here is the x264 crf 18 encode with the script: https://we.tl/t-rUxYcxDEys

Cheers,
Frank
FranceBB is offline   Reply With Quote
Old 22nd April 2021, 12:13   #17  |  Link
videoh
Useful n00b
 
Join Date: Jul 2014
Posts: 1,541
Thank you, FranceBB, for your kindness. Of course DGDecNV (DGIndexNV + DGDecodeNV) is preferred when an nVidia adapter is available, because one can use the PureVideo double-rate deinterlacer, which is quite effective and fast. The OP was concerned about the speed of his script so perhaps QTGMC may be an issue for him.

Always wondered why the alternative source filters cannot handle a relatively simple transport format. The specs are readily available and the authors are very smart people. Kinda baffling. A pathology of open-source development; multiple developers but no-one in particular wants to step up and fix things?

Last edited by videoh; 22nd April 2021 at 12:23.
videoh is offline   Reply With Quote
Old 22nd April 2021, 12:30   #18  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 9,631
Quote:
Originally Posted by videoh View Post
A pathology of open-source development; multiple developers but no-one in particular wants to step up and fix things?
Perhaps it requires someone who knows how to do it [subtle hint]
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 22nd April 2021, 13:14   #19  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Metropolitan City of Milan, Italy
Posts: 1,765
Quote:
Originally Posted by videoh View Post
Of course DGDecNV (DGIndexNV + DGDecodeNV) is preferred when an nVidia adapter is available, because one can use the PureVideo double-rate deinterlacer, which is quite effective and fast. The OP was concerned about the speed of his script so perhaps QTGMC may be an issue for him.
Oh right, he came here for speed in the first place...
Well, if he has an NVIDIA GPU, then yes, DGDecNV is great.


@abyss616... If you have a modern NVIDIA GPU and your concern is speed, you can definitely use it. I've been using it with my NVIDIA Quadro P4000 for some years now and it works like a charm. It ain't free, though, but you get lots of activation in a licence for a very small fee.
Anyway, if you wanna stick with the free stuff and the script I posted above is too slow, you can replace QTGMC with:

Code:
tdeint(mode=1, order=-1, field=-1, mthreshL=6, mthreshC=6, map=0, type=2, debug=false, mtnmode=1, sharp=true, cthresh=6, blockx=16, blocky=16, chroma=true, MI=64, tryWeave=true, link=1, denoise=true)
which is gonna be faster.



Quote:
Originally Posted by StainlessS View Post
Perhaps it requires someone who knows how to do it [subtle hint]
+1
FranceBB is offline   Reply With Quote
Old 22nd April 2021, 13:51   #20  |  Link
abyss616
Registered User
 
Join Date: Dec 2008
Posts: 98
Thanks everyone -- especially FranceBB -- for the help. My main concern, in regards to the speed, was why it was so slow (exponentially so) when compared to the 720p60 source material I had been using. Plus, the only interlacing/deinterlacing I had done before was on VHS captures that needed to have QTGMC applied to it to get rid of combing. But from what I'm seeing, this doesn't need QTGMC - just AssumeTFF in the script and the -tff flag in x264 and I'm good to go. Sometimes, simpler is better (and faster).

While I appreciate Donald's efforts in the way of his indexer (I am a capitalist, after all), I'll stick with SMASH since its free and demuxing into an MKV container isn't hard to do. My encodes are getting ~113 fps on the first pass, which isn't close to the ~300 fps for 720p60, but it's much better than the ~10 I was getting before. I do have an NVIDIA GeForce GTX 1650 SUPER, so maybe it's something to look into.
abyss616 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 13:21.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.