Possible to encode this faster?

[fabje]

Hello,

I'm encoding now quite a few AVC 1920 x 1080 recordings to 1280 x 720 mkv files.
Right now I'm doing it the following:
First I open de .ts file in DGAVCIndex and find the beginning and ending and cut it, and I demux the video.
After that I rename the original *.ts file to *_source.ts.
I also rename the *.demuxed.h264 file to *.h264. This file is created by DGAVCIndex.

Then I start TSMuxer and open the *.h264 with it and I mux it back to a .ts file.

Then I start Staxrip and select the source filter with the name DirectShow, this triggers FFMPEG. And uses FFDShow and the Haali Media Splitter.

Then I open the *.ts file that I created with TSMuxer.
When the loading is done I pick the audio file that has been created with DGAVCIndex.

Then I cut the commercials out of it and crop it.
I set the resolution on 1280x720 and I start then encoding.

This are the encoding results:
pass1 : +- 100 fps
pass2: +- 50 fps

And I wonder if this could go easier and faster, does someone have any ideas?

[Selur]

not sure if it helps since 100fps and 50fps for 1080p to 720p seems fast:
1. Why are you remuxing the file to .ts ? (why not use the raw files?)
2. Instead of DirectShowSource, better use DGDecNV or FFMpegSource
3. assuming you source has an average datarate of 10000 kbit/s that means with a frame rate of 24fps each frame has an average size of 417kBit since you decode 100 frames per second your hdd has to deliver around 41700kBit/s ~ 40MB/s reading and lets say 15MB writing, so you might want to make sure source and target lie on different hdds to be sure the hdd isn't a factor slowing down the process,..
4. use faster encoding settings ;)

[fabje]

not sure if it helps since 100fps and 50fps for 1080p to 720p seems fast:
1. Why are you remuxing the file to .ts ? (why not use the raw files?)
2. Instead of DirectShowSource, better use DGDecNV or FFMpegSource
3. assuming you source has an average datarate of 10000 kbit/s that means with a frame rate of 24fps each frame has an average size of 417kBit since you decode 100 frames per second your hdd has to deliver around 41700kBit/s ~ 40MB/s reading and lets say 15MB writing, so you might want to make sure source and target lie on different hdds to be sure the hdd isn't a factor slowing down the process,..
4. use faster encoding settings ;)
1. If I don't do that the audio will be out of sync. I always need to open the recording first in DGAVCIndex and cut it in there. If I load the .ts file directly into Staxrip the audio will be always out of sync. It took me hours to figure this out.
2. I can try DGDecNV and FFMpegSource.
3. We have 25fps here ;) And the avg bitrate is around 11500. And the read and write speeds aren't a problem I think because the specs of my disk are +- 65MB/s.
4. These settings are fine.

[Selur]

regarding:
1. I didn't say anything about not cutting, but why remux after that?
2. if the specs of your drive are only around 65MB/s your hdd might be a problem, since writing and reading needs to be done at the 'same time'

[nm]

3. assuming you source has an average datarate of 10000 kbit/s that means with a frame rate of 24fps each frame has an average size of 417kBit since you decode 100 frames per second your hdd has to deliver around 41700kBit/s ~ 40MB/s reading and lets say 15MB writing, so you might want to make sure source and target lie on different hdds to be sure the hdd isn't a factor slowing down the process,..

41700 kbps is only ~ 5 Mbytes/s

[Selur]

DOH, you are right forgot the div 8 go get to bytes :)

[fabje]

Ok I have tested DGDecNV and this where my results:
Pass 1: 74 fps
Pass 2: 63 fps

Pass 2 is faster but not fast enough to beat DirectShowSource with FFDShow and FFMPEG.

[nm]

Pass 2 is faster but not fast enough to beat DirectShowSource with FFDShow and FFMPEG.

And did you also try FFMpegSource?

[fabje]

And did you also try FFMpegSource?
I'm trying but cutting something with FFMpegSource sucks, if I want to make steps of 100 frames or more it uses a load of memory and it looks like Staxrip will crash. After 2 min of waiting a new frame will popup. But then I wanted to make a step of 100 frames and Staxrip crashes. And at this point Staxrip is using 1.7GB of memory.

Ok now I tested FFMpegSource:
Pass 1: 83 fps
Pass 2: 66 fps
CPU Usage: between 50 and 75%
Memory: 3.2GB at the end

Also something wierd in the DirectShowSource version of trimming it these are the values:
Trim(75,36471)
But now with FFMpegSource I needed to use:
Trim(152,72949)

So wtf is that about :S?

[MatLz]

Triming directshowsource correctly is nearly impossible...but something else seems to be wrong with these double funky numbers...

And you need to use the correct formula to calculate the overall speed :
(fpsPass1*fpsPass2)/(fpsPass1+fpsPass2)
So, your last test is the fastest.

[caaraa]

try FFMpegSource
http://www.primeaffiliate.com/track/images/13.gifhttp://www.ohiok.com/img/i59.photobucket.com/albums/g296/edge06/comment/gothic-04.gif

[Midzuki]

If you insist on using DirectShowSource() :

-- be sure to set the parameters "fps=xxx" and "convertfps=true" ;

-- do not use Haali Media Splitter for transport streams ;

[fabje]

Triming directshowsource correctly is nearly impossible...but something else seems to be wrong with these double funky numbers...

And you need to use the correct formula to calculate the overall speed :
(fpsPass1*fpsPass2)/(fpsPass1+fpsPass2)
So, your last test is the fastest.
Maybe the last test was with +- 3fps overall faster with your calculation, but the logfile that Staxrip creates it tells me something else:

First Test Pass 1:
Start: 18:21:19
End: 18:26:53
Duration: 00:05:34
Speed: 109 fps

First Test Pass 2:
Start: 18:26:53
End: 18:38:16
Duration: 00:11:23
Speed: 53 fps

Third Test Pass 1:
Start: 0:02:35
End: 0:16:20
Duration: 00:13:45
Speed: 88 fps

Third Test Pass 2:
Start: 0:16:20
End: 0:34:45
Duration: 00:18:25
Speed: 66 fps

try FFMpegSource
Read the thread before you say something please!

If you insist on using DirectShowSource() :

-- be sure to set the parameters "fps=xxx" and "convertfps=true" ;

-- do not use Haali Media Splitter for transport streams ;
I'm using the parameters "fps=xxx" and "convertfps=true" already ;)

And what else do I need to use then instead of Haali Media Splitter for transport streams?

And an other way to speeds this up is to encode on 64bit, but I have some problems with that as an example I use this avs script:
LoadCPlugin("E:\Staxrip\Applications\AviSynth\plugins\Yadif\yadif.dll")
SetMTMode(5,4)
DirectShowSource("E:\source.ts", audio=false, convertfps=true, fps=25.000)
Crop(0,0, -Width % 8,-Height % 8)
ConvertToYV12()
SetMTMode(2)
Yadif()
LanczosResize(1280,720)
Trim(52,38210)

And I keep getting an error with Yadif and yes all used dll's etc are 64bit versions when I tried this.

[MatLz]

If you feed the same thing to the encoder, that is the only formula to get the overall speed with only speed values.
But since you don't feed the encoder with the same source (script output), that can not work.

[nm]

First Test Pass 2:
Duration: 00:11:23
Speed: 53 fps

Third Test Pass 2:
Duration: 00:18:25
Speed: 66 fps

x264 encoded twice the number of frames with the FFMS script. "Double funky numbers" as MatLz said earlier.

I'd try muxing the source to MP4 or MKV, make sure it plays normally with VLC (for example) and then feed it to FFMS.



Yadif()

Note that with DGDecNV you could get higher quality same-rate deinterlacing (without CPU load, allowing you to use slower x264 settings). Yadif tends to smear details. Both Yadif and GPU filters are less than ideal for bobbing though.

And I keep getting an error with Yadif and yes all used dll's etc are 64bit versions when I tried this.

So where did you download the 64-bit yadif DLL?

[fabje]

x264 encoded twice the number of frames with the FFMS script. "Double funky numbers" as MatLz said earlier.

I'd try muxing the source to MP4 or MKV, make sure it plays normally with VLC (for example) and then feed it to FFMS.




Note that with DGDecNV you could get higher quality same-rate deinterlacing (without CPU load, allowing you to use slower x264 settings). Yadif tends to smear details. Both Yadif and GPU filters are less than ideal for bobbing though.



So where did you download the 64-bit yadif DLL?
So I first need to demux the source .ts file to a mp4/mkv file and then load it into Staxrip?

I downloaded that 64bit Yadif dll from here link (http://yo4kazu.110mb.com/)

[nm]

So I first need to demux the source .ts file to a mp4/mkv file and then load it into Staxrip?

Yes. Normally there should be no need to do that, but we're debugging the problem here. One possible issue is splitting the TS, which is done by Haali Splitter on your system. FFMS uses Haali for TS input, or lavf if Haali splitter is not installed.

So demux with eac3to, tsMuxeR or DGAVCIndex and mux to MP4 with Yamb (or MP4Box on command line). Then load the MP4 file to Staxrip.

I downloaded that 64bit Yadif dll from here link

Ok, I can't test that but maybe someone else knows if it should work. Also make double sure that your E:\Staxrip\Applications\AviSynth\plugins\Yadif\yadif.dll as well as the AviSynth instance are 64-bit versions.

[Didée]

Note that with DGDecNV you could get higher quality same-rate deinterlacing (without CPU load, allowing you to use slower x264 settings). Yadif tends to smear details.
"Higher quality", you say?
Both Yadif and GPU filters are less than ideal for bobbing though.
Not only for bobbing. Also for same-rate deinterlacing. The insufficiencies are more obvious during bobbing. But the general defects are exactly the same.

This is same-rate deinterlacing of a DVB capture:
http://img26.imageshack.us/img26/5295/nvyadiftgmc.th.jpg (http://img26.imageshack.us/i/nvyadiftgmc.jpg/)

Between NV-deinterlacing and Yadif, it's a close call which of them does a worse job. From a "quality" point of view, they're both unacceptable.

If people are concerned about (x264) encoding quality, they should not start out destroying the content with poor deinterlacers.

[fabje]

"Higher quality", you say?

Not only for bobbing. Also for same-rate deinterlacing. The insufficiencies are more obvious during bobbing. But the general defects are exactly the same.

This is same-rate deinterlacing of a DVB capture:
http://img26.imageshack.us/img26/5295/nvyadiftgmc.th.jpg (http://img26.imageshack.us/i/nvyadiftgmc.jpg/)

Between NV-deinterlacing and Yadif, it's a close call which of them does a worse job. From a "quality" point of view, they're both unacceptable.

If people are concerned about (x264) encoding quality, they should not start out destroying the content with poor deinterlacers.
I actually can't see de difference between the three of them.
Witch one do you prefer then?

[sneaker_ger]

Look at the edges of the table and the left part of the couch, the right eye of the woman etc.

[fabje]

Look at the edges of the table and the left part of the couch, the right eye of the woman etc.
Ah I see it now, and how much faster or slower is TGMC then Yadif?

[Selur]

I really like the results TGMC produces, now I just need a TGMC filter in mencoder/mplayer/ffmpeg that also uses the gpu and is fast. :)

[Didée]

I actually can't see de difference between the three of them.
Seriously?

left>right> DGDec - Yadif - TGMC
http://img846.imageshack.us/img846/7738/check1.png

http://img263.imageshack.us/img263/74/check2q.png

top>bottom DGDec - Yadif - TGMC

http://img560.imageshack.us/img560/4893/check4.png http://img148.imageshack.us/img148/8725/check3.png


how much faster or slower is TGMC then Yadif?
Depending on settings, 5 to 50 times slower. :D


Another point is "compressibility". Even if you can't spot differences easily, the encoder clearly sees every single pixel, and has to deal with those differences in all their glory.
Some over-the-thumb numbers: "standard" bobbing often requires 60 to 80% more bitrate than (Q)TGMC-bobbing, at the same quantisation level. "standard" deinterlacing requires 25 to 50% more bitrate than (Q)TGMC.

It all depends on context. For "mass processing", you probably won't have much fun with QTGMC, in particular with HD content. However, the point still stands that you need *more* bitrate, only to encode a file with *worse* quality.

The decision has everyone to make on its own. Still, one must at least be aware of the situation, in order to make a decision. ;)

[Groucho2004]

Have to agree with Didée, nothing even comes close to (Q)TGMC for de-interlacing.
I'd rather use QTGMC with preset "Fast" or even "Faster" than Yadif or NVIDIA GPU.

[fabje]

Ok what I did now is the following:
Demuxed the .ts file with DGAVCIndex
Demuxed the .264 file with Yamb to a .mp4 file.

Started Staxrip and opened the .mp4 file with this code:
FFVideoSource("%source_file%", cachefile="%working_dir%%source_name%.ffindex")
Then I trimmed the commercials out of it etc and started the encoding.

I also needed to use LeakKernelDeint, because Yadif keeps crashing in pass2 at 86%.

This where the results:
Pass 1:
Start: 18:41:26
End: 18:54:40
Duration: 00:13:14
Speed: 92 fps

Pass 2:
Start: 18:54:40
End: 19:17:02
Duration: 00:22:21
Speed: 54 fps

Also when i'm using the preview screen of Staxrip to cut out the commercials I need to push 2 times on the > or < button to go a frame forward or backward.
And because of that the mkv is crap, the audio is also out of sync. You can check the first 10mb of the mkv file here (http://www.fabje.eu/ffms.mkv).

First I want to get the encoding goes right to find the fastest way after that i'm going to find the best deinterlacer ;)