MultiCoreEnc & 2CoreEnc

[Mr_Odwin]

I recently created this for use with FAVC and thought that others may like access to it. Perhaps with a .bat file creation it could help with DVD2SVCD? I'm not a user, but the creation of helpful .bat files seems to be talked about on this board a lot.

File: http://homepages.nildram.co.uk/~pike/MultiCoreEnc.zip

MultiCoreEnc eases the process of splitting Avisynth scripts so that encoding to m2v (mpeg2) may utilise more than one cpu core. It supports HC and QuEnc encoders (my two favourite!) and although it may work for others it's not been tested at all.
MultiCoreEnc adds a 'Trim([Number],Number[])' command to the copied avs files that it generates and then merges the resulting m2v files - that's the top and bottom of it. Be sure that the command line syntax outlined below is followed correctly.

Command Line: (Order is important!)

MultiCoreEnc.exe -c [Number of threads to use ] -e [Full path to encoder in double quotes] -i [Full path to avs file in double quotes] -o [Full path to desired m2v file in double quotes] [Further desired arguments to pass to encoder]

For Example:

"C:\MultiCoreEnc.exe" -c 3 -e "C:\Quenc.exe" -i "C:\Title0.avs" -o "C:\Title0.m2v" -b 8000 -maxbitrate 8000 -dc 8 -1 -nohq -vbr -noscene -notrell -nocgop -nointerlaced -noextreme -gopsize 18 -maxbframes 2 -nocmatrix -aspectratio 4:3 -mpeg2mux noaudio -silent -auto -close

Note: There must be more frames returned by the Avisynth script than cores specified.

[Mr_Odwin]

I created something that should work with anything that works with QuEnc (I tried it with dvd2svcd, dvd rebuilder and favc - it seemed fine). Just rename your original QuEnc.exe to QuEnc_.exe and place the QuEnc.exe from the zip file into the same folder. 2 instances of QuEnc should then run.

http://homepages.nildram.co.uk/~pike/2CoreEnc.zip

[Mr_Odwin]

Just made a quick bugfix to 2CoreEnc where I'd forgotten to remove some MsgBox() commands that I'd been using for debugging.

[Darksoul71]

@Mr_Odwin:
Search for QuEnc^n :)

[Mr_Odwin]

@Mr_Odwin:
Search for QuEnc^n :)

I know about QuEnc^n. It was memories of your threads that lead me to do what I've done. I downloaded your work and liked it. However, I thought the inclusion of extra software for determining the number of frames wasn't necessary. (Correct me if I'm wrong, of course!) With a simple bit of Avisynth work it turns out okay.
QuEnc^n also incorporates some form of more complex bitrate distribution, but I just didn't need that. I made what I made with FAVC in mind which is more about simplicity than squeezing every ounce of quality from something.

[Darksoul71]

Ah, ok, I thought you didn´t know QuEnc^n and coded something similiar by coincidence. Such things happen quite often as I already experienced with my tool VCF2AVS when bb also developed something similiar. We didn´t know from each other. This was really funny :cool:

Yes, it´s true ! I used an external tool to determine the # of frames in the source video. Since AutoIt would have made it quite complicate to access the AVS file via VfW-interface I decided to outsource this to a separate file.

May I ask which "trick" you used in AVISynth to generate the right segment length ?

The bitrate adjusting stuff in QuEnc^n was only coded by me because a few folks pointed out that the resulting M2V segments could have lower quality if encoded with a constant bitrate.

MultiCoreEnc is very small. What language did you use ?
C++ ?

[Mr_Odwin]

It's written in vb.net. (I've not got the skills for anything more complicated.) Full code:


Module Module1

Sub Main()
If Command().ToLower.Contains("-c ") And Command().ToLower.Contains("-e """) And Command().ToLower.Contains("-i """) And Command().ToLower.Contains("-o """) Then
Console.WriteLine("Please wait while MultiCoreEnc processes your command ...")
Dim Commands As String = Command().ToLower()

Dim Enc_Index As Integer = Commands.IndexOf("-e")
Dim Enc_String As String = Commands.Substring(Enc_Index + 4)
Enc_Index = Enc_String.IndexOf("""")
Dim Encoder As String = Enc_String.Remove(Enc_Index)

If Commands.Contains(" -auto ") = False And Encoder.EndsWith("\quenc.exe") = True Then
Commands = Commands & " -auto"
End If
If Commands.Contains(" -close ") = False And Encoder.EndsWith("\quenc.exe") = True Then
Commands = Commands & " -close"
End If

Dim C_Index As Integer = Commands.IndexOf("-c ")
Dim C_String As String = Commands.Substring(C_Index + 3)
C_Index = C_String.IndexOf(" ")
Dim CoreNum As Integer = C_String.Remove(C_Index)

Dim i_Index As Integer = Commands.IndexOf("-i """)
Dim i_String As String = Commands.Substring(i_Index + 4)
i_Index = i_String.IndexOf("""")
Dim AVS_File As String = i_String.Remove(i_Index)

Dim o_Index As Integer = Commands.IndexOf("-o """)
Dim o_String As String = Commands.Substring(o_Index + 4)
o_Index = o_String.IndexOf("""")
Dim M2V_File As String = o_String.Remove(o_Index)

Dim Append As String = Commands.Substring(Commands.IndexOf("-o """) + o_Index + 6)

Dim j As Integer
For j = 1 To CoreNum
IO.File.Copy(AVS_File, AVS_File.Substring(0, AVS_File.IndexOf(".avs")) & "_" & j & ".avs", True)
If j = 1 Then
Dim AVS As New IO.StreamWriter(AVS_File.Substring(0, AVS_File.IndexOf(".avs")) & "_" & j & ".avs", True)
AVS.Write(vbNewLine & "Trim(0," & j & "*Floor(FrameCount()/" & CoreNum & "))")
AVS.Close()
ElseIf j > 1 And j <> CoreNum Then
Dim AVS As New IO.StreamWriter(AVS_File.Substring(0, AVS_File.IndexOf(".avs")) & "_" & j & ".avs", True)
AVS.Write(vbNewLine & "Trim(1+" & j - 1 & "*Floor(FrameCount()/" & CoreNum & ")," & j & "*Floor(FrameCount()/" & CoreNum & "))")
AVS.Close()
Else
Dim AVS As New IO.StreamWriter(AVS_File.Substring(0, AVS_File.IndexOf(".avs")) & "_" & j & ".avs", True)
AVS.Write(vbNewLine & "Trim(1+" & j - 1 & "*Floor(FrameCount()/" & CoreNum & "),FrameCount())")
AVS.Close()
End If
Next

Dim Enc_Process(CoreNum - 1) As Process
For j = 1 To CoreNum
Enc_Process(j - 1) = System.Diagnostics.Process.Start(Encoder, "-i " & Chr(34) & AVS_File.Substring(0, AVS_File.IndexOf(".avs")) & "_" & j & ".avs" & Chr(34) & " -o " & Chr(34) & M2V_File.Substring(0, M2V_File.IndexOf(".m2v")) & "_" & j & ".m2v" & Chr(34) & " " & Append)
Next

For j = 1 To CoreNum
Enc_Process(j - 1).WaitForExit()
Next

Dim Merge As String = "copy /b "
For j = 1 To CoreNum
If j <> CoreNum Then
Merge = Merge & Chr(34) & M2V_File.Substring(0, M2V_File.IndexOf(".m2v")) & "_" & j & ".m2v" & Chr(34) & "+"
Else
Merge = Merge & Chr(34) & M2V_File.Substring(0, M2V_File.IndexOf(".m2v")) & "_" & j & ".m2v" & Chr(34)
End If
Next

Merge = Merge & " " & Chr(34) & M2V_File & Chr(34)
Dim M2V_Merge As New System.IO.StreamWriter(M2V_File.Substring(0, M2V_File.IndexOf(".m2v")) & "_Merge.bat", False)
M2V_Merge.WriteLine(Merge)
M2V_Merge.Close()

Shell(M2V_File.Substring(0, M2V_File.IndexOf(".m2v")) & "_Merge.bat", AppWinStyle.MinimizedNoFocus, True)

For j = 1 To CoreNum
My.Computer.FileSystem.DeleteFile(M2V_File.Substring(0, M2V_File.IndexOf(".m2v")) & "_" & j & ".m2v")
Next
My.Computer.FileSystem.DeleteFile(M2V_File.Substring(0, M2V_File.IndexOf(".m2v")) & "_Merge.bat")
End If
End Sub

End Module


Using Avisynth the trim commands can be:
Trim(0,1*Floor(FrameCount()/3))
Trim(1+1*Floor(FrameCount()/3),2*Floor(FrameCount()/3))
Trim(1+2*Floor(FrameCount()/3),FrameCount())

(This is when three cores are specified.) This saves needing to know exactly how many frames there are and the text can just be appended to the end of a copy of the original avs file.

[Darksoul71]

Hi Mr_Odwin,

thanks for the sources.
It´s amazing how small console apps nowadays are. My old Delphi 5.0 had the tendency to blow up even the smallest code snippt to something > 400 kByte.

I like that AVISynth "trick". This would enable me to get rid of AVIInfoCLI.

Just let me add a few comments:
1) You could make the generated AVISynth scripts much more readable if you "outsource" the calculation of the segment length to a separate variable, e.g.:

SegLen = Floor(FrameCount()/3)

Your segment trim commands will then simply look like this:

Trim(0,SegLen)
Trim(1+1*SegLen,2*SegLen)
Trim(1+2*SegLen,3*SegLen)
Trim(1+3*SegLen,4*SegLen)
Trim(1+4*SegLen,5*SegLen)
Trim(1+5*SegLen,6*SegLen)
Trim(1+6*SegLen,7*SegLen)
Trim(1+7*SegLen,FrameCount())


2) If I understand your code correctly you are merging the M2V-segments via dos binary copy. I´m not shure wether this is really a smart idea. From what I know the mpeg-video has timestamps which will not be correct for the video. This could cause problems with DVD-Masteringprograms. That´s something the MPEG2 pro´s here have to comment. Binary copy works for merging AC3 streams though but for merging M2V segments I would always go the DGIndex route.

My AutoIt code should be pretty readable for you since AutoIt is also very basic-alike:

Func MergeM2V($SourceFiles, $TargetFile)
$D2V = StringTrimRight ($TargetFile,4)
$DGIndex = @ScriptDir & "\DGIndex\DGIndex.exe"
$DGIndex_CL = "-IA=1 -FO=0 -YR=2 -TN=1 -OM=0 -IF=[" & $SourceFiles & "] -DSD=0 -DRC=0 -DSA=0 -OFD=[" & $D2V & "] -EXIT -HIDE"

; Clean out temporary file if they exist from any crashed run
If FileExists ($D2V) Then FileDelete ($D2V)
If FileExists ($TargetFile) Then FileDelete ($TargetFile)

If ($Debug = 1) Then AddLog ($MyLog, "Commandline for DGIndex: " & $DGIndex_CL)

RunWait($DGIndex & " " & $DGIndex_CL, "", @SW_HIDE)
FileDelete ($D2V & ".d2v")
FileMove ($D2V & ".demuxed.m2v", $TargetFile, 1)
EndFunc


3) For compatibility reasons I think it´s always better to fully mimic the CLI of the wrapped
encoder and let the user configure stuff like # of threads used via ini-file. This ensures maximum
compatibility with other applications calling the wrapped encoder. All wrapper specific settings
either within the CLI or any used config file are problematic (as seen with AQE and the settings
for threads, QMatOp, etc).

As you stated yourself:
Perhaps with a .bat file creation it could
help with DVD2SVCD? I'm not a user, but the creation of helpful .bat
files seems to be talked about on this board a lot.


I think a cool thing would be a combination of a frontend similar to MultiCoreEnc, QuEnc^n, AQE^n, etc.
with a encoder indipendent configurable interface such as DIKO uses:
http://www.vmesquita.com/forum/index.php?topic=4377.0

Unfortunately I´m all out of time for coding such kind of stuff !

Later,
D$

[tamahome]

2CoreEnc works perfectly for quenc

I awaited a functional program of this type since last the version of quenc veiled me charmed

thanks you :)

[Adub]

is there a version of 2CoreEnc or something like it that will allow the use of multiple instances of HC with DVD rebuilder?

[Darksoul71]

DVD Rebuilder Pro does allow multiple instances of HC and for the free version of DVD there is DualDVDRB.

For DualDVDRB look here:
http://forum.doom9.org/showthread.php?t=97961

[Adub]

Thanks Darksoul71. I heard about this somewhere, but couldn't find it.

[Darksoul71]

You are welcome ! :)

[tamahome]

exist there a version compatible with hcenc ?

[Adub]

The pro version of DVD Rebuilder supports HC if that is what you want. DualDVDRB should as well I believe.

[MrC]

I have tried MultiCoreEnc with HCenc (0.21) and a dual core processor. The first thread (avs file lasts with _1) is correctly completed but the second one (lasts with _2) gets this error from HCenc:

ERROR, can't write to dbs file, aborting...

No problems instead with QuEnc.

Any idea?

:)

Bye
________
Triumph Speed Twin (http://www.cyclechaos.com/wiki/Triumph_Speed_Twin)

[Darksoul71]

@MrC: The problem is related to HCEnc. HCEnc always writes the 1st pass information into a dbs file in the local HCEnc directory. When you spawn a second HCEnc from the same dir they conflict.

I solved the problem with HCEnc^n by copying HCEnc in a separate dir for every instance per core which I spawn.

Regards,
D$

[MrC]

@MrC: The problem is related to HCEnc. HCEnc always writes the 1st pass information into a dbs file in the local HCEnc directory. When you spawn a second HCEnc from the same dir they conflict.

I solved the problem with HCEnc^n by copying HCEnc in a separate dir for every instance per core which I spawn.

Regards,
D$

Thanks a lot for the hint!

:thanks::cool:

Bye
________
VZR1800 (http://www.cyclechaos.com/wiki/Suzuki_VZR1800)

[Mr_Odwin]

What he said. Hank has said that the next version of HC will support a cli/ini argument for dbs placement so this will no longer be an issue.
Note that HC in 1 pass mode (quality/quant) will not result in an error as no dbs is written.

[MrC]

Let's wait for 0.22 then.

:thanks:

Bye
________
Honda CN250 (http://www.cyclechaos.com/wiki/Honda_CN250)

[Darksoul71]

@Mr_Odwin:
Good to hear :)
So I can simplify HCEnc^n

@MrC:
Or use the current HCEnc version plus
HCEnc^n (http://forum.doom9.org/showthread.php?t=130028) :)

Later,
D$