CCE Encoding speed?

[GGuyZ]

Hi everyone.

i have a Pentium D 3.5GHz system, with 2x512 dual channel RAM, and a 160GB SATA II harddrive.

I remembered CCE being quite fast, but for some reason when encoding a 2 hours movie previously encoded with Lagarith, CCE only works at 0.42x speed. Which means that a 3 pass vbr process will take 20 hours to complete.

Is it supposed to be that slow or am I doing something wrong?

With Regards,
Guy.

[buzzqw]

are you using avisynth ? so what script ?
or feeding cce directly with lagarith ? (maybe hd issue... try outputting in another drive)

give more info...

BHH

[GGuyZ]

No AviSynth script is used. I'm feeding the Lagarith AVI file directly into CCE.

I'm afraid I don't have another HD to try. only this one. but it seems to perform ok in other operations. As I recall, when I tried short clips in CCE it didn't have a problem to perform at 1.xx X speed.

By the way, checking my cpu usage - I only get around 50% usage on both cores. But it's not like one core is maxed out and the other isn't doing anything(like on most programs that don't support dual core). Both cores seem to be working, but just far from their maximum.

Some more details: The video was originally captured to DV using Canopus ADVC110. Then I filtered it using Vdub and saved the file with lossless Lagarith. And now I'm trying to reencode as MPEG2(for DVD burning in the end).

I hope this is enough info. if not please tell me what more I could provide.

[buzzqw]

i don't know ... sorry

i used both lagarith and cce but my file was no more 1gb... i think your problem maybe caused to paging. I think lagarith isn't feeding cce correctly

dumb question: are you using latest versione of lagarith ?
also: have you checked "Always suggest RGB..." ?

If you have installed Avisynth try this:
create a file temp.avs and write on it

avisource("movie.avi") # or name of your lagarith file.

and feed cce with this file

hope better news !

BHH

[GGuyZ]

Thanks anyway for your help :) I appreciate it.

Yes, of course I use the latest version of Lagarith.

Lagarith is lossless, so it's supposed to produce very big files. 1gb sounds like a little to me, were you creating a short clip?

I tried the avisynth tip you offered already, and it didn't help really. On the contrary - it made things even slower.

I tried playing some more with CCE. it seems that the encoding speed is dependant on the File Size. The lagarith file is 70gb so I get a 0.4x encoding speed.
I tried encoding the original DV file which is 30gb, and I received a speed of 1.6x.

Both cases are still extremely slow for my processor.

Can anyone please try and help?

Is there a way to make CCE use the entire 100& of my CPU? because it's far from it. And that might speed things up...

[buzzqw]

oh yes... i used a small capture, just some minutes !

one more test with avisynth:

1) try with MT (http://forum.doom9.org/showthread.php?t=94996&highlight=MT%2A)
2) try with setmemorymax(256)

best wishes

BHH

[Trixter]

No AviSynth script is used. I'm feeding the Lagarith AVI file directly into CCE.

If both cores aren't past 50%, it's very possible that your hard drive isn't fast enough to stream data to CCE to get it past what you're seeing. I experience this all the time on my broadcast rig: The uncompressed data requires no decompression, and it's only a 2.6GHz single core, yet CCE only stresses the CPU 80% because my hard drives can't quite feed 30GB per second (the datarate of my video) to the CPU.

[GGuyZ]

Thank you both :)

buzzqw:
I will try that. but it seems to me that the Multithread part is only for filters. I don't need any filters I'm afraid anymore, just to encode to mpeg2.

Trixter: It's more like 70% usage on 1 core and 30% on the second core.
Regarding the HD being the culprit - how could that be? My HD is SATA II(3GB theoretical rate). It's a pretty fast HardDrive. And the fact is - 0.4x is a speed rate people had gotten like a decade ago. I don't expect it to be super fast and I'd be extremely happy even 2x(although I should be getting even more than that).
I would say that the HD is faulty maybe? But it seems to work fine..

[bonehead66]

"can't quite feed 30GB per second (the datarate of my video) to the CPU."

I'm not surprised the average sata II hard drive can only sustain a maximum continuous data transfer rate of about 80 - 120 Mb's per second. 3Gb metioned by GGuyZ is more like it but that is the maximum to host.

On my 4800 x2 dual core my CCE speed averages about 4 - 5 using about 75 - 80% cpu so you should be seeing speeds reaching up to 3.5 - 4 with your cpu but even at that speed it will still take a while with the file sizes you are using.

[GGuyZ]

Bonehead66 - the speed dictates the time of operation. If I'd be getting 3.5-4x speeds like I'm supposed to, then the operation will only take about 4 hours to complete. right now at 0.42x it takes 21 hours!

0.42x is far far from being 'normal'. so something is seriously faulted here.

It seems that the encoding depends on the encoder type rather than the file size.
For example, with lagarith encoded files(even a 1 minute short clip) I'm getting 0.42x speed no matter what in cce

I tried a clean windows install. With windows default DV codec, I was getting a 4x speed cce trying to encode DV files! After installing panasonic dv codec the speed was reduced to 2.5x
Lagarith is still 0.42x even after formatting and reinstalling windows.

So Obviously my pc likes some encoders more and some less. But why is that? and how can I fix it?

[bonehead66]

All I can tell you is CCE is the fastest encoder on the planet, and I don't think the problem lies with your HD Although I have read about Lagarith I have not actualy used it myself and it's been some time since I worked with raw video data and such huge files (thanks to the dvd handycam)
so I'll leave the answer to someone with a bit more know how ie: someone whoe uses Lagarith and CCE in combination. Sorry to have been no help whatsoever. :D

[GGuyZ]

It's ok, Thanks for trying anyway :)!

Could any of the gurus(or anyone else with advice) jump in this thread and try to help me with this please?

Have a good day everyone!

[therat]

can you post more details about the video file please eg.

The original video before Lagarith

Aspect Ratio
Format - NTSC or PAL
Fps
Resolution
File size

and then the same info of the video file produced by Lagarith that you are feeding to CCE.

What Version of CCE?

cheers

[GGuyZ]

Original video is a DV capture from VHS:

Aspect Ratio - 1.250(FAR, DAR), 1.000 PAR
Format - PAL, interlaced, eventhough Gspot doesn't mention if it's either interlaced/progressive.
Fps - 25. Gspot doesn't mention how many fiields/sec exist for some reason(should be 50 I guess).
Resolution - 720x576
File size - 27.8gb

After conversion to Lagarith lossless(YV12-RGB when loading in Vdub and then lossless conversion to Lagarith - still RGB), I get the EXACT same reading in Gspot for the file specs.

The only difference is the file size which is 70.5GB

EDIT: Version 2.70 of CCE SP(trial).

[therat]

What format was the original video eg High Definition and what bitrate?.

I don't understand how a 2 hour movie could be over 27 gigs (unless it is HD). Also why convert it to AVI and then back to DVD (MPEG2) format, why not go from the original to DVD.

Compliant DVD format is 9mbits. Compressing a 70 gig AVI which must have an extreme bitrate to be that size, to fit on a 4.4gig DVD using CCE with 3 passes is going to take time, a lot of time.

If the movie was recorded with a HDTV card then it should have been recorded in SD format if you intended to put it on DVD because SD -> DVD is dead easy and quick and will give better quality than HD -> DVD.

If this isn't the case then I'm all out of ideas.

cheers

[GGuyZ]

Thank you therat,

The original footage was Analog VHS tape.

The process is this: Analog to digital(dv format), which is 13.5gb/hour. DV is somewhat lossy and of course compressed to some extent.

Then I use Vdub to filter the movie(noise reduction) and save it to a lossless format, in this case Lagarith, so I won't have any further decrease in quality. I don't go directly to MPEG2 because 3 pass mpeg2 will take way too long with filters.

It doesn't really matter what the source is. Lagarith is lossless and these type of encoders tend to produce huge files. especially since the movie is noisy(VHS...).

Anyway. The last process (Lagarith AVI-->Mpeg2 using CCE) is extremely slow - 0.42x speed in CCE. I get faster encoding times with files encoded in other lossless formats first, but I have some lagarith movies I need to convert too. Besides, it seems that I'm getting slow encoding times(less than 2) with other lossless encoded files as well.

So anyone, please, why does CCE speed depend so much on the previous file format, and why am I getting such slow speeds, which are far than being normal.

[buzzqw]

have you tryed with other encoder ? like HCenc or Quenc ?

also, try with xivd encoder at quant 1, no b frames, to create the avi to feed the encoder.

BHH

[GGuyZ]

xvid is lossy, and I don't want to go through 2 different lossy conversions. :/.

I have tried tmpgenc and got the same results.

[Trixter]

Trixter: It's more like 70% usage on 1 core and 30% on the second core.


Did you enable Use Multithreading in Lagarith's config box? If so, did you try DISabling it? I'm curious which is faster.

[PhillipWyllie]

Have you tried capturing your VHS video using HuffYUV( using YUV2 as the colourspace). I get ~1.2 when encoding 720*576 @25fps. Any filters(in VDub or Avisynth will slow down things, and they don't utilise dual cores). HDD speed (sustained data transfer rate) seems to the most important aspect (other than processor speed). This leads on to that file size (compression) is important. Trying uncompressed YUV2 may seem the best but HDD can't keep up so CCE speed decreases dramatically. Lastly CCE works in YUV2 and will do a conversion( that's why I'm suggesting YUV2). In a little test I got 1.3 with CCE doing the colourspace conversion and 1.41 with avisynth doing the colourspace conversion

Specs: Athlon XP 2600+, Win XP Pro SP2, 1*120Gb 2*80Gb HDD

[Trixter]

In a little test I got 1.3 with CCE doing the colourspace conversion and 1.41 with avisynth doing the colourspace conversion

The drawback on doing the colorspace conversion with avisynth is that you lose precision. See http://forum.doom9.org/showthread.php?p=871587#post871587 for some tests on the subject.

[PhillipWyllie]

The only conversion( I think) CCE does is RGB to YUV2. Any other(except UYVY) is done by the system codec(divX in my case). I found another thing about L2 cache(which apparently is very important), Win XP only uses 256. When I changed this using XP Smoker to 512 I got a 0.05 increase, maybe others could try this

[Trixter]

The only conversion( I think) CCE does is RGB to YUV2. Any other(except UYVY) is done by the system codec(divX in my case).

That's right, CCE will first try to decode YUY2; if it is unsuccessful, it will then try RGB. In the case of an RGB source, it will have to perform colorspace conversion. When doing the conversion, if you specify in advanced options you are using an 8-bit DCT, there is loss during the conversion because fractional values are rounded off... but there is less if you choose a 9-bit DCT, and hardly any if you choose 10-bit DCT.

I found this out when encoding a 0->255 luma gradient with CCE; with avisynth doing the colorspace conversion, or with CCE doing it into an 8-bit DCT, there was loss. When CCE did the conversion into a 9-bit DCT, there was less banding; when into a 10-bit DCT, there was no banding at all (when played on a set-top DVD player connected to a television). I'm making a commercial DVD with CCE right now and I'm feeding it RGB and choosing a 10-bit DCT, and the results are great.

[Blue_MiSfit]

I am very, very, very happy.

Using CCE SP, and the following AVS script:

SetMTmode(2,0)

qtinput("C:\Documents and Settings\dmf\Desktop\PAL Project\workout 3 - cardio strength.mov", audio=false)

bob(height=576)
bicubicresize(720,576)
ConvertFPS(50)
SeparateFields.SelectEvery(4,0,3)
Weave()


Source is a ~9GB QuickTime MOV that contains NTSC DVI. I am doing a PAL conversion, and outputting 3 pass VBR MPEG-2 with CCE.

I am running on a MacPro running Boot Camp with XP Pro. 4x2.66 GHz cores and 2 GB of FB-DIMM DDR2 533. The drive is a standard SATA drive, and the SATA drivers are correctly installed (had to make a special XP disc for this to work... fracking Apple..)

I am getting 100 fps. :D :D :D :D :D :D!!!!! To be exact, 3.95 - 4.05x realtime!

Thank you AviSynth!! Thank you CCE!! Thank you MT!!

I think this could be further improved, as I am currently seeing 60-80% utilization. With no multithreading it was only 20% or so.

~MiSfit

[lilhobo]

if u look a my thread in dvd rebuilder forum, i can only get 0.70 speed for a DVD-9 to DVD-5 conversion. On a 3Ghz P4 D ?? (486), with 1.5 dual channel RAM

i think it could be coz the motherboard setting to limit the CPU to maintain low temperature. I will need to go in and test this, coz CCE does tend to crank up the CPU use

[PhillipWyllie]

With my new specs I've been doing some more tests. Running an avs script I get ~1.7(as it doesn't take advantage of dual cores). Lagarith slows things even more. The surprising thing is is uncompressed YUY2 I get ~2.25 slowing down to ~1.2(must be my IDE HDD). Using the SATA drives I get ~2.5 which is fairly constant. Configuring the 2 SATAs in a Raid 0 array (HD Tach reports an average of ~110Mbytes/s) I get ~1.7. So my advice is to use uncompressed YUY2 and split your project up into several scenes. I can overclock by 25% quite easily and that doesn't increase things at all(so my bottleneck is still HDD speed)

Edit: I've tried an IDE Raid controller card and I've got a steady ~3.5!

[kolak]

Uncompressed is the best source for CCE, but you need lots of space and really fast array.
Other solution is to have source as a Canopus HQ codec, which is very fast, compressed (about 4-6 times) and very good quality.
Even with dual core 1.66 Ghz you will get 3.5 RT. With 2.9 Ghz it should be about 6-8 RT and with 2x dual core 3.0 GHz Xeons it's between 9-11 RT:)

[dvdboy]

Hope it's ok to jump onto this topic, but I'm also having speed issues with CCE SP.

I'm running a Dual Xeon, 3.06Ghz, 2Gb RAM, WindowsXP Pro SP2.
My source and destination is a Striped U320 RAID, and I'm using Huffy AVIs (Dialogue box says Huffy 2.2.0, About Box says 2.1.1 on the titlebar and 2.1.2 on the About Information...)

My Huffy's are configured to always suggest RGB, which is set to convert to YUY2, with YUY2 set to 'Predict Median'.

On a 2-pass CBR encode (4Mbps) I seem to get between 0.94 and 1.22.

Any suggestions?

[PhillipWyllie]

Are you using the same RAID 0 as the source drive and the destination drive? If so that's not so good. Also I'd try uncompressed YUY2 as your source( since you have a RAID 0). HUffyuv won't take advantage of your dual Xeons(it's a single thread thing). You're getting the same speed I got with Huffyuv on an Athlon XP 2600. Lastly, giving CCE RGB may give better results but the RGB -> YUY2 conversion it has to do is quite a speed hit.

[kolak]

On dual xeons 3.06 you should have 4-5 times faster than RT.
Don't use Huffy. Use Canopus Hq codec or Cineform or the best uncompressed, since you have fast Raid.

[dvdboy]

Thanks Guys, I will give that a go.

[Crabba]

I would've started a new thread, but this one fits my problem so I'll just try and post in here instead, hopefully someone has a solution for us...

I've been having problems with CCE SP encoding really slow for me lately for some reason. I used to be able to do 3-4 passes done in real-time, but now I can't get it anywhere near those speeds, right now about 2+ hrs for a 40 minute project or about ~0.3x or something...

I've been trying like crazy to fix this problem the last couple of days, reinstalling xp, different drivers, even installing my new SATA 300gb HD, but nothing has helped so far!

I have noticed some weird stuff though while testing:
* When pulling up the task manager while encoding I noticed that CPU Usage is only running between 0-5% (!!)
* I also noticed that apparently it only has a problem with my Lagarith and/or videos saved as "Uncompressed" with VirtualDub, because if I only use my original unedited DV files I get between 80-90% CPU usage all of a sudden and ~3.5x speeds!

Now, I really want to use a lossless codec for my edited video, to keep quality as high as possible, and I KNOW that I've used the VirDub Uncompressed format plenty of times before when encoding video with CCE without any problems, so why now?! Judging by the other posts it seems I'm not alone in having this strange problem either...

I'd be very greatful for any help on this!

[PhillipWyllie]

I suspect you're using uncompressed RGB24, which is not to good for speed. I have a SATA drive and when using uncompressed YUY2 on it I get ~3.5. Also I've found that Lagarith is very slow.

[Crabba]

PhillipWyllie: Yes, I suppose I am, since "Uncompressed RGB/YCbCr" is the only uncompressed option there is in VirtualDub, so how would I change it to use YUY2 or even check which color format the file uses? Are you also getting the same problem using Uncompressed RGB, with speeds at about 0.3x and ~2-10% CPU usage?

Also, the weird thing is that after even more testing over the last couple of days, I've realized that it's not even consistently slow! Not only does the speed differ quite a bit just from starting encoding, stopping it and then starting it again right after (!) but something even stranger: If I start an encode IMMEDIATELY after restarting the computer, I suddenly get speeds of 3.4x or something like that (!!) and CPU usage of 85-90% or something (with Raw files!) which is pretty identical to the speeds I get with standard DV files... AND, get this, if I right after try another encode of the exact same project, it's right back to sloooow again, like 0.3x and 0-10% cpu usage!

This is really driving me nuts, and I can't see any logic to it whatsoever. I have now also tried with several different versions of CCE, but with the same result. I just can't see how this isn't a really big bug in the CCE program, but I just havn't noticed until now because I might not have used as many raw files as I do in this particular project.

At least I CAN get it to encode at high speeds now finally, even if it's a real hassle to do a restart every time I want to start encoding something...

[PhillipWyllie]

Under Video/color depth... you can set the compression colour space. If I try uncompressed RGB24 I get around the same speeds as yopurself(HDD too slow). The best is uncompressed YUY2(no decompression and no colour space conversion). You'll notice that decompressing video is usally a single threaded affair and CCE's speed come from that it's optimised for use with multi processors.

[Crabba]

Thanks PhillipWyllie, I couldn't let it go after reading your last post, and noticed that the setting was right there in VirtualDub. I can't believe after all these years of encoding with VirtualDub, I havn't noticed that until now :eek:

Unfortunately though, I did some testing by encoding some of my original DV files to Uncompressed YUY2, but for some **** reason that didn't have any effect on the speed either! :confused:

Are you sure that your speed issues with RGB has to do with your HDD speed? It doesn't make much sense to me how the difference in size between uncompressed RGB and YUY2 would make such a huge difference in speed. Besides, if HDD speed had been the problem it wouldn't suddenly be able to encode my Uncompressed RGB files at full 3.4x speed just because of a Windows restart (which doesn't really make much sense at all anyway really).

[PhillipWyllie]

If you could post your log file(under options/outputs in CCE select logfile). In relation to HDD speed the important factor is how well your HDD can sustain the data transfer(try using HDTach). Have you tried compressing a short clip of your DV files using DivX or Xvid and trying them(making sure they are at the right size). Also could you post your specs.

[seunosewa]

Try lagarith lossless instead of uncompressed.