File size increases when recompressing Huffyuv file

[StephLG]

Hi,

I have a 22Gb NTSC (720x480@29.97fps) avi file captured using Huffyuv codec. When I process this file through an Avisynth script (which does IVTC to 23.976 fps) in VirtualDub, I notice that, using the same Huffyuv codec, the resulting avi is always bigger than the captured one. I thought it should be smaller since IVTC removed 1 frame every 5 frames. My 22Gb file turns to a 28Gb file. I selected 'Direct stream copy' for audio and 'Full processing' for video (with Huffyuv set to its default best gradient). Am I wrong with these settings? Should I choose 'Fast recompress' in Video or something like this?

Thanks for any input.

Steph.

[hakko504]

The captured file is probably in YUV mode, whereas VD convert it to RGB and then recompress it with Huffyuv in RGB mode. YUV only stores 16bits/pixel where RGB stores 24 bits/pixel. At the same compression ratio in both modes, you would expect the RGB file to be ~50% larger. Add the fact that you remove 1 frame in 5 this means that the new expected filesize would be 22*(4/5)*1.5~=26.5GB. So getting a 28 GB file is really within what you could expect from using VD's full processing mode, since it is only possible to process RGB data in this mode! I don't remember if you can do IVTC with VD if you use the 'fast recompress' mode, but why not use AVIsynth for doing the IVTC? Then you could use VD's fast recompress to create the new AVI and keep the data in YUV mode all the time. If you dl AVIsynth and Decomb from Doom9 (http://www.doom9.org) then you can use this script for doing IVTC:Loadplugin("decomb.dll")
SegmentedAVISource("file.avi")
Telecide()
Decimate(cycle=5)
If you are going to re-encode the file to something else later you could save some space (~20GB) by feeding the .avs directly to the encoder. hint:trim can be used to cut out the portions of the clip you want to use.

[StephLG]

Hi hakko504,

I thank you for your answer. As I was getting no reply last night, I did experiment a little bit myself and I did exactly what you say: IVTC in Avisynth and 'Fast Recompress' in VD. My 22Gb source file gave me a 17Gb resulting file. That's what I was expecting. I was doing IVTC in Avisynth since the beginning but I was not aware that VD was converting the YUV format to RGB when using 'Full processing' mode. Thank you for the tip.
As for saving hard drive space, I think feeding the encoder with an avs script would be too complicated since I'm using CCE 2.64 and my source file will be divided in 3 parts (22Gb+22Gb+6Gb captured from Laserdisc). I plan to trim, IVTC, crop/resize each part then join them in a final 40Gb (max) file to give it to CCE. It will be space consuming but I'm not sure using Vfapi would be easier and faster for a 4 pass VBR encoding.

Thanks again,
Steph.

[hakko504]

Actually joining files in AVIsynth is very easy, just use '+'! Your final script would be something like this:LoadPlugin("Decomb.dll")
video1=AVIsource("file1.avi")
video2=AVIsource("file2.avi")
video3=AVIsource("file3.avi")
video11=video1.trim(0,10000)
video12=video1.trim(11000,0) #remove frame 10001 to 10999 from first file
video21=video2.trim(0,50000) #remove some frames from the end of the second file
video31=video3.trim(100,10000) #Just keep the middle part of the third file
video=video11+video12+video21+video31
video.Telecide() #first part of IVTC
Decimate(cycle=5) #remove duplicates
Crop(8,0,704,480) #cut sides
BicubicResize(480,480,0,0.5) #soft Resize to NTSC SVCD size!
This way you would only get file to feed CCE with and you only need to run VD once.

[StephLG]

Thank you hakko504 for this Avisynth course; it is very clear!

I was not sure I could perform all those operations on the 3 files in one script; I guess I have to think in a non-linear editing way ;)

The other doubt came from Decomb; I was afraid that processing the IVTC on the 3 parts in one pass (once joined) would be hazardous because the 3 original files come from 3 capture sessions and I will trim each file a little bit (as your script does). I thought Decomb had chances to go wrong if the IVTC pattern changes between 2 files (and it will, for sure).

Thank you again,
Steph.

[hakko504]

Originally posted by StephLG
Thank you hakko504 for this Avisynth course; it is very clear!

Thank you again,
Steph.
You're welcome :)
Originally posted by StephLG
The other doubt came from Decomb; I was afraid that processing the IVTC on the 3 parts in one pass (once joined) would be hazardous because the 3 original files come from 3 capture sessions and I will trim each file a little bit (as your script does). I thought Decomb had chances to go wrong if the IVTC pattern changes between 2 files (and it will, for sure).
You have a point: Decomb will fail if you do cut things from the movie. It will however fail a lot less than most other adaptive IVTC's! On the other hand, you said you captured the files from LD and I assumed that each file corresponded to one side of the LD. In this case it is likely that if you find the exact cutting point between the sides of the LD, you won't have any problems with the IVTC. On the other hand, you can never be sure that the IVTC pattern doesn't change within the film itself(!) and in this case Decomb will of course fail.

If you do want to do IVTC before trimming then the script should look like this:LoadPlugin("Decomb.dll")
video1=AVIsource("file1.avi").Telecide().Decimate(cycle=5)
video2=AVIsource("file2.avi").Telecide().Decimate(cycle=5)
video3=AVIsource("file3.avi").Telecide().Decimate(cycle=5)
video11=video1.trim(0,8000)
video12=video1.trim(88000,0)
video21=video2.trim(0,40000)
video31=video3.trim(80,8000)
video=video11+video12+video21+video31
video.Crop(8,0,704,480)
BicubicResize(480,480,0,0.5)A point is that AVIsynth only produces frames on request by the encoder and can cache some frames so it will not actually do IVTC on more frames than the ones needed for each segment plus 5 (cycle) prior to the first one in each segment. Do remember to check the trim values, do not just reduce the number by 20%, you have to make sure which frame has been dropped and adjust the framenumber by ±1.

Good Luck!

[Jeff D]

StephLG or hakko504,

how well does the 3 segment code work? I'm in the process of moving some 9 disc laserdiscs over to dvd and this was a big question. Do I try to get one large 2hour capture to work without losing frames, or do I capture each disc independantly?

My first pass was the 1 cap per disc, that's done. I'm now working around with vdub and avisynth to figure out which does a better job.

My first test in avisynth did the trim BEFORE the telecide and decimate, which I was always curious if that would be a problem. I can't see how it would NOT be a problem.

I'll try hakko504's example tonight and see how that does.

I also wanted to do some 3d convolution flitering to get rid of garbage in the source, for example in the opening scene of one of the movies two space ships are flying over a planet, the planet has black flecks all over it. I was thinking a convolution3d filter would work good for this. But I couldn't find a conv3d that worked with avisynth 2.5 in YUY2 format, only YU12.

Would there be any other options for removing the "garbage"?

[hakko504]

Originally posted by Jeff D
StephLG or hakko504,

how well does the 3 segment code work? I'm in the process of moving some 9 disc laserdiscs over to dvd and this was a big question. Do I try to get one large 2hour capture to work without losing frames, or do I capture each disc independantly?
I've never had any problems of doing things like that. I once made four captures, cut individual scenes from each capture ( between 15&25 scenes/capture) and put everything back together at the end. Works like a charm.

My first test in avisynth did the trim BEFORE the telecide and decimate, which I was always curious if that would be a problem. I can't see how it would NOT be a problem.
If you do the trime before telecide you may end up with a different patternmatching than if you do the trim after telecide. If you are using guide=1 this may result in incorrect patternmatching if you trim before telecide.
I also wanted to do some 3d convolution flitering to get rid of garbage in the source, for example in the opening scene of one of the movies two space ships are flying over a planet, the planet has black flecks all over it. I was thinking a convolution3d filter would work good for this. But I couldn't find a conv3d that worked with avisynth 2.5 in YUY2 format, only YU12.

Would there be any other options for removing the "garbage"? FluxSmooth? Check the FAQ at Avisynth.org (http://www.avisynth.org/index.php?page=Section+3%3A+Filters+and+colorspaces#q3.4)

[Jeff D]

hakko, I'm posting a zip with this, maybe you can look at it.

I'm going to add, I'm new to this stuff, but I did fool around with tweaking the parameter with both fluxsmooth and tbarry's stmedian filters both failed.

The zip includes three sequential frames you can see garbage black spots on the planet's surface.

The only time I was able to get something close to removing the spots was cranking the values on fluxsmooth up to around 50. And that was a smaller black spot in frame 519. Are the large spots too big?

[Belgabor]

Attached zip file seemed to be invalid, please reupload.

[Jeff D]

Ok, I tried to upload the file again. It's only 3 24k images, maybe I should try them as seperate files...

While I'm posting. One quick question about YUY2 and captures. When I did my capture my desktop was 16bit (which I understood had no effect on the capture) huffyuv wants 24bit input, so I assumed the settings would be made for me, since I never saw a setting for source bitdepth.

There are many time in the captured file where the banding exists all over the image. It's really bad. Did I do something wrong in the cap too? I saw nothing in the ATI drivers that could be used to set this.

Just to compare I checked a DIVX rip of ep6 and damn it looked good. 25fps, pal obviously. It looked 100x better than my cap.

[Jeff D]

Just to update the obvious points I didn't make about viewing the cap in the previous msg...

desktop now 32bit for viewing, and vdub was used to view the captured avi.