Expression Encoder 2 help / suggestions?

[Koadic]

Ok, here is some background. When I back up my bluray movies, I convert them to a dvd5/9 size vc1 mkv w/ ac3 audio (usually 5 for 720p, 9 for 1080p). I have chosen vc1 for a couple reasons, I can convert the audio to 5.1 wma pro and create a wmv that I can stream or burn to a disc and play in the xbox 360, or I can convert to an AVCHD disc with 5.1 ac3 that plays in the ps3 if I want to take it to my buddies place, all with minimal conversion after the initial encoding.

All this works just fine as it is, but I started messing around with Expression Encoder 2 and found that it gives me a better looking encode in many cases vs TMPGEnc Xpress 4. But I am specifically shooting for a dvd5/9 size (including m2ts overhead for AVCHD), and the encoder misses the desired bitrate on every 2 pass vbr encode I have done so far.

Now this is fine if it is a little low, but when I spend 15-20 hours on an encode and it ends up being 250kbps (225 MB for a 2 hour movie) over my desired bitrate, it requires an entire recode since it will not fit on the dvd, and that irks me a little. I never quite know what I am going to end up with size wise. With TMPGEnc, I have never overshot the targeted bitrate/size.

I guess what I want to know is if there is something I could be doing that will result in better bitrate accuracy? or is this a problem I would have to live with should I purchase it? Granted it is a little overkill for just doing some simple conversion, but at $70 because of school, I would be willing to purchase it.

[benwaggoner]

All this works just fine as it is, but I started messing around with Expression Encoder 2 and found that it gives me a better looking encode in many cases vs TMPGEnc Xpress 4. But I am specifically shooting for a dvd5/9 size (including m2ts overhead for AVCHD), and the encoder misses the desired bitrate on every 2 pass vbr encode I have done so far.

Now this is fine if it is a little low, but when I spend 15-20 hours on an encode and it ends up being 250kbps (225 MB for a 2 hour movie) over my desired bitrate, it requires an entire recode since it will not fit on the dvd, and that irks me a little. I never quite know what I am going to end up with size wise. With TMPGEnc, I have never overshot the targeted bitrate/size.

I guess what I want to know is if there is something I could be doing that will result in better bitrate accuracy? or is this a problem I would have to live with should I purchase it? Granted it is a little overkill for just doing some simple conversion, but at $70 because of school, I would be willing to purchase it.
In my experience Expression Encoder 2 is extremely accurate in bitrate, with real-world variance of less than 1%.

First, in figuring out your target bitrate, remember it's video + audio + muxing overhead (overhead is probably 11 Kbps at your bitrates).

What kind of bitrate variance are you seeing? How are you evaluating the file's actual bitrate?

Don't forget that you EEv2 gives bitrates in Kbps, while most OS's will show free space in KiB.

GB v. GiB is almost an 8% difference...

[Koadic]

Ok, these are the basic forumlas that I use to determine bitrate from size and size from bitrate:

bitrate = size / time * 8 / 1000
size = time * bitrate * 1000 / 8

Bitrate as measured in kbps, time measured in seconds, and size measured in bytes gathered from the properties window when possible. For example, a 2:10 length file I have shows 16.7 GB, and is actually 18,034,316,506 bytes and 18,034,319,360 bytes on disk. I would use the 18,034,316,506 bytes.

When I do my encodes, I am ultimately shooting size wise for incorporating the vc1 stream into a m2ts container with ac3 audio for an AVCHD disc (largest overhead). Through my experimenting, the audio and file overheads are based on time. For a 5.1 640kbps ac3 stream the overhead is about 82kbps, and the file overhead is about 31kbps. The video overhead is different, it is based on size of the stream. From what I have seen, it is about 5% (varies from 4.5 to 4.8 in my tests, but I want a little 'slop') of the total video size. And in the few times I have checked a wmv file, I have seen about a 20kbps overhead as compared to the vc1 stream. With the above mentioned file, I can acurately calculate the ac3 size in a m2ts container using 753 (640+113 for m2ts overhead) as the bitrate and 7800 as the time with a variance of 0.13% vs the actual ac3 alone inside a m2ts file, so I know my theory works.

So, for example, I am trying to get the above file down to a dvd9 size assuming it will be in m2ts file, I use the following formula:

x = time in seconds, y = 94125 (for 640kbs ac3), dvd9 = 8,547,991,552 (bytes)

maximum video bitrate for BD9 = (dvd9 - (x * y)) / x * 8 / 1000 / 1.05

I end up with around 7632kpbs for the desired bitrate to fill out the dvd9 as much as possible. When I encoded this particular file, I had input 7630kbps as the bitrate average and 14000kbps as peak (trying to stay below the 360 max bitrate). When this was finished and striped from the wmv container, the resulting vc1 stream was 7,656,677,440 (or really close) which when I run through the bitrate equation equals about 7,853kbps, 233kbps above what I had specified, or about 3% greater that what I wanted. While close, this put the files size over what I could use for a BD9 disc. Every now and then I end up about 100MB smaller that what I am shooting for (same bitrate control problem), but is not nearly as big a deal than if I end up even 10MB or so over what I need. Whenever I do the same with TMPGEnc, I almost always end up no more than around 10kbps over or under the average bitrate I specify using the same method of calculating. I will admit that this particular encode was an extreme case of what I have seen and usually I am no more than 50kbps off what I am shooting for (usually meaning the 15 or so encodes I have done), but this happens almost everytime I encode with EE2.

[benwaggoner]

Huh. so you're really shooting for the bitrate of the elementary stream, and that's not hitting what you want? I'd expect that to be spot-on.

What encoding mode are you using?

[Koadic]

What info exactly are you looking for? I am using the 2 pass vbr peak contrained mode, with I frame dquant, In-loop filter, 1 b-frame, Adaptive True Chroma, SAD, and Adaptive search range. If there is any other information your looking for, let me know.

[benwaggoner]

What info exactly are you looking for? I am using the 2 pass vbr peak contrained mode, with I frame dquant, In-loop filter, 1 b-frame, Adaptive True Chroma, SAD, and Adaptive search range. If there is any other information your looking for, let me know.
That's what I was looking for.

FWIW, I-Frame DQuant is really only helpful with animation or CGI.

Can you take a look at the exported .WMV file with WMSnoop and see if it's reporting the video file size as being different than what you selected?

[Koadic]

Ok, here are the details of my most recent encode:

- Original wmv after encode with EE2
Finished Size - 3.74 GB (4,023,683,883 bytes)
- Ripped vc1 stream
Finished Size - 3.73 GB (4,012,080,755 bytes)
- Ripped vc1 stream remuxed into wmv container
Finished Size - 3.75 GB (4,032,284,028 bytes)

Input bitrate average - 4,070 Kbps
Time - 7,800 seconds
Calculated bitrate (from vc1 stream) - 4,115 Kbps

- Original wmv
Reported 'Current Bitrate' from WMSnoop - 4,070 Kbps
Under 'Snoop Details' Bitrate reported as - 3.881 Mbps
Reported Bitrate in MediaInfo - 4070 Kbps

- Remuxed wmv
Reported 'Current Bitrate' from WMSnoop - 4,113,942 bps
Under 'Snoop Details' Bitrate reported as - 3.923 Mbps
Reported Bitrate in MediaInfo - 4114 Kbps

Reported Bitrate in MediaInfo of vc1 muxed in mkv - 3928 Kbps
Reported Bitrate in MediaInfo of vc1 muxed in m2ts - 4180 Kbps

I went and ripped the vc1 stream out of the original wmv to see if there were any falsified attributes that the encoder had inserted that were being read by WMSnoop or MediaInfo. After remuxing, I found out that this was, in fact, the case.

If wanted I can include any screenshots of WMSnoop or MediaInfo, just ask.

[benwaggoner]

Hmmm. What were the average, peak, and buffer duration parameters you used? Are you using peaks and averages very close to each other?

And what are you using to mux into WMV? It's interesting that your remuxed files are larger than the original WMVs.

[Koadic]

For this encode, I used an average of 4070Kbps, peak of 14000Kbps, and a peak buffer size of 5 seconds.
The process I used to go from wmv->vc1->wmv is as follows:
1 - Wmv to mkv with DS Mux
2 - Then either of the following
-mkv to mkv with mkvmerge with timecode file (important for framerate as DS Mux messes this up) OR
-mkv to m2ts with tsMuxer (I can then rip the vc1 from the m2ts with tsMuxer)
3 - mkv or m2ts to asf file with Solveig's ASF Muxer
4 - asf to wmv with WMSE as the asf doesnt play on Xbox 360 (even when renamed to wmv)

Although, I need to either run the asf or final wmv through wmvmuxer before muxing audio with WMSE to eliminate any flickering when playing on the xbox 360. This only happens when I convert the original wmv to another format before converting back to wmv again, the original plays just fine on the xbox 360. I think that Solveig's ASF Muxer somehow breaks the stream. It would be nice if there was a microsoft tool like WMSE that accepted vc1 streams.

[benwaggoner]

Since you're getting the exact size elementary stream that you requested, this is presumably a mux issue.

Can you accurately predict the mux overhead with that tool? You may need to just reduce video bitrate by that amount.

[Koadic]

But I'm not getting the exact size elementary stream that I want. While WMSnoop and MediaInfo say that the bitrate is correct in the original wmv, the math tells me that it is wrong. And that is supported with the bitrate reported by those same tools when I insert the vc1 stream back into a new wmv. While the remuxed wmv is around 10MB larger, this doesnt account for 45 kbps increase between original and remuxed files. If I were a betting man, I would say that EE2 stored the value of 4070 kbps in the original wmv somewhere, and that is what was read by WMSnoop and MediaInfo. While when I remuxed it, there was nothing external telling the application what the bitrate was supposed to be, but was calculated from the size and time of the file.

In this instance, I am shooting for a vc1 stream of around 3,968,250,000 bytes (4070kbps * 7800 seconds * 1000 / 8) but I am geting a file of 4,012,080,755 bytes, which implies a bitrate of 4115 kbps. My whole point is that if I encode with TMPGEnc at 4070 kbps and demux the vc1 stream, I get the almost the exact size I am requesting. Going by past encodes, I would estimate that my file size would be approximately 3,960,000,000 bytes. While a little low, it is usually this close and never ends up higher than the average bitrate set.

With EE2, I can't determine any repeatable variance, whether a constant bitrate overhead, or a percentage increase in final bitrate over requested bitrate. Sometimes it is high, usually around 50-75 kbps (with an experienced high of more than 200 over), but sometimes it is up to 100 kbps low.

As per the first post, I was just wondering if there was just something that I was missing or if I could do something different to get more predictible results. I will most likely still purchase it, as I do get a better looking encode vs. TMPGEnc, but I guess I will just have to shoot for a bitrate thats a little lower than I want and hope that it doesnt end up being too big, or so small that I would have 200 MB or more unused space on the dvd should I burn it. I would really like to be able to use every last bit of space on a dvd though, whether it be a single or dual layer disc.

Regardless, I appreciate you taking the time and trying to help me out. If you have any other suggestions, I will gladly try them out.

Update: I just finished an encode without using dquant, and the resulting bitrate was spot on (within 3kbps after calculations) I will try to complete another one soon and see if this holds true. Hopefully I have found a solution and the next one works just as well.

Secondary Update: I just got done scanning through the wmv without dquant and the one with, and I think I will just try to take my chances guessing on the bitrate because while this movie wasn't animation or cgi, dquant helped out quite a bit in the dark and smoky areas to reduce blocking. I may not need to use dquant on my larger encodes as I will have a larger bitrate to work with, but when I am trying to get a 720p 2 hour movie on a dvd5, the extra bitrate for the dark areas helps. If it really is the dquant that is messing up the bitrate calculation when encoding, maybe it is a bug that can be fixed. I will wait to see how my second encode without dquant turns out and will report back. While only 2 good encodes won't really prove anything beyond a doubt, it will atleast give me something to work with. I may just end up having to encode something twice, and hope the 'dquant difference' in the bitrate stays the same to properly encode the second time.

[benwaggoner]

Interesting. You may have found a bug in the DQuant code! I'll have the team check it out.

[Koadic]

Well, I have finished a couple more tests. I have redone the same file @ 7630kbps that was 200+ kbps over when I had dquant enabled, and the new encode was 7633kbps, only 3kbps off. I also ran a 16min 40sec clip through @ 8000kbps, encoding it twice, first with dquant and then without, everything else being equal. The first encode with dquant finished with 8263kbps and the second finished with 8014kbps. The second encode is only .175% higher than what was entered, and falls within my .3% cushion that I give myself because my m2ts overhead calculations aren't exact. So, as these extra tests imply, there is a bug concerning the bitrate calculation when using dquant.

[benwaggoner]

And bear in mind that we're only talking a 1.1% variance even with using I-Frame DQuant, so it's really not that big a deal (I remember getting 20-30% data rate overshoots with Cinepak back in the day. And VP6 can do nearly as bad at low bitrates today).