Any MPEG-2 encoder capable of 1080p @60fps ???

[Roco]

I know what am I asking is out of standards, (but ...I'm always seeking for more than what the mega-companies suggest and what the main mass accepts :rolleyes: and I believe this is more or less the case with many others in this forum as well -in different areas of course)!

So here is my problem:
I have some video clips created with my hdv camera, processed and converted to 1920x1080 progressive at 60 fps (via avisynth) for ultra-smooth motion. So it would be perfect to convert to mpeg2 at the same frame rate and frame size at about 50Mbit, because mpeg2 converts fast compared to any other format, while at that bitrate quality remains very high -almost lossless! (I intent to view them on a 23" PC display for now at full resolution and as a second choice on a 720p projector).

BTW, my PC is capable of doing the same job in realtime with the 8800 Nvidia card + cyberlink BUT, it can't adjust the colors and process the video to the degree avisynth can, while each clip needs different adjustments, so this is not what I really want.

in other words, I prefer encoding speed + quality, over file size.
So, is there an MPEG-2 encoder that can be configured to encode to the ...forbidden 1080p60???

[Sulik]

Note that 1080p60 is NOT compliant with MPEG-2 Main Profile @ High Level.

[Roco]

Note that 1080p60 is NOT compliant with MPEG-2 Main Profile @ High Level.

Yes, this is why I said "out of standards"... I don't care about any compliance anyway, as long as the encoded video can playback on a software player on a PC...

I just need a flexible encoder that can accept and encode 60fps (instead of the maximum 25fps at 1080p).

[akupenguin]

libavcodec has no restrictions on framerate.
But really, any mpeg2 encoder should work. Framerate is none of the encoder's business, it just takes frames and compresses them. Only the muxer needs to know what timestamp is assigned to each frame.

[drmpeg]

Note that 1080p60 is NOT compliant with MPEG-2 Main Profile @ High Level.
There is an effort to add 1080p@60 to MPEG-2.

http://ftp3.itu.ch/av-arch/jvt-site/2007_10_Shenzhen/JVT-Y086.zip

Ron

[Roco]

akupenguin, I've tried many MPEG-2 encoders with no luck, including the Tmpgenc and Mainconcept via Premiere 2, I tried yesterday. it seems like they are all ...safe guarded against higher progressive frame rates at 1080!
I will try libavcodec tomorrow, do you suggest a particular gui or app? I'd have to pass avs scripts to it -thx for the suggestion.

drmpeg, thx for the link. We are heading to an era of very high frame rates and mpeg2 although old, is still a cheap way to compress/decompress huge amounts of video data.

[akupenguin]

I've tried many MPEG-2 encoders with no luck, including the Tmpgenc and Mainconcept via Premiere 2, I tried yesterday. it seems like they are all ...safe guarded against higher progressive frame rates at 1080!
I was going to say, just don't tell the encoder the true framerate. But it turns out that mpeg2 stores timestamps in the video stream in addition to the container, so if you do that you risk making a self-contradictory file and there's no guarantee what a player will do with it. How lame.
I will try libavcodec tomorrow, do you suggest a particular gui or app?
I use mencoder. But that doesn't mean I recommend it to new users, just that I've memorized all the commandline options. If you want GUIs, I can't help.

[Sulik]

MPEG-2 stores the frame rate within the elementary stream, and most encoders will apply strict conformance in order to guarantee MP@HL compliance and VBV restrictions.

Note that this is not really different from a commercial H.264 encoder that would enforce HRD restrictions (frame rate info is also stored indirectly within VUI)

[drmpeg]

MPEG-2 stores the frame rate within the elementary stream, and most encoders will apply strict conformance in order to guarantee MP@HL compliance and VBV restrictions.
If the bitstream is VBR (that is, vbv_delay is not used), then you can patch the stream to be VBV compliant by just setting the bitrate higher by the same factor as frame rate. I've made an example with a 23.976p bitstream patched to 59.94p (bitrate is increased from 17,608,000 Mbps to 44,020,000 Mbps).

http://www.w6rz.net/1080p60.mpv

The VBV dump of the first 10 frames.


Found Sequence Start Code at byte position 0.
* -------- Sequence Start Code at 0 ---------
* Picture size: 1920x1080 Frame rate: 59.94 Bitrate: 44.02 Mb/s
* VBV size: 9552 Kb Intra QM: loaded Non Intra QM: loaded
* Progressive sequence.
* Main Profile, High Level MPEG-2 bitstream.
* -------- GOP Start Code at 152 ---------
* Closed GOP at 00:00:00.00 drop flag: 0
* Using VBR VBV model.
* -------- Intra frame at 160 ---------
* Temporal Reference: 2 VBV: 9781248 (+ 0) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 457024 bits VBV after decoding: 9324224 bits
* Average rate: 22.279 Mb/s
* Frame number: 2 Average mquant: 1.01
* Zero stuffing: 636 bits
* -------- Bidirectional frame at 57288 ---------
* Temporal Reference: 0 VBV: 9781248 (+ 277377) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 14656 bits VBV after decoding: 9766592 bits
* Average rate: 21.209 Mb/s
* Frame number: 3 Average mquant: 1.75
* Zero stuffing: 781 bits
* -------- Bidirectional frame at 59120 ---------
* Temporal Reference: 1 VBV: 9781248 (+ 719745) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 14592 bits VBV after decoding: 9766656 bits
* Average rate: 20.192 Mb/s
* Frame number: 4 Average mquant: 1.00
* Zero stuffing: 778 bits
* -------- Predicted frame at 60944 ---------
* Temporal Reference: 5 VBV: 9781248 (+ 719809) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 16768 bits VBV after decoding: 9764480 bits
* Average rate: 19.233 Mb/s
* Frame number: 5 Average mquant: 1.02
* Zero stuffing: 656 bits
* -------- Bidirectional frame at 63040 ---------
* Temporal Reference: 3 VBV: 9781248 (+ 717633) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 14336 bits VBV after decoding: 9766912 bits
* Average rate: 18.314 Mb/s
* Frame number: 6 Average mquant: 1.00
* Zero stuffing: 508 bits
* -------- Bidirectional frame at 64832 ---------
* Temporal Reference: 4 VBV: 9781248 (+ 720065) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 9792 bits VBV after decoding: 9771456 bits
* Average rate: 17.428 Mb/s
* Frame number: 7 Average mquant: 1.00
* Zero stuffing: 433 bits
* -------- Predicted frame at 66056 ---------
* Temporal Reference: 8 VBV: 9781248 (+ 724609) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 16384 bits VBV after decoding: 9764864 bits
* Average rate: 16.606 Mb/s
* Frame number: 8 Average mquant: 1.00
* Zero stuffing: 570 bits
* -------- Bidirectional frame at 68104 ---------
* Temporal Reference: 6 VBV: 9781248 (+ 718017) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 12736 bits VBV after decoding: 9768512 bits
* Average rate: 15.814 Mb/s
* Frame number: 9 Average mquant: 1.00
* Zero stuffing: 550 bits
* -------- Bidirectional frame at 69696 ---------
* Temporal Reference: 7 VBV: 9781248 (+ 721665) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 11136 bits VBV after decoding: 9770112 bits
* Average rate: 15.056 Mb/s
* Frame number: 10 Average mquant: 1.00
* Zero stuffing: 397 bits
* -------- Predicted frame at 71088 ---------
* Temporal Reference: 11 VBV: 9781248 (+ 723265) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
Found Picture Start Code at byte position 72928.


The original 24p bitstream.

http://www.w6rz.net/1080p24.mpv

The VBV dump


Found Sequence Start Code at byte position 0.
* -------- Sequence Start Code at 0 ---------
* Picture size: 1920x1080 Frame rate: 23.98 Bitrate: 17.61 Mb/s
* VBV size: 9552 Kb Intra QM: loaded Non Intra QM: loaded
* Progressive sequence.
* Main Profile, High Level MPEG-2 bitstream.
* -------- GOP Start Code at 152 ---------
* Closed GOP at 00:00:00.00 drop flag: 0
* Using VBR VBV model.
* -------- Intra frame at 160 ---------
* Temporal Reference: 2 VBV: 9781248 (+ 0) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 457024 bits VBV after decoding: 9324224 bits
* Average rate: 8.912 Mb/s
* Frame number: 2 Average mquant: 1.01
* Zero stuffing: 636 bits
* -------- Bidirectional frame at 57288 ---------
* Temporal Reference: 0 VBV: 9781248 (+ 277377) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 14656 bits VBV after decoding: 9766592 bits
* Average rate: 8.484 Mb/s
* Frame number: 3 Average mquant: 1.75
* Zero stuffing: 781 bits
* -------- Bidirectional frame at 59120 ---------
* Temporal Reference: 1 VBV: 9781248 (+ 719745) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 14592 bits VBV after decoding: 9766656 bits
* Average rate: 8.077 Mb/s
* Frame number: 4 Average mquant: 1.00
* Zero stuffing: 778 bits
* -------- Predicted frame at 60944 ---------
* Temporal Reference: 5 VBV: 9781248 (+ 719809) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 16768 bits VBV after decoding: 9764480 bits
* Average rate: 7.693 Mb/s
* Frame number: 5 Average mquant: 1.02
* Zero stuffing: 656 bits
* -------- Bidirectional frame at 63040 ---------
* Temporal Reference: 3 VBV: 9781248 (+ 717633) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 14336 bits VBV after decoding: 9766912 bits
* Average rate: 7.326 Mb/s
* Frame number: 6 Average mquant: 1.00
* Zero stuffing: 508 bits
* -------- Bidirectional frame at 64832 ---------
* Temporal Reference: 4 VBV: 9781248 (+ 720065) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 9792 bits VBV after decoding: 9771456 bits
* Average rate: 6.971 Mb/s
* Frame number: 7 Average mquant: 1.00
* Zero stuffing: 433 bits
* -------- Predicted frame at 66056 ---------
* Temporal Reference: 8 VBV: 9781248 (+ 724609) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 16384 bits VBV after decoding: 9764864 bits
* Average rate: 6.642 Mb/s
* Frame number: 8 Average mquant: 1.00
* Zero stuffing: 570 bits
* -------- Bidirectional frame at 68104 ---------
* Temporal Reference: 6 VBV: 9781248 (+ 718017) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 12736 bits VBV after decoding: 9768512 bits
* Average rate: 6.325 Mb/s
* Frame number: 9 Average mquant: 1.00
* Zero stuffing: 550 bits
* -------- Bidirectional frame at 69696 ---------
* Temporal Reference: 7 VBV: 9781248 (+ 721665) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
* Picture: 11136 bits VBV after decoding: 9770112 bits
* Average rate: 6.023 Mb/s
* Frame number: 10 Average mquant: 1.00
* Zero stuffing: 397 bits
* -------- Predicted frame at 71088 ---------
* Temporal Reference: 11 VBV: 9781248 (+ 723265) bits
* Intra DC: 9 bits Quant scale: non-linear
* Scan: alternate Intra VLC table: 1
* Top field first: 0 Repeat first field: 0
Found Picture Start Code at byte position 72928.


As you can see, the VBV occupancy is identical. The hex patch is:

replace
00 00 01 B3 78 04 38 31 2A FD 32

with
00 00 01 B3 78 04 38 37 6B 78 B2

But it turns out that mpeg2 stores timestamps in the video stream in addition to the container, so if you do that you risk making a self-contradictory file and there's no guarantee what a player will do with it.

GOP headers are optional in MPEG-2. The GOP timecode is not used by the decoder for any purpose. It's only there for editors.

Ron

[hank315]

I just need a flexible encoder that can accept and encode 60fps (instead of the maximum 25fps at 1080p).
HCenc can do it.

[SeeMoreDigital]

I have just generated the following MPEG-2 1920x1080 60p sample using FFdshow's MPEG-2 encoder: -

http://rapidshare.de/files/38289928/MPEG-2_1920x1080_60p.mpg.html

It seems to play okay in my hardware player.

[Roco]

Thank you ALL for your suggestions!!! :thanks:

Ron, your idea was very interesting. If I understand you well, you're saying that someone could use *any* encoder at a valid framerate and VBR and then hack the headers to make it 60fps, increasing also the bitrate. How about a tiny app to do this automatically, something like the "Avi Frame Rate" utility???
EDIT: Ron, your website is a real treasure!!

SeeMoreDigital,
I did tried, FFDSHOW which uses libavcodec (I think) but unfortunately it restricts bitrate at 10Mbits max which results in very poor mpeg-2 quality. This is not enough bitrate for 1080p60 -for any encoder IMO.

hank315,
Your suggestion was perfect!
HCenc is the first MPEG-2 encoder I have tested, truly capable of a quality 1080p60 encode, at high bitrates (I tested it up to 100Mbit) and even freeware!
It was stable, fully compatible with the latest avisynth 2.5.8 and easy to setup.

In case someone's interested, I had the best results so far (for 1080p60 input) with these settings:
-average kb/s 50000,
-maximum kb/s 100000,
-dc prec 11,
-progressive

Also, I used:
-last I-frame
-colorimetry BT.709
-normal

I left all other settings at default.

(Single pass encoded faster, but it was not adequate for difficult footage, with fine detail in motion)


Compared to the only competitor for the task (for alternative formats), VC-1, HCenc gave me much faster and decent results (better) than Windows Media Encoder which restricts VC-1 (Advanced Profile) to 20Mbits. (Off topic: from my testing, x264 and Nero's h264 have some issues with color gradients + fast fps, compared to VC-1 -I don't speak about expensive encoders of course)

In one encoding for example, HCenc compressed at 1080p60 with the above settings in 8 x realtime, while WM encoder in 23 x realtime (the encoding time varies with content). In WM encoder I had to remove "ReduceFlicker" function that some scripts had, because it made the WM encoder crash.
Also, when I gave WM encoder a really noisy footage (sea) It produced a stupid error half way of the encode saying "Not enough storage is available to complete this operation (0x8007000E)" while there was more that 10GB available.


The only con is that HCenc produces a video-only file.
So I had to extract the sound (eg with VirtualDubMpeg2), convert it to ac3 with "Wav To Ac3 Encoder" and use the built in MPEG-2 multiplexer of Tmpgenc 2.5 Plus, to multiplex to program stream.
I feel there must be a faster way to have the sound multiplexed (imagine you have to encode 50 clips). If anyone knows, I'd appreciate it!

About playability, the resulted clips played flawlessly and smoothly only on Media Player Classic (this is what I was praying for). They were stuttering on KMplayer, VLC Media player, and Windows Media Player. I think this has something to do with the way Tmpgenc multiplexed the footage.

In conclusion, the first really flexible non-restrictive, high bitrate, high framerate, fully Avisynth 2.5.8 compatible, easy to use and even freeware MPEG-2 encoder capable for 1080p60 has been found: HCenc! :) :) :) :) :)
(suggested by hank315) http://hcencoder.bitburners.com

LATE EDIT: Looking at the other threads I realized that hank315 is the author of HCenc! Congratulations Hank!!!

[SeeMoreDigital]

SeeMoreDigital,
I did tried, FFDSHOW which uses libavcodec (I think) but unfortunately it restricts bitrate at 10Mbits max which results in very poor mpeg-2 quality. This is not enough bitrate for 1080p60 -for any encoder IMO.If you use the "one pass - quality" option, it is possible to generate MPEG-2 streams at far higher bit-rates than 10Mbps :)

[Roco]

If you use the "one pass - quality" option, it is possible to generate MPEG-2 streams at far higher bit-rates than 10Mbps :)

That was an interesting discovery! (Who knows what else is hidden in its open source! :) )

Pros:
-There was no need for multiplexer, I used VirtualDubMpeg2 for the encode which received the avs script (I installed the latest FFDSHOW Tryout beta 1723 Dec 24 2007).
-Encode could be fast if additional features were'nt enabled.

Cons:
No matter what I tried, the encoding quality wasn't on par with HCenc dual pass (see above settings). In one encode FFDSHOW took 107sec (single pass), produced a file size of 266MB, vs 150sec (dual pass) of HCenc which produced a file size of 115MB and still the quality of FFDSHOW was much worse. On another encode, I saw crawling noise on a shaking backround when it almost didn't exist on HCenc's encode.
Also, playability was worse.

So, in slow moving, very clean footage it might be an option when you are ...in a hurry! :)

[Roco]

Much more testing for FFDSHOW... :)

The default settings of FFDSHOW were disastrous for 1080p60. In previous tests I didn't change the "Maximum I frame interval" which was at 250 and the internal logic caused all the trouble.

So, with the following settings, FFDSHOW (using libavcodec) did produce video very close (some times with the same quality) to HCenc's 2 pass, most of the time, including the "noisy sea" clip and with about the same speed, EXCEPT for one clip with color gradient on the sky and subtle detail ( smoke). On that clip, FFDSHOW produced excessive banding + noise (while HCenc's encode was perfect). It is possible that someone who knows FFDSHOW's settings better than me, would be able to improve its encoding on such cases...

(Encoder=MPEG-2)
Mode: One pass - quality
Quality : between 70 and 97 so that average bitrate remains at least 50Mbps. (The trouble with this setting is that you have to run for a while the encoder and see the average bitrate, then stop re-adjust and restart, otherwise, bitrate might vary from 17Mbps up to 120Mbps -and visual quality too!!!)
Maximum I frame interval 16
Minimum I frame interval 1
bframes, 1, dynamic
Macroblock decision mode: rate distortion
ME compare function: SAD, Chroma enabled
Subpix. ME comp. function: SAD, Chroma enabled
Macroblock ME comp. func: SAD, Chroma enabled
Quantization type h.263, Trellis searched quant. disabled

The above settings are a compromise for speed and quality at MPEG-2 1080p60.
If anybody knows how to eliminate color banding (if possible) please inform us...

I also changed the HCenc's setting to "BEST" and forced SSSE3 (That wasn't necessary for that clip).

Still, playback with mpeg-2 1080p60 is not completely smooth with FFDSHOW and I noticed that this happens to a lesser degree with HCenc too now and then (while cpu load was equal or less than 50% and VC-1 @20Mbit I mentioned previously, does playback at 1080p60 totally smooth...)

Does anybody know why is that happening???

[Ranguvar]

Perhaps your hard disk can't read the data fast enough? :P

If that's true, maybe a different player (still use ffdshow) would be better optimized.

If the skips are every once in a while, try pausing for a little bit before you think a skip may happen and then playing, so it can buffer.

[Roco]

I found some answers, solutions and ...mistakes:

Ranguvar thanx, but that was not the case...

* HCenc's 1080p60 MPEG-2 jerkyness on Media Player Classic was caused by Mainconcept's (Adobe's) video decoder which I had as "preferred" in "External Plugins" (it was available because of Premiere Pro 2).
So, with MPC's internal MPEG-2 decoder (if enabled) or with FFDSHOW on "External Plugins", playback is always 100% smooth. (Average CPU utilization when playback 50Mbit 1080p60 footage on dual core 6600 2gb ram, EN8800, is about 35%).

Still, jerkyness exist on other players (VLC, KM WM).

* By a stupid mistake (or a stupid assumption) I was encoding to a very original format:
Mpeg-2 in ...AVI container! :p
This was the output of VirtualDUB-MPEG2 with FFDSHOW MPEG-2 compression!
This might explain the jerkyness, but it doesn't explain the banding artifacts (vs HCenc and VC1).

So, at least one quality solution for 1080p50/60 MPEG-2 exists, with Avisynth->HCenc->Multiplexer and playback with MPC.

[2Bdecided]

It's probably a bit late to note that MPEG-1 might be an easy solution. Most players seem happy with this.

Cheers,
David.