Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Video Encoding > MPEG-2 Encoding

Reply
 
Thread Tools Search this Thread Display Modes
Old 15th December 2014, 18:17   #1  |  Link
rwill
Registered User
 
Join Date: Dec 2013
Posts: 343
y262

So over the last four X-Mas when I was on PTO I wrote an mpeg2 encoder.

It was mostly finished last year but I have added some small features in the last few days like pulldown and pipe support.

It would be nice if some people check it out and give feedback.

I have run some limited tests myself and picture quality appears to be good - but the encoder is somewhat slow as it currently lacks multithreading.

*edit 1
Its now up on github. I still do not provide binaries.

https://github.com/rwillenbacher/y262/

Last edited by rwill; 22nd August 2022 at 21:58. Reason: updated repository link
rwill is offline   Reply With Quote
Old 17th December 2014, 18:18   #2  |  Link
hank315
HCenc author
 
Join Date: Nov 2003
Location: Netherlands
Posts: 570
A first test using y262.
Test is a progressive PAL 3200 frames clip (720x576) using low bitrate and 2 passes.
For testing I added the next matrix:
Code:
 8 16 19 22 26 27 29 34 
16 16 22 24 27 29 34 37 
19 22 26 27 29 34 34 38 
22 22 26 27 29 34 37 40 
22 26 27 29 32 35 40 48 
26 27 29 32 35 40 48 58 
26 27 29 34 38 46 56 69 
27 29 35 38 46 56 69 83

16 17 18 19 20 21 22 23 
17 18 19 20 21 22 23 24 
18 19 20 21 22 23 24 25 
19 20 21 22 23 24 26 27 
20 21 22 23 25 26 27 28 
21 22 23 24 26 27 28 30 
22 23 24 26 27 28 30 31 
23 24 25 27 28 30 31 33


Adaptive Quantisation works fine.
Ratecontrol is good.
Intra/inter decision for MacroBlocks in P and B frames seems OK.
MacroBlock skipping in P and B frames works OK if there's real content but doesn't work in black bars.

Video quality is good, major drawback it's slow.
Metrics (PSNR and SSIM) for this test are high, for this low bitrate encoding metrics beat all commercial encoders I know.

For now it sure looks promising.
__________________
HCenc at: http://hank315.nl
hank315 is offline   Reply With Quote
Old 17th December 2014, 20:49   #3  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,531
A build, a build, a kingdom for a build....
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."
Emulgator is online now   Reply With Quote
Old 17th December 2014, 20:53   #4  |  Link
hank315
HCenc author
 
Join Date: Nov 2003
Location: Netherlands
Posts: 570
I will upload one soon, also some files I used for testing.

Here is an exe and some files used for testing.
The matrices used can be activated with -mpeg (see run.bat)
A raw intermediate file is used, pipes are also possible but I didn't try that.
Success and thanks to rwill.
__________________
HCenc at: http://hank315.nl

Last edited by hank315; 17th December 2014 at 22:41. Reason: extra info + link
hank315 is offline   Reply With Quote
Old 18th December 2014, 11:59   #5  |  Link
rwill
Registered User
 
Join Date: Dec 2013
Posts: 343
Thank you hank315 for trying out my encoder

Ok, regarding the failure of the encoder to use skip mode in unicolour macroblocks: lambda was 0 for quant = 1 so the bit count of the macroblock was not taken into account when determining modes. -Fixed in SVN. Black bars should be skip now.

I also added the ability to specify text files containing 64 values as Intra or Inter Quantization Matrices to the application so there is no need to make changes in the source code anymore when trying QMats out.
rwill is offline   Reply With Quote
Old 18th December 2014, 12:12   #6  |  Link
Sparktank
47.952fps@71.928Hz
 
Sparktank's Avatar
 
Join Date: Mar 2011
Posts: 940
Just when I thought there wasn't any more toys, Christmas kicks in.
__________________
Win10 (x64) build 19041
NVIDIA GeForce GTX 1060 3GB (GP106) 3071MB/GDDR5 | (r435_95-4)
NTSC | DVD: R1 | BD: A
AMD Ryzen 5 2600 @3.4GHz (6c/12th, I'm on AVX2 now!)
Sparktank is offline   Reply With Quote
Old 20th December 2014, 18:11   #7  |  Link
hank315
HCenc author
 
Join Date: Nov 2003
Location: Netherlands
Posts: 570
@rwill
If input is 1920x1080, output is 1920x1088 in the sequence header and 1920x1080 in the sequence display extension.
IMO in the sequence header, height should also be 1080 using 8 black lines in the lower MacroBlocks, see also here, table 6.2.
__________________
HCenc at: http://hank315.nl
hank315 is offline   Reply With Quote
Old 20th December 2014, 18:12   #8  |  Link
rwill
Registered User
 
Join Date: Dec 2013
Posts: 343
I just added slice threading. Sadly it does not scale linear.
rwill is offline   Reply With Quote
Old 20th December 2014, 18:32   #9  |  Link
hank315
HCenc author
 
Join Date: Nov 2003
Location: Netherlands
Posts: 570
I also tried that but the thread overhead is pretty large so no linear scaling.
I ended up with frame threading which is almost linear.
Maybe input and lookahead can be threaded?
__________________
HCenc at: http://hank315.nl
hank315 is offline   Reply With Quote
Old 20th December 2014, 18:39   #10  |  Link
rwill
Registered User
 
Join Date: Dec 2013
Posts: 343
@hank315

I fixed that vertical size signaling.

Lookahead is threaded. Problem is that different parts of pictures take a different amount of time

*edit* Regarding frame threading, I am scared of the bitrate control when its set to CBR.
rwill is offline   Reply With Quote
Old 21st December 2014, 01:17   #11  |  Link
hank315
HCenc author
 
Join Date: Nov 2003
Location: Netherlands
Posts: 570
@rwill
Threads.c and threads.h seem to be missing in SVN repository?
__________________
HCenc at: http://hank315.nl

Last edited by hank315; 21st December 2014 at 01:32.
hank315 is offline   Reply With Quote
Old 21st December 2014, 09:14   #12  |  Link
rwill
Registered User
 
Join Date: Dec 2013
Posts: 343
Whoops, they are there now.
rwill is offline   Reply With Quote
Old 21st December 2014, 17:43   #13  |  Link
hank315
HCenc author
 
Join Date: Nov 2003
Location: Netherlands
Posts: 570
New build, threading gives an average speed boost factor of 2.5 on my quad core.
You will need at least SSE2 for this one.
The new avs2yuv will pipe raw data into y262app, so no intermediate raw file is needed.
See readme.txt and example pipe.bat.

Download link


EDIT:
Package updated, should work also with XP now.
__________________
HCenc at: http://hank315.nl

Last edited by hank315; 22nd December 2014 at 00:32. Reason: package updated
hank315 is offline   Reply With Quote
Old 23rd December 2014, 10:56   #14  |  Link
Emulgator
Big Bit Savings Now !
 
Emulgator's Avatar
 
Join Date: Feb 2007
Location: close to the wall
Posts: 1,531
Wow, not bad for this low version count. Potential.
__________________
"To bypass shortcuts and find suffering...is called QUALity" (Die toten Augen von Friedrichshain)
"Data reduction ? Yep, Sir. We're that issue working on. Synce invntoin uf lingöage..."
Emulgator is online now   Reply With Quote
Old 21st April 2021, 05:26   #15  |  Link
rwill
Registered User
 
Join Date: Dec 2013
Posts: 343
y262 is now on github:
https://github.com/rwillenbacher/y262

Not much changed but I added Mpeg-1 support, fixed some bugs and switched to cmake as a build system.
rwill is offline   Reply With Quote
Old 21st April 2021, 20:16   #16  |  Link
Blue_MiSfit
Derek Prestegard IRL
 
Blue_MiSfit's Avatar
 
Join Date: Nov 2003
Location: Los Angeles
Posts: 5,988
Cool, good to have more MPEG-2 encoders in the world Maybe start a new thread? This one is 7 years old
Blue_MiSfit is offline   Reply With Quote
Old 22nd April 2021, 00:20   #17  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Since we're talking about MPEG-2: can it encode compliant XDCAM-50 files?
FranceBB is offline   Reply With Quote
Old 22nd April 2021, 06:17   #18  |  Link
rwill
Registered User
 
Join Date: Dec 2013
Posts: 343
Quote:
Originally Posted by Blue_MiSfit
Cool, good to have more MPEG-2 encoders in the world Maybe start a new thread? This one is 7 years old
Thanks for making me feel old.

Quote:
Originally Posted by FranceBB
Since we're talking about MPEG-2: can it encode compliant XDCAM-50 files?
Maybe, i don't know the attributes and constraints a XDCAM-50 file has. Can ffmpeg or x262 generate XDCAM-50 files ? If yes, can someone post a respective commandline here ?
rwill is offline   Reply With Quote
Old 26th April 2021, 14:50   #19  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Royal Borough of Kensington & Chelsea, UK
Posts: 2,883
Sure, there you go:

Quote:
avs2yuv.exe "Z:\AVS Script.avs" -csp AUTO -o - | ffmpeg.exe -i - -pix_fmt yuv422p -vcodec mpeg2video -s 1920:1080 -aspect 16:9 -vf setfield=tff -flags +ildct+ilme+cgop -b_strategy 0 -mpv_flags +strict_gop -sc_threshold 1000000000 -r 25 -b:v 50000k -minrate 50000k -maxrate 50000k -bufsize 36408333 -g 12 -bf 2 -profile:v 0 -level:v 2 -color_range 1 -color_primaries 1 -color_trc 1 -colorspace 1 -y "\\mibctvan000\Ingest\MEDIA\temp\output.mxf"
As you can see it's basically an MPEG-2 with a closed GOP M=3 N=12 profile High, Level 2, yv16 (so 4:2:2 planar 8bit) either 25i or 29.970i TFF or BFF at 50 Mbit/s CBR.



The reason why I'm asking you this is that the current MPEG-2 encoder in FFMpeg was made back when monocore and single socket CPU/PCs were still a thing, so it's terribly non-optimized/parallelized and it doesn't really make use of instruction sets, so having a Dual Xeon 60c/120th AVX512 configuration is useless unless I encode several files at a time. I generally encode 40-60 files at once on such a configuration to make use of the whole CPU power, but it's still slow as it's SSE2 only...
Where does y262 stand in terms of intrinsics/assembly optimizations and parallelization?

As Derek said, the more the better when it comes to open source MPEG-2 encoders, however the only real use case for these encoders nowadays is not consumer-distribution but rather professional/enterprise driven so it would be nice to get this thing working properly with professional formats like XDCAM.
Question: any particular reason why you made your own encoder instead of contributing to the one already implemented in FFmpeg or to x262? (Just a curiosity, 'cause if your encoder gets these things right it might as well be integrated in FFmpeg one day).


Anyway thank you for making this available to everyone.
FranceBB is offline   Reply With Quote
Old 26th April 2021, 18:12   #20  |  Link
rwill
Registered User
 
Join Date: Dec 2013
Posts: 343
Yeah that would translate to something like this. Maybe drop -flatmat for default quant matrices ?

Code:
./y262.exe -in test_interlaced_422_8b.yuv -size 1920 1080 -frcode 3 -profile main -level high -chromaf 422 -bitrate 50000 -vbvrate 50000 -vbv 9600 -interlaced -quality 0 -nump 3 -numb 2 -closedgop -flatmat -videoformat 709 -arinfo 3 -out test.m2v -mpout test.stats -rcmode 1 -threads 1 6 -cbr

./y262.exe -in test_interlaced_422_8b.yuv -size 1920 1080 -frcode 3 -profile main -level high -chromaf 422 -bitrate 50000 -vbvrate 50000 -vbv 9600 -interlaced -quality 0 -nump 3 -numb 2 -closedgop -flatmat -videoformat 709 -arinfo 3 -out test.m2v -mpin test.stats -rcmode 2 -threads 1 6 -cbr
I am somewhat confused by -bufsize 36408333 and i saw somewhere else it being -bufsize 17825792 for xdcam 422. But I am stuck with the first release of the mpeg2 spec from ~1995 which is lacking the 422 profile specs so I do not know the level limits for that. So right now y262 is capped at around ~9600k video buffer because that is what 420 has. I might remove the buffer size check for 422 and 444 but I would feel safer when I have some sort of standard to reference to when checking level limits. Google got me nothing and ISO wants money. y262 currently only accepts rates and buffers in 1000 units so this might be a problem too.

y262 currently does have basic slice threading. I have been thinking about adding frame based threading but this would require some changes to bitrate control and picture re-encode logic in addition to the threading itself. This comes with extensive corner case testing. I think right now it will utilize around 6 cores if told to do so for HD. MPEG2 with pixels being 8 bit, macroblocks being 16x16 and the transform being 8x8 does not benefit much from SIMD registers above 128bit width. Not much one can do with AVX2 or AVX512.

I started y262 before x262 was announced and I think bolting mpeg2 support onto x264 is the wrong development approach anyway. The ffmpeg encoder design is too old. Always starting fresh might not be the fastest way to get some sort of bitstream producing library out but it brings the oportunity to avoid mistakes of the past.
rwill is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 01:17.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.