h264info alpha 0026 lite test

[Trahald]

This program lets you save nal info to disk from a h264 .es stream. it also will do pulldown and add sei time data. it is very alpha and not that tested. i havent had a chance to work on it for a while so im just putting it out there. at a minimum the info part seems to work and is very tested. i dont have alot of reference streams but the ones i have work fine. The program will add the data to allow a .es that is not accepted by scenarist to be accepted. when adding data it uses some dummy data so be careful. It is also slow.

http://batchccews.sourceforge.net/ is where you can get it under h264info package. source is included (although its not pretty.) Much of the parsing part is from mpeg4ip source rewritten. as such it is being released under the MPL. i fixed some parts from info i got from googling and aku's posts.

see sourceforge.net changelog for latest.

** Thanks to musicman2311 for help with testing

<0009> fixed variable change caused bad reconstructing of slice nal in some streams on 59.94/60 fps. (green blocks)

<0008> cpb removal delay calc revised to match reference encoders output. so far testing matches 100%. Result should be equal output durations.

<0007> split buffer period sei from pic timing sei into their own nalus. mimics characteristic of all reference encoders.

<0006> fixed error caused by me cutting and pasting one of my routines but not changing the variable. was a length variable so caused occasional crashing

<0005>
updated dpb_output_delay to be more accurate on 1080p w/pulldown sources.
added ability to set the level (no compliancy checking is done at this time)

<0004>
a little tweaking... 720p/480p(59.94fps with pulldown) seems ok but sca doesnt account for the pulldown so you get wrong timecodes and exceeds bitrate errors (since its muxing the frames too fast) it has the same prob with othersamples ive found so maybe its sca? anyways ...
1080p 29.97fps pulldown seems ok. waiting to hear :)

<0003>
fixed (hopefully) cpbremovaldelay values to get better timing on pulldown. also writing dpboutputdelay has been updated. for dpboutputdelay i use max values (0 for b frames and for all other types the max using max_dec_frame_buffering as reference

<0002>
added - allows for pulldown for ntsc 720p and 480p 24fps->59.94fps by using frame doubling and tripling.
(25p->50p PAL frame doubling support will be added later)

[Sagittaire]

Really interessing. Test in progress ... thx

Possible to add these flags:
- Change level and profil
- add sequence end code

[b_squared]

I know you can change the level easily (one of these days I should work on a patch for x264 that will automatically set the level according to the input stream size and desired bitrate, and not be user selectable), but I don't think you can change the profile without breaking some major stuff. Am I wrong?

[Sagittaire]

Well I test with x264 with hrd patch and this command line:

x264.exe --keyint 14 --min-keyint 1 --bitrate 4000 --vbv-maxrate 24000 --vbv-init 1.0 --vbv-bufsize 14745 --nal-hrd --mvrange 255 --level 4.1 --bframe 2 --b-rdo --bime --weightb --ref 3 --mixed-refs --direct auto --deblock -1:-1 --pass 3 --stats "x264_stat.log" --qcomp 0.75 --ipratio 1.10 --pbratio 1.30 --partitions "all" --8x8dct --me "hex" --subme 6 --trellis 1 --aud --sar 1:1 --progress -o H264_6Mbps.h264 Encodage_HD_NTSC.avs

Source is 1920*1080*23.976 with 2257 frames. I use h264info alpha 0001 for make pulldown flags.

I check the ES stream and I have a problem. The total duration for ES is not good. I obtain Tf = 24/30 * Ti.

[Sagittaire]

You want valide stream (with correct pulldown & framerate) to compare the result ... ???
The way for HDDVD/BD compatibility is really not far now.

[bond]

Possible to add these flags:
- Change level and profili dont think this should be doable. it will only lead to borked files

good would be a tool that checks the streams features and set automatically the lowest profile/level possible (or lets the user choose to set profiles/levels himself that fit the stream)

[Trahald]

@bond . yeah.. thats not a bad idea .. i can easily add changing the profile and level as the entire stream is rewritten, but avoided it for that reason (making a bad stream)

@sagittaire how far off is it? i used main concept to make my reference stream. but made only short ones as 1) pc was really slow then and 2) short streams easier to test. i have to make some bigger ones for reference as my pc now is a little less slow . as soon as i get some time.

[b_squared]

I have a PS3 and it is very picky about what it can play. If a stream is wrongly marked as level 5.1 (oh God why do people do that? That's for UltraHDTV or whatever it will be called), it will refuse to play it, even though it might perfectly capable of.

The only way of fixing this is by setting the level correctly. The PS3 will happily take anything up to level 4.1.

It would be very interesting though to see what if I got something that was marked High Profile and mislabeled it as Main Profile. Maybe the PS3's decoder is perfectly capable, but just refuses to take anything marked as High Profile. Obviously, somewhere in there there is code that decodes High Profile streams, as that's required for Blu-Ray discs. It could be just a silly limitation.

[bond]

Trahald

i am having another feature idea: generally merging two avc streams is only creating a new correct avc stream if the two parts use exactly the same encoding options

now merging two streams is also possible with different options BUT they need to have different sps ids (are there pps ids too?)! normally encoders always write the same ids (so you cant merge that streams), so it might be good to have a tool that lets you change the ids so you can create a spec compliant merged stream

[Trahald]

0002 binary available

[bond]

:thanks:

btw the standard extension for raw avc streams is .264 (as outputted by the reference encoder). you might want to add it to your list of extensions

[Sagittaire]

Unfortunaly always the same problem : after 23.976p@29.970i with pulldown flag convertion, Streameyes don't report the good duration for the stream. Importation in scenarist SCA will not report good duration too ...

[Trahald]

sag - k.. will probe further

[Trahald]

0004 up.. will add level profile changing next.

[Sagittaire]

a little tweaking... 720p/480p(59.94fps with pulldown) seems ok but sca doesnt account for the pulldown so you get wrong timecodes and exceeds bitrate errors (since its muxing the frames too fast) it has the same prob with othersamples ive found so maybe its sca? anyways ...
1080p 29.97fps pulldown seems ok. waiting to hear

Sorry for the delay but I make long test with complete 1080p encoding movie.

I think that pulldown for 59.94 fps must be pulldown for frame and not for field (progressive pulldown). I will make little 720p sample like for 1080p ...

[Trahald]

for the 29.97, short tests seem to be ok. (ie scenarist has correct time.. buffer analyzer doesnt work at all for me in vista) .. im working on longer source too (but will be probably a 3-4 days or so before i have a full movie encoded.)

as per 59.94, i used a frame based source. scenarist gets it wrong.. but like i said.. i even fed it 59.94 straight from cinemaster and it gets the time wrong and ignores the flags during compile.

[Sagittaire]

Well this time x264 in HDDVD NTSC work (1080p29.97 with pulldown flag).

I test x264 with this command line ...
x264.exe --keyint 14 --min-keyint 1 --vbv-maxrate 24000 --vbv-init 1.0 --vbv-bufsize 14745 --nal-hrd --mvrange 511 --level 4.1 --bframe 2 --b-rdo --bime --weightb --ref 3 --mixed-refs --direct auto --deblock -1:-1 --crf 24 --qcomp 0.75 --ipratio 1.10 --pbratio 1.10 --partitions "all" --8x8dct --me "hex" --subme 6 --trellis 1 --aud --sar 1:1 --cqmfile Sagittaire.cfg --deadzone-inter 6 --deadzone-intra 6 --aq-strength 0.25 --aq-sensitivity 5 --progress -o ED_24.h264 Encodage_HD_NTSC.avs

... and the mux in Scenarist SCA work really well. It's now possible to make NTSC HDDVD with x264.

[Trahald]

woohoo.. i only need 2 more days to finish encoding to compile my first. ;) if all works well after after a disk or 2 i may make a x264 binary from the hrd patch that also adds the pulldown flags for 1080p. until i figure out whats up with 720p i'll stick with 1080p

[Sagittaire]

Here little trailer with x264 and HDDVD NTSC:
http://dl.free.fr/F0pYxMwy/DiscImage.rar

Test on your HDDVD player ...

[Sagittaire]

i may make a x264 binary from the hrd patch that also adds the pulldown

Good idea ... :thanks:

[Trahald]

alpha 0005 available

[Sagittaire]

alpha 0005 available

I will test that. alpha 0004 work very well for 1080p23.976 source (test for complete movie at 200 000 frames).

[woah!]

thx for this :)

it seems multi cores doesnt work once the hrd cmd is used in my script :( 3 cores sitting idle is painful to see heh..

is it possible to even use this cmd on multicores?

ACA seemed to take a test encode just fine tho so it works here so far.

[quantum]

Scenarist ACA won't take a 1280x720 23.976 to 59.94 pulldown. However it takes 1280x720 23.976 to 29.97 pulldown and will mux and build despite h264info's warning message that its not HD-DVD compliant.

I could not get either 1280x720 23.976 to 59.94 or 23.976 to 29.97 to work in SCA.

[woah!]

i have a weird issue.

it seems after the pulldown is done, the file once imported into ACA has the right timeline except the frames at the end of the timeline are different.

eg:

orig vc1 file: [01:39:09:23]

orig ec3 file: [01:39:09:23]

pulled x264: [01:39:09:04]

encoded ac3 file : [01:39:09:23]

so even tho the time is correct the frames arent, does this maybe have something to do with non-drop and drop frames ??? when the pulldown is written does it do anything to that part of the file maybe??

its a weird one for sure and of course causes sync issues.


NOTE: if i mux the non-pulldown .264 with the ac3 file to mkv it is perfect in sync by the way.

[survivant]

I'm trying to create movies playable for the PS3. Sometime the file play well and sometime it's not. I'm using the same setting. The files are encoded with 4.1 level. 23.976 or 29.97 fps. I'm using the official x264 build.

there is something that I have to check to ensure that the movie will play fine ? (and suppose that the movie don't have sound, will be easier to test).

[Trahald]

@woah
0008 should match duration up closer if not right on.. let me know.

big warning tho.. if you used min key int of 1 then there is probably going to be duration issues that wont fix. it seems consecutive I frames mess up duration (thats even when using a pro app like cinevision)

@survivant
does the psp need files that have pulldown? i would assume that a native 23.976 should work fine which really means h264info isnt needed. I guess im not sure why you are using this app.

[Trahald]

0009alpha -- green blockies should be gone for 59.94/60 fps.

[woah!]

ok i will try to be understandable here :)

i tested your alpha 0009 and your 663G.x264.exe

1920x1080 @ 29.97 = works correctly on A2 player

1280x720 @ 29.97 = works correctly on A2 player <-- very nice result actually i am impressed. i might need this program to work correctly now heh...

1280x720 @ 59.94 = does not work on A2 player error code (408bc504) seems to mean it cant play this disc type or some sh*t.

only thing i see that "might" seem weird is that at 59.94 it chooses a 120000 time scale? the other 2 options its 60000

when i play back the 60fps pulldown output from h264info in powerDVD, it runs at 2x the speed it should.

[woah!]

hmm you know it could just be the a2 cant play this style of disc plain and simple. your program could well be working perfectly, as i can play the authored 60fps folder ok in powerdvd?

does a 720p 29.97fps really matter that much to me? it plays ok? maybe i will use this for long movies until more info is found out..

talking to myself here heh...

[Trahald]

hehe... well there was some bad code that needed fixing that caused green gobblies when 59.94 was used. (was reported in another thread) that was the only change..

120000 (/1001) is how we get 59.94 .. thats ok

and yeah.. if 29.97 works thats cool.. i'll keep tweaking 59.94 but dont kill too many dvd-9s on it if 29fps works.

[Sagittaire]

@ Trahald

you nal_hrd patch don't write correctly the hrd flag. StreamEyes (buffer analyser) report big overflow problem but I'am sure that the stream is vbv compliant.

[Sergey A. Sablin]

@ Trahald

you nal_hrd patch don't write correctly the hrd flag. StreamEyes (buffer analyser) report big overflow problem but I'am sure that the stream is vbv compliant.

if it is overflow in VBR mode, that this is 100% error with initial_cpb_removal_delay's/cpb_removal_delay's. If it is CBR, than it is absence of padding - in case of overflow in CBR case encoder shall write Filler SEI messages to prevent overflow.

[Trahald]

@ Trahald

you nal_hrd patch don't write correctly the hrd flag. StreamEyes (buffer analyser) report big overflow problem but I'am sure that the stream is vbv compliant.

not sure if you mean x264 patch or h264info. x264 hrd patch uses the max value (still Ian Caulfields original calc) for initial cpb removal delay. that value has the biggest effect on the buffer. during pulldown i only change cpb_removal_delay which has no effect... (to be specific, no matter the state of the buffer, cpb_removal_delay follows the same pattern strictly based on decode order/frame type.) hours of looking at pro apps output have confirmed this. dpb_output_delay also is strict to time and not buffer. i concede initial cpb removal delay is inaccurate (its set at the max value depending on bitrate but should adjust gop to gop and doesnt). ive only been tweaking the code i added to the patch.

h264info must be fed a stream with hrd there already, what it uses for values on streams w/out hrd were just fillers i put in when originally coding it.

can you make a stream available i can look at that will show the effects in buffer analyzer?

[Sagittaire]

not sure if you mean x264 patch or h264info. x264 hrd patch uses the max value (still Ian Caulfields original calc) for initial cpb removal delay. that value has the biggest effect on the buffer. during pulldown i only change cpb_removal_delay which has no effect... (to be specific, no matter the state of the buffer, cpb_removal_delay follows the same pattern strictly based on decode order/frame type.) hours of looking at pro apps output have confirmed this. dpb_output_delay also is strict to time and not buffer. i concede initial cpb removal delay is inaccurate (its set at the max value depending on bitrate but should adjust gop to gop and doesnt). ive only been tweaking the code i added to the patch.

h264info must be fed a stream with hrd there already, what it uses for values on streams w/out hrd were just fillers i put in when originally coding it.

can you make a stream available i can look at that will show the effects in buffer analyzer?


Well complete report:
- Your build x264_663g work perfectly now but not x264_663f
- Dark Shikari make x264 build for test SSIM optimisation with your patch but the NAL_HRD are not correct. There are big overflow problem exactly like for your x264_663f build (scan with Streameyes). Can you update your patch ?

[Trahald]

ahh.. ok... i'll have a 'g' patch out in an hour or so

[bob0r]

You found x264's G-spot?

[Kurtnoise]

:D



In fact, trahald uses a,b,c,d,e,f,g,....for his versioning.

[Kurtnoise]

Trahald: could you upload the sources of the last release please ?

[Trahald]

Are you sure? If you look at my code, youll go blind! :) Sure... give me a day or 2.. i dont really keep special copy so will be .11 with anything i coded since (not much)

[Kurtnoise]

Well...I'm not a MFC fanatic you know. It's just to implement command line args in it. If you have some time for this feature, that could be great.

[Trahald]

Source for 12 is up. will have binary available later.

[lazyn00b]

Hello Trahald, and thank you for h264info - I have been using it with success to apply pulldown to progressive streams encoded with x264. However, there seems to be a problem authoring these streams to Blu-Ray format with either Scenarist HDMV or Adobe Encore CS3. For example, if I encode a 480p23.976 source with x264, apply pulldown with h264info, and then attempt to author the stream with Scenarist, Scenarist accepts the file as an asset but complains that the file can not be authored as the "Primary" video stream. This is because Blu-Ray format allows 480i as the Primary Video Stream, while 480p is only allowed to be a Secondary stream (e.g. for PIP). Similarly, Adobe Encore accepts a pulldowned 480p stream as an asset, but refuses to author it without re-encoding first.

Out of curiousity, I re-encoded my source with the HC MPEG-2 encoder, which applied pulldown to make the m2v output file DVD compliant. This file was accepted by Scenarist HDMV, which had no problem muxing it as the Primary video stream. I looked at the properties of the m2v in Scenarist and noticed that it was flagged as "interlaced", whereas the x264 stream with pulldown applied by h264info was flagged as "progressive". This disparity is interesting because both streams were 23.976 fps progressive video with pulldown applied.

As a final test, I re-encoded with x264, but this time with interlaced encoding turned on. After applying pulldown to this stream, Scenarist and Encore were happy to mux it! However, this method of encoding is bad for quality and encoding efficiency.

This raises a question: is it possible to flag an H.264 stream as interlaced when you apply pulldown? If so, would it be possible to include this as an option in h264info?

[Sagittaire]

@ Trahald

You can make new patch for x264 with new PPS picture flag ... thx

[woah!]

Hello Trahald, and thank you for h264info - I have been using it with success to apply pulldown to progressive streams encoded with x264. However, there seems to be a problem authoring these streams to Blu-Ray format with either Scenarist HDMV or Adobe Encore CS3. For example, if I encode a 480p23.976 source with x264, apply pulldown with h264info, and then attempt to author the stream with Scenarist, Scenarist accepts the file as an asset but complains that the file can not be authored as the "Primary" video stream. This is because Blu-Ray format allows 480i as the Primary Video Stream, while 480p is only allowed to be a Secondary stream (e.g. for PIP). Similarly, Adobe Encore accepts a pulldowned 480p stream as an asset, but refuses to author it without re-encoding first.

Out of curiousity, I re-encoded my source with the HC MPEG-2 encoder, which applied pulldown to make the m2v output file DVD compliant. This file was accepted by Scenarist HDMV, which had no problem muxing it as the Primary video stream. I looked at the properties of the m2v in Scenarist and noticed that it was flagged as "interlaced", whereas the x264 stream with pulldown applied by h264info was flagged as "progressive". This disparity is interesting because both streams were 23.976 fps progressive video with pulldown applied.

As a final test, I re-encoded with x264, but this time with interlaced encoding turned on. After applying pulldown to this stream, Scenarist and Encore were happy to mux it! However, this method of encoding is bad for quality and encoding efficiency.

This raises a question: is it possible to flag an H.264 stream as interlaced when you apply pulldown? If so, would it be possible to include this as an option in h264info?

i would like an interlaced option added aswell, but to your other issues, why are you adding a pulldown to the 23.976fps video? in bluray (i believe) it doesnt need a pulldown as it reads 24fps and 23.976fps directly.

[Trahald]

@ Trahald

You can make new patch for x264 with new PPS picture flag ... thx

yeah i was planning on it... should have something out in the next few days.. (simple to code actually but computer acting up)

[lazyn00b]

i would like an interlaced option added aswell, but to your other issues, why are you adding a pulldown to the 23.976fps video? in bluray (i believe) it doesnt need a pulldown as it reads 24fps and 23.976fps directly.

Unfortunately, Blu-Ray supports 1080p at 23.976 fps, but not 480p. I thought applying pulldown to 480p23.976 would be enough, but apparently the stream must be flagged as "interlaced" as well to be a compliant Primary Video Stream. Secondary Video Streams allow 480p, but this doesn't help because you must have a Primary before you can author a Secondary :(

[bond]

trahald, can you please add the possibility to your tool to change the cropping values in the SPS? this way people can crop video streams without needing to reencode them

:thanks:

[Trahald]

hmm.... shouldnt be an issue. hopefully have some time this week.

[Sagittaire]

yeah i was planning on it... should have something out in the next few days.. (simple to code actually but computer acting up)

... some news ... :helpful: