GPL command line M2V transcoder

[Eyes`Only]

DVDRFreak: no problem ;)

[dragongodz]

Nic - do you have a working email for rejig.org ? i have done a little more tweaking (to both engines) and thought people would like to try/test them out. basically original engine should down size more accuratly(and lower) while new engine should look better(less blocky) and still hold goof sizing for 70%(but 50% is FUBAR more than it already was). since mike never put the last changes and he hasnt messaged here for a while i was wondering if sending it to you via rejig.org email would be possible so you could put it on the development page ?

the question about P frame quantisation probably came about because i said it isnt done. well with the old engine P frame non-intra does not change quant but drops last coefficient. so i basically said proper requant of P frames should help produce better sizing and better quality by taking some of the work/reduction off B frames. yes the new engine does do it but too heavily which is causing the blockyness. sorry for any confusion over this. :)

[dragongodz]

"However, this topic definitely doesn't belong in a pro-Rejig thread"
not at all. its the truth that transcoders will not and can not give the same quality for high compression amounts. if you are talking reducing the size a small amount then transcoders can do a fantastic and fast job but for large compression proper re-encoding is the only real way to get the best results.

[JvD]

Just sitting here waiting, nothing to do (accept working ;-) so it would be nice to try it out.

[dragongodz]

hahaha well jvd if you email me (my email is on rejig.org) i will send it to you.

its 0.4i modified meaning you need to do the registry change to change which engine to use of course.

please dont everyone ask for it though unless maybe you have somewhere you can make a link for others to download from until its put on rejig.org.

[JvD]

Well, I sent u an mail but all versions should be up on rejig.org for everyone to evaluate (as soon as possible). Ill try the same move with the old end the new (your) version and compare the too.

[dragongodz]

jvd - you've got mail. :)

yes i agree it should be on rejig.org for all who want to to try. problem is i cant do that. mike(mmgrover) started the site and put the first version i made changes to (0.4b) online but he hasnt been around and never put up trhe last changed version i did. i remember nic asking for access to do it so hopefully he has it. i will try emailing it to his rejig.org email in hopes that he can put it up anyway.

[Nic]

@dragongodz: You can always get me at nic@nic.dnsalias.com nic@rejig.org should work and get to that address too, but nic@nic.dnsalias.com is the direct one.

As DVDRFreak said, they seemed to update source on Saturday, have you looked at making changes to that one? I didn't do a diff so I cant tell the difference between the newest and the one in Rejig...

-Nic

ps
Uploaded to development section at www.rejig.org

[mmgrover]

@dragongodz

I have been sick so I haven't been able to add your code to the
development section.


@Nic

Thanx for uploading that
Yes, I hope to feel better today :-)


mike

@dragongodz, JvD
Yes, Nic has full access to rejig.org to make changes as he see's
fit.

[Nic]

Did a diff of the version released on the 30th of november by metakine...

Directly after the line:
else threshold = greedyFactor;

This has been added:
if (threshold < 1024) threshold = 1024;

Thats the only difference :(

Adding that line into dragongodz code will bring it up to date, ill add that in tonight into a new release
(Was up until half past midnight fixing the SUP stuff (PTS reset/wraparound issue) all fixed now :) )

@mmgrover: Hope you get better soon :)

-Nic

[TCmullet]

Originally posted by Eyes`Only
TCMullet: CCE is my tool of choice for reencoding .m2v files. You can get excellent quality even at low bitrates (something no transcoder can do currently). Though there's an upside (quality) there's also a downside (speed) so it's basically dependent on how much you value quality. I personally don't mind the extra time for the output I get. However, this topic definitely doesn't belong in a pro-Rejig thread :)
Actually it kinda does. I'm here (in ReJig-land) because on the Dazzle board I was remorsing that I had just finished reencoding some big CBR files down to smaller VBR ones via Tmpgenc, had deleted the originals, then found out that my CD-R dealer had half-lied to me about the capacity of a DVD-R. He said they are 4.7GB. What he should have said was 4.7 billion or 4.37GB. I haven't corrected him yet, but my Dazzle friends helped me to see that "all is not lost". And here I am.

Now as far as reencoding... Isn't CCE many hundreds of $? I have very close to absolute zero. Also, my experience with reencoding is that the bitrate of the source has to be significantly greater than the target bitrate. It's not the speed of Rejig that I crave so much (although every bit of speed helps!), it's the idea of throwing away all that valuable encoded data by decoding to raw video and totally encoding from scratch. A LOT of picture quality is lost by that. (As well as time.) This is why the idea of ReJig is so thrilling to me--keeping most of the 1st generation encoded data. I think I realize that using Rejig with a low percentage (30-50%) is probably a lost cause (someone prove me wrong), but at least above 80%, it appears to be a godsend.

So back to my original question.. Is rejig the only transcoder (that doesn't totally reencode like CCE, Tmpgenc, and bbMpeg) that can accept mpeg2 PS or ES streams?

[JvD]

OK guys, now I have tested dragongodz version and compared it to the new one. First the size. I did a movie only, with one audio stream and one subpicture stream, both at approx 69%. The old engine produced a complete set at 4370MB, and the new (dragongodz version) end at 4300MB (2-pass). So according to my little test, the old engine is still better to give an accurate result (size)
.
Then the picture.

Old engine (use old engine=1): http://www.morck.se/old.jpg

new engine (use old engine=0): http://www.morck.se/new.jpg

This is in a medium-fast-moving scene (from the movie 9 Queens). In this frame the old version still produces a less blocky picture. Sorry, i didnt have the time to watch the to versions on my standalone so I cant really say anything about an overall impression. These pictures are exampels.

[dragongodz]

jvd - thanks but this is the sort of information i am after.

old engine(nic build) compared to old engine(my build) resizing accuracy. so what is the original .m2v size, what ratio used, what each size each build produced.

new engine(nic) compared to new engine(me) blockyness. that means which build produces blockier output ? does mine look better or the same ? no quality will not be as good as old engine yet but if there is a noticable improvement then it lets me know i am working on the correct area.

Nic - cool, i will send any other changes to both you and mike when i make them. oh and yes i had a quick look after there was the mention of the new engine changed but didnt notice any thing(no time to do a diff then) so didnt worry about it for now.

MMgrover - hi mike, hope you are feeling better. i was just explaining why my changed version was not on rejig.org. :)

[E-Male]

judging from my own tests (not many) i'll stick to the old engine for now

one thing i noticed is that rejigs quality depends highly on the source:

-noisy, badly filtered input ==> ugly output but i daubt reencoding would have gotten much more of that source (really watchable on TV)
-lil grainy input ==> some noise-/artefact-patterns (should look fine on TV) [depends on original bitrate and quality of the original encoding]
-clean input ==> quality varies between good and perfect (should always be great on TV) [and i'm talking about <60% here!]
(all sources where "real" movies, no aniamtion of any kind]

about the rejig vs CCE discussion, i don't have the time/resoruces/nerve for CCE, i either use rejig or split the disc, depending on what DVD it is

EDITed after testing the transcode of the bad source on TV

[JvD]

dragongodz, this was only a small test of the version you sent me. I ran your version with the variablem "use old engine" set to either "1" or "0". I can compare it later today with other versions?

[mmgrover]

I'm feeling better today, back at work :-)

let see if I can catch up to what's going on.. lol


mike

[TomSem]

hi,

i use rejig and ifoedit author.
The movie plays fine on windvd.
But my homeplayer pauzes a the end of EVERY chapter.
I have to press next to continue to next chapter and at the end i have to press next again to see chapter three ... and so on ..
This is my second movie that has this problem.
Both movies Fast forward and fast rewind also doesn't work.

Anybody can help me out ?

[layer3maniac]

NTSC?

[Rombaldi]

A Question for the Transcoding Gurus.....

Is it POSSIBLE to modify any of the video parameters while transcoding... in specific the change the black level?? I've
got a coupla of hundred hours of MPEG2 files that were
recorded with the black level set to high, I would like to be
able to subtract 7.5 IRE globally thru the entire file...

(I suppose that the same question could be asked for any of
the RGB values as well.. be able to do any modifications
to them either??)

[int 21h]

Originally posted by Rombaldi
A Question for the Transcoding Gurus.....

Is it POSSIBLE to modify any of the video parameters while transcoding... in specific the change the black level?? I've
got a coupla of hundred hours of MPEG2 files that were
recorded with the black level set to high, I would like to be
able to subtract 7.5 IRE globally thru the entire file...

(I suppose that the same question could be asked for any of
the RGB values as well.. be able to do any modifications
to them either??)

Sorry, no.

[JvD]

Yesterday, i did the test again to try to keep as many factors as possible constant. Since i was sitting on a train (and nothing else to do) i wrote all info down in html and published it this morning. Its really nothing new but since i done it i might even publish it ;-) As stated before, the old engine is still quiet good and gives very good size (when remuxed).

Go here for the results (http://members.fortunecity.se/jvd1/index.htm)

Since it soooo quiet in this thread i assume you guys are doing some hard working on Rejig. Or are you suffering from a cold?

[unixfs]

I didn't know of such parameter: "black level".
Can anyone explain what it is, please?
Is it responsible for those nasty greyish blocks that
appear in black areas using ffmpeg?

Thanks.

[dragongodz]

JvD - thanks that tells me more. hmm was that my build or Nics that you used for old engine ? can you try the other build old engine aswell ?
well i can see a little more fine detail in my build new engine (nose and ear are slightly blurred more with Nics build) but they are still very similar. i will do some more playing with it tommorow.
as for output size, hmm strange for the new engine to undersize so much. try using 1 pass with new engine set to 68.48% and see what that gives you please.

i havent been working on it because i have been busy with other things and really didnt want to do anything until i got some feedback. Nic no doubt has also been busy, read all the things he has been working on in previous posts.

unixfs - did you try the resizing with the old engine in my build ? resize smaller ?

[Mug Funky]

@unixfs:

black level isn't an MPEG issue... just a matter of what level the darkest luma value is. (ie. 8 for CCIR, 0 for regular RGB, and if something's captured wrong or there's generational level clamping you could get up to 24-32 as a black level).

kinda like "noise floor" i suppose...

[unixfs]

Originally posted by dragongodz

unixfs - did you try the resizing with the old engine in my build ? resize smaller ? [/i]

Yes, but the smallest size is still 33% of the original (sorry, I forgot to report).

I made sure to enable the old engine.

[TomSem]

to layer3maniac
yes , you're right.
I ve problems running NTSC on my player
No problems with original NTSC discs.
Only troubles after rejig and ifoedit author.

Anybody having same problem ?

[JvD]

Originally posted by dragongodz
well i can see a little more fine detail in my build new engine (nose and ear are slightly blurred more with Nics build) but they are still very similar. i will do some more playing with it tommorow.
as for output size, hmm strange for the new engine to undersize so much. try using 1 pass with new engine set to 68.48% and see what that gives you please.

Ill try it later today. Ill run a one-pass with the new engine tonight and fill that page with more information. The undersizing problem, is it because of the second pass?
As I said before, I havent made an overall judgement on a standalone player.

[Nic]

Yup im busy on writing a backup wizard for the impending 0.5 release...

A real Newby question: Are NTSC disks always the same framerate? if so what is that exact number? (I dont own any NTSC disks...so I dont have any first hand evidence)

-Nic

[unixfs]

there are NTSC discs that change back and forth to and from 23.976 and 29.976

[JvD]

Cant remember what you guys said before but if I use the one-pass option in the latest version do rejig use the old engine or the new? When I did tests I assumed that Nics and dragongodz versions containd the SAME "old" version? Or are there different "old" versions?

Nic, looking forward to the new version. *very excited* ;-)

[Rombaldi]

Might I make two (probably not so small, but I can hope) suggestions.

1) mayhap a simple 'check off' on the interface as to which engine to use? (pair of radio buttons where one or the other must be checked). Perhaps simpler than reg hacks.

2) I suppose I'm in the minority, but I'm using ReJig mostly (99%) for M2V files. Archiving TV down to disc and it's handy to give them a light squeeze (no more than 10% on most) to get a better stacking of episodes (the difference between putting 5 and 6 eps on a disc is sometimes less than 500mb). It would be nice if you could just DRAG/DROP files into the window (I may do 10 or 15 files at once). If you open multiple files in the OPEN dialog, it concats the files together and it makes one big output file instead of 10 individual ones). There a bunch of folks that have found ReJig and are using it for non-dvd backups, but just as a 'raw' file compressor.

[int 21h]

Originally posted by Nic
Yup im busy on writing a backup wizard for the impending 0.5 release...

A real Newby question: Are NTSC disks always the same framerate? if so what is that exact number? (I dont own any NTSC disks...so I dont have any first hand evidence)

-Nic

NTSC generally come in 3 flavors that I've seen commonly:

24fps progressive with pulldown for 29.97 (i.e. FILM)
29.97fps interlaced (TV stuff, low budget things, animation)
24fps and 29.97fps mixed (TV stuff, things with visual effects sometimes)

[sundance]

@nic:
...and while we're at suggestions:
Maybe it's easy to implement an option (like the one in DVD-Decrypter) which allows you to reset the time stamp of the m2v file generate to 0:00:00. That could make life a lot easier to import those files (into Maestro e.g.) together with chapter marks/subs.
If that's a tough one then forget it, since it's just "nice to have"; there are several work-arounds for that issue...

[TCmullet]

Please help me out somebody. I don't understand how it can be meaningful to concatenate .m2v files. When I want to rejig my program stream, I demux out the .m2v video stream and the .mp2 mpeg audio stream. Then I rejig the .m2v multiple times to my satisfaction, then I mux it back together with the .mp2 to form a new smaller mpeg program stream. I know of no way to concatenate mp2 streams. I only can use mpegvcr, or Tmpgenc tools to concatenate PS files, not elementary files. What good is m2v concatenation if there's no way to also do the mp2 files so I can mux the joined audios back in as one step?

[unixfs]

with TMPG you can merge 2 mp2 files in a single PS and then demux the resulting mp2.

[E-Male]

can't besplit do that in one step?

[dragongodz]

new mod(3) of version 0.4i has been sent to Nic and mike so should be up on rejig.org soon.
should be slightly better quality for old engine and slightly better quality for new engine aswell.
for tests can you check actual playback and not just a single frame ? this will be better to judge quality.

JvD - can you also compare the last version using playback rather than the 1 frame ?

and people some more feedback would be nice even if you only do a couple of short tests.

[E-Male]

i'm low on time, but i'll try to do some comparison with different sources

i'm right when i think that the amount of grain is the main factor for the "transcodeability" of a source?

[TCmullet]

Originally posted by dragongodz
new mod(3) of version 0.4i has been sent to Nic and mike so should be up on rejig.org soon.Can we trust that the version number will be bumped up, such as 0.4j?

[mmgrover]

@TCmullet

I released dragongodz's version and called it 0.4i-mod3.


mike

[JvD]

@dragongodz: Ill do some testing as soon as possible.

I did some more testing using the one-frame-method and the result is not too exiting. I tested IC8, DVDShrink (with and without deep analisys) and Nero Recode and looked on the exact same frame*. My only conclusion was that that method is really bad in saying something about the overall picture quality of the movie. One single frame doesnt say much, you have to look at several frames and different kind of scenes. The best way IS to watch the movie on the standalone or the monitor, then you can make an overall judgement. Ill do that in the future. I might also pick a movie that is considered to have REALLY GREAT image (any suggestions?).


*By this simple (read not-too-good) comparisment IC8 gave the best picture, very close to the original. But then again, it cost money and is rather slow (and I dont like the interface).

[E-Male]

as i already said
i think the better the soruce the easier the transcoding

for example fightclub transcoded much better then thunderbolt because of it's (MUCH!) cleaner image

EDIT (forgot my conclusion):
so movies with great image quality are not the best sources for tests, well not alone at least
for good test sources of different quality are required i think

[Nic]

@dragongodz et al:
http://nic.dnsalias.com/PSNRCheck.zip

Use PSNRCheck infile1.m2v infile2.m2v

Feed it two raw M2V files (one the original and one newly rejiged) and it will give you a PSNR rating.

Code by Makira, ported to win32 and newest libmpeg2 by me :)

Should be useful in testing quality, but of course PSNR isn't a definite value of quality, but it's a start...

-Nic

ps
Im not sure how accurate/useful this will be...the numbers out don't seem quite right...hmmm

[JvD]

@Nic: What is it PSNRCheck really do and what does it measure? What "factors" are important?

[dragongodz]

Nic - PSNR can be funny at times so maybe using SSIM would be more acurate. i will have a look at PSNRcheck tommorow anyway and see what it does.

JvD - in simplest terms it compares 2 images/frames(original and compressed) to see how similar they are. the closer they are they better the compressed picture should look(since closer to original) so higher PSNR. problem is that something like a near black picture or static scene can have more done to it and still look good but can get a low PSNR value(as example) so it isnt always to be trusted. thats why i said watch an actual scene play, your own eyes are generally the best judge.

E-Male - it depends also on how the oriuginal was compressed(not just noise etc). so yes multiple examples is better than 1.

[TCmullet]

ReJig is giving me information I haven't seen in my 3 years of using the Dazzle DVCII. Specificly, when I add a program stream to the work list, it shows the 2 elementary streams in the upper right window. Because my program streams are often the output of an older (and buggy) version of Mpegvcr (that I haven't been able to upgrade yet), the 2nd line says "MPA Audio Stream - StreamNo: 0xC0 - AudioDelay: -67". A more normal stream would not have the "AudioDelay" clause. Every file that is output from this mpegvcr, when loaded into ReJig, shows either -34, -67, or -101. (I may have seen a -100.) These number are interesting in a side-respect: They virtually equal to 1, 2 and 3 whole frames respectively (assuming the numbers refer to milliseconds).

But here's my main concern. How should I apply my awareness of this number when remuxing the elementary streams back together with bbMpeg? That is, how should we interpret the '-67'? It either means "the audio starts 67 ms earlier than the video" or "the audio starts 67 ms later than the video". In the context of bbMpeg remuxing, the video and audio delay defaults are 180 and 180. I'm inclined to think I should change the audio delay to 113 (which is 180 minus 67, which is 'advancing' the audio). I tried adding 67 getting 247, but the audio was definitely delayed too much. I also tried leaving it alone. (I'm using a volleyball game with those ball smacks as material for testing this sync issue.) 113 seems to sound the best to me of the 3.

So if my original file says '-67' in ReJig, how should I change the value of 180 for bbMpeg's audio delay?

[int 21h]

PSNR values should be between 20 and 40. A value of 40 will be nearly identical to the source. In general the MPEG committee specifies that any difference less than 0.5db is to be considered insignificant. Just remember higher numbers do not necessarily mean better quality, just that its closer to the original source.

[dragongodz]

Nic - PSNRcheck just crashes for me. function trying to access illegal area of memory.

my system - duron 1ghz, 256meg ddr400(100mhz fsb for duron of course), motherboard is gigabyte 7s748(sis748 chipset), windows xp

[E-Male]

crashes for me aswell
i'm using win xp pro (no SP)
i ripped/demuxed chapter 2 of Das Boot superbit rc1 disc 2 with dvddecrypter and rejiged it to different percentages
psnrcheck gives 0 values of teh secodn file doesn't exist and crashes on any second file, even if it's ther same as the first, but only when both files are .m2v

EDIT: same with Ice Age pal riped/demuxed with rejig

EDIT2: tried another disc and i'm now sure it's not source related
a result containign just -0 is given when on-existing or non-m2v files are used, but when there are 2 m2v file used and teh program should give a result, then it crashes

[Nic]

:( Dont know why it crashes...then again I only tested it on 4 minute files (the numbers where wrong anyway I think...).

@TC: The delay value will be correct. I think your assumption of how to correct it is right. I dont know bbmpeg well, your best bet is to try and see if you can tell the difference...

-Nic