FairUse development

[UMP]

I would like to add chapter ripping to FairUse, and I need to convert LBA values to frame numbers.

What would be the right way to determine the frame index for a given LBA value ?

[fu2k]

Once the index has been created and you have an Index::Section object describing the video of interest, work through the 'chunk' array, and then the 'video' array inside that, counting the frames as you go. The 'block' field in the VideoIndexUnit is the LBA relative to the start of that title/type VOB set. So when you get to the block you want, the frame number should be the number of previous frames you have counted (skipped over).

[UMP]

Thank you very much fu2k !...

Your hints were very useful because after I looked at the code in Summary::Create to implement these, I finally realized that no LBA conversion was needed at all (stupid me!) :/

Thanks to you, chapter positionning is nearly ready. :)

ump

[Ookami]

</Lurking>

Thanks to both of you, for your great work!

Nice to see that someone has the time and motivation to continue the work on one of the best programs in the scene...

@UMP

Could you please make an official homepage or something where interested people can download the binaries (and/or the sources)?

Thanks and bye,

Mijo.

<Lurking>

[UMP]

Hello fu2k,

I would need some light on image processing code :D

In the following code, I don't understand what these strange "41 / 16" do mean ?

FrameQueue &FrameQueue::Create(const Vector &dim, unsigned elements, unsigned encoders)
{
[...]
_data = new unsigned char [dim.x * dim.y * 41 / 16 * _elements];
[...]
}

void FrameQueue::_Data(unsigned element, const unsigned char *&ycrcb, const unsigned char *&grey, const unsigned char *&mask)
{
ycrcb = _data + element * _dimSize * 41 / 16;
}

EDIT:

In the same way, what is the "* 5 / 4" for ?

void ResizeVideo::Resize(const VideoDecoder::FrameBuffer &top, const VideoDecoder::FrameBuffer &bottom, unsigned char *dst)
{
_luma.Resize(top.luma, bottom.luma, dst);
_chroma[0].Resize(top.chroma[0], bottom.chroma[0], dst + _resize->dst.x * _resize->dst.y);
_chroma[1].Resize(top.chroma[1], bottom.chroma[1], dst + _resize->dst.x * _resize->dst.y * 5 / 4);
}

Thanks a lot,

ump

[fu2k]

Each 'FrameQueue' element holds 5 things:

(a) luma (size = dim.x * dim.y)
(b) chroma0 (size = dim.x / 2 * dim.y / 2 = dim.x * dim.y / 4)
(c) chroma1 (size = dim.x / 2 * dim.y / 2 = dim.x * dim.y / 4)
(d) grey-scale representation (size = dim.x * dim.y)
(e) quality masking information (size = dim.x / 4 * dim.y / 4 = dim.x * dim.y / 16)

The total size of all of these together is (dim.x * dim.y) * (1 + 1/4 + 1/4 + 1 + 1/16), which is (dim.x * dim.y * 41 / 16). So this term is used when allocating the memory initially, and also for finding the offset of a particular element in the queue.

The '* 5 / 4' is used to get access to the second chroma plane when dealing with luma/chroma0/chroma1 planes that are stored back-to-back in a memory region:

The luma plane begins at offset = 0 and has size = dim.x * dim.y.
The chroma0 plane begins immediately after this, at offset = dim.x * dim.y, and has size = dim.x * dim.y / 4.
The chroma1 plane begins immediately after this, at offset = (dim.x * dim.y) + (dim.x * dim.y / 4) = dim.x * dim.y * 5 / 4, and has size = dim.x * dim.y / 4.

Hope that helps.

[UMP]

From this point of view it becomes clear as crystal. :)

Thank you very much.

[UMP]

Hi fu2k,

I'm now trying to implement stream demuxing to be able to :

1. demux subtitles (to use them either the DVobSub way or even to use SubRip OCR)
2. process non-AC3 audio streams

ATM, I understand that FU is ripping the whole filesystem, and then parses it with internal routine to find where the files are located, etc.) but I still don't deeply understand the way data is structured.
I'll keep on tracing the code, but I was wondering if maybe I don't see that there is a trivial way to do a simple stream extraction ?

Thank you for your support,

ump

[fu2k]

In "Demux::_DemuxPack(...)" you will find code that demuxes the vob stream by stream id. Currently it processes video (0x0E), audio (0xBD/0x80-0x87) and subpicture (0xBD/0x20-0x37). If you want other streams, you will have to add it in here.

From there it goes to "DemuxIndexed::_DemuxData(...)" where the desired subsection of each stream is selected depending on the ranges provided. The video, audio and subpicture streams are then passed to "_DemuxIndexedVideo(...)", "_DemuxIndexedAudio(...)" and "_DemuxIndexedSubpicture(...)" respectively.

Inside the more derived class "Decode", these streams are fed to decoders of the appropriate type. If you wanted to write a subpicture stream to disk, you could do this with a different type of subpicture decoder at this point.

If you want access to different types of audio streams, you will need to demux them inside "Demux::_DemuxPack(...)". You can then either map them to different audio stream numbers and feed them into the current audio path, or create a parallel path for this new stream type.

[UMP]

I implemented quick and dirty subtitle demuxing and IFO file extraction. With a few more work to cleanup the code and to add some controls in the GUI that will do the job. :)

Thank you very much !

ump

[UMP]

Hello,

quote from FU's readme file :
0.23 beta Added a key fallback mechanism and more diagnostic output to try
to resolve the problem with a few DVDs where the key can't be
found

0.24 beta Removed the key fallback mechanism that was added in 0.23 beta
because it didn't fix the key problem

0.25 beta Added an automatic key retry mechanism to fix a problem getting
keys from some DVDs using some DVD drives; you need to create a
new project if you want to take advantage of this fix


This problem still happens sometimes to a few people, not with every DVD. May I ask for more info about this issue, and maybe even a hint or a clue about why this would happend and/or how this could be solved please ?

I first thought I broke something after implementing SPTI, but it even happens with FU 0.31 (built from the original released source without any modification).

Thank you very much for your support,

ump

[fu2k]

To solve this, you will need to gather more information about the problem. If you have a DVD exhibiting this problem yourself then this will be easier. If this problem is only seen by other users then it's going to be more difficult.

First, I am assuming that you are getting "no key for encrypted block" messages in the diagnostic window. This is caused when an encrypted block is encountered and no decryption key is currently known.

In "Demux::_DemuxChain(...)" there is code that tries to find the decryption key at the start of each cell. Really, it should only be necessary to get the key once per title, rather than per chain or per cell, but a few years ago there were a few incorrectly mastered DVDs that changed keys in the middle of a chain. I don't know if there are many of these DVDs still in circulation.

Trying to get the decryption key at the start of each cell might be causing this problem, especially if the first block of the first cell is not itself encrypted.

You should gather more information about the failure case by adding diagnostic messages to "DvdUdf::Key(...)", "DvdCache::Key(...)", "Css::TitleKey(...)" and "DvdIo::CtlReadTitleKey(...)".

Also be aware that the ".fud" files can only store 512 keys, so if you are trying to process video with a total of more than 512 cells then you will run into problems ("Key map table full").

[Nic]

@UMP: Do you happen to have a list of DVDs that cause this problem? Or a link to any more info. I'm always interested in all things decss and i'll try to look into it if you get stumped or need any help :)

-Nic

[UMP]

@fu2k: Thank you very much for these informations and hints, yes I was talking about the "no key for encrypted block" error.

@Nic: One FU user was experiencing this issue with "LOTR - fellowship of the ring", but I was not able to reproduce it myself. The problem occurs when trying to get keys "from some DVDs using some DVD drives", this is something I can't explain.

Trying to get the decryption key at the start of each cell might be causing this problem, especially if the first block of the first cell is not itself encrypted.
FU looks for a key at each new cell, but it was always using the first block of the cell. I changed the code to use a random block from the cell, which is the way SmartRipper proceeds. I then submitted this to the user experiencing the "no key found" issue. He still has to give some feedback.

Regards,

ump

[UMP]

Hello,

why does the code "correct" the aspect ratio by 54/59 or 11/10 ? Is it related to a difference between TV and PC monitor aspect ratio ?

dar = 720.0 / (_summary.IsPal() ? 576.0 * 54 / 59 : 480.0 * 11 / 10) * (_summary.IsAnamorphic() ? 4.0 / 3 : 1.0);

Best regards,

ump

[fu2k]

This adjustment is because the pixels stored on a DVD are not square, but the pixels stored in an AVI need to be square (usually). The actual pixel aspect ratio on a DVD depends on NTSC/PAL and anamorphic/not. See ITU-R BT.601 for much aspect ratio fun.

[UMP]

Originally posted by Nic
@UMP: Do you happen to have a list of DVDs that cause this problem? Or a link to any more info. I'm always interested in all things decss and i'll try to look into it if you get stumped or need any help :)

-Nic
Several CSS related issues were solved recently, but I came through an issue I don't know how to deal with :
FU won't be able to index (rip) from a DVD that has a mismatched region code on an RPC2 drive. :(
AFAIK, only DVDDecrypter is able to rip a DVD in such case.

Nic, some help on solving this issue would be very nice :)

Regards,

ump

[fu2k]

The problem is that the DVD drive firmware is refusing to hand over keys because it has noticed the region mismatch. You can still authenticate with the drive and get the encrypted data though, so it's possible to apply the CSS known plaintext attack if you can find any vulnerable blocks.

[UMP]

Originally posted by fu2k
The problem is that the DVD drive firmware is refusing to hand over keys because it has noticed the region mismatch. You can still authenticate with the drive and get the encrypted data though, so it's possible to apply the CSS known plaintext attack if you can find any vulnerable blocks.
I fully understand this concept, but the point is that DvdIo::Read() fails to read the blocks, even through it seems to me that the disc is authenticated. What am I missing ?

ump

[fu2k]

Originally posted by UMP
I fully understand this concept, but the point is that DvdIo::Read() fails to read the blocks, even through it seems to me that the disc is authenticated. What am I missing ?

I don't know. From what I recall (it's a long time ago now), I thought that DvdIo::Read would work, but some key fetches wouldn't. Maybe I'm wrong, or maybe the RPC2 drives work differently now days.

If DVDDecrypter is working, then you could analyse what it is doing and then try using that technique.

[UMP]

Originally posted by fu2k
If DVDDecrypter is working, then you could analyse what it is doing and then try using that technique.
Unfortunately DVDDecrypter is not open source software, but I'll try again, there obviously IS a way to read encrypted data.

Regards,

ump

[UMP]

Well, I got the point : after a failed title authentication, the RPC2 drive locks the sector reading, until the disc is authenticated again. This means I'm now able to read encrypted blocks.

ump

EDIT :

@fu2k : could it be that the currently unused _KeyDiscovery() routine is CSS brute force attack code I'm looking for ?

[Nic]

Plain text attack is pretty small.... ( http://nic.dnsalias.com/MiniDec.zip )

Have you thought about implementing libdvdcss ( www.videolan.org/libdvdcss ) into FU as a back up? It never fails for me....I was going to do it myself, but after looking at FU's decryption, it looked advanced enough and I assumed only a little bug was causing your problems...

-Nic

[UMP]

I think my previous problems were related to the fact that FU relied on some header flags to decide if a DVD was encrypted or not. I slightly changed the way this works and it seems it was successful, only the RPC2 drive issue should remain.
I'll use your sample code as a basis, thank you very much.

I thought too about implementing libdvdcss, but I fear I'm not skilled enough in C++ programming to reuse such complex code.

ump

EDIT : can I consider that the blocks from one single cell are enough to brute force, or should I look for one single global CSS key through all the cells ?

[fu2k]

Originally posted by UMP
could it be that the currently unused _KeyDiscovery() routine is CSS brute force attack code I'm looking for ?

No, this is code to extract possible player keys.

Originally posted by UMP
can I consider that the blocks from one single cell are enough to brute force, or should I look for one single global CSS key through all the cells ?

You may not find a vulnerable block within a particular cell, so you can't necessarily determine one key per cell. However, as I said previously, I believe that DVDs that change keys within a title are outside the spec, and very uncommon these days.

Honestly, I am surprised that the plaintext CSS attack is still effective. It would take only a minor change during mastering to detect vulnerable blocks and ensure that they weren't encrypted. I predicted this about 3 years ago, but it hasn't happened. I guess TPTB aren't that smart.

[UMP]

Good news, I finally succeeded in reading a full chain from a RPC2 drive with a mismatched region DVD. :cool:

You may not find a vulnerable block within a particular cell, so you can't necessarily determine one key per cellExperience showed this too.

I guess TPTB aren't that smartmmmm... what are TPTB ?

fu2k and Nic, many thanks to both of you for your support :)

[UMP]

FU stores encrypted data to the HD, and decrypts it on the fly, re-determining keys at each time, isn't it ? Would there be any drawback to decode data on the fly and always store decrypted files to the HD ?
This would allow for faster processing once the data is indexed ?

ump

[fu2k]

what are TPTBThe Powers That Be
Would there be any drawback to decode data on the fly and always store decrypted files to the HD ?There's no drawback really. I didn't do it this way since I didn't want FairUse to be able to be used as a general purpose decryption tool, thus it never stores the decrypted data on the HDD. But you could change this if you wanted.

[UMP]

I have a "special request" to anyone who think could help on this :

FairUse currently implements two resizing algorithms : a C++ written bicubic resizer which provides very good visual quality, and a faster "simple" MMX optimized bilinear resizer.

The bilinear resizer is fine for 1 CD backups since it slightly softens the picture, thus allowing for higher compressibility; but for 2 CD backups the picture could look much better with a bicubic resize.

The problem is that I don't know much about assembly, and I have to say I failed to either convert the existing routine to assembly, or to reuse existing bicubic resizing code from other apps.

Any help on this part would be more than welcome :)

Best regards,

ump

EDIT : this help request is now obsolete.

[UMP]

I added some code in FU to demux MPEG audio streams from a DVD, but the resulting MPEG audio file is too short (2 sec missing on a 80 sec clip). The start of the audio clip is in sync with the video, but progressively gets desynced.

BeSweet transcoding show syncing issues too :

[00:00:00:048] Stream error : Sync found after 308 bytes
[00:00:00:768] Stream error : Sync found after 284 bytes
[00:00:07:872] Stream error : Sync found after 284 bytes
[00:00:08:736] Stream error : Sync found after 284 bytes
[00:00:10:872] Stream error : Sync found after 284 bytes
...

All this makes me think that I'm missing some bytes while demuxing. What the code does is simply to extract all data from the 0xC0 stream after stripping the PES header. The strange thing is that I get exactly the same behaviour if I extract streams 0xBB and 0xC0 to a new VOB file, while the original VOB file doesn't expose this problem ?

EDIT : The problem was on the parsing of the chain, not on the data extraction itself.

[UMP]

@fu2k:

In Demux::_DemuxPack(...) why is it needed to state that

_demuxMatch = _demuxChain->cell[cell].vobId == ((pes[25] << 8) | pes[26]) && _demuxChain->cell[cell].cellId == ((pes[27] << 8) | pes[28]);

and then to ensure that

if(_demuxMatch && (pes[-3] == 0xE0 || pes[-3] == 0xBD)) {
...

Since the data passed here has already been split and demuxed as a cell/block, the _demuxMatch check should be useless, shouldn't it ?

Thanks,

ump

[fu2k]

Originally posted by UMP
Since the data passed here has already been split and demuxed as a cell/block, the _demuxMatch check should be useless, shouldn't it ?

It's been a long time since I've looked at this so I'm a bit hazy about the details, but I am pretty sure that this additional check is necessary in the general case.

It's because in some cases different streams can be interleaved within the same cell/block range. I think this is the case when "multiple angles" are present, but I'd have to look back at earlier versions (that didn't handle multiangle properly) to be sure.

Let me know if you want me to do this.

[UMP]

Well, that would be nice, since this additional check seems to cause some data to be dropped, at least with some MPEG-1 audio streams.

Thank you for your support,

ump

[fu2k]

Originally posted by UMP
Well, that would be nice, since this additional check seems to cause some data to be dropped, at least with some MPEG-1 audio streams.
I have checked this and it is as I had described. Use of '_demuxMatch' in this way is necessary to ensure that only a single angle is demuxed. When I did a test with this removed, a multiangle scene had all angles combined (about 1/2 second of each angle and then on to the next, in a cycle). Also there were lots of video and audio synchronisation diagnostics generated.

So I don't think that removing '_demuxMatch' is the answer...

[UMP]

Thank you for taking the time to check this :)

I think the problem is elsewhere. I'll keep on testing, I have probably missed some point in the demuxing process.

Best regards,

ump

[vio_man]

Now we have to pay for the FU installation package :mad:

The author could add a Paypal donation option, but I don't think the solution is forcing users to pay.

[UMP]

It appears that the link to FU's source code was lost during the website's relooking.

I'll correct this soon. In the meantime, here is the direct link : http://fairuse.free.fr/Release/fu-src.zip

Regards,

ump