x264 progress indication

[LoRd_MuldeR]

All discussed values are estimates, not 100% accurate values. And you can compare two encodes by SSIM if you take accuracy of metric into account.
Well, the problem is not that the metric is inaccurate, the problem is that it completely fails sometimes.
This means even if the metric says "the quality has become worse", the perceived quality might actually be much better than before.
Of course it can also happen that both, metric and perceived quality, have decreased (or increased).

So how do you want to decide? :confused:

[akupenguin]

Even aside from the inaccuracy of metrics, you can't compare the partial average SSIM of two encodes if you take the partial average from different points. So what are you going to do, store the SSIM of each frame of the first encode, look at the partial average reported by the 2nd encode in progress, then sum up the right number of frames from the 1st log file to compare?

[Quark.Fusion]

The metric (if look at value in correct way) doesn't say "the quality has become worse", it says "the metric value has become worse", then you transform non-linear metric value to linear, apply error and get an advise that quality become worse if values doesn't intersect by error value. Maybe x264 should do that and say that quality is somewhere in range of X to Y (and not exactly in the middle)?

During encode you can look at metric value and if it is unusually low then you get a reason to check your encode settings, maybe you specify "--crf 48" instead of "--crf 18"? Or maybe video is too complex for that bitrate?

[Dark Shikari]

The metric (if look at value in correct way) doesn't say "the quality has become worse", it says "the metric value has become worse", then you transform non-linear metric value to linear, apply error and get an advise that quality become worse if values doesn't intersect by error value. Maybe x264 should do that and say that quality is somewhere in range of X to Y (and not exactly in the middle)?

During encode you can look at metric value and if it is unusually low then you get a reason to check your encode settings, maybe you specify "--crf 48" instead of "--crf 18"? Or maybe video is too complex for that bitrate?As far as I know no metric exists that can compare quality between different videos; e.g. SSIM 0.99 can look great for some videos and terrible for others, depending heavily on the type of source. So its impossible to automatically state "this is good quality" or "this is bad quality"; at least not accurately.

[Quark.Fusion]

I know that SSIM 0.987 can be here for image with 55% quality and 0.88 for 7%, but it's exceptional cases. I don't know real video that can result in this. I think that you almost always can say that SSIM < 0.83 isn't good and < 0.73 is poor quality for encodes. (If I understand metric right, didn't do many test with it) (I'm actually newbie here).
Seems that SSIM is vulnerable to noise, maybe it case for lower values with PSY-optimisations (since preserved noise in encode can be shifted and metric can think that it is structural change, but human doesn't care about that)

Where we compare diffirent videos? Or I something don't understand, we compare original source with encoded source. Or you mean to compare how much video A differs from encoded video A with how much video B differs from encoded video B?

I think that SSIM metric at least can separate quality in 5 areas, we can name borders as "very bad", "bad", "poor", "medium", "good" and "very good" :) Assuming accuracy at ±10% (yes, it very low, but it's because quality is subjective)
So if metric says "poor" and we wanted "good" that it's a reason to check if all going right.

[akupenguin]

I think that you almost always can say that SSIM < 0.83 isn't good and < 0.73 is poor quality for encodes.
Except when SSIM<0.98 isn't good, SSIM<0.83 is holy crap there's nothing left of the content, and SSIM<0.73 is impossible to achieve at any settings for some videos.
Assuming accuracy at ±10%
No, how about you assume an accuracy of +/- 1000%.

I have said it before: the relative value of metrics comparing two encodes of the same source with different settings of the same codec, is vaguely useful. The absolute value of metrics measure how much noise was in the source, nothing else.

[Quark.Fusion]

I assume accuracy by this (http://www.cns.nyu.edu/pub/eero/wang03-reprint.pdf) paper. If you (x264 author as your forum rank says) think that SSIM/PSNR is vaguely useful why they here in x264 codec and ON by default? Doesn't that slow encoding?

And you right, I make an error with values.

[akupenguin]

A certain amount of the x264 code and/or defaults are designed to be useful to me as a developer, rather than to the end user.

[recover]

I still haven't managed to find out what SetConsoleTitle does when combined with xterm, wine, screen, etc.
I'm not sure if I can test this as properly and thoroughly as is required, and I'm not when I get the time, so I hope that anyone of you can do this for me.
I still think that the patch is stable as a rock, but I understand that these tests are necessary.

[LoRd_MuldeR]

I still haven't managed to find out what SetConsoleTitle does when combined with xterm, wine, screen, etc.
Grab some Linux Live-CD (Ubuntu, Knoppix, etc.), boot from that disc and see what your patched build does ;)

Since "SetConsoleTitle" is a Win32 API function, I think it cannot be used in a native Linux build and should be disabled. Calling SetConsoleTitle in a Win32 build running under Wine should be no problem, because Wine will either emulate the function call (change the title of console window) or it will ignore the call...

[ficofico]

I know that my request it's very unusual but for me it's a very usefull things.

Like others in this great forum, I've made a gui that, with avisynth, x264, mp4box, hcenc etc etc etc do some things..
I've ported with autoit v3 all my batch script to visual basic, and now I'm able to use these tool without need dos shell, in " hide" mode. The ony tool that I don't want to hide at this moment it's only x264, because eta, fsp etc are too important, and don't see "how many hours" avisynth + x264 need to end process it's very tedious.

Like megui, it's possible to create a progress bar that show x264 %, but I'm not able to replicate that, and I think that I've find a better way to have this "very important info" :p

My crazy request is:

no one want to compile the "at this moment" latest version of x264 with the ability to write in a text file, every time in the same line, overwriting previous text, with the same refresh rate like title of the shell, the fps, eta, time remaining ?

I don't want to read that no one is able enought to do this !!!! ahahaha, I think that it's a work that take less that 30 s........ for a x264 diff, patch,and compiler...


Ok, I write the post, if someone want to aid me, maybe in pvt or here..........mmmmm, I'll pray for you... hih

[Sharktooth]

isnt it easier to read the x264 stdout/stderr so to make your progress bar?

[ficofico]

ok, now I read about this function. In megui there's also fps indicator, it's only a division of stdout/stderr ?

[Sharktooth]

stdout and stderr are standard streams: http://en.wikipedia.org/wiki/Standard_streams

[ficofico]

I've solved using stdout and stderr but only for print on the gui x264 info and mencoder... no progress bar, but it's what I want, thanks sharktooh

[LoRd_MuldeR]

I've solved using stdout and stderr but only for print on the gui x264 info and mencoder... no progress bar, but it's what I want, thanks sharktooh

You can parse the text from the stdout/stderr in order to "extract" the current progress. This info can then be used to update your progressbar...

[Selur]

yup create a progress bar and update it if the infos yo get via std change,...

[stanjr]

I am compiling/encoding on a 64-bit Ubuntu quad-core machine.

Patching with x264_progress.indication_r957.diff causes the following error:
x264.c: 27:21:
error: windows.h: No such file or directory
x264.c: In function 'Encode':
x264.c: 867: warning: implicit declaration of function 'SetConsoleTitle'

Is there a workaround?

[wyti]

x264_progress.indication_r957.diff is a windows only patch.
you can't use it on linux.

[recover]

This patch is only helpful in windows.
The first version of the patch (first post) fail to compile on any other OS, but there's a second patch in one of the posts in this thread which contains code that tells the compiler to ignore the code if compiling for any other OS than windows.

[stanjr]

That's what I figured. Is there a patch that would work for Linux, or I guess I might have to figure this out?

Nice idea. Personally I wouldn't use it. fyi, the same can be accomplished in linux for terminals that support it with escape sequences, like this:

echo "^[]2;Sample Titlebar^G"

Where ^[ is ascii code 27 (escape character)
and ^G is ascii code 7 (bell character)
both numbers given in decimal

[recover]

I have not made one for Linux, and I don't think anyone else have either.
So I guess you'll just have to do it yourself ;)

Please post here if you manage to get it working. :)

[Inventive Software]

Has this been commited to the trunk yet? Dark Shikari said he was happy to commit useful patches, so e-mail/PM him with the necessaries, and either he can do it, or give write access to you.

[recover]

No, it haven't been commited yet.
It has been a while since I last compiled x264, so as soon as I get home I'll just verify that it compiles cleanly on the trunk, then I'll send a pm to Dark Shikari.

[recover]

Okay.
I've tested the patch on the trunk and it didn't cleanly compile. It was pretty simple to fix it, and it now cleanly compiles. I also added the "kb/s" since it was recently added to the terminal progress indication.

The updated patch can be found here (http://recover89.googlepages.com/x264progress.v3.patch). I am now going to pm Dark Shikari with the request.

[Inventive Software]

commit bb11e37f87fe53633f531bd3b9d331f987852ed3 r966
Author: Jason Garrett-Glaser <darkshikari@gmail.com>
Date: Tue Sep 9 23:12:20 2008 -0700

Add title-bar progress indicator under WIN32
Also add bitrate-so-far output when piping data to x264 (total frames not known)
Patch mostly by recover from Doom9.



It made it! Good work. :)

[recover]

Yes, it's done.
But as you might have noticed Dark Shikari edited the patch a bit, the title is now exactly the same as the terminal output.
That means that the first text in the title is "x264 - encoded frames: ". That text takes up all the visible space in the taskbar entry, to see the rest you have to hover the mouse over it. My patch had the text "x264 [50.2%]" first, so you could easily see how far it had progressed.
I have sent a pm to Dark Shikari with my thoughts as well.

[kemuri-_9]

If you're attached to the old style then something like

diff --git a/x264.c b/x264.c
index f472fd7..34ea576 100644
--- a/x264.c
+++ b/x264.c
@@ -838,15 +838,15 @@ static int Encode( x264_param_t *param, cli_opt_t *opt )
if( i_frame_total )
{
int eta = i_elapsed * (i_frame_total - i_frame) / ((int64_t)i_frame * 1000000);
- sprintf( buf, "x264 - encoded frames: %d/%d (%.1f%%), %.2f fps, %.2f kb/s, eta %d:%02d:%02d",
+ sprintf( buf, "x264 %d/%d (%.1f%%), %.2f fps, %.2f kb/s, eta %d:%02d:%02d",
i_frame, i_frame_total, 100. * i_frame / i_frame_total, fps, bitrate,
eta/3600, (eta/60)%60, eta%60 );
}
else
{
- sprintf( buf, "x264 - encoded frames: %d, %.2f fps, %.2f kb/s", i_frame, fps, bitrate );
+ sprintf( buf, "x264 %d, %.2f fps, %.2f kb/s", i_frame, fps, bitrate );
}
- fprintf( stderr, "%s \r", buf+7 );
+ fprintf( stderr, "encoded frames:%s \r", strstr( buf, " " ) );
SetConsoleTitle( buf );
fflush( stderr ); // needed in windows
}

will cause it to be more like the old style.

[Quark.Fusion]

This will not fit % to taskbar.

[kemuri-_9]

then does anything else besides the progress % actually matter in the title bar?
obviously not seeing much of anything in the taskbar if you don't even see the % with that small patch.

because the utmost simplest solution for that would be to skip over the i_frame/i_frame_total and go straight to the (%progress)
and only keep data starting from there, looking like:
(% done), fps, kb/s, eta

if you don't want that, then it would just be likely more efficient to create a new string for the title bar and sprintf generate that like in the old version.

[Quark.Fusion]

All matter in the title bar as you can mouse-over taskbar button to see full content in a tooltip. With number of frames you can estimate new time remaining if you double fps for example. Although I think that is better show how many frames left. Like «x264: 10.1% done, fps: 101.21, eta: 12:13:11, 187962 frames left, current bitrate: 10236.37kbps», in that order as tooltip's space is also limited and it will be cropped. Or simplier «x264: 10.1%@101.21fps, eta: 12:13:11 (187962 frames), bitrate: 10236.37kbps»

[Quark.Fusion]

Also % inicator needs leading zero for better difference between 6% and 60%, especially when string is cropped and you see only first digit :)

[kemuri-_9]

Or simplier «x264: 10.1%@101.21fps, eta: 12:13:11 (187962 frames), bitrate: 10236.37kbps»
to generate something mostly like this then it's:

diff --git a/x264.c b/x264.c
index f472fd7..9c13918 100644
--- a/x264.c
+++ b/x264.c
@@ -778,6 +778,7 @@ static int Encode( x264_param_t *param, cli_opt_t *opt )
int i_frame_size;
int i_update_interval;
char buf[200];
+ char title[100];

opt->b_progress &= param->i_log_level < X264_LOG_DEBUG;
i_frame_total = p_get_frame_total( opt->hin );
@@ -838,16 +839,20 @@ static int Encode( x264_param_t *param, cli_opt_t *opt )
if( i_frame_total )
{
int eta = i_elapsed * (i_frame_total - i_frame) / ((int64_t)i_frame * 1000000);
- sprintf( buf, "x264 - encoded frames: %d/%d (%.1f%%), %.2f fps, %.2f kb/s, eta %d:%02d:%02d",
+ sprintf( buf, "%d/%d (%.1f%%), %.2f fps, %.2f kb/s, eta %d:%02d:%02d",
i_frame, i_frame_total, 100. * i_frame / i_frame_total, fps, bitrate,
eta/3600, (eta/60)%60, eta%60 );
+ sprintf( title, "x264 %04.1f%%@%.2ffps, eta %d:%02d:%02d (%d), %.2f kb/s",
+ 100. * i_frame / i_frame_total, fps, eta/3600, (eta/60)%60, eta%60,
+ i_frame_total - i_frame, bitrate );
}
else
{
- sprintf( buf, "x264 - encoded frames: %d, %.2f fps, %.2f kb/s", i_frame, fps, bitrate );
+ sprintf( buf, "%d, %.2f fps, %.2f kb/s", i_frame, fps, bitrate );
+ strcat( strcpy( title, "x264 " ), buf );
}
- fprintf( stderr, "%s \r", buf+7 );
- SetConsoleTitle( buf );
+ fprintf( stderr, "encoded frames: %s \r", buf );
+ SetConsoleTitle( title );
fflush( stderr ); // needed in windows
}
}

[recover]

I think the reason the patch was remade was to only have one sprintf() which computed all the values.
I have tried rearranging the order of the output, and this doesn't look bad at all:
sprintf( buf, "x264 - %.1f%%: %d/%d encoded frames, %.2f fps, %.2f kb/s, eta %d:%02d:%02d",
100. * i_frame / i_frame_total, i_frame, i_frame_total, fps, bitrate,
eta/3600, (eta/60)%60, eta%60 );
Output will be something like this:
7.9%: 2390/30437 encoded frames, 67.77 fps, 749.24 kb/s, eta 0:06:53
And in the title:
x264 - 7.9%: 2390/30437 encoded frames, 67.77 fps, 749.24 kb/s, eta 0:06:53

I'm pretty confident that the only way we can get any change into the trunk is to come up with something that doesn't require another sprintf().
Please try it and tell me what you think.

[Quark.Fusion]

Please add also average quantizer value if you added bitrate as with various settings CRF isn't mean anything useful when you want certain quality. SSIM will also be useful if it was enabled. Currently I need to parse log in excel to get average values before encode end.
I.e. «x264: 10.1%@101.21fps, eta: 12:13:11 (187962), 10236.37kbps, Q:21.75, S:0.98765» = 79 chars.

And why another sprintf() is bad? Time in it is nothing compared to encode time and it can be turned off when there is no --progress switch. Maybe format numbers and then strcpy() them in different orders?

Also why you turn it off when verbose log is on? («opt->b_progress &= param->i_log_level < X264_LOG_DEBUG;») There is no word anywhere that --progress is not compatible with --verbose.

[J_Darnley]

Please add also average quantizer value if you added bitrate as with various settings CRF isn't mean anything useful when you want certain quality. SSIM will also be useful if it was enabled. Currently I need to parse log in excel to get average values before encode end.
I.e. «x264: 10.1%@101.21fps, eta: 12:13:11 (187962), 10236.37kbps, Q:21.75, S:0.98765» = 79 chars.
As akupenguin said (http://forum.doom9.org/showthread.php?p=1178833#post1178833), current average QP cannot be added with ease. My understanding of the source code is that you would need to move progress reporting into encoder/encoder.c.

[Dark Shikari]

As akupenguin said (http://forum.doom9.org/showthread.php?p=1178833#post1178833), current average QP cannot be added with ease. My understanding of the source code is that you would need to move progress reporting into encoder/encoder.c.Or you could make an API function to fetch the data.

[Quark.Fusion]

As akupenguin said (http://forum.doom9.org/showthread.php?p=1178833#post1178833), current average QP cannot be added with ease. My understanding of the source code is that you would need to move progress reporting into encoder/encoder.c.
I don't understand why not — it writes it in verbose log. Just move code to encoder.c

[Dark Shikari]

I don't understand why not — it writes it in verbose log. Just move code to encoder.cBut that would cause problems for library apps that don't want x264 to print its own progress (such as ffmpeg and mencoder) but still want x264 to print warning messages.

[Quark.Fusion]

Why not set log-level to 1?

Also you can make define that will turn off code if x264 compiled as library.

[Dark Shikari]

Why not set log-level to 1?Because sometimes the libraries might want other information too?Also you can make define that will turn off code if x264 compiled as library.Ugly, you shouldn't have to compile twice to make both an exe and a lib.

[Quark.Fusion]

Brrr, no --progress switch and all is alright with libraries.

[Chabb]

Did someone notice?

Since bitrate indication appeared
x264 window freezes several seconds after encoding start
and text doesn't update until encoding is finished.
(window becomes inoperable, except closing it)

Since progress indication appeared
the same bug occurred except one updating string.

This happens at least on my system.

[Dark Shikari]

Did someone notice?

Since bitrate indication appeared
x264 window freezes several seconds after encoding start
and text doesn't update until encoding is finished.
(window becomes inoperable, except closing it)

Since progress indication appeared
the same bug occurred except one updating string.

This happens at least on my system.Interesting, what operating system?

[Chabb]

Interesting, what operating system?

WinXP SP3 Rus x86

Also I've noticed that all indication is normal
unless I do any clicks in x264 window

[DarkZell666]

WinXP SP3 Rus x86

Also I've noticed that all indication is normal
unless I do any clicks in x264 window
Do you happen to have "Quick edit" enabled in the console preferences ? If so, pressing enter or doing a right-click inside it should unfreeze the console. "Quick edit" is what enables copy/paste operations, and when enabled, clicking tells the console to halt refreshing the display so you can actually copy some text (you can't capture everchanging text right ? :p).

[Chabb]

Do you happen to have "Quick edit" enabled in the console preferences ? If so, pressing enter or doing a right-click inside it should unfreeze the console. "Quick edit" is what enables copy/paste operations, and when enabled, clicking tells the console to halt refreshing the display so you can actually copy some text (you can't capture everchanging text right ? :p).

No. Nothing like "Quick edit", cause in this case it can be "unfreezed" :)

The bug I've described occurs even if I switch tasks.
In latest builds (r969) x264 window not only "freezes"
with text updates,
but goes completely white inside, while encoding continues.

Also the last notice about my problem is that initially x264 window
looses focus and continues to update it's text quite normally,
unless I'll try to bring the window on top of others
(by clicking on it or by using Alt-Tab)

[Chabb]

Ooops, I did it again :)
Seems I have bad luck in all concerning bug-search job.
I've posted possible bug in progress indication,
blaming x264, but it's not that simple.

All I thought it was only x264, but there is another participant
in this case. The coder seems to be all right.
But DECODER is causing the bug mentioned in my previous posts.
I'm using CoreAVC by default while decoding AVC-containing streams via DirectShowSource in AviSynth scripts,
and until version 1.8.0.0, when systray icon appeared, all things was fine.

I've updated CoreAVC to 1.8.0.0 same time as progress indication
was introduced - that's it.

This bug disappeared when I switched decoder to FFDShow.

So those posts should be addressed to BetaBoy or someone else
of CoreAVC developers.

[Quark.Fusion]

How CoreAVC DECODER can be related to x264 encoding process, especially to progress indication?

[Chabb]

How CoreAVC DECODER can be related to x264 encoding process, especially to progress indication?

Simple - it used to decode streams containing x264 encoded video
in AviSynth script (via DirectShowSource)

If only you know how to do such things another way, you are welcome to do advices.