x264 progress indication

[recover]

Hi!

I have a suggestion, and a patch, to enhance the progress indication on windows. :rolleyes:

What the patch does is updating the console title with the progress, very simple but quite handy.

The patch is very simple, and will probably need to be modified a bit to get into the official builds.
The patch was made from the current x264 trunk a few minutes ago.

Please let me know what you think.
:thanks:

Note: There's an improved patch later in this thread.

[Inventive Software]

I like that idea. I hope it makes it into the SVN. :)

[jase99]

Nice idea. Personally I wouldn't use it. fyi, the same can be accomplished in linux for terminals that support it with escape sequences, like this:

echo "^[]2;Sample Titlebar^G"

Where ^[ is ascii code 27 (escape character)
and ^G is ascii code 7 (bell character)
both numbers given in decimal

[recover]

Are there any x264 developer that can tell me their thoughts on this?
I will not maintain my own branch for this change alone.

[Dark Shikari]

Sounds like a good idea as long as pengvado has no objections. Requiring windows.h might be annoying, but meh.

[Trahald]

it will only have a chance if there are #ifndef's to check to see if its being compiled for vis studio

#ifdef _MSC_VER
#include "windows.h"
#endif

im sure there is a way to check for mingw also (if windows.h works with it?)

[recover]

I compiled it with MinGW and MSYS, which worked perfectly.
There should be some proper way to check which OS it is being compiled for.
Are there no other windows specific code in x264?

Edit: what I meant was that I compiled x264+my patch with MinGW and MSYS. I do not think the _MSC_VER define will work in MinGW.

[buzzqw]

any slowdown in encoding ? (due to refreshing window)

BHH

[recover]

I can't say for sure how much slower it is, or if it's even of any importance.
Though I can say that the title will only update if you provide the --progress argument.
I guess the only real way to know is to try encoding a video two times, one with SetConsoleTitle() and one without, and then look if there was any significant slowdown.
I'm by no means an expert on these matters, I'd rather let you guys determine this.

[buzzqw]

just launch one encode with --progress then the same without --progress

check the faster...

BHH

[recover]

I think the proper way of doing it would be to use two different binaries, one with my patch and the other without it, since when you supply --progress, x264 also outputs the progress to the terminal, which should have an impact.
The two binaries must be compiled with the exact same settings as well.

[Dark Shikari]

Speed cost is likely basically zero. Don't worry about it.

[Gabriel_Bouvigne]

Perhaps the inclusion of windows.h should be put into osdep.h instead?

[akupenguin]

But osdep.h is part of libx264. libx264 doesn't depend on windows.h, only x264cli does.

[recover]

(back from a vacation)

Since this is a windows specific feature, not compiler specific feature, there should be some other way to do this than enclosing the code in #ifdef _MSC_VER.
I'm thinking of the x264.nl builds, which are built with mingw/gcc, but built for windows.

Perhaps --windows should be a ./configure option, I don't know what would be the best solution.

[MasterNobody]

There is _WIN32 define which is supported by MSVC, MinGW, Cygwin (and it is used in x264 for Windows specific code)

P.S. It is defined for applications for both Win32 and Win64.

[recover]

So, the question remains, where should windows.h be included?

I created the title variable to be used when sprintf()'ing the text which is then passed to SetConsoleTitle(), could this be done more efficiently somehow?

When (and hopefully not if) this is commited, a thanks to recover in the commit log/changelog would be cool. :D

[recover]

I've updated the patch (http://recover89.googlepages.com/x264progress.patch) and enclosed the code in #ifdef _WIN32.
Everything is working fine, it successfully compiles and runs for me when compiling with MinGW.
windows.h is still included in x264.c.

I think it's ready to find its way into the trunk. :D

[techouse]

Great idea :D

EDIT: i made a build here http://techouse.project357.com/builds/x264_x86_r859_progress_techouse.7z

Source: x264 r859 GIT (git://git.videolan.org/x264.git)

Applied patches (current versions):

x264_2pass_vbv.9.diff

x264_fix_win_stdin.diff

x264_FGO.diff (fixed by Dark Shikari & techouse)

x264.gaussian.cplxblur.01.diff

x264_hrd_pulldown.04_interlace.diff

x264_me-prepass_DeathTheSheep.diff

x264_progress.diff

x264_rd-optimze_DeathTheSheep.diff


Please check http://forum.doom9.org/showthread.php?t=130364 and http://git.videolan.org/gitweb.cgi?p=x264.git;a=shortlog for more info

Compiled by techouse on May 24rd 2008, 17:05:14 CEST with GCC-4.3.0 on Windows Vista Business SP-1 32-bit.

Commandline used: ./configure&&make

Platform: X86
System: MINGW
avis input: yes
mp4 output: yes
pthread: yes
gtk: no
debug: no
gprof: no
PIC: no
shared: no
visualize: no

[recover]

I just gave it a try and it works beautiful.

<pengvado> what does SetConsoleTitle do when combined with xterm? wine? screen? output redirection?
I suggest we find this out.

[stax76]

StaxRip uses redirection and don't have problems with it.

[salehin]

Thanks a lot, recover :)

[recover]

No problems. Our work ain't done yet though, we still need to get it into the reps. :)
I'll try to get the time to test the patch with xterm, wine and screen, etc., feel free to help me with this.

[martino]

Just wanted to say thanks for this patch. Makes it much easier to see the progress while working on other stuff. ^^

Keep up the good work. :)

[recover]

Thanks.
I would like to get it into trunk, but unfortunately I've been busy and haven't had the time to check the things I wrote about in a previous post.
I might be able to do it next week, but no promises - you guys can help and try the things as well.

And yea, it's a pretty nice patch :)

[techouse]

THANK YOU, recover!!!

[ajp_anton]

When using 2>logfile.txt, this is the only way to see how much is left (besides reading the end of the .txt file). Thanks!

[Snowknight26]

Speaking of which, is there any way to pipe the output of the command prompt to a file (> file.txt) AND have the command prompt display the info as well?

[skystrife]

Speaking of which, is there any way to pipe the output of the command prompt to a file (> file.txt) AND have the command prompt display the info as well?

Use tee.

x264 --pass1 SETTINGSANDSTUFF --output NUL "VIDYA_SOURCE" 2>&1 | tee "LOGFILE"

Or, something like that.

[Snowknight26]

For Windows, forgot to mention.

[J_Darnley]

And tee.exe is available for Windows: http://unxutils.sourceforge.net/ or http://cygwin.com/

[recover]

When using 2>logfile.txt, this is the only way to see how much is left (besides reading the end of the .txt file). Thanks!

I haven't thought of that. That's kinda cool.

[Quark.Fusion]

Is this patch Vista x64 - compatible? (Doesn't work for me)

[recover]

Is this patch Vista x64 - compatible? (Doesn't work for me)

I have no idea. As far as I can see, msdn (http://msdn.microsoft.com/en-us/library/ms686050(VS.85).aspx) doesn't mention anything special about x64.

It can have something to do with the ifdef's thought. The patch is using #ifdef _WIN32. According to a post in this thread, that should be defined for Win64 as well.
Anyone else know anything about this?

[Quark.Fusion]

I'm using Win32 version under Vista x64. (this one (http://forum.doom9.org/showpost.php?p=1149474&postcount=411))

[recover]

It seems as if they are using the first patch I made (first post). There's an updated patch (http://forum.doom9.org/showthread.php?p=1117683#post1117683) which #ifdef's the code, but I don't think that should cause the problem.
I assume this have something to do with you running Win32 version under x64, but I don't have any idea how to fix this other than compiling an x64 version.
Maybe google holds some more information.

[Quark.Fusion]

Seems that it's my fault :) I run x264 with --verbose switch, where x264 doesn't output ETAs. Also it start output only on first 1‰ frame, which can take long time on long slow encode.

Can you modify patch that it still count ETA with --verbose switch? Also it will be good to check timer on every frame and update once per second, not every 1‰ frame, which can result in long pauses between updates.

[Quark.Fusion]

And just for fun, can you also include average QP and SSIM/PSNR in title if they are enabled? :)

[recover]

Seems that it's my fault :) I run x264 with --verbose switch, where x264 doesn't output ETAs. Also it start output only on first 1‰ frame, which can take long time on long slow encode.

Can you modify patch that it still count ETA with --verbose switch? Also it will be good to check timer on every frame and update once per second, not every 1‰ frame, which can result in long pauses between updates.
Hmm... You sure you don't accidentally omit --progress?
Also, I'm only updating the title when the --progress updates the progress in the terminal.

And just for fun, can you also include average QP and SSIM/PSNR in title if they are enabled? :)
Whoa, I have no idea how to do that. I actually don't know exactly what/how QP/SSIM/PSNR are or how they work. I think I would need some help to do that.

[Quark.Fusion]

Did you look into code?
817 /* update status line (up to 1000 times per input file) */
818 if( opt->b_progress && param->i_log_level < X264_LOG_DEBUG &&
819 ( i_frame_total ? i_frame * 1000 / i_frame_total > i_progress
820 : i_frame % 10 == 0 ) )
So, if "param->i_log_level" = X264_LOG_DEBUG then all code will not run and progress will not be displayed.

About QP and SSIM/PSNR values: look here (http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob;f=encoder/encoder.c#l1687). You can make two more objects for counting average values and use them on progress indicator update.

[Quark.Fusion]

At first sight it may be good idea to show progress indicator in log file only limited amount of lines, but implemented aproach is just plain stupid. If i have 1000-frames files then it will write every frame, if i have 1000000 frames file then it will write every 1000 frames. On other side if it doesn't know how many frames video have then it will be write every 10 frames.

The "correct" way is to update every X time (every second for example) and have progress counter as a float, not integer. (We need higher resolution on long encode.)

If there is perfomance question on how often update, then answer is in "--progress" switch itself, not in what precision to use for progress. In the end one frame encoding is many times longer that progress update, just not updating 1000 times in a second is enough for fast perfomance.

[Underground78]

About QP and SSIM/PSNR values: look here (http://git.videolan.org/gitweb.cgi?p=x264.git;a=blob;f=encoder/encoder.c#l1687). You can make two more objects for counting average values and use them on progress indicator update.

There is something I don't understand, why these informations are useful during the encoding ? :confused:

[LoRd_MuldeR]

There is something I don't understand, why these informations are useful during the encoding ? :confused:

They can be used to objectively compare the "quality" of encodes. This is mainly useful for developers that are testing and fine-tuning new features.
Generally SSIM/PSNR are not a good measure for the quality of an encode. You have to see it with your own eyes.
For example AQ and Psy-RDO will hurt the SSIM/PSNR values, but in fact they will improve the quality of your encode significantly (most of the time).

[Underground78]

They can be used to objectively compare the "quality" of encodes. This is mainly useful for developers that are testing and fine-tuning new features.
Generally SSIM/PSNR are not a good measure for the quality of an encode. You have to see it with your own eyes.
For example AQ and Psy-RDO will hurt the SSIM/PSNR values, but in fact they will improve the quality of your encode significantly (most of the time).

Yes, I know this but Quark.Fusion has said :

And just for fun, can you also include average QP and SSIM/PSNR in title if they are enabled? :)

so during the encoding if I understand well and I can't understand the interest of displaying these values before the end ...

[LoRd_MuldeR]

so during the encoding if I understand well and I can't understand the interest of displaying these values before the end ...
Well, if you care about those values for some reason, you might want to see the current state of those values during the encode.
For example if you test a new method and it produces horrible SSIM/PSNR values, you could abort after 50% of your encode and don't wait for the end.

But I think for most users it will be useless to display PSNR/SSIM values during encode (most of the time).
Also it would encourage people to use those metrics for quality measurement, instead of doing a proper visual comparison...

[Underground78]

Ok, that's true, it's more understandable now ...

[Quark.Fusion]

To be able to do proper visual comparison you need to wait until encode will end and you can't use slow preprocessing script as a source. We really need some proper visual quality metric, but since we don't have one we use approximate metrics.

As a solution there can be gui-addon to display input and output frames, surely it must contain some additional features like contrast enchancement or showing visual difference by some inproper metric since you can't spot small quality differencies with unarmed eye (and later you will spot them on very-large super-bright incredibly-contrast futuristic-display).

[akupenguin]

To be able to do proper visual comparison you need to wait until encode will end
Or just use a media player that can read partial files.

[LoRd_MuldeR]

We really need some proper visual quality metric, but since we don't have one we use approximate metrics.

Still the PSNR/SSIM metrics can't help you to judge the quality of the result in a useful way.
They even can't be used to approximate the quality result, because the metrics often become worse, when the result actually looks better...


SSIM and PSNR are relatively useless for comparisons IMO, especially since x264 has been moving more and more towards the realm of psy optimizations.

[Quark.Fusion]

In the past there was no SSIM metric and we look at PSNR value, but since PSNR doesn't represent visual quality in proper way, we got new SSIM metric — maybe someday someone invent better metric for visual quality.

To finish discussion about practicability of those metrics I want to say that this thread isn't about it. It's about showing us some maybe-useful internal values that x264 already produces. (In the end we can't completely trust ETA time as frames take different amout of time to encode and others factors as user activety can affect it)
All discussed values are estimates, not 100% accurate values. And you can compare two encodes by SSIM if you take accuracy of metric into account. (And that SSIM isn't linear).
PSY optimizations just make accuracy of metrics less, not makes metrics useless. Surely it's useless to say that encode A better that B if metric differs less that it's accuracy (which again isn't linear).