Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Announcements and Chat > General Discussion

Reply
 
Thread Tools Search this Thread Display Modes
Old 31st January 2023, 15:31   #1  |  Link
LStation
Registered User
 
Join Date: Sep 2020
Posts: 19
Scene selection for encoder roundup

Hello, I think benchmarking encoders is pretty neat but I'm a little skeptical about the nature of the video gaming footage people use for the benchmarking so I thought I would start my own dataset of sorts.

I started with two game titles, different art styles, that I could complete within an hour, a once through at a reasonable high quality*, used ffprobe to export pkt_size and excel to give me a moving average (in MB/s) that graphs pretty well.. but I'm scratching my head here with scene selection criteria that might make sense in practice, for example, an average scene given the content (imgur.com).
*I'll go back and attempt to recreate the scenes, recording lossless

So if you have any ideas surrounding what makes a good video dataset, or how I could approach scene selection differently feel free to leave a comment below.
LStation is offline   Reply With Quote
Old 4th February 2023, 07:18   #2  |  Link
LStation
Registered User
 
Join Date: Sep 2020
Posts: 19
Some backstory, I couldn't relate to the footage presented in some hardware acceleration comparisons using VMAF so I followed a guide on how to operate the VMAF tool myself. Along with finding that guide I found other things like studies on LCEVC by Jan Ozer and Nabajeet Barman.

In Barmans evaluation(lcevc.org) they use a plot of SI & TI values as a way to identify a spectrum of complexity (pg18&19 https://www.itu.int/rec/T-REC-P.910-202207-I/en), which I had trouble with compiling its alternative called VCA(bitmovin.com), and eventually found Barmans github where they shared a MATLAB script (github.com) I could run on .yuv files but the problem with .yuv files is I'd have to split a 4.5hr playthrough into like 28 jobs due to their total size (25TB). That lead me to looking at what Ozer did which was reviewing the datarate of CRF 23 recordings.

Time passed, and I had changed it to CQP 14 instead because I was under the impression smaller details like foilage might want more bitrate to be precise but I suspect I pushed it into diminishing returns territory because I believe THPS1&2(drive.google.com)'s datarate(imgur.com) should've been much higher than this content Vampire Survivors(drive.google.com)'s datarate(imgur.com, bottom) or perhaps using the datarate to predict complexity can only get you so far.
LStation is offline   Reply With Quote
Old 5th February 2023, 21:11   #3  |  Link
LStation
Registered User
 
Join Date: Sep 2020
Posts: 19
did some more playthroughs and probes (attachment(imgur.com) pending approval), here's a folder(drive.google.com) with the probes and example xls for anyone to play with.

I think I should tackle SI&TI values next, see what they have to say about complexity for these two titles.

Edit: UPDATE
Success compiling the VCA project(imgur.com) in a vm, then copied missing dependencies(imgur.com) to my system but now I have to figure out my plumbing problem(imgur.com).

my previous headache was "installing cmake", you don't use the installer.. you unzip the portable and edit your system environment variables PATH to include that portables /bin directory

Edit: UPDATE2
Success, VCA ran.. but I didn't see 300fps(imgur.com), ran slower than real time in my case, here's two plots I made with its data(imgur.com) on vampire survivors (threw the -VCA.csv in with the probes), I'll throw THPS1&2 on next then take nap.

Edit: UPDATE3
--no-chroma doubled the fps* for VCA and perhaps --no-lowpass will have an impact as well, I'll look into benchmarking the matlab script either with a yuv file or get virtualdub2 to frameserve
*it doubled less complex footage(imgur.com)

Edit: UPDATE4
I did a benchmark with 5 minutes of 1440p60 yuv420.yuv footage(imgur.com), the MATLAB scripts output is much larger than VCAs csv, and only had 33% utilization on my system.
after further consideration, the MATLABs output is big because of the extra millions of data entries it had for each 2560x1440 pixel multiple times, not necessarily extra math, also scripts not fully utilizing a system is prevalent in googles search results on why that might be the case. rec p910 or siti-tools mentioned how things near the edges of a frame can throw off the calculations, so I can look into placing a black boarder over footage to feed VCA to see what sort of result we can get but I have my doubts the TI scale will be under 100.

I haven't directly compared avg framesize to siti yet, SI appears to correlate pretty strongly with framesize for all those vampire surivor ffprobes.
Attached Images
 

Last edited by LStation; 8th February 2023 at 15:15.
LStation is offline   Reply With Quote
Old 11th February 2023, 07:00   #4  |  Link
LStation
Registered User
 
Join Date: Sep 2020
Posts: 19
Threw some commands through powershell, replotted some familiar graphs, and I'll share my thoughts. As a reminder these were four separate recording sessions..

THPS1&2 CQP14 Plots, CQP23 Plots; CPQ14 Shell, CPQ23 Shell(imgur.com)
Vampire Survivors CQP14 Plots, CQP23 Plots; CQP14 Shell, CQP23 Shell(imgur.com)

If I played through either title again I'm confident the framesize or spatiotemporal (SI&TI) values would continue to plot in a similar manner.
A lower CQP appears to decode faster, any forced -hwaccels did not help this.
A little puzzled by THPS1&2 results, the decode speed is lower so it must be complex? I interpret the SITI plots don't agree and point to Vampire Survivors being more complex instead*.
*I'm aware it's recommended to look at the SITI mean, I can update this later.
I'll fix the scale of THPS1&2 plots later.
I'll throw the re-ran csv's into the folder later.
I left a bugreport on github for VCA, if its supposed to mirror siti-tools output or if theirs stands alone.
I currently feel using ffprobe would be the way to identify scenes of interest, and the siti stuff could be a way to show how those scenes plot in a spectrum of complexity.
No thought paid towards "an avg scene" yet, watching lets plays on the phone(procrastinating) while tests run.
LStation is offline   Reply With Quote
Old 8th March 2023, 11:14   #5  |  Link
LStation
Registered User
 
Join Date: Sep 2020
Posts: 19
Still around,

I wanted to address my overplotting problem* (attached image, pending ofc, are plotted playthroughs that are essentially unresponsive in practice), Left side being the mess, Right side being a clearer picture of what I've plotted.

I recently picked up RStudio, it allows me to publish chart figures you can interact with in a limited fashion, and have a standardized chart formatting**.. still learning but here's a 10 second example (drive.google.com) in a self contained html file you can view with your web browser. I could also just post png's but what's the fun in that?

I've got footage submitted that exceeds expected ranges for VCA in a bugreport, so either my compile was bad or they have some work to do there.

So I'll be tinkering, and getting good at games I pick up. May not be able to grade the complexity (quickly, and reliably) yet but that shouldn't stop me from pointing out where to find the demanding scenes. I think I've given up on the idea of an average scene.

*Banging my head against the wall with other tools in regards to the bins/bucket size for the creation of the 2D histogram, rstudio didn't require that I have the bins figured out already but I'll likely want to.. to keep bin ranges standard. I read 2D histograms can produce bad data if your bin ranges are suboptimal which is just more time verifying my tools and not playing games.

Tried gnuplot but it wanted bins in another column, and I thought that would be a LOT of busy work in Excel (could probably vba macro the binning with data analytics), but figured someone might suggest using a better tool than Excel so that's what I've been looking for.

Read that Vaex is good at dealing with >1B data points AND visualizing it, but I'm intimidated and it looks like it requires the bin data to already be there also.

**Can you believe Excel in 2023 can't paste formatting of a chart? I'd have to format a chart, save that as a form of preset, and then swap data on it.


April 24th
few wrinkles being ironed out,
1 OBS had a bug with custom ffmpeg output, v28~29 were always tv-range output until 29.1 betas
1a I'm using rawvideo since my nvme drive can handle it fine (1440p60 bgra 7078 Mb/s)
2 recording your screen has issues you can minimize to an extent, so after engineering a scenario and verifying dupe detection methods, then after the trials I settled on OBS game capture(imgur.com)
2a if it's a single player game and chance/time commitment is a major factor I'll pause the game before the scene then run it at half speed (CE) and sus out the unique frames with winrar(imgur.com)
3 learned how to use named pipes on windows
4 learned how to install siti-tools, and get it to accept named pipes, launch multiple instances, and tidy&stitch multiple logs into one, with a few scripts (this attempts to avoid reliance on MATLAB)
4a siti-tools doesn't like noise from quantization, avisynth has a limiter so it stays in rigid compliance with tv-range
5 played some games and recorded footage
6 replaced excel tasks with rstudio, trying to extract more from my logs(imgur.com)
7 my posts don't bump this thread until someone else posts, which is a good spam deterrent, so I'll continue to edit if need be
Attached Images
 

Last edited by LStation; 24th April 2023 at 08:14.
LStation is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:37.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.