Capturing 1080p60 lossless to a single mechanical HDD?

[ChiDragon]

Continuing discussion from the MagicYUV thread (http://forum.doom9.org/showthread.php?p=1730825#post1730825) because I don't want to go further off-topic.

This is definitely a job for RAID.
I was hoping lossless would allow me to avoid the need for a speedy RAID as required for uncompressed. Given the cost/hassle/maintenance/lower speed, I think I would actually opt for a 512GB SSD if forced to abandon single HDDs. That would give me more write speed headroom, I never need to capture more than 1-hour increments of video this large (usually much less), and transferring the files after capture wouldn't be tooooo bad. Maybe.

Some say RAID is required even for 720p/1080i capture, but I've managed it okay. Finding the right voodoo settings for VirtualDub's "Disk I/O" buffer settings was the trick for that, as the defaults are no good for the huge bitrates required -- at least with the "green"/low-power 5900RPM hard drives I've written to. (AmaRecTV has the same buffer issue.)

I wonder whether using a SSHD hybrid rather than an HDD would lesson the need for the RAM buffer to avoid random drops? Or is the cache only used for reads and not writes?

Certainly others (http://www.thethrillness.com/2014/10/1080p60-battle-magicyuv-vs-x264vfw.html) have managed to capture 1080p60 4:2:2 to a single spinning hard disk. RGB imposes even more throughput, but a guy can dream, can't he?

The random nature is likely explained by the framedrops occuring when you hit a slow sector on the drive.
And yes, sectors are not equal on a drive - especially later in their life.

You can take a look at it by running a "read" surface test using HDDScan (http://hddscan.com/).
The "RD-Read" task with 256 block size is going at around 130KB/sec, so it's gonna take a while! I assume this is expected. But I think I see what you mean. The initial scan that I started and cancelled had one sector that was "< 500 ms [orange]" and while the vast majority of the current test is "< 5 ms" there are still odd sectors here and there that show other counts.

[benwaggoner]

I've never even tried it. I've had 3-drive RAID0 arrays in a workstation not be able to keep up with 1080p60 444 just a couple of years ago.


-Ben Waggoner (via TapaTalk)

[benwaggoner]

Also, lossless sounds great until you hit something relatively incompressible like the HBO Logo. It helps reduce total storage requirements MUCH more than peak throughout.


-Ben Waggoner (via TapaTalk)

[ChiDragon]

That's funny, the HBO logo was the example I used a few years ago when I described situations where my 720p setup can't keep up. I've never actually captured it, but it's a lot more well-known than whatever actual "static" scene I encountered.

I've never even tried it.
Of course not, you're a real pro. :)

[luquinhas0021]

You ought to see what is the write/read speed of your desired sshd, because a hdd can has more read/write speed than a sshd. The same is true for hdd vs ssd.
Is equal you answer me: what is better: a 8 GB RAM or a 16 GB RAM? Well, you must compare more technical aspects, beyond GB!

[ChiDragon]

Naturally, I had my eye toward the write speed benchmarks above all else. They aren't completely consistent across different reviews, though. And I'm not sure whether I should aim for the max speed or the average, as it changes the ranking quite a bit.

UserBenchmark (http://hdd.userbenchmark.com/) max write speeds (large data set with many different PC configurations)
Sorted by average write speed, page 1 (http://s1.postimg.org/zazuoo84f/UBM_sort_write_avg_pg_1.png)
Sorted by average write speed, page 2 (http://s1.postimg.org/lg1k67dpb/UBM_sort_write_avg_pg_2.png)

The Tech Report (http://techreport.com/review/25425/seagate-desktop-sshd-2tb-hybrid-drive-reviewed/5) review (single PC configuration)

According to the graph, the Barracuda 3TB slightly exceeds the SSHD 2TB's sequential write speed except for a few brief dips where it merely matches it, averaging 162MB/sec vs 159MB/sec. Both drives have a dip for perhaps the first 2% before they hit their peak speed, followed by a gradual stairstep down a curve. The burst write speed is also 145MB/sec higher on the Barracuda.

The VelociRaptor 1TB has its peak at the very start and follows an almost linear descent to 50%, where it begins to stairstep more and curves. It only beats those Seagates noticeably in the range from 0-5% and ~62-100%, which is where it earns its 3MB/sec better average speed.

So if you believe The Tech Report's average for the Barracuda over UBM's 12MB/sec lower average, a Barracuda 3TB partitioned to only use the disk region from 10-60% == VelociRaptor sequential write speed and with slightly better burst speed as a bonus? That would give me 1.5TB for $128 vs 1TB for $283 vs cheapo SSD 512GB for $215 (all CAD, current at NCIX).

[raffriff42]

1080p/60 RGB requires 356 MB/s uncompressed
MB/s = W x H x FPS x bytes-per-pixel / 1024 / 1024
...and somewhere around 240 MB/s lossless, depending on the compressibility of the images.

Even one of the fastest HDDs, a 1 TB Velociraptor, "only" gets around 210 MB/s (writing to the outer tracks, the best case scenario)
http://www.storagereview.com/western_digital_velociraptor_1tb_review

I think you need to look at a RAID as Ben mentioned, or one of the better SSDs.

On the low end, a Kingston SSDNow, 240GB can do nearly 300 MB/s for under $100...
http://www.pcstats.com/articleview.cfm?articleid=2718&page=2

...but an Intel 750, 400GB can do 1800 MB/s for $400...
http://www.storagereview.com/intel_ssd_750_review

(EDIT performance numbers are CrystalDiskMark "sequential write"; prices are from Amazon.com)
(EDIT sorry I missed your post, ChiDragon, as I was typing)

[QBhd]

I agree with riffriff42... if this is a serious endeavor for you and you want reliable performance (i.e. the drive not being a bottleneck in any way) you either need to go RAID0 with it's inherent risks or go with a top notch SSD (Samsung 850 Pro for SATA) or best option for performance is a PCI-e SSD as mentioned in the above post.

QB

[ChiDragon]

:o It's a hobby project, so nothing mission-critical.

...but an Intel 750, 400GB can do 1800 MB/s for $400...
Hmm, the guy at the store yesterday said the Intels were $700+ CAD. The exchange rate shouldn't cause that much of a difference.

somewhere around 240 MB/s lossless, depending on the compressibility
1.48:1? That seems incredibly low for a clean digital source.

MagicYUV allowed me to capture 1 minute of this type of pixel insanity (http://postimg.org/image/wu9indm81/) without any dropped/inserted frames, using a 1GB buffer size (click the preview if you care to see the 2.6MB PNG). I would use more RAM if VirtualDub could handle it, but the version I'm using hangs or crashes upon hitting Capture with anything larger. I've attempted to capture this demo several times and every other time it can only grab around 48-55 fps, so I guess I got lucky and the randomly-generated game content wasn't too complex this time.

Perhaps this isn't the most challenging content, but I think it's pretty tough. The filesize is 8,060,243,860 bytes = 135MB/sec = 128MiB/sec. I assume the hard disk benchmarks are in MiB and not true MB.

ULRG can compress this capture even better, down to 126MiB/sec, but during realtime capture of the sequence it doesn't seem to be able to keep up as well as MagicYUV even though the CPU usage has plenty of headroom.

It turns out the problem I was having before, getting drops when the content wasn't very complex, was partly down to being connected through my Denon AVR-890 with "Video Convert" set to On. The menu claims that it simply passes through my "RGB 4:4:4 8-bit" signal, but it turns out that it applies ridiculous color subsampling, possibly with dithering that killed the compression ratio. I can't believe I've never seen this mentioned anywhere in the reviews or forums I read prior to buying it back then.

EDIT: Somehow I had forgotten about the existence of VirtualDub's Test Video Capture mode that compresses without writing to disk. It appears that demo scene needs about 146MiB/sec to be captured without hiccups. Meanwhile, the highest peak I've seen while playing around with the PS3 is about 185MiB/sec with ULRG and about 3MiB higher when MagicYUV is used. I've gotten some "Inserted frames" with ULRG even in test mode. Resource Monitor doesn't hint me to any CPU spikes, so I'm not sure why it's doing this.

[raffriff42]

guy at the store yesterday said the Intels were $700+ CAD.Here's the link (I hope I got the right model etc)
http://www.amazon.com/Intel-Solid-State-Drive-SSDPEDMW400G4R5-PCI-Express/dp/B00UHJJQAY/

1.48:1? That seems incredibly low for a clean digital source.
It's true, I used a very conservative estimate there, but not impossibly so. A while back some tests were made on games compressed with Fraps (https://encodingtalk.com/threads/variance-in-bitrate-between-game-titles-while-recording-with-fraps-why-it-matters.2691/#post-6798) in YV12 (4:2:0) mode, and the "worst" games came in at nearly 7.0 Bits/(Pixel*Frame), so I generally assume 8.0 for estimating hard drive requirements. To account for the doubled bits-per-pixel in RGB, I doubled my size estimate to 16.0 Bits/(Pixel*Frame). Yes, MagicYUV has a better compression ratio than Fraps - in fact most lossless codecs that aren't Huffyuv are better than Fraps, but again, I was being very conservative :)

[luquinhas0021]

I use H.264 lossless for compression. Looks like to be more stable than H.265. Let a video uncompressed is fool when I can compress it losslessly, not?

[Atak_Snajpera]

On the low end, a Kingston SSDNow, 240GB can do nearly 300 MB/s for under $100...
http://www.pcstats.com/articleview.c...id=2718&page=2
I would never suggest anything from Kingston because You never know what type of NAND you will get in this SSD. Kingston always erases original labels on chips. Besides Have you heard about this scam ?
http://www.anandtech.com/show/7763/an-update-to-kingston-ssdnow-v300-a-switch-to-slower-micron-nand

Crucial BX100 250 GB for 85 bucks is a much better choice than this kingston.
http://news.mynavi.jp/articles/2015/02/26/crucial_bx100/images/014.jpg

[wonkey_monkey]

Let a video uncompressed is fool when I can compress it losslessly, not?

I don't think anyone's talking about recording uncompressed, are they? MagicYUV, fraps, etc, are all designed specifically for lossless compression, and I believe they have advantages over H.264 for this particular kind of use - namely CPU usage.

This is what my Crucial M550 gave me on the same benchmark program, by the way (done under normal conditions while I browsed):

http://snag.gy/bxgOB.jpg

[kolak]

CPU should not be a problem for magicyuv at all. HD will use tiny % of current eg. i7 CPU. The one think which you have to make sure is that there are no pixel format conversions happening.
Single modern SSD should have no problems with HD 60p.
Single HDD may be not fast enough, depending on the footage nature.
Your disk has to provide at least simultaneous speed of uncompressed/1.5.
If very complex scene arrives magicyuv may end up with even lower than 1.5x compression, but average it's around 2.5x. For clean CGI source may may even get 4x.

[Revgen]

I capture 1080p60 lossless without RAID using a single traditional 7200rpm SATA HDD drive and the UTVideo codec in VirtualDub.

I have an Intel Core i7-3960X & X79 chipset.

Very rarely do I encounter dropped frames.

An SSD drive should work fine too, but it seems like overkill unless you're capturing uncompressed.

[easy2Bcheesy]

I capture 1080p60 lossless without RAID using a single traditional 7200rpm SATA HDD drive and the UTVideo codec in VirtualDub.

I have an Intel Core i7-3960X & X79 chipset.

Very rarely do I encounter dropped frames.

An SSD drive should work fine too, but it seems like overkill unless you're capturing uncompressed.

It will depend on the content. Try capturing Super Stardust HD on PS3 or PS4 using UT Codec - that's my throughput stress-test and even YUY2 can hit 140MB/s, though 60MB/s-100MB/s is more commonplace. RGB will be more rigorous

I've gotten some "Inserted frames" with ULRG even in test mode. Resource Monitor doesn't hint me to any CPU spikes, so I'm not sure why it's doing this.

Yeah, UT codec needs a fair bit of overhead. In a multi-thread environment, your CPU may be under-utilised if, for example, one thread is waiting for tasks to complete on another thread. Downsampling from RGB to another colourspace can introduce inserted frames, though I've never had that problem going from YUY2 to YV12.

For the record though, I've been able to capture 1080p60 on a 1.6GHz Core i5 Sandy Bridge ultrabook (max turbo 2.3GHz), in both YUY2 and YV12 (converted from YUY2). It's when you go to 1440p and 4K that the CPU requirement becomes onerous.

[ChiDragon]

@Revgen - 4:2:2?

After seeing a 200MB/sec spike in Test Video Capture mode with some particularly challenging RGB content, I ended up ordering an SSD for this project. The Samsung 850 EVO 500GB on sale for $190 CAD (= $146 USD but they only ship to Canada) plus our lovely 12% tax. I was gonna say I'll report back once it arrives, but I suppose that would be off-topic in my own thread. :p

[Revgen]

@Revgen - 4:2:2?

After seeing a 200MB/sec spike in Test Video Capture mode with some particularly challenging RGB content, I ended up ordering an SSD for this project. The Samsung 850 EVO 500GB on sale for $190 CAD (= $146 USD but they only ship to Canada) plus our lovely 12% tax. I was gonna say I'll report back once it arrives, but I suppose that would be off-topic in my own thread. :p

4:2:0 mostly.

But 4:2:2 hasn't been an issue for me either in a few tests I've done. I haven't done any long term, so maybe some frame skipping will occur during longer caps.

If so, I can always resort to using X264VFW lossless, which does an even better job of compressing the data. My CPU has the horsepower for it. But it's easier for me to edit my vids using UTVideo in VirtualDub so I stick with it.

[Revgen]

It will depend on the content. Try capturing Super Stardust HD on PS3 or PS4 using UT Codec - that's my throughput stress-test and even YUY2 can hit 140MB/s, though 60MB/s-100MB/s is more commonplace. RGB will be more rigorous

That very well could be true. But I mostly stick to 4:2:0 for capping, and UT works for what I'm doing.