PDA

View Full Version : Lossy replacement for lossless encodes


Quark.Fusion
26th October 2008, 06:43
There is times when you want to transfer your video to another person for editing and lossless isn't option here. (Think about one night uploading over internet).
So there is question what settings to use for consistent quality between frames and visual transparency for editing and further lossy encoding. Should we use one pass or two-pass if we target certain uploading time? If one pass then QP or CRF? Should we set ipratio and pbratio to 1, what about b-pyramids in this case?
Consider encoding time is also matters, as we have one time pool for encoding and uploading. Is there any solution to start uploading before encoding ends? (it should support resuming transfer in case when connection breaks)

Guest
26th October 2008, 06:55
Mail them a data DVD.

Quark.Fusion
26th October 2008, 06:58
Didn't I say that transfer time is limited? Mail fails in speed :)

Guest
26th October 2008, 07:05
Snail mail.

Selur
26th October 2008, 09:07
if available: "overnight curier/express mail"
alternatively: move somewhere next to a datacenter with a ultrafast upload, buy a server there and upload there directly. (output splitting every x MB also helps, if possible)

Blue_MiSfit
26th October 2008, 10:33
Mailing hard drives works well :)

I do 20-30 terabytes a week like this!

~MiSfit

Quark.Fusion
26th October 2008, 12:33
Guys, the topic question is how to use time-limited lossy compression when lossless isn't an option. Consider free internet connection with 768kbit upload capacity as example, source resolution near 720p. Direct transfer is prefferable as dedicated host usually needs to complete upload before download will be available.
How much lossless is better, is it that much to justify complex transfer and fees?

kemuri-_9
26th October 2008, 13:56
Direct transfer is prefferable as dedicated host usually needs to complete upload before download will be available.

that depends on the ftpd running on the server, there are several daemons that can allow for downloading a file as it is being uploaded.

cogman
26th October 2008, 16:02
Guys, the topic question is how to use time-limited lossy compression when lossless isn't an option. Consider free internet connection with 768kbit upload capacity as example, source resolution near 720p. Direct transfer is prefferable as dedicated host usually needs to complete upload before download will be available.
How much lossless is better, is it that much to justify complex transfer and fees?

It really isn't that hard to figure out. Step one is to get how much time you have available to transfer. step two is to figure out the length of your video in seconds. Step 3 is to figure out the average bitrate of the internet.

Now, take time to transfer / video length and then multiply that by average bitrate. to be safe you'll probably want to subtract like 10 kb/s from that bitrate. Then do a two pass encode on the video at that bitrate and viola, you have optimum quality in said time constraint.

However, snail mail is actually the fastest way to transfer large amounts of data great distances. Think of how much time it takes you to download 8 GB, on some internet connections it can take several days. However, if you overnight ship a dvd you have exceeded the transfer speed of your internet connection. If you talk about 40GB, 100GB, or 250GB, then shipping a hard-drives easilly becomes the fastest method

Zarxrax
26th October 2008, 17:20
Now, take time to transfer / video length and then multiply that by average bitrate. to be safe you'll probably want to subtract like 10 kb/s from that bitrate. Then do a two pass encode on the video at that bitrate and viola, you have optimum quality in said time constraint.

But depending on the speed of your computer, your encoding settings, and the length of your video, the encoding time could also be a major factor that you need to consider. In that case I might advocate doing a 1pass bitrate based encode at a little bit higher quality and using faster encoding settings.

Sagekilla
26th October 2008, 17:52
On a 800 kbps connection, if you have 14 hours to upload the file, then you have at -best case scenario- about 5 GB of data to transfer. Realistically? You won't have anywhere near that much, since you might average around 500 - 700 kbps.

So if we say we can get about 3.5 GB transferred, depending on how long the clips are you may be able to get some very good quality out of it.

Quark.Fusion
27th October 2008, 01:13
It really isn't that hard to figure out. Step one is to get how much time you have available to transfer. step two is to figure out the length of your video in seconds. Step 3 is to figure out the average bitrate of the internet.

Now, take time to transfer / video length and then multiply that by average bitrate. to be safe you'll probably want to subtract like 10 kb/s from that bitrate. Then do a two pass encode on the video at that bitrate and viola, you have optimum quality in said time constraint.

However, snail mail is actually the fastest way to transfer large amounts of data great distances. Think of how much time it takes you to download 8 GB, on some internet connections it can take several days. However, if you overnight ship a dvd you have exceeded the transfer speed of your internet connection. If you talk about 40GB, 100GB, or 250GB, then shipping a hard-drives easilly becomes the fastest method
You forget about encoding time, as it cut out transfer time, especially in case when you can't start uploading before encoding ends. Snail mail isn't an option in this example as we consider fast free internet connection available. It's one-to-one transfer with known capacity, not one-to-many with unknown.

Why you press so much to snail mail solution? Is quality loss really worth it?

So back to original question: what you think is better in this scenario — figure out what CRF value to use with some safety margin or cut out some of bitrate with 2pass when waiting for encode ends? What settings to optimize for speed for this transfer bitrate, assuming we have 3GHz Core2 Duo CPU?
In case of 2pass encode is b-adapt 2 worth speed loss? What setting to use if we target video editing instead of viewing? What about 1pass ABR?

Guest
27th October 2008, 01:18
This is such a stupid strawman question. Do what works best for you. Any sensible person would ship a hard drive with the source files, as has been very clearly explained. Listen to Blue_MisFit! He's a professional in that very business of sending large streams around for processing/editing by others. And listen to cogman! He's done the math for you on effective transfer rates.

Internet transfer here is a red herring. You're asking how to make a good compression with minimal quality loss. It's been discussed over and over again. Why do we need another thread about it?

Sagekilla
27th October 2008, 01:30
Unless you have a decently large up pipe like I do that doesn't get throttled (FiOS ;)) Don't even bother considering the internet route...

The more time wasted trying to figure out what to do is less time you could've possibly encoded the video and started uploading it, if you really absolutely had to.

LoRd_MuldeR
27th October 2008, 01:37
Is it that hard? If encoding time matters, use 1-Pass CRF. And choose the highest CRF value that you can accept quality-wise.
The result will be as "small" as you can get it. So you'll have to accept the uploading times that result from it.

Also "slower" encoder settings may be helpful to get a "smaller" file at the same quality. It takes more time to encode, but less time to upload.
Unless you use overkill settings, you'll safe more upload time than you'll add encoding time...

BTW: If you choose the HDD shipping method, SSD's may be(come) a less fragile alternative to "classic" HDD devices.
Maybe even a very big USB Stick could serve for the transfer. There are 16 GB, 32 GB and even 64 GB models available now ;)

Quark.Fusion
27th October 2008, 01:45
I wasn't talking about slow internet connection, it's slow only for upload, for download it's 2-6mbit. The real need for lossy transfer was sometime ago for game video material and in the end my friend come up with some default encoding settings of video editor to transfer output for editing to another person. For lossless solution he say that result file is just to big. I want to find better solution for that situation as it happends for time to time. Buying and mailing hard drive can be sensible if you making money with your work, but for some non-pro editing it's not always an acceptable solution.

In the first post I was asking about what settings is better for lossy encoding for editing purposes, not what method to use for lossless transfer. But you press that lossless is the GOD without any argument. Even when I ask if it really is noone answer why.

LoRd_MuldeR
27th October 2008, 01:51
In the first post I was asking about what settings is better for lossy encoding for editing purposes, not what method to use for lossless transfer.

What more than "1-Pass CRF at highest CRF you can accept quality-wise" do you want to hear? :confused:

Lowering the key-frame interval may be of interest, in case you are going to cut the material without re-encoding...

Quark.Fusion
27th October 2008, 01:52
Is it that hard? If encoding time matters, use 1-Pass CRF. And choose the highest CRF value that you can accept quality-wise.

So you say if CRF 16 gives acceptable visual quality with acceptable size, so it's no need to seek for better? There is no need to change ip- and pbratio?

LoRd_MuldeR
27th October 2008, 01:54
So you say if CRF 16 gives acceptable visual quality with acceptable size, so it's no need to seek for better? There is no need to change ip- and pbratio?

There are a lot of options to tweak for an individual source, but it's hard to give a general recommendation, I think :o

But I'd consider using slower settings (higher me, higher subme, higher merange, etc.) to get a smaller file at same quality (same CRF) and further reduce upload times...

Guest
27th October 2008, 01:56
Buying and mailing hard drive can be sensible if you making money with your work, but for some non-pro editing it's not always an acceptable solution. You mean "the scene"?

You don't have a USB stick you can stick in a mailer for a few measly bucks?

Like I said, strawman city.

In the first post I was asking about what settings is better for lossy encoding for editing purposes, not what method to use for lossless transfer. But you press that lossless is the GOD without any argument. Even when I ask if it really is noone answer why. Ask me if I'm still beating my wife, and see if you get an answer.

If lossless is actually faster to transfer (via mail) then why mess around with re-encoding and sending over the internet, which is bound to diminish quality?

LoRd_MuldeR
27th October 2008, 02:03
If lossless is actually faster to transfer (via mail) then why mess around with re-encoding, which is bound to diminish quality?

Depends on the distance. Mailing a HDD might not be a good idea, if you need to get your data from one side of the world to the opposing side. Also upgrading to a faster internet connection might be more convenient than transferring digital data on a physical disc all the time, except you really have a HUGE amount of data to transfer (more than just a few GB).

Quark.Fusion
27th October 2008, 02:03
At the start I was thinking that because frames will be reused and reencoded it's better to make them same quality and set ip- and pbratio closer to 1 for it and was wondering how to specify value for b-pyramids — is it all useless and better to get different quality between frames, but higher overall?

Guest
27th October 2008, 02:06
Depends on the distance. Mailing a HDD might not be a good idea, if you need to get your data from one side of the world to the opposing side. Also upgrading to a faster internet connection might be more convenient than transferring digital data on a physical disc all the time, except you really have a HUGE amount of data to transfer (more than just a few GB). Let's ask QF how much data he's transferring and how far. Then we can do the definitive calculation.

The point is he keeps saying that lossless transfer isn't an option. But it clearly is an option. And it's likely to be faster for 10GB or more based on the internet transfer rate he quoted.

And look at his last post. Does it make any sense to you?

Sagekilla
27th October 2008, 02:10
I wasn't talking about slow internet connection, it's slow only for upload, for download it's 2-6mbit. The real need for lossy transfer was sometime ago for game video material and in the end my friend come up with some default encoding settings of video editor to transfer output for editing to another person. For lossless solution he say that result file is just to big. I want to find better solution for that situation as it happends for time to time. Buying and mailing hard drive can be sensible if you making money with your work, but for some non-pro editing it's not always an acceptable solution.

In the first post I was asking about what settings is better for lossy encoding for editing purposes, not what method to use for lossless transfer. But you press that lossless is the GOD without any argument. Even when I ask if it really is noone answer why.

So was I: I have FiOS which has 5 mbps upload. That's much faster and far more feasible than an 800 kbps connection for transferring a movie.

As neuron2 said, if quality is really a huge concern, buy a USB stick (you can get a decent sized 8 GB one for dirt) and stick the movie on there. But if you plan on even bothering with transferring a movie over the internets, don't expect to have quality if you have an 800 kbps connection and maybe 8 - 12 hours deadline to transfer.

foxyshadis
27th October 2008, 06:33
Okay, the obvious part of the first post that's missing is "I absolutely refuse to spend any money on this whatsoever, so don't even bother suggesting spending anything." Overnighting a USB stick can still cost $13 in the US, and sending hard drives overseas can be hundreds of dollars. Given that, mailing is just out, and the next best choice, upgrading the internet speed, is also out.

You can work around this, some FTP clients can start transferring a file as it's being written, I believe SmartFTP and FileZilla can. The fella on the other side just sets up a free FTP server, so it's a direct transfer. As long as the encoding is a little faster than the transfer you're fine.

Don't even THINK about touching ip and pb ratios, you don't really know what how they work and you'll just mess up overall quality. They're already both as low as makes sense. B-pyramid always increases, never decreases quality. Use the slowest encoding parameters that are still faster than your transfer speed, with a VBV that restricts maximum bitrate, so that your transfer time isn't too high if your chosen CRF is a little too low. In case your transfer ends early because you encoded too slow, you can resume it any time.