Samplerate and Sync.

[zerowalker]

Well when that is the matter, i don't know what to do.
Let's say i capture from a VCR, which clock should i sync after, the video clock, or the audio clock.
That's the dilemma here, as sadly, i don't have hardware that can keep everything sync.

I have read some documentation there, and it's very interesting, learned about video clock there, and how Virtualdub handles it.

http://i41.tinypic.com/2z5sr6h.png

For those who want to test a sync issue, i learned a way to easy see it.

Set up a graph like this, choose your video source and audio source however you want, and codec.

Then click on AVI Mux, go to Avi Specific, enable Capture Drift. And choose the audio pin (in this case it's pin 2, which is number 1 (1 is 0)).
Now capture and stop after awhile, maybe 1-2 minutes.

Now check the file, it should not have the correct framerate, it should be synced to the audio, and either han slightly faster fps or slower.

[Ghitulescu]

Finally I got it: you record the games by capturing them, like you'd do with say VHS. I didn't realize this before as I do not "do games" nor I like watching other people playing (I never understood why would one look at someone playing a game).
In this case, follow the advice of IanB (two posts above), and see how to capture properly when audio gets out of synch.

[IanB]

Fundamentally you have 3 disparate clocks sources. The incoming video frame rate, the PC clock and the audio card capture clock.

The goal is to correct the sync between the audio and the video.

The problem is you have audio rate and video rate measured from the PC clock. To use a standardise video playback rate you need to translate the skew between the video and pc clock through to extra virtual skew between the audio and pc clock.

For example you have a VCR or Gaming console that when measured from the PC clock appear to be running at 24.99875 fps. We also measure the audio card capture clock from the PC clock and it appears to be running at 48002 Hz.

So if we choose a standard 25.0 fps playback rate we need to include the 25 to 24.99875 fps skew into the audio skew thusly 48002 * 25.0 / 24.999875 = 48004.40.

So now we have normalised the video clock skew to the PC clock to zero and have an aggregate skew between the PC clock and the audio clock of 4.40, which is the actual skew between the video and audio clocks, which is the value we need to work with in doing the sample rate and sync correction.

Which clock was correct does not matter, it is the relative skew between the video source and the audio capture clock that matters. As I said earlier using a good integrated audio video capture card handles this automatically. So for the above video source the observed clock rate of an integrated audio clock should have been 47999.76 Hz

[creaothceann]

Even a real SNES will have slight differences in its video and audio rates. Real oscillators are affected by temperature etc. so their frequency varies from the norm.

PC monitors and sound cards don't run at exactly 60fps and 32/44.1/48/...kHz either. Emulators like ZSNES or SNES9x will drop or repeat a frame after some amount of time. The solution that byuu (author of higan, formerly known as bsnes) came up with is to dynamically resample the audio (http://board.byuu.org/viewtopic.php?f=8&t=3960&p=89174#p89174) (if VSync is enabled).

The only place you'll get perfect sync is in the world of mathematics and theoretical physics.

[raffriff42]

Capture timing and capture sync - VirtualDub blog (comment)
http://www.virtualdub.org/blog/pivot/entry.php?id=78

I worked on this problem in the late '90s for several years. The core problem is when you have more than one crystal, both trying to give you timing information -- there will always be some rate slew (CPU clock / video clock / sound card crystal / etc).

The only way to really solve this problem is to use a house clock, and make sure your audio and video gear all syncs off the same clock (for more on this, google "blackburst"). Needless to say, cheap consumer hardware won't go through the trouble of implementing this crucial feature, so we'll get wow and flutter from "compensating" software forever to come.

If you actually CARE about your multimedia, you need to get gear that can sync right, but expect an audio+video capture rig to set you back over $1,000. On the bright side, you'll be able to play back OUT with the same sync :-)

Jon

[zerowalker]

Totally missed all these posts.

But extremely interesting.

IanB, if i understand it correctly, for example.

If i capture from a VCR with PAL.

The audio should be 48khz, and the Video should be 25fps.

So i capture it without any sync to anything.

The result is, 25fps, But the audio is off sync, so if i sync the video to audio, it will be off, 24.9996 for example.
And as this is not correct, i will have to fix the audio, as the Video is supposed to be 25 as PAL is 25.

So in this scenario i will have to sync the audio to the video to get the correct video/audio.

Hope that's correct.


I have tried to make some experiments with my webcam, but even if i use it's internal microphone, it's of sync, it doesn't seem to record at 30fps, but a bit lower, so i am syncing the video to the audio there.
But i guess the webcam just have bad timestamps which causes even the internal audio and video to go off sync.

Cause as you say, when the Audio and Video are on the same device (not a PC though) they will often be synced to each other (take a video camera for example).


PC monitors and sound cards don't run at exactly 60fps and 32/44.1/48/...kHz either. Emulators like ZSNES or SNES9x will drop or repeat a frame after some amount of time. The solution that byuu (author of higan, formerly known as bsnes) came up with is to dynamically resample the audio (if VSync is enabled).

Funny that you mention that, i am currently investigating Higan a bit, and just got knowledge about the "repeat drop frame" thing.

raffriff42, i guess what's mentioned there is one of those "Master Clocks", and they seem to cost hell, and aren't really suited for "home work".


IanB, a question, as you never get an exact sync form something through the methods i use, and you have to fix it afterwards by syncing to it's intended format (fps/frequency).
Is there any Real loss of quality?

I mean, if there is dropped/repeated frames, then there is a real quality loss. But resampling the sync of audio, seems to be pretty transparent with a good resampler.
So is there anything else that's going on, or is it just there 2 things you need to think about?


EDIT:

Is this a correct way to sync audio to the video length, while maintaining it's samplerate?

b=36000000
AdjustDenominator=36000000
AR=AudioRate()
TR=Float(AR)*AR*FrameCount()/(AudioLengthF()*FrameRate())
AC=FrameCount/(Audioduration*FrameRate)
BB=int(b*AC)
AssumeSampleRate(1)
ResampleAudio(BB, AdjustDenominator)
AssumeSampleRate(AR)

Or can it be better, or is it way off?

[IanB]

From the bad old days when I used a Matrox rainbow runner and and a SBawe32 on a P-II 450 with a 40GB disk. I tried many many solution from that era and I found the best was to just capture the streams and collect a VirtualDub capture Timing Log (Capture -> Enable time log). I would import the log into Excel and use the Linear Regression (LINEST) to calculate the true video and audio rates and the initial start capture delay. My JVC deck ran slow, my Panasonic desk ran fast, my sound card ran fast, my P-II's clock ran slow. Now I use a usb capture device called a LiDiC it produces ~5Mb/s H.264 with stereo 48KHz AAC @192K, the audio clock is locked to the Video Hsync.

Fixing things after the capture allows you to choose much higher quality tools, during capture everything has to run in real time. And by not loading up the cpu during capture with "fix it" tools you can get a better capture (less dropped frames, less audio resync hunting).

And re-clocking the audio is so much easier and higher quality than trying to re-clock the video.
The goal is to correct the sync between the audio and the video.
Which clock was correct does not matter, it is the relative skew between the video source and the audio capture clock that matters.

[creaothceann]

i am currently investigating Higan a bit

This (http://web.archive.org/web/20130315081624/http://byuu.org/bsnes/accuracy) might be interesting (unfortunately much of the site's content is offline because of a recent server move).

[zerowalker]

I found the best was to just capture the streams and collect a VirtualDub capture Timing Log (Capture -> Enable time log). I would import the log into Excel and use the Linear Regression (LINEST) to calculate the true video and audio rates and the initial start capture delay.

Interesting, will look at this.
Sadly i am not all that familiar with how Excel works, i used it in school a bit, but it was just to make a diagram of thing, (like votes, which one got highest, that kind of stuff).

Fixing things after the capture allows you to choose much higher quality tools, during capture everything has to run in real time. And by not loading up the cpu during capture with "fix it" tools you can get a better capture (less dropped frames, less audio resync hunting).

That's very good to hear, as before i knew about this, i always captured with Virtualdub using dynamic resampling, and i think it has it's drawbacks, as it doesn't know the true difference (like the entire video vs audio), it just changes in realtime, which isn't always that accurate i think. And as you say, it's very limited in quality, you can't choose what you want etc, and it takes CPU cycles.

And re-clocking the audio is so much easier and higher quality than trying to re-clock the video.

how do you even re-clock a video?

I mean, if i capture a PAL VCR band, shouldn't this script be a way to perfectly sync it (if there is no capture delay).

CLIP
AssumeFPS(25, sync_audio=false)
ConvertAudioToFloat()
b=36000000
AdjustDenominator=36000000
AR=AudioRate()
TR=Float(AR)*AR*FrameCount()/(AudioLengthF()*FrameRate())
AC=FrameCount/(Audioduration*FrameRate)
BB=int(b*AC)
AssumeSampleRate(1)
ResampleAudio(BB, AdjustDenominator)
AssumeSampleRate(AR)

//Is this possible, or worthless

AdjustDenominator=ContinuedNumerator(PI())
b=ContinuedNumerator(PI())

////
Oh and btw if you didn't see that i edited my post.
Can this be improved (the audio sync to video length), as i pretty much copied a script i found from you, and edited it to fit into the new things you have allowed (AdjustDenominator etc).

And also, sadly, by doing this, i am limited to Avisynth's resampler, while it's not bad, atleast while working in Float. There is better.
Is it possible to make use of another resampler somehow?

This might be interesting (unfortunately much of the site's content is offline because of a recent server move).

Will look into that, thanks:)

[IanB]

With double precision, your adjustment calculation might be right, but Avisynth script floats are only 32 bit IEEE single precision. So you will probably hit a wall like you did earlier in this thread. I suggest you do all your calculations with integer arithmetic. Google for Rational Arithmetic.

Also VirtualDub capture is at the mercy of the video and audio capture drivers, so even though both are instructed to start and stop capturing at the same time sometimes they invariably do not. IIRC on my old P-II the Matrox driver could take up to 200 milliseconds to respond to both the stop and start commands. So yes the streams were different in length, but mostly not from the clock skew.

[zerowalker]

With double precision, your adjustment calculation might be right, but Avisynth script floats are only 32 bit IEEE single precision. So you will probably hit a wall like you did earlier in this thread. I suggest you do all your calculations with integer arithmetic. Google for Rational Arithmetic.

Double precision??

Do you mean that as Avisynth works in 32bit IEEE single precision (what does single/double precision mean?) the numbers it can work with is limited in this task?

I googled Rational Arithmetic, but i don't really understand it, i am not good at math at all sadly, and programming, even though i try to be.

But i am guessing you it's some kind of highest int number that Avisynth can work with, or something?

Also VirtualDub capture is at the mercy of the video and audio capture drivers, so even though both are instructed to start and stop capturing at the same time sometimes they invariably do not. IIRC on my old P-II the Matrox driver could take up to 200 milliseconds to respond to both the stop and start commands. So yes the streams were different in length, but mostly not from the clock skew.

Interesting, has happened to me on some occasion that the delay will be wrong, and the only way to solve it is to manually adjust the audio/video start.
Does the timing log record all this stuff, so you know exactly what's going on so you can fix it afterwards based on the log?

also.

AdjustDenominator=ContinuedNumerator(PI())
b=ContinuedNumerator(PI())

With using this, i am not that mathematical as i said earlier.
but i do know that PI is infinite, so will this be the biggest precision that Avisynth can offer with int?
(this can probably be way out of hand, as said, i am not that good with int math and stuff, so sorry if i am talking crap here)

[Ghitulescu]

If i capture from a VCR with PAL.

The audio should be 48khz, and the Video should be 25fps.

So i capture it without any sync to anything.

The result is, 25fps, But the audio is off sync, so if I sync the video to audio, it will be off, 24.9996 for example.
And as this is not correct, i will have to fix the audio, as the Video is supposed to be 25 as PAL is 25.

So in this scenario I will have to sync the audio to the video to get the correct video/audio.
You got pretty much the idea.
Now, almost every card can synch itself to VHS, provided the VHS signal is steady (eg ran through a TBC). I am not talking here about the possible bottlenecks that might hinder this, like the speeds of the CPU, memory, HDDs. The PC clock is not important here.
The failure to synch to VHS (I keep saying VHS, but in fact I refer to any other analogue source like camcorder) is immediately noticed by most people, thus the main time has to be provided by the video. We also know that VHS has a fixed and reproducible fps of 25Fps (or 50fps), or a granularity of 20 millisecond.
On the other hand, the audio is more forgiving. At normal sampling rates (44k1 or 48k) the granularity of audio is of only ~50 microseconds. People won't miss one audio frame, as they hardly notice a 20ms lipsynch, provided the dropped audio frame was correctly removed (no "Click" no "Pop"). And even if the audio has to be resampled (eg from 44k1 to 44056) this is far from being objectionable, as it would be the interpolation of video frames to match 25Fps.
Therefore audio has always to be synched to video, not viceversa. Again the PC clock is not at all important.

The PC clock becomes important however if the playback would take place on the PC. The PC clock would also become important if you've chosen to use time codes, as these are given by its clock.

[IanB]

@zerowalker, You really need to find some way to come to grips with the arithmetic here. Maybe a friend or college can explain it enough to get you started. And Google is your friend!

Double precision is a 64 bit floating point representation.

More than highest number, it's the smallest difference that can be represented.

If you have the required numerical skills, you can extract the required information from the log.

You said you knew how to do graphs with Excel. Well do a short capture with a log and graph the audio and video timestamps versus real time. You will get 2 roughly straight lines, jitter and drops will distort the lines. Where the lines intersect the 0 real time axis is the actual start delay for each stream, the slope of the lines is determined by the true audio and video rates. Methods like Linear Regression allow you to fit a "best" straight line to a set of jittery data points. With the "best" straight lines you will have the 2 startup delays and the 2 rates needed to sync the audio and video tracks. I used this method very successfully but it was well more than 10 years ago.

If you cannot get a grip on the maths involved here, perhaps you should consider getting a video capture device with synchronised and integrated audio capture.

=================================================

As for the continued fraction stuff, Pi=3.1415926535897932384.... this is a representation to 20 digits.

If we examine under the covers Avisynths version of PI()=3.141592741 which matches for the first 7 digits, which is the 6 to 7 digits you expect for a 32 bit float.

Using the continued fraction functions we can get a rational pair of integers that match the precision of the underlying 32 bit floating point number :-

ContinuedNumerator(PI())=93343
ContinuedDenominator(PI())=29712

And if we resolve the rational pair back to a 32 bit floating point number we get the original value back.

Float(93343) / 29712 = 3.141592741

But if we resolve the rational pair back at a higher precision we can see a deviation :-

93343 / 29712 = 3.1415926225094238018...

[zerowalker]

@zerowalker, You really need to find some way to come to grips with the arithmetic here. Maybe a friend or college can explain it enough to get you started. And Google is your friend!

Will try with google, as sadly, no one of my friends or that mathemetical;P
But hopefully i will grasp it sooner or later!

I will also do some tests to see if i get the Excel timing log, from your explanation, i think i should grasp it.

And with using AdjustDenominator=(ContinuedNumerator(PI()))
in the calculation for syncing the audio to video, i took it as it was very high.

Cause isn't a higher number better?
Meaning, the highest possible Integer number that IEEE 32bit allows, will be the highest precision it allows when calculation this, isn't that correct?

So can't i just use AdjustDenominator="highest int number", or something like that, if you get what i mean here.


AR=AudioRate()
TR=Float(AR)*AR*FrameCount()/(AudioLengthF()*FrameRate())
KScale=Int($7FFFFFFF/ TR)
ResampleAudio(Round(TR*KScale), KScale)
AssumeSampleRate(AR)

Here is the original script i took the idea from, which you posted.
And this one is before you fixed the limitations when working with small samples.


If i use function to get the correct AdjustDenominator/Numerator, i don't need any real knowledge about that mathematical going on.
Though of course i do want to know what's going on.

[pbristow]

It's horrifying to me that domestic capture devices are still unable to properly sync to a typical analog video source, given how many of them have been sold for the express purpose of hooking up to an existing VCR and "preserving your precious video memories" - a job that usually involves capturing an entire 2, 3 or 4 hour VHS tape in one go!

All that's required is for the hardware to derive a "theoretically 48kHz" audio clock from the incoming video line frequency, either directly through a PLL and divider network, or through a crystal-clocked timer that adjusts the rate of the audio clock. That way, if the video source is running slow (say), then the audio capture clock is slowed to match, and every block of 48000 audio samples stays reasonably well matched up to the corresponding second (25 frames) of video. Or, you could just over-sample the audio at a simple multiple of the line frequency (e.g. 8x) and then resample it down to the required rate. (With that much oversampling you can simply drop the unwanted samples, as any resulting jitter artifacts will be ultrasonic anyway. Let the audio purists pay for their expensive versions with "proper" interpolation, if they wish... ;) )

None of these should be expensive circuits to produce, if they were churned out en masse. Is there *really* no domestic or sub-£200 gear that does this?!?


Example:

25fps x 625 lines/frame = 15625 lines/second,
So the incoming (PAL) video is expected to contain Hsync pulses at a rate of 15625 Hz, but with some expected variation.

Multiply this by 8x using a PLL: 125kHz (but with the same, proportional variations). As long as the range of speed variation is les than about 5%, a PLL should have no problem tracking it - And I don't know of any VCR that runs that badly under/over speed!

Use this derived 125kHz signal to clock the audio capture, then downsample the audio by a fixed factor of 5.208333333... And hey presto! Audio is output at (theoretically) 48kHz, with the actual frequency from moment to moment automatically following any variations in the incoming video. You end up with a file on disk with an average of 48000 audio samples captured for each 25 frames of video, and with no individual moment of audio being more than a few samples (a tiny fraction of a frame) adrift of where it should be.

[zerowalker]

pbristow, indeed, it's very weird how the consumer market has been suffering with this all this time.


IanB, i am trying to figure out the higher int number i can use in AdjustDenominator, though i am not coming that far.

in a clip that has:

Audio length: 26422399 samples

I can use: 1721028160, anymore will break the calculation.
But this number is only correct for this clip, i am trying to find a way to get a function that works for every clip.
I am guessing the audio length has something to do with the number.

[IanB]

@pbristow,

Most integrated devices that have both audio and video connectors in to the devices are well behaved and lock the audio clock to either the video line (15625Hz/15734.27Hz) or field rate (50Hz/59.94Hz).

The problem comes when people try to use a free standing sound card for the audio capture. (Watch for capture cards that internally link to the free standing sound card).


@zerowalker,Audio length: 26422399 samplesA piece of string is 5 long :devil: (you have only provided part of the information).

From the context of past posts, we can assume you want to re-sample the audio track to make it the same length as the video track. Okay accepting the limited accuracy of a single precision float, this will give you the best pair of integers from the floating point ratio :-AR=AudioRate()
TR=Float(AR)*FrameCount()/(AudioLengthF()*FrameRate())
ResampleAudio(ContinuedNumerator(TR), ContinuedDenominator(TR))
AssumeSampleRate(AR)Be aware that floats only have 24 bits of precision, so the finest ratio that can be represented is 8388608 / 8388607 = 1.00000012 or 8388607 / 8388608 = 0.99999988

[zerowalker]

A piece of string is 5 long (you have only provided part of the information).

Well what i meant was, the highest int number that can be used , for the highest precision, is limited depending on the audio samples, i think (you probably know this better than me).

From the context of past posts, we can assume you want to re-sample the audio track to make it the same length as the video track. Okay accepting the limited accuracy of a single precision float, this will give you the best pair of integers from the floating point ratio :-

Yes, that function just aims for clocking the audio to the video length. And for that i would like to have the highest possible precision.

And i know that Avisynth is limited to using Int for calculating the variation in the end.


AR=AudioRate()
TR=Float(AR)*FrameCount()/(AudioLengthF()*FrameRate())
ResampleAudio(ContinuedNumerator(TR), ContinuedDenominator(TR))
AssumeSampleRate(AR)


Though from this, you only use Float?
It "works" when i tried, meaning it didn't complain on the float (or maybe it's int?). But the samples are way of, it's making a 7 hour clip becoming 180ms.
Fixed it, i forgot Assumesamplerate(1)!

Or maybe i have missunderstood something here?

[IanB]

I would not have thought scaling the audiorate would make any difference, 0.99999988 or 47999.99424 both have 24 bit precision. I will have to think on this for a bit.

Ding! :sly: Ah yes it will make a difference, because using unscaled will crimp the style of the 32 bit rational denominator, by almost 15.5 bits.AR=AudioRate()
TR=FrameCount()/(AudioDuration()*FrameRate())
AssumeSampleRate(1) # Scale to maximise available precision
ResampleAudio(ContinuedNumerator(TR), ContinuedDenominator(TR))
AssumeSampleRate(AR)

7 Hours is 1209600000 samples, 180ms is 8640 samples. As float 1209600000.-8640. = 1209591296. or 1.209591296E9 not the anticipated 1209591360 an error of 64 samples or 1.33ms.

[zerowalker]

Okay wait, this is getting ahead of me, let's see if i get it correctly.

Ding! Ah yes it will make a difference, because using unscaled will crimp the style of the 32 bit rational denominator, by almost 15.5 bits.

So, by using AssumeSampleRate(1), it will increase the available precision, while not using it, will decrease it by "15.5bits"?

AR=AudioRate()
TR=FrameCount()/(AudioLengthF()*FrameRate())
AssumeSampleRate(1) # Scale to maximise available precision
ResampleAudio(ContinuedNumerator(TR), ContinuedDenominator(TR))
AssumeSampleRate(AR)

Is this function the highest possible precision that avisynth allows for calculation the length of video/audio?
Is ResampleAudio(ContinuedNumerator(TR), ContinuedDenominator(TR)) the highest number that can be used?

7 Hours is 1209600000 samples, 180ms is 8640 samples. As float 1209600000.-8640. = 1209591296. or 1.209591296E9 not the anticipated 1209591360 an error of 64 samples or 1.33ms.

Not sure what you mean here, do you mean that without Assumesamplerate(1) i will generate the error instead of the assumed correction?
Meaning if i calculate a 7 hours audio to a video, i will get 64 samples wrong.?

[IanB]

So, by using AssumeSampleRate(1), it will increase the available precision, while not using it, will decrease it by "15.5bits"?Of the denominator!

Because we are talking about numbers around 1, to maximise the available precision of the rational pair we want both numerator and denominator to be the same order of size. Including a factor of 48000 in the numerator reduces the range of denominator values to MAXINT/48000 = 44739.
Is this function the highest possible precision that avisynth allows for calculation the length of video/audio?
Is ResampleAudio(ContinuedNumerator(TR), ContinuedDenominator(TR)) the highest number that can be used?
I have already answered this, and with an example. Please go and learn about floating point numbers and the precision of rational numbers. If you use single precision floating point numbers, the precision is 24 bits. If you use integer arithmetic with rational pairs you can calculate an exact result, but that result may not fit within a pair of 32 bit integers, however you can use the continue fraction method to calculate the closest pair of 32 bit integers.
Not sure what you mean here, do you mean that without Assumesamplerate(1) i will generate the error instead of the assumed correction?
Meaning if i calculate a 7 hours audio to a video, i will get 64 samples wrong.?With 32 bit floating point numbers the result is only accurate to 24 bits. For a 7 hour stream than means approximate to within 64 samples. Including a relativly large scale factor in the numerator will further reduce the available precision.

[zerowalker]

Of the denominator!

Because we are talking about numbers around 1, to maximise the available precision of the rational pair we want both numerator and denominator to be the same order of size. Including a factor of 48000 in the numerator reduces the range of denominator values to MAXINT/48000 = 44739.

Sorry, as said, i am bad at this.
I am reading around like mad and trying out stuff, i hope i am slowly getting more knowledge.
It's hard with the mathematics i have never used, especially when the words are in english (i am from Sweden), so there is some problematic there as well.

I have already answered this, and with an example. Please go and learn about floating point numbers and the precision of rational numbers. If you use single precision floating point numbers, the precision is 24 bits. If you use integer arithmetic with rational pairs you can calculate an exact result, but that result may not fit within a pair of 32 bit integers, however you can use the continue fraction method to calculate the closest pair of 32 bit integers.

Really;S
Sorry once again. I do know a bit about floating point from my programming classes. But the thing i know is pretty much, Float is much higher then int. that's pretty much it sadly, will have to really read up on the rational numbers.
Really need to know how to calculate with integer arithmetic, as if what you say is true (which it is), i can use integer arithmetic with rational pairs, get the exact value, and then use "continue fraction method to calculate the closest pair of 32 bit integers", and that will be higher precision than floating point, as it's 24 bit.

I did a bit of reading on that function, and i guess that's 24 bit single precision, as you stated in an earlier post (was pretty focused that it used Float when it must be int in the Numerator/Denominator).

With 32 bit floating point numbers the result is only accurate to 24 bits.

This proves how little i know.
As when i read this, it doesn't make much sense to me.

32 bit is only accurate to 24 bit.

So once again, i must really read up on the floating point.

Thanks as always, sorry for wasting your time with worthless questions when i don't get some stuff.

[IanB]

Back in post #119 there was an error I used AudioLengthF instead of AudioDuration, I have edited the post to make it right. :o

From Sweden, I would never have guessed, your written English is superb, better than most native speakers on this board. ;)

If we try a simplistic approach we run into integer overflow for quite small clips, less than 30 minutes for pal.# Typical values and limits
# FrameCount() 900000 for 10 hours with pal
# FrameRateNumerator() 25 or 50 for pal, 30000 or 60000 for ntsc
# FrameRateDenominator() 1 for pal, 1001 for ntsc
# AudioRate() 44100, 48000, 96000, 192000
# AudioLength() MAXINT for 12.4 hours at 48000
#
AR=AudioRate()
TRnum=AR * FrameCount() * FrameRateDenominator() # Goes bang at frame 44740 for pal, 45 for ntsc
TRden=AudioLength() * FrameRateNumerator() # Goes bang at 00:29:49.5 with pal, 00:00:1.5 with ntsc
AssumeSampleRate(1)
ResampleAudio(TRnum, TRden)
AssumeSampleRate(AR)
Lets dust of some mathemagical tricks to see what we can squeeze out.
AR=AudioRate()

VDnum=FrameCount() * FrameRateDenominator()
VDden=FrameRateNumerator()

ADint=AudioLengthHi(AR) # AudioLength64 / AudioRate
ADrem=AudioLengthLo(AR) # AudioLength64 % AudioRate

Snum=MulDiv(VDnum, AR, MaxInt)
Sden=MulDiv(ADint * VDden, AR, MaxInt)
Scale=Max(Snum, Sden) + 1 # Divisor to ensure MulDiv does not overflow

TRnum=MulDiv(VDnum, AR, Scale)
TRden=MulDiv(ADint * VDden, AR, Scale) + ADrem * VDden / Scale

AssumeSampleRate(1)
ResampleAudio(TRnum, TRden)
AssumeSampleRate(AR)
For a 12.4 hour clip Scale will be 26, which results in precision to within 38.4 ms, which is not quite as good as the floating point version, using continued fraction technology might improve on this.

The above code is untested!

[zerowalker]

From Sweden, I would never have guessed, your written English is superb, better than most native speakers on this board.
Well thank you. Though i use the inbuilt spell check on my browser to prevent bad spelling, as i am crappy on that part.

If we try a simplistic approach we run into integer overflow for quite small clips, less than 30 minutes for pal.

Luckily i know the normal stuff, audiorates/framerates and framecount, don´t get what you mean here though: # AudioLength() MAXINT for 12.4 hours at 48000. Do you mean that at 12.4 hours on a PAL video (25 fps) the audiolength if 48khz is used will be the maximum integer allowed?

And what does that code do, is it to show the limitation of Int, meaning when a certain length is reached with a certain framerate, it goes out of the boundaries for which is allowed, breaking the calculation?


For a 12.4 hour clip Scale will be 26, which results in precision to within 38.4 ms, which is not quite as good as the floating point version, using continued fraction technology might improve on this.

The above code is untested!

Just to see if i understand it correctly here.

When you say, precision within 38.4ms, do you mean that if the real answer it for example, 55.9 ms, it will off that answer by 38.4ms?
If that's what i means, it's very accurate. How accurate will floating point be then, as that's supposed to be less accurate then Integer with Rational Numbers with Continued Fraction.

Also, i can't test the code, i don't know what i am supposed to do with "MaxInt", there is no such function, so am i supposed to calculate or know the MaxInt on my clips?

And, i have done some testing, and noticed something,

If i use:

TR=float(AR)*AR*FrameCount/float(AudioLengthF*FrameRate)
it seems to be either more accurate, or less acurate, then when using

TR=float(AR)*FrameCount/float(AudioLengthF*FrameRate)
Assumesamplerate(1)

i actually think it's more accurate, though you can probably see if it is or not.


And hate to bring up Excel, but i can't get the graph to work as intended.
I can get 2 lines, but Audio and Video aren't even close to each other, the Audio goes way faster than Video, but they end at roughly the same size.
It seems that i was able to calculate the estimated framerate though, by taking the framecount / Vglobaltime (the last in the log), and then i would get the frame rate, then i could use that to recalculate the difference in the audio, and resample it to sync to 30fps.

Though it seems it doesn't work every time, so it may not be accurate at all, and simple just happens to be close, but this is just me testing.

[creaothceann]

MaxInt is 2^(32-1), i.e. 2147483648.

[zerowalker]

Okay thanks, added: MaxInt=Int(2147483648). Hopefully that's correct.
But the code still doesn't work as intended i think, 9 hours becomes 40 hours.
But either i am doing it wrong, or it's cause of it being untested.

[creaothceann]

Why the Int() ?

[zerowalker]

If i didn't use it, the script didn't work at all.
I think the maxint changes depending on the clip. I think that the longer the video, the less size is the MaxInt.
Not sure though, but it atleast seems to matter for some reason.

[Gavino]

MaxInt is 2^(32-1), i.e. 2147483648.
MaxInt (meaning the largest positive integer representable in signed 32 bits) is actually 2^(32-1)-1, or 2147483647 ($7FFFFFFF)

If 2147483648 is used in a script, it is converted to float since it is too big to fit in an int.

[zerowalker]

Ah that explains it, now it works.

Though it seems to be quite limited (guess that was the purpose?), and breaks on to long videos.

Also: Timing Log (http://www.sendspace.com/file/i79wfz)

I am trying to use this timing log to calculate the sync issue (in this particular capture).
But i am not able to do it, i have done an estimated calculation through listening, and that is:

ConvertAudioToFloat()

AR=AudioRate()
TR=float(AR)*(32260240)/(32400000)

ResampleAudio(ContinuedNumerator(TR), ContinuedDenominator(TR)) #AssumeSampleRate(1)
AssumeSampleRate(AR)


But that is just from me listening and calculating from that.
There must be a Correct way to do it, atleast when a timing log is available, but sadly i don't see it.

[IanB]

precision within 38.4ms,Uncertainty of the answer, the result can be off by up to that amount in either direction. It might be right, it might be low, it might be high, but it will be within that range.

And yes MAXINT is $7FFFFFFF

And geez I said start small. 46.6 MB capture log, 9 hours 23 minutes, 1012250 frame capture and NTSC not PAL like you have been implying.


Linear regression [y=mx+b] of the first 65k lines of the log. (The PC I am borrowing only has Excel 2002.)

=INDEX(LINEST(C2:C65536,A2:A65536,TRUE,FALSE),1), M value is 33.47367754
=INDEX(LINEST(C2:C65536,A2:A65536,TRUE,FALSE),2), B value is 55.7123293

=INDEX(LINEST(H2:H65536,G2:G65536,TRUE,FALSE),1), M value is 0.010415426
=INDEX(LINEST(H2:H65536,G2:G65536,TRUE,FALSE),2), B value is 15.8849696

Video capture started at 55.7123293 ms and we have 33.47367754 ms per frame.

Audio capture started at 15.8849696 ms and we have 0.010415426 ms per byte.

So the audio capture started 39.8273597 ms before the video capture.

Use a DelayAudio(-0.0398273597)

The measured video rate is 29.87421979 fps, the measured audio rate is 48005.717673 Hz.

For a standard NTSC rate of 30000/1001 fps the relative audio rate is 48159.6777261 Hz.

So the correction factor should be 0.99668441041

[zerowalker]

Uncertainty of the answer, the result can be off by up to that amount in either direction. It might be right, it might be low, it might be high, but it will be within that range.

And yes MAXINT is $7FFFFFFF

Great:)

And geez I said start small. 46.6 MB capture log, 9 hours 23 minutes, 1012250 frame capture and NTSC not PAL like you have been implying.

Sorry for that, i do have lower, why i uploaded a "massive" capture log was cause i thought, the longer the clip, the higher precision could be made.
And it's a Webcam, it's not PAL or NTSC really, it's 30fps (not 29.976), though Avisynth says it´s 30/33333333 or something like that, not sure which is correct there.

(why i used a webcam was just for testing, as it was an own device "outside" of the PC, and using a separate mic, so i could produce a similar scenario as when capturing other stuff).

Linear regression [y=mx+b] of the first 65k lines of the log. (The PC I am borrowing only has Excel 2002.)

Not really sure how you even do that, but will try to find my way there.

Cause from your information, i can get all the information out of that log, making a perfect sync. I am Amazed!


EDIT:

Okay was able to understand the function now, took some time.

But i wonder, is it possible for the audio to start at a negative number?
I get this with a test clip: -65.4628313190

Also, how do you calculate the relative rate of audio/frame?


EDIT 2:

Okay i got how you calculate the Framerate, 1000/(Frametime).
But i can't find anything about calculating the sample rate from ms and bytes:(

[IanB]

Because Linear Regression is about fitting a single straight line, y=mx+b, to a set of data points, such that the line gives the lowest overall error, a negative "b" is always possible.

However given the way capturing works a negative "b" is unlikely and I would be double checking my maths very carefully.

From the audio format you need to know how many bytes per sample to calculate sample rate from bytes and milliseconds.

For 16 bit mono it is 2 bytes per sample, for stereo is is 4 bytes.

So 0.010415426 ms per byte is 0.020830852 ms per sample => 1000/0.020830852=48005.717673 samples per second.

[zerowalker]

Because Linear Regression is about fitting a single straight line, y=mx+b, to a set of data points, such that the line gives the lowest overall error, a negative "b" is always possible.

However given the way capturing works a negative "b" is unlikely and I would be double checking my maths very carefully.

From the audio format you need to know how many bytes per sample to calculate sample rate from bytes and milliseconds.

For 16 bit mono it is 2 bytes per sample, for stereo is is 4 bytes.

So 0.010415426 ms per byte is 0.020830852 ms per sample => 1000/0.020830852=48005.717673 samples per second.

It seems to be correct. Did another and that one was 13ms.
What am i to do with a negative number?
Am i supposed to calculate from negative to positive?
Example:

-60 on audio
40 on video.

Audio = 100 delay ?


And also, how did you get a factor from the wanted framerate compared to the relative sample rate?

EDIT:

I think i got it:

TR=float(AR)*("Relative framerate")/float(framerate)*(AR/"Relative audiorate")

But i don't know if i am supposed to Delayaudio before or after resampling. I guess before though.

[zerowalker]

=INDEX(LINEST(C2:B65536,A2:A65536,TRUE,FALSE),1), M value is 33.47367754



I Noticed that i overlooked this. Is this a typo?
Should it be B2:B... or C2:C...?
Cause i can´t get anything from C2:B...

I am making a VBA script that will post everything, making this alot easier, will post it here once i am done:)

[IanB]

If the LR gives a negative B it just means the "start time" needs to considered to be earlier in following calculations. So -60 and 40 is 100.

It could also mean the audio capture driver is not starting with an empty buffer, so the first chunk of data arrives much earlier than expected, because there are samples from before you pushed the start button. It does not matter because the methodology discard audio samples before the start of the video and stretches/squeeze the remaining audio track to make the ends match.

And yes you just include any video scaling into the audio scaling calculations.

The DelayAudio goes first, but doesn't really matter, the change between before or after is so minuet it's lost in the precision available. i.e. -0.0398273597 versus -0.0396953085 thus 132 microseconds.

C2:B... yes it's a typo, I have fixed the post above.

[zerowalker]

If the LR gives a negative B it just means the "start time" needs to considered to be earlier in following calculations. So -60 and 40 is 100.

It could also mean the audio capture driver is not starting with an empty buffer, so the first chunk of data arrives much earlier than expected, because there are samples from before you pushed the start button. It does not matter because the methodology discard audio samples before the start of the video and stretches/squeeze the remaining audio track to make the ends match.

And yes you just include any video scaling into the audio scaling calculations.

The DelayAudio goes first, but doesn't really matter, the change between before or after is so minuet it's lost in the precision available. i.e. -0.0398273597 versus -0.0396953085 thus 132 microseconds.

C2:B... yes it's a typo, I have fixed the post above.

I didn´t quite get what you meant there.
If the audio has samples from before i started, and those are discarded, won´t that alter the way i should calculate the Delayaudio?
Shouldn´t i calculate as the audio was started as 0ms then?

And good to hear, will use DelayAudio before, even if it doesn't matter, it´s "better" to do it correct:)

Okay, good, was surprised how it was supposed to work if it wasn't.

But is is really C, i though B was the captured time frame, and C was the "wanted" time frame?

So shouldn't i compare the Audio to the actual video capture?



And here is the Macro:)
Took quite some time as i didn't understand the scripting that well, and got help with it.

It is extremely poorly optimized, and you don´t want to see the code;P

But the results should be precise.

Though i am trying to solve that Excel keeps truncate calculations, making them a bit less precise than they can be.

But still, from my testing, i can keep anything in sync by following the script (at least with what i have added).

Please try it, and if you have any thoughts of it, please tell.

btw: Webcam is my Webcam's FPS, which is 30.0000 (10000000/333333).
Also, i don´t know how to calculate 24 bit, so it´s probably not accurate at all*

[IanB]

The Linear Regression is about fitting a single straight line, y=mx+b, to a set of data points, such that the line gives the lowest overall error.

Note! Lowest error not necessarily zero error.

When processing the .CSV logfile be careful, the audio data ends much earlier than the video data, do not include the ",,,,," cells in your audio LINEST range.

If you get a negative B's just fold them into the calculations.

If you get an Audio B later than the Video B just fold it into the calculations.

The method is right. It just handles the numbers.

Your Webcam's FPS is purely nominal. It is stating 10000000/333333=30.00003 fps but you are measuring 29.87421979 fps. The claimed 30fps is bullshit.

Assign the most sensible rate for you final play back, i.e. 30000/1001 (ntsc) or 25/1 (pal) and scale everything to suit.

The 2 video times, B & C, are the lie reported by the driver and the actual time measured by VirtualDub. Analysing the driver reported time and comparing it measured time can be instructive when looking for dropped or duplicated frames.

[zerowalker]

The Linear Regression is about fitting a single straight line, y=mx+b, to a set of data points, such that the line gives the lowest overall error.

Note! Lowest error not necessarily zero error.

I see, why isn´t it possible to get Zero error?
Is it because of the limitation of numbers that Excel allows, or how Linest works, or simply the log that Virtualdub makes?

When processing the .CSV logfile be careful, the audio data ends much earlier than the video data, do not include the ",,,,," cells in your audio LINEST range.

I don´t understand this. I know that the Audio data ends much earlier, which is why i had to make a script that looks for every cell Until it´s blank. I guess that correct´and ",,,," = blank?

If you get a negative B's just fold them into the calculations.

If you get an Audio B later than the Video B just fold it into the calculations.

So, no matter if it´s negative or positive, i shall always look for the difference, (Lower audio = - delay, higher audio = + delay)?
Example: video start = 10ms, Audio start 100 ms, , that´s 90 ms Later, meaning Delayaudio(0.090) - Actually when i look at this, it makes more sense in Delaying audio with + if it´s starting earlier to make it the same as the Video, but i guess that´s wrong.

The method is right. It just handles the numbers.

Great:)

Your Webcam's FPS is purely nominal. It is stating 10000000/333333=30.00003 fps but you are measuring 29.87421979 fps. The claimed 30fps is bullshit.

So, the 30fps it´s saying it will do, is actually just wrong?
But if i change the FPS, the video will become longer/shorter, won´t that make it, inaccurate?

The 2 video times, B & C, are the lie reported by the driver and the actual time measured by VirtualDub. Analysing the driver reported time and comparing it measured time can be instructive when looking for dropped or duplicated frames.

When you say "lie" is that only for this device cause you said the "30" fps is way off?
And so it´s possible to compare B and C to look for dropped frames?

[IanB]

You cannot get zero error because the data is bumpy, it has jitter.

If the data was 10, 20, 30, 40, 50, 60, ... i.e exactly 10 each time then it would be possible.

But the data is like 9, 21, 29, 42, 49, 50, ... i.e sort of 10 each time but with jitter. You can get a best line but not all the data points are on that line.


For the ",,,," if you look at the .csv in notepad you will see about 1/15th down the file the audio columns become ",,,," which excel loads as blank cells. For video you get a line every frame, for audio you get a line every ~0.5 seconds.


Yes, lower audio = - delay, higher audio = + delay


The 30fps is nominal. i.e. the value specified and aspired to. In real life it is something close, measured at 29.87421979 fps using your PC's clock as a reference. Maybe it is truly 30.0 fps and your PC clock is wrong, maybe the driver code is slack. It does not matter. We just want to match the video and the audio. Then we want to play them back and again we need to specify a useful nominal frame rate. If you are smart you choose a nominal frame rate the same as everybody else does, i.e. 30000/1001 (ntsc) or 25/1 (pal).


I say lie because many capture drivers return bogus numbers. Your webcam driver appears to return a version of the PC clock, and given it is a webcam that is probably expected. The webcam driver probably uses the PC clock to wait for an interval of 1/30 of a second and then takes some 0.0001403442 seconds to give some instructions to the camera and have the camera respond with a frame and then simplistically waits for another interval of 1/30 of a second, hence the slightly slow measured 29.87421979 fps.

A helpful capture driver would return the count for the internal PLL clock synchronised to the analogue video sync pulses.

[zerowalker]

You cannot get zero error because the data is bumpy, it has jitter.

If the data was 10, 20, 30, 40, 50, 60, ... i.e exactly 10 each time then it would be possible.

But the data is like 9, 21, 29, 42, 49, 50, ... i.e sort of 10 each time but with jitter. You can get a best line but not all the data points are on that line.[/QUOTE

I see, isn´t it possible to make the log jitter free?
By increasing the intervals on the logging or something?

[QUOTE]For the ",,,," if you look at the .csv in notepad you will see about 1/15th down the file the audio columns become ",,,," which excel loads as blank cells. For video you get a line every frame, for audio you get a line every ~0.5 seconds.

Ah, well that is solved, the script ignores the blank cells.

Yes, lower audio = - delay, higher audio = + delay

Great:)

The 30fps is nominal. i.e. the value specified and aspired to. In real life it is something close, measured at 29.87421979 fps using your PC's clock as a reference. Maybe it is truly 30.0 fps and your PC clock is wrong, maybe the driver code is slack. It does not matter. We just want to match the video and the audio. Then we want to play them back and again we need to specify a useful nominal frame rate. If you are smart you choose a nominal frame rate the same as everybody else does, i.e. 30000/1001 (ntsc) or 25/1 (pal).

Well it matters also that i get hte correct time. For example, if i record, and i say "hi" at 140 seconds when i record. And on playback i have synced to NTSC, it will be off.
So the nominal is the most "correct", though it´s just this scenario, as if what you say how the capture driver acts. The this isn´t really something that i will run into.

The webcam was also just something i could capture to try with, as i currently don´t have any VCR and stuff up and running, other capture devices normally have something they are supposed to be running at (PAL = 25 etc).

I will go with syncing to the "Capture framerate" with the webcam, as if my thought are right, that will be the correct way if i would like to sync to something else.

For example:

I Have a video i record which is 25 fps. And along with that, i record webcam and talk about the video.

So, to make those synced to each other, the 25fps will have to be 25fps, and the webcam would have to be "webcam fps", and of course the audio should be synced to the respective device.

Hopefully i explained that correctly;P

[IanB]

No the jitter is in the real world. The log records when things actually happened.



Yes you can sync two clips the same as you sync the audio and video of one clip.

Syncing a 25 fps pal with a ~30 fps webcam will be a challenge. You need to make them both the same first then fine adjust the sync.

[zerowalker]

No the jitter is in the real world. The log records when things actually happened.

Don´t quite understand this,
Is there no jitter on the actual recordings, but when logging it appears compared to what´s expected?

Syncing a 25 fps pal with a ~30 fps webcam will be a challenge. You need to make them both the same first then fine adjust the sync.

It should work fine as long as i use the same Fps as the capture driver wants on both videos, and sync the audio to both videos.
The i should just need to put them together and adjust the timing (delay) of the two, then they should run in sync with each other (except the difference between the frames, as 30fps is more then 25, so 30 should be more precise per ms, but it´s negligible)

[zerowalker]

Okay, i am having a bit of a problem when doing calculations.

I was currently using a way to invert a calculation, but it seems it isn´t accurate for some reason.

For example:


25/24 = 1.041666666666667
1 / 1.041666666666667 = 0.9599999999999997
1 / 0.9599999999999997 = 1.041666666666667

There it works perfectly, i can invert it back and forth.
But now the problem appears.

24/25 = 0.96
1 / 0.96 = 1.041666666666667
1/1.041666666666667 = 0.9599999999999997

I get the same results afterwards, but they must be incorrect.
As by inverting, i should get from 1.04..... to 0.96, but instead i get 0.9599999999999997.


I am not mathematical so it won´t surprise me if there is a logical explanation for this.

[zerowalker]

Hate to open an old topic, but easier than starting a new and explaining.

I would like to know, is the code that stretches the audio portable to c++?

No programming skills to mention though, but as it's very stable and works as expected, i hoped that i could learn something from it so use in , well c++ for example.

Thanks

[foxyshadis]

You mean ResampleAudio? The implementation is in audio.cpp (https://github.com/AviSynth/AviSynthPlus/blob/MT/avs_core/core/audio.cpp#L1166). Or do you mean the Excel line-fitting?

[zerowalker]

Probably, and wow that was complicated.

I can work with Audio in a float array, like stereo to mono, 32bit to 16bit, such stuff. This is on a completely different scale.

Thanks though:)