Limiting amplitude of audio spikes

[Poutnik]

Is there recommended automated way to remove audio spikes,
based on limiting of amplitude of WAV/MP2 of audio record ?

I often do processing of DVB-T videos for personal archive purposes.
I use ProjectX to fix transmittion errors and process m2v and wav/mp2 files with MeGUI..

There are occasional spikes of maximum amplitude, some dozens ms in duration.
The problem with them is they ruin listener comfort.
They make squeezing noise and player often set volume temporarily weaker.

I fix it eventually by manual way in Audacity by setting silence there.

Is there an automated way, best by command line,
that either selectively limit amplitude under given values,
or fix/replace such a piece ?

I have not found in Audacity, or elsewhere, automated way, but I am not familiar with it audio processing so much.

[Poutnik]

I tried to search Int/Ext Avisynth filters, but I am afraid nothing usable, possibly some clever script......

There are some filters in Audacity to remove something like that, but it is probably aimed to much shorter analog spikes.
These ones seem to be corrupted 24/48 ms mp2 audio chunks, and filters do not catch them at any parameter settings.
But I can be wrong, as I am novice in AC.

But best way would be command line filter.

[Poutnik]

Hm, I got an idea, it should be simple.

Audacity has amp scale -1.0 .. 1.0, the signal amp is usually within -0.5 .. 0.5, with spikes -1 .. 1.
If I amplify it in Avisynth by proper coef, it should discriminate the spikes, and not yet the signal.
Then I could attenuate it back again.

[pandy]

foobar2000 hard -6dB limiter or advanced limiter - but... what is reason for such spikes? Perhaps improving reception will make stream glitch free - solving spike issue should start from root.

Hm, I got an idea, it should be simple.

Audacity has amp scale -1.0 .. 1.0, the signal amp is usually within -0.5 .. 0.5, with spikes -1 .. 1.
If I amplify it in Avisynth by proper coef, it should discriminate the spikes, and not yet the signal.
Then I could attenuate it back again.

Spikes ALWAYS will be on +-1 for digital samples - digital signal can't be higher or lower than +-1 - spikes are errors when decoder produce error sample value due lack or due of corrupted data.

[Poutnik]

foobar2000 hard -6dB limiter or advanced limiter - but... what is reason for such spikes? Perhaps improving reception will make stream glitch free - solving spike issue should start from root.

I am interested rather on encoding side than playback side.

I agree with improving idea, but this would be problem. It is a block of flat common wall source of TV signal, based on DVB-T receiver.

Sometimes there is none, sometimes 2-3, sometimes 20 in case of near thunderstorm. I would say ProjectX would eliminate most of them, but it seems not.

Spikes ALWAYS will be on +-1 for digital samples - digital signal can't be higher or lower than +-1 - spikes are errors when decoder produce error sample value due lack or due of corrupted data.

I am afraid you are wrong here. They are not digital 1-0.
+/- 1 is relative scale of sound amplitude stored in WAV format.
Of course wave format cannot produce more than what can produce.

[turab]

I think this is what you want: http://manual.audacityteam.org/man/Low_Pass_Filter. Your approach isn't so good because of the scaling.

[Poutnik]

I think this is what you want: http://manual.audacityteam.org/man/Low_Pass_Filter. Your approach isn't so good because of the scaling.

Thanks. I am aware about existence of low pass filters,
but I am afraid it cannot be distinguished by frequency from signal.
It would probably attenuate the majority of signal.

I think my the approach is acceptable
as there are not such quality demands as for CD audio.

I can try it.
Or, I can look for Avisynth alternative for
mp2/wav -> Avisynth -> wav
or
mp2/wav -> Avisynth -> BEpipe/wavi -> NeroAAC CLI encoder

[markanini]

Please upload a short clip (audio only) containg the spikes.

[Poutnik]

Well, currently I have none, no DVB-T MPG to process.
Later at home I will try t get one and upload it. I bet none will occur if I want to find it :-)

In Audacity the Hard Limiter within LADSPA Effects does exactly what I want.

Even better would be some automated Avisynth processing.

[Poutnik]

As I said . :-)
I have remotely checked some 55 min freshly recorded document, and no spikes there in demuxed ProjectX processed WAV,
not even in original MPEG2 PS.

[pandy]

I am interested rather on encoding side than playback side.

foobar can decode and encode with DSP processing in real time.


I agree with improving idea, but this would be problem. It is a block of flat common wall source of TV signal, based on DVB-T receiver.

Better antenna? Like Biquad for example:

https://www.youtube.com/watch?v=gHwZEEFvnVM
http://www.cnet.de/praxis/wochenend/41001557/die_beste_eigenbau_dvb_t_antenne_doppelquad_fuer_5_euro_basteln.htm
http://wirbel.htpc-forum.de/electronics/biquad/index2.html


Sometimes there is none, sometimes 2-3, sometimes 20 in case of near thunderstorm. I would say ProjectX would eliminate most of them, but it seems not.


ProjectX can't help you to improve reception and signal with errors can be decoded correctly - key to solve your issue is improve RF signal reception which should be made with proper hardware (antenna and antenna localization)


I am afraid you are wrong here. They are not digital 1-0.
+/- 1 is relative scale of sound amplitude stored in WAV format.
Of course wave format cannot produce more than what can produce.

No, i use +-1 as float values - in fact it should be or +-0.5 or 0 - 1 for integer PCM but i understand your convention and i use this convention.

[Poutnik]

foobar can decode and encode with DSP processing in real time.
I suppose so, for sound it should not be difficult. But I would prefer Avisynth solution I am more familiar with.

Better antenna? Like Biquad for example:

https://www.youtube.com/watch?v=gHwZEEFvnVM
http://www.cnet.de/praxis/wochenend/41001557/die_beste_eigenbau_dvb_t_antenne_doppelquad_fuer_5_euro_basteln.htm
http://wirbel.htpc-forum.de/electronics/biquad/index2.html

I would not say the problem is the signal strength, but some unclear disturbations in the signal.

ProjectX can't help you to improve reception and signal with errors can be decoded correctly - key to solve your issue is improve RF signal reception which should be made with proper hardware (antenna and antenna localization)

Yes, PX cannot improve reception, this is matter of antenna systems.
But No, Antenna systems would not help you with signal path, and PX does help to fix the record at least to look nicer for
video processing systems, keeping A/V synchronization, and at the worst case to omit 1/2s - 12 frame GOP from MPG.

Typical value is 2 clicks per 90min movie, that is too few for just weak signal.

[Poutnik]

Interesting option for automation would be CLI swiss knife for sound processing

SOX ( Sound Exchange from sourceforge) (http://sox.sourceforge.net)

and/or with

Avisynth External_filters#Audio_Filters - Sox Audio Effect Filter by Sh0dan (http://avisynth.org/mediawiki/External_filters#Audio_Filters)
Sox Audio Effect Filters v1.1 beta (http://forum.doom9.org/showthread.php?t=104792)

[Poutnik]

Better antenna?

The local city DVB-T transmitter is about 2-3 km at direct visibility.
The signal must be strong enough. Stronger signal could make things even worse, with possibility of over saturation of receiver input.

Even in opposite case,
I cannot imagine I would persuade the council of housekeepers to buy a new antenne.

[pandy]

The local city DVB-T transmitter is about 2-3 km at direct visibility.
The signal must be strong enough. Stronger signal could make things even worse, with possibility of over saturation of receiver input.

Oh please - typical cheap RF frontend is able to receive signals with level up to 80dBuV (microvolts), most of RF frontends deals quite well with signal even up to 95dBuV, i know frontends that can't be saturated even for 108dBuV (customer equipment not studio) - we talking about very high level signal - most likely active antennas can be saturated thus AVOID active antennas (equipped with amplifier).
If your current configuration is susceptible to interferences search for better one even at a cost of signal level, if not use directional antenna to improve immunity for interferences, you can reduce level of signal with simple fixed 10 - 20dB attenuator if You think that this will help. However solve root cause not search for method to fix corrupted signal.


Even in opposite case,
I cannot imagine I would persuade the council of housekeepers to buy a new antenna.

I can imagine - they buying new home appliances, washing liquid etc - sometimes if current configuration is Not OK you need to fix problem by buying antenna not searching algorithms for fixing corrupted signal (imagine that you persuade housekeepers to fix corrupted signal with DSP algorithms).

Of course there is possibility that broadcaster distribute corrupted signal but even today usually broadcaster care about technical side of the RF emission.

[Poutnik]

Oh please - typical cheap RF frontend is able to receive signals with level up to 80dBuV (microvolts), most of RF frontends deals quite well with signal even up to 95dBuV, i know frontends that can't be saturated even for 108dBuV (customer equipment not studio) - we talking about very high level signal - most likely active antennas can be saturated thus AVOID active antennas (equipped with amplifier).
If your current configuration is susceptible to interferences search for better one even at a cost of signal level, if not use directional antenna to improve immunity for interferences, you can reduce level of signal with simple fixed 10 - 20dB attenuator if You think that this will help.


OK, I am not expert in TV antenna receiver, i was just guessing it is possible to saturate sensitive input amplifiers, getting them to non-linear stage, or cross interference dues non linearity.


However solve root cause not search for method to fix corrupted signal.
Fixing signal can be much easier in this scenario, than finding root cause. Especially if there can be no error during the whole hour.

And, NO real signal is error free, and any processing of real signals must count with errors and their fixing.

Note that the cause need not to be in antenne system at all.


I can imagine - they buying new home appliances, washing liquid etc
Not here.... It is not about housekeepers of my flat, it is about housekeeping of block of 100 flats.

- sometimes if current configuration is Not OK you need to fix problem by buying antenna not searching algorithms for fixing corrupted signal (imagine that you persuade housekeepers to fix corrupted signal with DSP algorithms).

No, I cannot imaging I make them to replace antennas, to replace recivers to replace cables in the whole block of flat,
to realize nothing helped. Especially if nobody complains about TV quality.
It is a common system for the whole block.

I would be treated as an overdemanding fool.

[Poutnik]

Another 2h10m recording, not even one spike.

[pandy]

Once again - Czech use same standards as surrounding countries, perhaps you can replace common antenna with local antenna - DVB-T use lot of error corrections methods, there COFDM modulation, there is convolutional codding + Reed Solomon, but at some point none of them is efficient enough to correct uncorrectable errors - perhaps antenna and amplifier receive strong EM signal from some electrical motor with poorly done decoupling - such flash of energy can blind (saturate) amplifier and mask signal.
And one thing additional - perhaps something is wrong with your receiver? If no one complain except you.

[Ghitulescu]

I assume there are video errors as well :)
I'd rather spend some hours in adjusting the antenna, rather than in encoding the audio (and/or video) with editing. DVB-T has anyway such a low bitrate that makes transcoding into a "more compact" format merely a waste of time.

[Poutnik]

........... ,but at some point none of them is efficient enough to correct uncorrectable errors - perhaps antenna and amplifier receive strong EM signal from some electrical motor with poorly done decoupling - such flash of energy can blind (saturate) amplifier and mask signal.

That I think either. Or from microwave oven, magnetrons or klystrons or whatever is inside can be very powerful in side effects.
And one thing additional - perhaps something is wrong with your receiver? If no one complain except you.

Rare error people would not almost notice, will forgot,
or if not forgot, not remember to mention.

It would have be much worse for case otherwise.

BTW vendor TV application shows 7/7 dots of signal power.

[Poutnik]

I assume there are video errors as well :)

Yes, but the sound ones are more disturbing, if occurs.

I'd rather spend some hours in adjusting the antenna, rather than in encoding the audio (and/or video) with editing.


The other hundredds of inhabitants could try to kill me if I would tweak antennas, not mentioning I would not be allowed to.

DVB-T has anyway such a low bitrate that makes transcoding into a "more compact" format merely a waste of time.

I would not say so. Encoding mpeg2 to H264 with comparable quality has about 1/3 - 1/4 size, with reasonable resize even more than 8times smaller. I would be out of space soon, not mentioning ads cutting.

[pandy]

That I think either. Or from microwave oven, magnetrons or klystrons or whatever is inside can be very powerful in side effects.


Probably not - microwave oven is made as a Faraday cage - also magnetron is relatively narrow band RF source and front-end attenuate everything above 1GHz so for 2.4GHz overall attenuation should be quite high.

Switching large electric power without proper decoupling create spikes which have high energy and very high bandwidth. This can be compared at some point to EMP.


Rare error people would not almost notice, will forgot,
or if not forgot, not remember to mention.

It would have be much worse for case otherwise.

BTW vendor TV application shows 7/7 dots of signal power.

Signal power is less important than BER (or "quality") - there can be high power signal with BER over 1.0E-4 and this mean visible and audible glitches.
My advice - instead of building common antenna installation, use home antenna, passive, no amplifier.

And also seems that You should be able to raise this problem to government authority responsible for quality of the RF spectrum - it is forbidden to corrupt RF signal - they should help you - this is mandatory for them - usually there special agency in telecommunication ministry - they searching for illegal transmitters but also they search for sources of such spikes.

[Poutnik]

Probably not - microwave oven is made as a Faraday cage - also magnetron is relatively narrow band RF source and front-end attenuate everything above 1GHz so for 2.4GHz overall attenuation should be quite high.

Switching large electric power without proper decoupling create spikes which have high energy and very high bandwidth. This can be compared at some point to EMP.

I have meant rather along the power lines noise.
E.g. using digital video output from videocard, to monitor picture is fine. But during temporal period I used integrated intel graphics with analog output. At that time monitor picture was very eye-disturbing, if my microwave oven was working.

My advice - instead of building common antenna installation, use home antenna, passive, no amplifier.

I am afraid my low floor location ad position away from signal,
without direct visibility and with reflection from other buildings will not be satisfactory.

I used initially a simple dipole, coming with my receiver,
but strength and quality was very poor. Than I bought room active antenna,
strength and quality better, but still not excellent.
The common source was much better.

All that struggles are not worthy for me to solve.
Either I can take the signal as it is,
either I fix it and I already know how.

Thanks all for the support.

[manolito]

Just at the end of this discussion I'd like to add my two cents...:p

DVB-T receptionwise I think I am in a similar situation as Poutnik. I live on the first floor of an older 6-story building in the middle of Berlin, Germany. Basically good signal coverage, but my low location plus being surrounded by other high buildings make the reception a little problematic.

I uploaded a small sample of a typical audio glitch which neither ProjectX nor PVAStrumento can correct:
http://www.sendspace.com/file/xqzzkk

So far I have not found an automatic way to correct such glitches, the best I could get was using Sonic Foundry plugins (Vinyl Restoration and Click & Crackle Removal). But it is much easier to remove those spikes manually in a WAVE editor.


Still I'd like to share some tips how I improved my DVB-T captures considerably. The cost to put an antenna on my roof was prohibitive, so I did a lot of testing different antennas and some different tuners.

First of all I strongly disagree with Pandy when he discards active antennas. In all my tests active antennas delivered a higher quality signal compared to passive ones.

The real breakthrough for me was to use a diversity tuner. That's two separate tuners in one case which can be used in a way that one of the tuners corrects the signal of the other one. One tuner uses a directional antenna (Technisat Digitenne), the other one a simple omnidirectional antenna. Both antennas are spaced apart by about 2 meters.

Another problem for me was my slow computer. I don't know if this applies to you, but for a slower machine you should observe some things:

Capture in renderless mode. DVBViewer and DVBDream are my favorite capture programs. Also set priority to High and use a large recording buffer (Older versions of DVBViewer support this setting).

During capture disable your Internet connection and terminate all background software which could compromise I/O (like resident virus scanners).



Cheers
manolito

[Ghitulescu]

Yes, diversity antennas are excellent for location with a lot of interferences (like your flat). However, one could try and fail :) to find a better position for the antenna (my antenna had such a strange position until I renounced at all at DVB-T).

[pandy]

First of all I strongly disagree with Pandy when he discards active antennas. In all my tests active antennas delivered a higher quality signal compared to passive ones.


don't get mi wrong manolito but active antenna = antenna + amplifier - issue with amplifiers in active antennas is that they are very cheap and without knowing parameters - if You really need amplifier go for some modern IC RF amplifiers or buy decent transistor amplifier (for me one of the most important parameters for RF amplifiers - "1dB compression point" also very important IP2 and IP3 - but good amplifiers are more expansive than cheap ones embedded in antenna). DVB-T is sensitive for saturated amplifier more than a analog receiver. Also power of signal (analog level) is less important than "signal quality" - medium level signal but with "quality" equal 100 is better than strong signal with "quality" 30%.

Not sure how you performed you tests but i know how antenna works and there is no magic - high gain passive antenna is ALWAYS better than low gain antenna with amplifier (there is no sens in amplifying noise and add noise from amplifier itself).


The real breakthrough for me was to use a diversity tuner. That's two separate tuners in one case which can be used in a way that one of the tuners corrects the signal of the other one. One tuner uses a directional antenna (Technisat Digitenne), the other one a simple omnidirectional antenna. Both antennas are spaced apart by about 2 meters.


DVB-T is designed to deal and USE multipath reception - this means that there is no need eliminate reflections and at some special cases reflection can be better than direct reception.

COFDM with correct Guard Interval values can gain from many paths of signal. http://en.wikipedia.org/wiki/Orthogonal_frequency-division_multiplexing#Guard_interval_for_elimination_of_intersymbol_interference