Two Channel Audio Zooming System for Smartphone

A general wideband array data model can be written in STFT domain as

$$\mathbf{Y}(f,\nu)=\mathbf{D}(\Psi,\nu)\mathbf{S}(f,\nu)+\mathbf{N}(f,\nu)$$

(1)

where $\mathbf{Y}(f,\nu)$ is the received array signal, and $\mathbf{N}(f,\nu)$ is zero mean, uncorrelated sensor noise. For $M$ microphones and $L$ sources, $\mathbf{D}(\Psi,f)$ is $M\times L$ array manifold given by

$$\mathbf{D}(\Psi,f)=[\mathbf{d}(\Psi_{1},f),\mathbf{d}(\Psi_{2},f),\dots,\mathbf{d}(\Psi_{L},f)]$$

(2)

where $\mathbf{d}(\Psi_{l},f)$ represents the steering vector for $l^{th}$ source given by

$$\mathbf{d}({\Psi_{l},f})=\begin{bmatrix}e^{-j2\pi f\tau_{1l}}&e^{-j2\pi f\tau_{2l}}&\dots&e^{-j2\pi f\tau_{Ml}}\end{bmatrix}^{T}$$

(3)

$\Psi_{l}=(\theta_{l},\phi_{l})$ is the incident direction of the $l^{th}$ source. Here $\tau_{ml}$ represents the time delay of arrival of the $l^{th}$ signal at the $m^{th}$ microphone with respect to a reference microphone. MPDR beamforming problem is equivalent to minimizing the output power with a distortionless response in the target direction given as

$$\underset{\mathbf{W}}{min}\hskip 2.84544pt\mathbf{W}^{H}\mathbf{R_{Y}}\mathbf{W}\hskip 8.5359ptsubject\hskip 2.84544ptto\hskip 8.5359pt\mathbf{W}^{H}\mathbf{d}({\Psi_{l},f})=1$$

(4)

The solution to the above optimization problem under diagonal loading is given by

$$\mathbf{\hat{W}}_{(\Psi_{l},f)}=\frac{(\mathbf{\hat{R}}_{{Y},f}+\alpha\mathbf{I})^{-1}\mathbf{d}(\Psi_{d},f)}{\mathbf{d}^{H}(\Psi_{d},f)(\mathbf{\hat{R}}_{{Y},f}+\alpha\mathbf{I})^{-1}\mathbf{d}(\Psi_{d},f)}$$

(5)

where $\alpha$ is the diagonal loading factor. The co-variance matrix is estimated as

$$R_{Y,f}=\frac{1}{K}\sum_{k=0}^{k=K}Y(f,k)*Y^{*}(f,k)$$

(6)

where K is total number of time frames.

Audio zooming application is additionally, explored using two channel time domain beamformer. The target is assumed to be incident from broadside, resulting in identical delays at the two microphones. The $n^{th}$ snapshot of the received signal at the $m^{th}$ microphone is written as

$$y_{m}(n)=s(n)+n_{m}(n),\textrm{ }n=\{0,1,\cdots,N_{s}-1\},m=\{1,2\}$$

(7)

where $s(n)$ is the target signal to be zoomed in, and $n_{m}(n)$ is the total noise and interferences.

As the target signal undergoes identical delays at the two microphones, no phase adjustment is required herein when compared to the original GJBF [5]. The constrained weight $w_{c}=\frac{1}{2}[1,1]^{T}$ for the target will result in simple addition of the two channel signal providing signal plus interference $y_{f}$ in the upper branch of GJBF in Figure 2. The blocking matrix $B=[1,-1]^{T}$ subtracts the two channel data thus it does not allow the signal from the constraint direction in the lower branch of GJBF. The adaptive weight $w_{a}$ in the lower branch is chosen to estimate the signal at the output of $w_{c}$ as a linear combination of the data at the output of the blocking matrix $B$. As blocking matrix does not allow the signal from the constraint direction, the signal estimated by $w_{a}$ is the interference close to interference present at the output of $w_{c}$.

The overall output of the two microphone GJBF is

$$z(n)=y_{f}(n)-y_{b}(n),$$

(8)

where $y_{f}$ has the target signal, noise and interference with response determined by $w_{c}^{H}$, and $y_{b}$ has only the noise and interference. The filter weights $w_{a}$ is found based on minimizing the power contained in $z(n)$. The power will be minimum when $y_{b}(n)$ closely models the interference and noise present in the $y_{f}(n)$. In particular, Frequency Domain Adaptive Filter (FDAF) approach [7] using overlap-save method has been utilized for computing $w_{a}$. The FDAF approach has logarithmic complexity when compared to polynomial complexity for the classical LMS as utilized in the original GJBF.

In this Section, adaptive block thresholding based post-filtering is proposed for Audio Zooming effect creation. The estimate obtained by adaptive beamforming recovers the target with some amount of noise and interference. Additionally, the beamformed output might show transient or tonal behavior, or combination of the two. The residual interference along with transient and tonal behavior is now simply treated as single additive interference term for further processing. Mathematically, the beamformed output can now be written in STFT domain as

$$Z(f,\nu)=S(f,\nu)+I(f,\nu)$$

(9)

where $S(f,\nu)$ is STFT coefficient of desired signal and $I(f,\nu)$ is STFT coefficient of residual noise and interference. The total variance of such additive interference can be computed as

$$\hat{\sigma}^{2}(f,\nu)=\frac{1}{2}\{(Y_{1}(f,\nu)-Z(f,\nu))^{2}+(Y_{2}(f,\nu)-Z(f,\nu))^{2}\}$$

(10)

where $Y_{1}(f,\nu)$, $Y_{2}(f,\nu)$ are the STFT coefficients of microphone 1 and 2 respectively. This is the best residual interference variance that can be estimated considering the beamformed output is close to the target signal. This estimate works well with practical scenarios. More accurate estimation can provide better result. It is to be noted that because of limited degrees of freedom (two, utilized both) in the spatial domain, post-filtering is now attempted in frequency domain. For this purpose, the output of the either beamformer is considered in short-time Fourier transform (STFT) domain, with suitably chosen block sizes in the time and frequency, as detailed below.

The time-frequency coefficients are modified by multiplying each of them by an attenuation factor to attenuate the interference dominated components as

$$\hat{S}(f,\nu)=a(f,\nu)*Z(f,\nu)$$

(11)

and creating the zooming effect.

The attenuation factor $a(f,\nu)$ depends upon the values $Z(f^{\prime},\nu^{\prime})$ for all $[f^{\prime},\nu^{\prime}]$ in the neighborhood of $[f,\nu]$. The signal estimate $\hat{S(f,\nu)}$ is computed from the noisy data $Z(f,\nu)$ with a constant attenuation factor $a_{i}$ over the sub-block $B_{i}$ as

$$\hat{S}(f,\nu)=a_{i}\hskip 2.84544ptZ(f,\nu)\textrm{ }\forall(f,\nu)\in B_{i}$$

(12)

Selection of the block is done as follows. The entire STFT matrix $\mathbf{Z}$ is divided into macro-blocks of size ${P\times Q}$. Each macro-block is further divided into sub-blocks of size $2^{H-v}\times 2^{v}$ where $2^{H}=L$ and $2^{V}=W$ and $v\in\{0,1.....H\}$. Various ways of dividing a macro-block into sub-blocks is shown in Figure 4. A Threshold block SNR is chosen to differentiate higher and lower block SNR values. Out of the various structure of sub-blocks, the one having highest SNR is chosen.

The mean square error can be written as

$$r=E(|\hat{S}-S|^{2})\leq\frac{1}{A}\sum^{I}_{i=1}\sum_{f,\nu\in B_{i}}E\{(a_{i}\hskip 2.84544ptZ(f,\nu)-S(f,\nu))^{2}\}$$

(13)

The error can be minimized by choosing [8]

$$a_{i}=(1-\frac{1}{\zeta_{i}+1})_{+}$$

(14)

where $\zeta_{i}=\frac{\bar{S}^{2}}{\bar{\sigma}^{2}}$ is the average apriori SNR in the sub-block $B_{i}$, that is computed from

$$\bar{S}^{2}=\frac{1}{B^{0}_{i}}\sum_{f,\nu\in B_{i}}S^{2}(f,\nu)\textrm{and },\hskip 14.22636pt\bar{\sigma}^{2}=\frac{1}{B_{i}^{0}}\sum_{f,\nu\in B_{i}}\sigma^{2}(f,\nu)$$

(15)

Here $B_{i}^{0}$ is number of coefficients in the sub-block. As $S(f,\nu)$ is unknown, the apriori SNR ${\zeta}_{i}$ can be computed alternatively using

$$\hat{\zeta}_{i}=\frac{\bar{Z_{i}}^{2}}{\bar{\sigma_{i}}^{2}}-1$$

(16)

that can be derived from (9). $\bar{Z_{i}}^{2}$ can be computed as in (15) using the beamformed signal.

It is to be noted that the selection of $a_{i}$ ensures that the target signal is enhanced corresponding to the high SNR sub-blocks while suppressing the low-SNRs sub-blocks. This results in the required audio-zooming effect.

Two microphones separated by $10$ cm were taken for the simulation experiments. The objective parameters were computed for fifty target speech files from TIMIT database [13]. The target source was taken at azimuth $90^{\circ}$, while the interference was assumed to be at $60^{\circ}$. Signal-to-Interference Ratio (SIR) was taken to be $0$dB. The experiments were performed in reverberant condition. For reverberant condition, the reverberation time was taken to be $100$ms. The results are presented in Table 1. It is to be noted that the proposed MPDR beamforming based system outperforms the other methods. The performance of GJBF based system is comparable to RMVDR based system.

The data recording was done was in anechoic chamber and open field. Subjective listening tests were conducted. Mean Opinion Score (MOS) measure is presented to evaluate the proposed audio zooming system. Fifteen subjects in the age group of $20-25$ years were invited for listening the zoomed audio. The subjects listened the mixed received signal and the output. The rating was given for the quality of the output audio on the scale of $0$ to $5$ [14].