Time-Frequency Analysis and Its Applications to Multimedia Signals

Time-frequency analysis has been intensively investigated and developed in the last two decades. A variety of time-frequency distributions have been developed to provide efficient analysis of signals with a time-varying spectral content. In most cases, signal analyses in the joint time-frequency domain outperform the traditional frequency-domain approaches. Generally, the time-frequency distributions have found fruitful applications in many important fields dealing with nonstationary signals, such as biomedical, radar, seismic, telecommunications, and mechanical engineering. Additionally, a large number of applications are related to multimedia signals in speech, audio/music, image, and video signal processing, where time-frequency analysis can be employed to broaden and enhance the signal processing capabilities. Because each type of multimedia signals has its specific nature that may significantly differ from others, the applicability and method of time-frequency analysis depend on the multimedia data to be processed. This fact opens a number of challenging directions for research in the field of time-frequency analysis and its applications to multimedia signals. For instance, having the different dimensionalities of multimedia signals in mind, time-frequency analysis for one-, two-, and three-dimensional signals should be used.

Since there is no single time-frequency distribution that can be used for efficient representations of all kinds of nonstationary signals, novel theoretical formulations that may lead to more practical solutions are still challenging. Moreover, improved forms of time-frequency distributions allow us to further expand and diversify their applicability.

This special issue aims to help readers to understand how time-frequency distributions could be used for the analysis of multimedia signals, with emphasis on their specific nature, complexity, and multidimensionality. Toward this end, various specific distributions have been examined and highlighted in terms of their appropriateness for different multimedia applications.

A set of two review papers and eight research articles are collected in this special issue.

At the beginning, the review paper "Time-frequency analysis and its application in digital watermarking," by S. Stanković, provides a detailed theoretical overview of various time-frequency distributions, discussing their main advantages and drawbacks when applied to multimedia signals. The goal is to facilitate the choice of an appropriate distribution in a specific multimedia application. The second part of this paper is dedicated to time-frequency-based digital watermarking and its application to digital audio, image, and video signals. Here, the main focus is on the unified concept of using time-frequency approaches to shape the watermarks according to the host signal components. The watermark is embedded as well as detected in the time-frequency domain.

The second review paper, "Audio signal processing using time-frequency approaches: coding, classification, fingerprinting, and watermarking" by K. Umapathy et al., discusses different applications of time-frequency analysis in audio signal processing. Currently, a great number of audio processing applications require sophisticated algorithms for compression, classification, and digital audio protection. Hence, this review paper presents several time-frequency-based techniques that has been used to provide efficient extraction of information from audio signals for the purpose of audio coding, music classification, classification of environmental sounds, audio fingerprinting, and watermarking. It is important to emphasize that the universal time-frequency approach provides modeling of audio signals in the joint time-frequency domain, which further allows one to process model parameters based on the application requirements.

The eight research articles can be roughly categorized into two general areas: (1) robust and fragile digital data protection algorithms and (2) time-frequency-based algorithms for data analysis, classification, and compression.

In the first four papers, different time-frequency-based watermarking procedures for multimedia data protection have been proposed. Generally, it has been shown that time-frequency analyses and representations can provide very flexible algorithms that are capable of retaining the high fidelity of multimedia signals while achieving secure data protection.

The paper "A robust image watermarking in the joint time-frequency domain," by M. Öztürk et al., proposes a robust, secure, and high-capacity image watermarking procedure based on spatiofrequency representation. The suitable representation is obtained using the discrete evolutionary transform (DET) calculated by the Gabor expansion. By combining the advantages of the spatial and spectral domains, the proposed procedure provides robustness to commonly used attacks.

In order to characterize the time-varying spectral content of speech signals, the S-method-based time-frequency analysis has been considered in the paper entitled "Time-frequency-based speech regions characterization and eigenvalue decomposition applied to speech watermarking" by I. Orović and S. Stanković. The eigenvalues decomposition is applied to the representation obtained by the S-method to separate speech components. Different components can be combined to create an arbitrary time-frequency mask that is used to shape the time-frequency characteristics of watermark. Both watermark embedding and detection are performed in the time-frequency domain. This procedure provides great flexibility in implementation and is characterised by reliable detection results.

An approach for the optimization of digital audio watermarking based on the genetic algorithm is presented in the paper "A genetic algorithm optimization technique for multiwavelet-based digital audio watermarking" by P. Kumsawat. The watermark is embedded in the discrete multiwavelet transform domain using the quantization index modulation technique. The genetic algorithm provides four optimal watermarking parameters, thus improving both the audio signal quality and watermark robustness.

The fragile watermarking methods for image authentication have been proposed in the paper "Time-frequency and time-scale-based fragile watermarking methods for image authentication" by B. Barkat and F. Sattar. The first method is based on the time-frequency analysis and uses the nonstationary watermark with known time-frequency characteristics. The time-frequency signature of the extracted watermark is used to identify whether the image content has been modified. The second fragile watermarking approach is based on the hierarchical image decomposition using wavelet analysis. The special features of watermark, created as a complex chirp signal, are used for content authentication.

The remaining four papers are related to different time-frequency-based algorithms for efficient multimedia data representation, classification, and compression.

In order to obtain high-resolution reassigned time-frequency representations, the use of fuzzy clustering for Bayesian regularized neural network model has been explored in "Validity-guided fuzzy clustering evaluation for neural network-based time-frequency reassignment" by I. Shafi et al. The resulting time-frequency distributions provide high resolution and do not contain interference terms between different signal components. This approach can provide good discrimination between known patterns for nonstationary signal classification, even when signals are corrupted with additive Gaussian noise with a small variance.

The paper "Parametric time-frequency analysis and its applications in music classification," by Y. Shen et al., deals with analysis and classification of music signals. Music signals are decomposed into atoms using an adaptive time-frequency-based matching pursuit method with Gabor dictionary. The discriminant classification features are obtained by analyzing the atoms parameters. It has been shown that the proposed method provides good classification accuracy.

Time-frequency analysis and the Hermite projection method have been combined in the paper "Video frames reconstruction based on time-frequency analysis and Hermite projection method" by S. Stanković et al. to provide a method for temporal analysis and reconstruction of digital video sequences. The S-method is used to examine the stationarity/nonstationarity of the video coefficients. The reconstruction of stationary coefficients is done using the first coefficient in the temporal sequence, while the nonstationary coefficients are reconstructed using the Hermite projection method. The proposed method can be combined with the existing video compression algorithms to further reduce the volume of data for high-quality video reconstruction.

The paper "Fuzzy morphological polynomial image representation" by C.-P. Huang et al. combines the advantages of optimum fuzzy fitting and morphological operators to extract geometric information from signals. The geometrical decomposition is achieved by windowing and sequentially applying fuzzy morphological opening with structuring functions. The resulting representation can be used in data compression and fractal dimension estimation of temporal signals and images.