Time-Frequency Analysis and Its Application in Digital Watermarking

Abstract

A review of time-frequency analysis and some aspects of its applications in digital watermarking are presented. The main advantages and drawbacks of various time-frequency distributions are first discussed. The aim of this theoretical overview is to facilitate an appropriate distribution selection in a specific application. Different aspects of the time-frequency analysis when applied to digital watermarking are then presented. In particular, the method that maps time-frequency characteristics of a host signal to the pseudo noise watermark sequence is thoroughly discussed. This approach is presented in the multidimensional form and then applied to digital audio, digital image, and digital video watermarking. Finally, the theoretical considerations are illustrated by various numerical and real-life examples.

1. Introduction

Theoretical aspects of time-frequency analysis have been intensively studied over the last two decades [1–27]. In parallel, their various applications have been exploited as well. Namely, for an efficient analysis of nonstationary signals, such are radar, sonar, biomedical, seismic, and multimedia signals, time-frequency representations are required. Time-frequency distributions are most commonly used for this purpose. Many of the researchers have made significant efforts in defining a distribution that is optimal for a wide class of frequency-modulated signals [8–11]. As a result, a number of time-frequency distributions have been proposed. However, the efficiency of each of them is more or less limited to a specific class of signals and, consequently, to a specific application. One of the goals of this paper is to highlight the most important features of some popular time-frequency distributions and to give an idea of how to choose the most appropriate distribution depending on the signal form. The linear, quadratic, higher-order, and multiwindow time-frequency distributions are considered. The short-time Fourier transform, as the most commonly used linear transform, is firstly discussed. Next, the Wigner distribution, as the best known quadratic distribution, is presented. Also, the Cohen class and some specific distributions belonging to this class are considered [1, 5–7]. It is shown that the quadratic distributions are optimal for a linear frequency-modulated signal. However, if the instantaneous frequency variation within the analysis window is faster, multiwindow, or higher order distributions should be used, [14–16, 19–27]. The Hermite functions-based multiwindow approach is also discussed. Finally, highly concentrated distributions with complex-lag argument are presented. To facilitate in better understanding of the presented theoretical considerations, numerous illustrative examples have been provided.

The second part of the paper considers time-frequency-based watermarking techniques. The watermarking of digital audio, digital image, and digital video is discussed [28–37]. A short overview of some existing approaches is first given and they are related to digital audio and digital image [28–50]. Watermarking using time-frequency techniques is usually employed in either of the following two ways. The first one uses the time/space domain of the host signal to embed the watermark with specific time-frequency characteristics. The time-frequency analysis is then used for detection. The second way uses time-frequency distributions to create or embed watermark in the time-frequency domain. A flexible procedure that can be used for different kinds of signals is discussed more extensively. Therein, the watermark is shaped according to the time-frequency characteristics of the host signal. The detection is performed in the time-frequency domain. This particular approach is presented in the multidimensional form, and it is applied to digital audio, digital image, and digital video. It provides a high degree of robustness and imperceptibility. Hence, even when the watermark is very weak, a reliably detection can still be achieved. Also, the watermark gets completely hidden by the time-frequency characteristics of the host signal.

2. Time-Frequency Analysis

The Fourier transform provides spectral content of a signal. It has been a valuable tool in various applications. However, for nonstationary signals the Fourier transform cannot give satisfactory results since the information about frequency components variations in time is required.

Furthermore, it can happen that two different signals have the same spectral contents, as illustrated in Figures 1(a) and 1(b). Based on Figure 1(c), however, we can conclude that the time-frequency representations of the two signals are quite different. This example is a simple illustration of the importance of time-frequency analysis for signals whose spectral contents vary with time. Various time-frequency distributions are used for this purpose. The ideal time-frequency representation can be described as [19–21]

(1)

where a signal of the form is considered. This representation provides the signal local energy distribution, as well.

Figure 1

Nonstationary signals. (a) Time domain representations. (b) Fourier domain representations. (c) Spectral components' variations along the time axes.

A question that naturally arises at this point is whether there exist a single representation that would be ideal for any signal at hand. The answer is no, hence a number of time-frequency distributions have been introduced.

2.1. The Short-Time Fourier Transform

The simplest and most commonly used time-frequency representation is obtained by using the short-time Fourier transform (STFT) defined as [2]

(2)

Thus, it is a windowed version of the Fourier transform. The sliding window function is denoted by , where τ is the lag coordinate. An illustration of the STFT calculation is shown in Figure 2.

Figure 2

Illustration of the STFT calculation.

Note that the spectral content is calculated for each windowed part of the signal. The central point of the sliding window is the time instant for which the spectrum is calculated. The influence of the window size is critical, as it will be discussed later.

The energetic version of the STFT is known as the spectrogram. It can be written as

(3)

where is the Fourier transform of the time domain window , is the first derivative of the phase function, the frequency convolution is denoted by , while is the spread factor which depends on the second and higher order phase derivatives. Note that an ideal representation will be obtained if the signal is constant frequency modulated (   for  ) and if , that is, for a large time domain window. However, if the signal is not constant frequency modulated, a large window size can produce a low time resolution and vice versa. In general, there is a trade-off between the time and frequency resolution, and it is best described with the uncertainty principle, as

(4)

where,

(5)

are the measures of duration in time and frequency, respectively. The signal should satisfy .

An illustration of the window size influence on the spectrogram resolution is given in Figure 3. A four-component signal is considered. In order to achieve a good time resolution, two short sinusoidal components should be analyzed by using a narrow window. However, to obtain a good frequency resolution the third component (a sinusoid with a long duration) should be analyzed by using a wide window. In Figure 3(a), a narrow window is used, and it results in a good time resolution, while the frequency resolution is low. However, when a large window size is used, a low time resolution is obtained. Hence, the two short sinusoids have almost merged into one (see Figure 3(b)). Due to the presence of a linear frequency-modulated component (chirp signal), the spread factors are present in both cases.

Figure 3

STFT of the four-component signal calculated by: (a) narrow window, (b) large window.

The spectrogram satisfies the marginal properties

(6)

The total energy is obtained as

(7)

An important property of the STFT is its linearity. Namely, the STFT of a multicomponent signal is . Consequently, if the signal components do not intersect in the time-frequency plane the spectrogram will be equal to the sum of spectrograms of each of the signal components. This is evident from Figure 3.

2.2. Quadratic Time-Frequency Distributions

Quadratic time-frequency distributions have been introduced in order to improve the time-frequency resolution. Namely, they remove the spread factors for linear frequency-modulated signals. Among them, the Wigner distribution is the most commonly used. It has also been used as a base to define several interesting time-frequency distributions.

The windowed version of the Wigner distribution is called the pseudo-Wigner distribution. It is defined as [1, 2]

(8)

The spread factor in this case is .

The Wigner distribution satisfies the marginal conditions. Note that it is always real, while the numerical realization requires oversampling with factor 2. However, the Wigner distribution is not linear. Namely, for a multicomponent signal , the Wigner distribution is of the form

(9)

Thus, beside the autoterms, the interactions between different signal components (), called cross-terms, appear. This is a major drawback of this distribution. The Wigner distribution of the four-component signal, from the previous example, is given in Figure 4.

Figure 4

Wigner distribution of a four-component signal.

In this case, the auto components are well concentrated. However, the cross-terms presence is significant. Namely, the time-frequency representation contains frequency components that do not exist within the signal itself. It could lead to a wrong analysis result.

The cross-terms could be reduced by using a filter function in the ambiguity domain. The ambiguity function is the two dimensional Fourier transform of the Wigner distribution, that is,

(10)

In the ambiguity domain, the auto-terms are located around the origin. Thus, a two-dimensional filter, called the kernel function, is used to filter out the cross-terms (that are generally located away from the origin) [1, 5–7]:

(11)

where is the ambiguity function, and is the kernel function. The time-frequency distribution based on the filtered function is obtained as

(12)

This is the definition of the Cohen class of distributions. By choosing the corresponding kernel functions, some specific distributions belonging to the Cohen class can be obtained. They satisfy the marginal properties if holds. For example, the Choi-Williams distribution [5] is obtained for the kernel function where is a scaling factor to control its attenuation rate. The kernels for some distributions are defined in Table 1, [7].

Table 1 Some distributions from the Cohen class.

The four-component signal analyzed by the Choi-Williams and Born-Jordan distribution is shown in Figure 5. The Choi-Williams distribution with two values of its kernel parameter is used. The parameter results in a wider width of the kernel assuring good auto-terms concentration, but a significant amount of cross-terms is still present (Figure 5(a)). The Born-Jordan distribution producing almost the same auto-terms concentration is shown in Figure 5(b). Note that the side-lobes and cross-terms are more emphasized than in the Choi-Williams distribution with [7]. However, if the Choi-Williams distribution with the attenuation parameter is used (Figure 5(c)), the cross-terms get suppressed, while the auto-term concentration becomes reduced (for the chirp component).

Figure 5

Time-frequency representations by: (a) Choi-Williams distribution ( ), (b) Born-Jordan distribution, and (c) Choi-Williams distribution ( ).

Note that there is a trade-off between the cross-terms reduction and auto-term concentration (it depends on parameters of the kernel function). Obviously, distributions belonging to the Cohen class lie in between the two extreme cases: the spectrogram that eliminates the cross-terms with a low auto-term concentration and the Wigner distribution that provides high resolution, but with emphatic cross-terms. It is possible to obtain an ideal time-frequency concentration only if signal dependent kernels are used [8, 10].

Next, we can ask the following question. Is there a distribution that provides the auto-terms concentration as good as in the Wigner distribution, while eliminating the cross-terms like the spectrogram does? In order to define a distribution with these properties, let us start with the following relationship between the short-time Fourier transform and the Wigner distribution:

(13)

Clearly, the convolution along the frequency axis improves the auto-terms concentration, but it introduces the cross-terms. Thus, the convolution should be performed only over the same auto-terms, avoiding different signal components being convolved. It can be performed by introducing a frequency domain window (see Figure 6).

Figure 6

Illustration of the STFTs convolution.

A distribution that allows for such convolution is called the S-method [17]. It is defined as

(14)

The frequency domain finite window is denoted by . Observe that for the spectrogram and the Wigner distribution are obtained, respectively.

The discrete version of the S-method is

(15)

The discrete window width is . It determines the number of summation terms in (15). Note that they improve the spectrogram concentration toward that of the Wigner distribution. An illustration of the S-method calculation is shown in Figure 7.

Figure 7

Illustration of the S-method calculation for a given time instant .

The calculation of the S-method is illustrated for the central point of an auto-term as well as for the point located in between the two auto components (Figures 7(a) and 7(b)). It is important to observe that the summation has to be performed only over the auto-terms. If the other terms are included, the concentration will not be improved. In addition, the noise could be also picked up. The window has to be narrower than the minimal distance between the auto-terms. If this is not the case, the interactions between the auto-terms will produce the cross-terms. Namely, as illustrated in Figure 7(b), the cross-terms appear if the window includes the summation terms marked by red color.

The adaptive S-method with a variable window adjusted to the auto-terms is introduced in [18]. However, in many applications the fixed window size of has been shown to provide very good results, since the convergence within the window is fast, and it is mostly achieved after a few summation terms.

The S-method of the four-component signal is given in Figure 8.

Figure 8

The S-method of the multicomponent signal (various window sizes are used).

Note that all signal components (with constant and linear frequency modulations) are well concentrated even for . By increasing , for the cross-terms start to appear (the minimal distance between the auto-terms becomes less than ).

The spectrogram and the S-method of a real speech signal are shown in Figure 9.

Figure 9

Time-frequency representations of a speech signal: (a) spectrogram, (b) S-method.

The speech signal time-frequency resolution is improved by using the S-method.

Observe that the spread factor in the quadratic distributions will be present if the instantaneous frequency contains third and higher order phase derivatives. Hence, further concentration improvement can be obtained by using the multiwindow approach or by using higher order time-frequency distributions, as discussed below.

2.3. Multiwindow Time-Frequency Distributions

The concept of multiwindow time-frequency distributions has been developed by using optimally concentrated orthogonal windows [25–27]. The Hermite functions that are localized in both time and frequency domain can be used as orthogonal windows. The multiwindow spectrogram is defined as a weighted sum of the spectrograms:

(16)

The total number of the spectrograms and Hermite functions is , while are the weighting coefficients. The order Hermite function is defined as

(17)

This function can be obtained by using a recursive realization as follow

(18)

while:

(19)

The weighting coefficients are calculated by

(20)

where is the signal amplitude. The weighting coefficients for a signal with a constant amplitude are given in Table 2.

Table 2 The weighting coefficients for .

It is important to emphasize that the spread factor is reduced proportionally to the highest order of Hermite functions used in the multiwindow spectrogram:

(21)

Thus, the first term in the spread factor comes from the phase derivative. An additional concentration improvement can be achieved by introducing the multiwindow S-method [51]. It can be written in the discrete domain as

(22)

In this case the spread factor is

(23)

An example, where the multiwindow spectrogram and the multiwindow S-method are used is shown in Figure 10 (the components are with constant amplitude and each of them is treated separately).

Figure 10

Time-frequency representations by using: (a) spectrogram, (b) S-method, (c) multiwindow spectrogram and (d) multiwindow S-method.

From Figure 10 it is obvious that the multiwindow versions of distributions outperform their standard counterparts. The multiwindow approach reduces the noise influence as well [27].

This multiwindow approach can also be interpreted by using the Cohen class of distributions, where it can be written as a two-dimensional convolution of the Wigner distribution and the kernel function: . The kernel function producing the multiwindow Wigner distribution is obtained as

(24)

where is the order Laguerre function, and it is the Wigner distribution of the order Hermite function.

According to the previous consideration, the spread factor can be gradually reduced by increasing the number of Hermite functions, that is, by increasing the number of spectrograms in (16) and (22). Similar resolution improvements can be obtained by using polynomial time-frequency distributions, where each additional order of the distribution results in a removal of one more term within the spread factor [3].

2.4. Distributions with Complex-Lag Argument

A significant components concentration improvement can be achieved by introducing higher order distributions with the complex-lag argument [19, 21]. A general form of these distributions is [22]

(25)

A special case follows for [19]

(26)

The fourth-order distribution is obtained for , that is, Thus, it has the form

(27)

The term with the complex-lag argument is calculated by

(28)

For a multicomponent signal, it is calculated for each component separately, where the STFT is used to separate them [20]. This procedure can be generalized for an arbitrary distribution order [24].

For and we have . It defines the sixth-order distribution. Observe that, regarding the spread factor reduction, each distribution order is related to the previous one in the same way the Wigner distribution is related to the spectrogram. Namely, the spread factors for the fourth- and the sixth-order distributions are

(29)

The complex-lag distributions are in particular useful when the instantaneous frequency variation within the window is very fast. The examples where the distribution order is increased in order to improve time-frequency resolution are shown in Figure 11.

Figure 11

Time-frequency representation of signals with fast instantaneous frequency variation obtained by using: the Wigner distribution, the fourth-order complex-lag distribution, and the sixth-order complex-lag distribution.

Note that the Wigner distribution produces poor results for both signals, since it cannot follow the instantaneous frequency variations.

3. Digital Watermarking

Digital watermarking has been used to protect multimedia data. Demands in this area increase proportionally with the number of internet applications. Namely, these applications are associated with a need for copyright protection of digital audio, digital image, and digital video. Note that the cryptographic methods could be used for this purpose. However, once the data are decoded they can be unlimitedly copied. This has been one of the primary reasons for developing the watermarking techniques. The watermarking, in general, consists of embedding a secret information that can be reliably detected within the host signal. Obviously, this information should be imperceptible within the host data. Depending on the application type, the watermarking can be robust, fragile, or semifragile. The robust watermark should be resistant to various nonmalicious or malicious attacks. Nonmalicious attacks are commonly used signal processing techniques such as compression algorithms, filtering, and so forth, while the malicious attacks are the signal processing techniques that are intentionally used to remove the watermark. The fragile watermark is used to prove data authenticity. Thus, if the content of a signal has been changed, the watermark should no longer exist. The semifragile watermark should be robust to a slight modification, such as for example a certain degree of compression.

Depending on the type of host signal (speech/audio signals, image, video, etc.) various watermarking approaches are developed. Also, different domains have been used: the time domain (or the space domain), the spectral domains such as DFT, DWT, and DCT domain, and a joint time/space-frequency domain. The existing watermarking techniques are mainly based on either the time or frequency domain. However, in both cases, the time-frequency characteristics of the watermark do not correspond to the time-frequency characteristics of the host signal. It may result in the watermark being not imperceptible, because it is present in the time-frequency regions where the signal components do not exist.

3.1. An Overview of Some Time-Frequency-Based Watermarking Techniques

The time-frequency domain can be very efficient regarding the watermark imperceptibility and robustness. This section presents some key time-frequency-based watermarking procedures with the aim to inspire more contributions on this topic.

Here, we will classify the time-frequency-based watermarking techniques into two categories. The first one is the approaches based on watermarks with specific time-frequency characteristics, where the detection procedure is performed within the time-frequency domain. The second one uses the time-frequency domain to embed or to shape the watermark.

• (A) Image Watermark with Specific Time-Frequency Characteristics

(A.1) Among the first time-frequency-based image watermarking procedures is the approach introduced in [38]. Although the watermark is embedded in the space domain it is chosen to have a specific space/spatial-frequency characteristic. Namely, a two-dimensional chirp signal is used as watermark:

(30)

Observe that the Wigner distribution provides an ideal representation for this signal.

The watermark is embedded within the entire image: .

The watermark detection is performed by using

(31)

where

(32)

The variable parameters ,, and are used. Different values of those parameters (, and ) produce a set of projections. The additional term can be used to detect some geometrical transformations, as well. Note that the detector has the form of the Radon-Wigner distribution, which ensures that the energy of the watermark is distributed over the hyper plane defined by ( is the phase function of the watermark). In order to make a decision weather the watermark exists in the image or not, the maxima of the Radon Wigner distribution

(33)

are compared with an assumed reference threshold. Also, multiple chirp watermarks with small and randomly chosen amplitude are used to increase flexibility of the proposed procedure. The parameters of the chirp signal as well as the random sequence that defines the amplitudes of chirp signals serve as the watermark key. Since the watermark is embedded within the entire image in the space domain, a proper masking that provides imperceptibility should be applied. An analysis of the performances giving an estimation of the detectable watermark amplitude level is provided in [38]. The robustness is tested on various attack, some being a median filter, geometrical transformations (translation, rotation and cropping simultaneously applied), a high-pass filter, local notch filter, and Gaussian noise.

(A.2) Mobasseri et al. [44] have proposed a scheme for robust watermarking based on the polynomial phase. The algorithm combines the approach in [45] (where bits are embedded in the image) with the 2D chirp-based methods. Here, the image of size is partitioned into blocks. A 2D chirp of the form

(34)

is used, where are taken. The watermark is embedded in the block located at the pixel () according to

(35)

The constant that controls the watermark strength is , and the integer part is denoted by "", while are watermark bits taken from . The knowledge of the pair is required in order to recover . It can be obtained by using the chirp transform

(36)

where:

(37)

Finally, the total detection over all blocks can be obtained by

(38)

This provides a possibility to use the watermark that cannot be detected by considering a single block only. Thus, in such case it would be necessary to integrate all of them over the entire image. Note that it is also possible to generate different chirps for different blocks instead of using the same chirp for all blocks. It would make the detection even more difficult for unauthorized users.

The embedded bits are recovered by

(39)

The proposed method is adapted to be robust to the JPEG compression algorithm. The watermark is embedded within the blocks by using the quantization matrix . Namely, the DCT coefficients and the 2D chirp are quantized by this matrix. However, choosing an appropriate pair is necessary to ensure that the watermark survives this quantization. The watermark survival degree can be quantified by

(40)

where is the unmarked compressed block. The watermark is completely removed by compression if is obtained. The quality of the proposed technique is tested on the image Lena, and it is proven that, for this case, it outperforms the standard spread spectrum technique.

(A.3) The watermarking in the fractional Fourier domain belongs to the time-frequency-based algorithm as well. This approach is defined in [39], and it uses a combination of the space and spatial-frequency domain. Namely, the image is transformed in the fractional Fourier domain for the angles :

(41)

where FRFT denotes the one-dimensional fractional Fourier transform. The FRFT can be treated as a rotation in the time-frequency plane for an angle , while the inverse transform can be considered as a rotation for the angle . Thus, the FRFT domain is a combination of the time and frequency domain (the Fourier transform is a special case for ). Depending on the angle , the FRFT assures that the time or the frequency domain is dominant. For close to the frequency domain is dominant, while for small the FRFT is dominantly in the time domain.The watermark is embedded in the FRFT coefficients reordered into a non-increasing sequence . By analogy with the watermarking in the DCT domain, the first coefficients are omitted, while the next coefficients are used. The watermark is embedded as

(42)

A real valued watermark key composed of and is used. The detection is performed by

(43)

The performance analysis providing the detection threshold is done, the threshold being chosen as

(44)

where the watermark is a Gaussian white noise with the variance . The watermark key consists of the watermark sequence and the angles (₁, ₂). Thus, the algorithm provides two more degrees of freedom, and it offers more possibility to generate watermarks. The watermarking procedure is tested on various images and attacks.

(A.4) Barkat and Sattar have proposed a fragile watermarking procedure for image authentication [43]. The watermark with a particular time-frequency signature is inserted in the image pixels. Although, in general,   pixels (according to the image size) exist, a significantly lower number of them is used. The pixels location can be chosen arbitrarily. The authors have used diagonal pixels, modulated by a pseudonoise sequence as a secret key. Various frequency-modulated nonstationary signals can be a watermark, as well. However, the features that could be easily identified should be used. Consequently, different time-frequency distributions should be used for watermark detection. Barkat and Sattar have used a quadratic frequency-modulated signal. It is detected by using the Wigner distribution. The proposed scheme is tested on the following attacks: cropping, translation, JPEG compression, and scaling. Very week and imperceptible attacks were applied (e.g., JPEG with 99% quality is used). It is shown that the watermark cannot be identified after these attacks.

(B) Watermark Created in the Time-Frequency Domain

(B.1) An image watermarking approach is proposed by Al-khassaweneh and Aviyente in [49]. The image rows are used to create a set of one-dimensional signals. Then, the Wigner distribution is calculated for each of them. Also, the watermark sequence is transformed to the time-frequency domain by using the Wigner distribution. Finally, the Wigner distribution of the watermark sequence is embedded in the Wigner distribution of each image row as follows:

(45)

where is a set of the time-frequency dependent weighting coefficients. The watermarked image is obtained by using the inverse transform. Having in mind that we deal with a real and positive signal, it is defined as

(46)

However, the previous equation holds only if the two-dimensional function (45) is a valid Wigner distribution. Namely, it is well known that any two-dimensional function cannot be the Wigner distribution. It introduces a very restrictive condition on the function . In the proposed method it is determined by using the time-frequency representation of the corresponding row and taking the middle frequency region.

Al-khassaweneh and Aviyente have suggested a nonblind detection procedure. Namely, the second part of the function in (46) that depends on the watermark is selected. The detection is performed by using the standard correlation detection. A threshold that provides a minimal probability of error is derived. The proposed method is tested on different images and under various attacks. The average probability of error was found to be 0.03.

(B.2) Foo et al. in [48] have defined a method for digital audio watermarking based on the time-frequency domain. Here the audio frames are changed, so that the logical value of 1 is assigned. If the original frame is lengthened or shortened, the logical value 1 is assigned, otherwise the "normal frames" correspond to the logical value 0. The watermark is a sequence obtained as a binary code of the alphabet letters, converted to the ASCII code (the example with the binary code 010001100101001101010111 for the letters FSW is used). The crucial part of this method is the selection of frames that will be lengthened or shortened (the frame size of 1024 samples is used). The frames with signal energy level above the masking threshold are selected (the psychoacoustic model is used to determine the masking threshold in each subband). The frames length is changed by adding or removing samples with amplitudes that do not exceed the masking threshold. Four samples are added or removed within the frame of 1024 samples. It ensures that a perceptual distortion will not appear. In order to preserve the total length of the watermarked audio signal, the same number of the lengthened and shortened frames is used. The pair of frames called Diamond frames is used to represent the binary 1, while the logical values 0 are assigned to the unaltered frames.

The detection procedure is nonblind, that is, the original signal is required. A significant difference between the watermarked and the original signals will appear only if a pair of changed frames exists. Thus, it is used for logical values detection. The proposed watermarking scheme has been tested on various musical signals, as well as on a speech signal, and a set of different attacks has been applied (filtering, resampling, noise, cropping, and MP3 compression). Although the results vary for different signals and attacks, in general they are good. The worst results are obtained for the rock and pop music signals with MP3 compression. However, in all cases the owner can be identified.

(B.3) Esmaili et al. have proposed a spread spectrum based watermarking in the time-frequency domain [46]. This technique is used for watermarking of music signals. The watermark is created as

(47)

where is the watermark before spreading, is the spreading code or the pseudonoise sequence, while is the time-varying carrier frequency. The parameter controls the watermark strength. The masking properties of the human auditory system are used to shape an imperceptible watermark. The pseudonoise sequence is low pass filtered according to the signal characteristics (the Butterwort filter is used). Two different scenarios of masking have been considered. The tone- or noise-like characteristic are determined by using the entropy

(48)

The probability of energy for each frequency (within a window used for the spectrogram calculation) is denoted by , while is the maximum frequency. A half of the maximum entropy is taken as a threshold between noise-like and tone-like characteristics. If the entropy is lower than it is considered as a tone-like, otherwise it is a noise-like characteristic.

The time-varying carrier frequency is obtained as the instantaneous mean frequency of the host signal, calculated by

(49)

Finally, after the watermark is modulated and shaped, it is embedded in the time domain as .

A simple watermark detection procedure is applied. First, demodulation is performed by using the time-varying carrier, and then the watermark is detected by using the standard correlation procedure with the pseudonoise sequence.

The proposed method has been tested on several music files. It has been shown that, under various attacks, the bit error rates are mostly between 0.02 and 0.08.

(B.4) An interesting audio watermarking approach based on linear chirps has been proposed in [47]. The watermark is created as a chirp signal, which is perceptually shaped according to the host signal samples. Different chirp rates, each representing a unique watermark message, produce different slopes in the time-frequency domain. The efficient time-frequency representation is obtained by using the Wigner distribution. The extracted chirps are postprocessed in the time-frequency plane by an optimal line detection method based on the Hough-Radon transform. It can correctly estimate the slope of the watermark signal despite the broken lines caused by attacks. The simulation results show that the Hough-Radon transform applied to a time-frequency distribution can detect the watermark message correctly at bit error rates up to 20%.

3.2. Watermaking Approach Based on the Time-Frequency-Shaped Watermark

The approach that will be presented can be used either for audio signals or images [41, 42]. Thus, the embedding and detection procedures for both kinds of signals will be defined and discussed simultaneously, by using the multidimensional notation.

In order to ensure imperceptibility constraints, the watermark should be modeled according to the time-frequency characteristics of the signal components. The concept of nonstationary multidimensional filtering [52] is adapted and used to create a watermark with time-frequency characteristics that correspond to the characteristics of the host signal. The corresponding algorithm consists of the following steps:

()selection of the nonstationary parts of signal suitable for watermark embedding;

()watermark modeling according to the multidimensional time-frequency characteristics of the host signal;

()watermark embedding and watermark detection procedure within the multidimensional time-frequency domain.

Multidimensional time-frequency distributions are employed in order to determine the nonstationary regions. As it will be shown later, the S-method can be efficiently used to analyze dynamics of the regions of speech signals and images. Although the cross-terms are usually undesirable in the time-frequency analysis, they have found to be useful in watermarking. Namely, they may increase performances of a speech watermark detector, and also, increase the efficiency of dynamic regions selection within an image.

The watermark is obtained at the output of a nonstationary filter as follows:

(50)

where is the short-time Fourier transform of a multidimensional random sequence . The function contains the information about the components within the region . It is used to create the watermark that will be adjusted to these components. Thus, we may start with an arbitrary random multidimensional sequence and, by using , its multidimensional time-frequency characteristic is modeled.

The region will be used for watermarking if a time-frequency distribution contains a sufficient number of components whose energy is above a floor value:

(51)

The function returns a number of components that satisfy the condition within the parenthesis, while is the reference number of points used to make the decision about the region nonstationary. The parameter is an energy floor that can be determined as a portion of the TFD maximum:

(52)

A value of between 0 and 1 is taken.

The components' positions within are identified by using the support function:

(53)

An additional function is defined in order to consider the significant components only:

(54)

The energy threshold is denoted by . Thus, the resulting support function is defined as

(55)

The watermark embedding is done according to

(56)

where and are the short-time Fourier transforms of the multidimensional watermarked data, the host data, and the watermark, respectively.

Note that when compared to the signal domain, in the multidimensional time-frequency domain the number of coefficients that contain the information about the watermark is significantly increased. Consequently, the detector response will be enhanced. The standard correlation detector in the multidimensional time-frequency domain is defined as

(57)

The multidimensional time-frequency domain-based detector provides a low probability of error, even when the number of watermarked samples in the signal domain is small.

3.2.1. Digital Audio Signal

Let us consider the voiced part of a speech signal. The region

(58)

is determined by the start and the end instances and of the voiced speech, as well as by the interval that contains the strongest formants. The S-method is used to define the support function [41, 53]:

(59)

The region appropriate for watermarking is shown in Figure 12(a). The corresponding support function (Figure 12(b)) is created by using the value = with .

Figure 12

(a) Region selected for watermarking. (b) Support function. (c) Time-frequency representation of the watermark. The S-method is used.

The watermark is embedded in the time domain: . The time-frequency representation of the watermark is shown in Figure 12(c).

As expected, the time-frequency characteristics of the watermark follow those components of the speech signal. Consequently, the watermark is inaudible within the speech signal.

Next, a music signal of the flute is considered, Figure 13(a).

Figure 13

(a) S-method and the corresponding support function. (b) Fourth-order complex-lag distribution and corresponding support function.

In this case, the fourth-order complex-lag distribution is more appropriate for the region selection than the S-method, because it better follows the frequency variations in the signal (Figures 13(a) and 13(b)). Thus, by using this distribution an inaudible watermark is created.

Note that an important improvement in the watermark detection is obtained if the cross-terms are included [41]. Namely, the watermark is present within the cross-terms, as well. A standard correlation detector in the time-frequency domain that includes the cross-terms can be written in the form

(60)

The second term in (60) is the result of cross-terms.

Note that this form of detector can be used in other existing detector structures.

The following measure of the detection quality

(61)

is used. The mean value and standard deviation of the detector response are denoted by and . Indices and indicate the right and the wrong keys, respectively.

Efficiency of the proposed procedure is demonstrated on various examples. The results for speech signals with maximum frequencies of 4 kHz and 11,025 kHz are presented in [41]. This approach provides a reliable detection for a high SNR (SNR = 32 dB has been used) and under various attacks. The watermark sequence was created by using a pseudorandom Gaussian sequence of 1000 samples.

The probability of error was of order 10⁻⁷ for: MP3 (constant bit rate 8 kbps and variable bit rate 75–120 kbps are considered), delay monolight echo (180 ms, mixing 20%), echo 200 ms, deep flutter (deep 10, sweeping rate 5 kHz), amplitude (normalize 100%), and additive Gaussian noise (SNR = −35 dB). The worst case is obtained for pitch scaling and it is of order 10⁻⁵. The results for other attacks (time stretch wow delay 20%, wow delay 10% and bright flutter, MP3 variable bit rate 40–50 kbps) are of order 10⁻⁶.

3.2.2. Digital Image

The space-spatial-frequency analysis (two-dimensional time-frequency analysis) is used to select pixels that belong to the image nonstationary regions [42]. The two-dimensional S-method is used as a space-spatial frequency distribution [54]:

(62)

By increasing the size of a two-dimensional window, the cross-terms start to appear. Thus, when compared to the spectrogram, the number of frequency components increases, hence making the region characterization easier. A pixel that belongs to the dynamic region can be selected by using the following procedure.

()The S-method is calculated for a window (windows of size up to are used). The middle frequency range is used.

()The energy floor is obtained by using the experimentally determined .

()The region is considered as nonstationary if:

(63)

where , while is the total number of the points within the region .

The examples where the pixels belong to the dynamic and stationary regions, respectively, are shown in Figures 14(a) and 14(b).

Figure 14

(a) Dynamic region. (b) Stationary region (the spectrogram—based characterization-the second column; the S-method-based characterization—the third column).

The procedure for watermark embedding is just a two-dimensional case of the presented multidimensional approach. Namely, a two-dimensional support function is used:

(64)

The S-method as a two-dimensional time-frequency distribution is applied, while the energy threshold is . The watermark is shaped by the space-spatial frequency characteristic of the image components:

(65)

A two-dimensional pseudorandom sequence is used.

The watermark embedding and detection are performed in the space-spatial frequency domain:

(66)

This procedure is tested on several images (Lena, Peppers, Boat, F16, and Barbara), under various attacks (JPEG80-JPEG40, Median , Median , Average , Impulse noise, Gaussian noise, Lightening, and Darkening). The PSNR was around 50 dB. The number of the selected pixels varied from 3304 for F16 to 7833 for Barbara. The probability of error was compared with the standard DCT-based procedure (with different detector forms), where 22050 coefficients are used. It was shown that the proposed procedure significantly outperforms the standard DCT procedures.

3.2.3. Digital Video

Observe that the proposed approach can be also used for video signal watermarking. The two-dimensional and one-dimensional time-frequency distributions are combined in this case. Namely, the stationary pixels and stationary regions around them are selected by using the two-dimensional analysis, as it was described in the previous subsection. Then, the time dependent sequence is produced by taking the stationary pixels at the position , along consecutive frames. Based on the frequency modulated signal is created as

(67)

where , while is a constant. The stationarity of the selected pixels, along the time axis, is examined by using the one-dimensional S-method. The experiments show that the minimal number of pixels for reliable watermark detection is about 600. This can be easily achieved, even for a very short video sequence (note that more than 2500 stationary pixels are obtained for a signal of duration of 2 s in the example provided in [42]). This approach was tested under the presence of MPEG4 compression. The obtained probabilities of errors were found to be within the range .

4. Conclusion

An overview of most important time-frequency analysis techniques is presented. An appropriate distribution selection procedure for a specific type of signal is discussed. Time-frequency-based watermarking algorithms for digital audio, digital image, and video are reviewed, as well. The watermark is either a signal with specific time-frequency characteristics or a pseudonoise sequence shaped according to the time-frequency characteristics of the host signal. The main advantages of the time-frequency domain over the Fourier, DCT, and signal domain are emphasized. Finally, the presented theory could be used to generalize the existing watermarking approaches defined in either the Fourier or the DCT domain.