A methodology for time-frequency image processing applied to the classification of non-stationary multichannel signals using instantaneous frequency descriptors with application to newborn EEG signals

Abstract

This article presents a general methodology for processing non-stationary signals for the purpose of classification and localization. The methodology combines methods adapted from three complementary areas: time-frequency signal analysis, multichannel signal analysis and image processing. The latter three combine in a new methodology referred to as multichannel time-frequency image processing which is applied to the problem of classifying electroencephalogram (EEG) abnormalities in both adults and newborns. A combination of signal related features and image related features are used by merging key instantaneous frequency descriptors which characterize the signal non-stationarities. The results obtained show that, firstly, the features based on time-frequency image processing techniques such as image segmentation, improve the performance of EEG abnormalities detection in the classification systems based on multi-SVM and neural network classifiers. Secondly, these discriminating features are able to better detect the correlation between newborn EEG signals in a multichannel-based newborn EEG seizure detection for the purpose of localizing EEG abnormalities on the scalp.

1 Introduction to non-stationary EEG signal processing for diagnostic of abnormalities

This article is a novel contribution in three aspects. The first is a review of time-frequency (T-F) image processing techniques, i.e. image processing techniques that process the T-F representation of a signal considered as an image. The second aspect is the extension of traditional mono-channel T-F analysis to multichannel analysis. The third is to combine the previous two aspects and propose a more complete multichannel T-F image processing approach to the problem of Electroencephalogram (EEG) abnormality diagnostic and localization. This is important because these affect both adults and infants in a significant way. For example, newborn congenital anomalies, often referred to as birth defects, have a variety of causes ranging from pregnancy or birth complications to genetic malformations to viral infections in utero [1]. Epileptic seizures are among the most observed abnormalities, and result from problems such as lack of oxygen, haemorrhage, meningitis, infection and strokes [2]. The best tool for diagnosing the different abnormalities is the EEG which uses special sensors (electrodes) placed on the surface of the scalp to measure the electrical activity of the brain [3]. An illustration of EEG multichannel measurements using EEG Scalp electrodes is shown in Figure 1.

Figure 1

EEG scalp sensor electrodes (from [3]): (a) Lateral view of the electrode positions in a realistic display. (b) Electrode position and labels for various systems. Black circles indicate positions of the original 10-20 system, gray circles indicate additional positions in the 10-10 extension and small dots indicate additional positions in the 10-5 extension [3].

The EEG is a representative signal containing information about the electrical activity generated by the cerebral cortex nerve cells; it is the most utilized signal to clinically assess brain activities, and is widely used to detect abnormalities such as epilepsy and seizure [4]. Figure 2 shows an example of newborn EEG seizure and non-seizure waveforms. The manual detection of these EEG abnormalities includes visual scanning of EEG recordings which is very time consuming especially in the case of long recordings [5]. It also requires skilled interpreters; i.e. a neurophysiologist, who could be prone to subjective judgment and error. So, instead, it is desired to get the EEG signal parameters extracted and analyzed using computer based digital signal processing (DSP) techniques. Such an approach is highly useful in diagnostics and more suitable for automatic EEG abnormalities detection and classification [6, 7].

Figure 2

Newborn EEG seizure and non-seizure patterns (1st row) with their power spectral densities (2nd row).

1.1 EEG abnormalities detection using DSP

The focus of this study is to analyze abnormal newborn EEG signals in order to detect the different abnormalities and to locate their spatial distribution on the scalp. This abnormality detection and localization problem can be solved by analyzing newborn EEG signals and extracting features, which are then classified.

A typical scheme for EEG abnormalities detection using DSP includes the following three steps: (1) signal analysis in either time, frequency, time-scale or joint time frequency domain, (2) features extraction which characterize the different abnormalities and (3) classification of these features in order to assign the signal to one abnormality class with a relevant degree of severity: mild, moderate or severe abnormality.

Step 1, signal analysis, can be either a single channel analysis or a multichannel analysis, depending on the nature of the data and the application considered. The use of single channel analysis for complex systems description can produce incorrect system model and get false results. So, it is often more desirable to use instead a multichannel analysis which offer a much more accurate model although it may complicate the system model and lead to an increase in computation load. In the multichannel case, EEG data can be represented by a d dimensional vector [s₁(t)s₂(t) ... s _d (t)], where d represents the number of channels. Figure 3 shows an example of array EEG for newborn in the normal and seizure case, and their corresponding electrode placement on the scalp.

Figure 3

An example of array EEG for newborn. (a) Normal and seizure newborn EEGs. (b) Electrode placement for measuring neonatal EEG: the measurement of brain activity via the EEG is performed by attaching several electrodes (or antennae) to the head of the newborn. These electrodes are placed according to the International 10-20 system. This example shows a montage which is built from recordings of F3, F4, Cz, C3, C4, T3, T4, O1, O2.

For a single channel, EEG data would be represented by only the symbolic notation s₁(t). Then, the signal analysis permits to exploit all information necessary to characterize different abnormalities in the newborn EEG signal. However, this is in general only an approximation aimed at simplifying the problem and this leads to inaccuracies that are detrimental to the application, and it is therefore preferable to consider the multichannel case when possible.

Step 2, feature extraction, then extracts this information in order to assign it to one abnormality type using the classification method used in Step 3.

Research on EEG abnormalities detection is wide and varied, but almost in all previous studies, a single channel signal analysis approach is used (see Section 1.3). Although, the EEG abnormalities detection and classification system receives an input of more than one EEG channel, the analysis is made for a single channel only. Hence, one main goal of this study is to not only examine the EEG abnormality detection problem but also consider the localization of this abnormality in the brain and on the scalp, using a multichannel analysis of the multichannel EEG signal.

1.2 EEG abnormality localization using multichannel analysis

The EEG abnormality localization can be achieved by: (1) extracting features, which are able to detect and characterize the abnormalities, from the multichannel EEG signals; and (2) using these features to then identify the channels or electrodes in which the abnormalities are present. This procedure locates the sources of the abnormality on the scalp. Figure 4 presents the relevant block diagrams of the two discussed signal analysis schemes in the joint T-F domain. Figure 4-(a) illustrates the single signal analysis scheme and Figure 4-(b) exhibits the multichannel analysis scheme.

Figure 4

EEG-based automated abnormalities detection and classification scheme: (a) Block diagram of traditional single analysis. (b) Block diagram of multichannel analysis.

1.3 The T-F approach to non-stationary EEG analysis

Several automated techniques have been proposed for detecting EEG seizures [5, 8–16]. These techniques are based on time, frequency, non-stationary and nonlinear characteristics of EEG signals. Most of these methods use single channel EEG as their input and are based on a methodology similar to the one previously defined and illustrated in Figure 4-(a)[14–16].

Among these methods, we note those which use time domain statistics [8], spectral features [9], autoregressive modeling [17], and linear prediction error energy [18]. Other methods use a combination of time and frequency features [19], chaotic features [20–22], fast Fourier transform (FFT) coefficients [10]; more advanced methods use the coefficients of the discrete wavelet transform (DWT) of EEG signals [11], combination of DWT coefficients and chaotic measures [23], energy distribution of EEG signals in the T-F representation [24, 25], and T-F matched filtering methods [26, 27]. A comparison of the performance of different feature sets in classifying EEG signals can be found in [5]. In [12, 13], a combination of features in time, frequency, and time-scale domains and chaotic measures are used for neonatal seizure detection.

Due to the non-stationarity of EEG signals, T-F and instantaneous frequency (IF) based methods seem naturally more suitable for seizure detection and classification [7, 26, 28–32] and [[6], Section 15.5]. Recently, a study exploited the additional information provided by signal variations in terms of non-stationarity observed with the T-F approach, and then, developed novel features extracted from the T-F representation of EEG signals for the purpose of classifying them [14–16].

A variety of methods have already been applied for automatic seizure detection using multichannel EEG. This includes, for example, the use of T-F matched filters [7, 33, 34], multichannel matching pursuit [35], spatial and temporal contextual information obtained from multichannel EEGs [36].

One of the aims of this article is to extend the proposed approach in [14–16] using the full multichannel EEG signal in order to define a new multichannel abnormalities detection and localization approach for the newborn EEG.

The article is organized as follows. Section 2 presents a brief review of T-F signal analysis and IF estimation using quadratic time-frequency distributions (QTFDs). Section 3 describes a review of T-F image processing techniques applied to the time-frequency distribution (TFD) considered as an image. The methodology employed to extract new T-F features using T-F image processing tools for classifying EEG signals is described in Section 4. Experimental results with discussion are reported in Section 5. Finally, the article concludes with Section 6.

2 T-F analysis of multichannel non-stationary EEG signals

This section introduces the concept of T-F signal analysis applied to both mono-channel and multichannel non-stationary EEG Signals using quadratic TFDs.

2.1 Representing EEGs with TFDs and their IFs

Time-frequency signal analysis can be seen as the analysis of non-stationary signals with time-varying frequency content. Such signals are best represented by a TFD, which is intended to describe how the energy of the signal is distributed over the two-dimensional (2D) T-F space. The TFD not only shows the start and stop times of signal components and their frequency range, but also shows the component variation in frequency with time (which is called the IF) [37, 38]. The IF can be estimated using a peak detector in the T-F domain that selects the frequency with the maximum value in the T-F representation as a function of time. The use of a TFD for EEG abnormalities detection and classification is therefore the most natural approach, given their properties [6]. Figure 5 shows an example of TFDs of newborn EEG signals given in Figure 2, using the modified-B distribution (MBD).

Figure 5

The TFD of newborn EEG signals with seizure (left) and non-seizure (right) patterns given in Figure 2, using the MBD. The TFD in the left illustrates clearly the non-stationary nature and multi-component characteristics of the EEG signal.

2.1.1 EEG time-frequency based feature selection approach

A T-F based feature selection approach can improve the classification of EEG signals (illustrated in Figure 4-(a) acquired from healthy and epileptic subjects [14–16]. This is achieved by exploiting the link between the non-stationary features of the signal and considering their impact on the performance of the classification scheme. This new classification method is based on a feature vector composed of ten T-F features extracted from the TFD of EEG segments. These include: the IF and singular value decomposition (SVD) based features, the T-F complexity, and the energy concentration measures (see detail in Section 4).

Statistical analysis and experimental results given in [14–16], have shown that the proposed T-F features extracted from all TFDs perform well and achieve a highly significant discrimination between the EEG signals collected from healthy and those from epileptic patients. Also, the use of reduced interference TFDs (RIDs)-such as MBD, Smoothed Wigner-Ville distribution (SWVD), Gaussian Kernel distribution (GKD), Spectrogram (SPEC), Separable Kernel based RID (SEPK) [[6], Chapter 2 & Section 5.7] slightly improves the performance of the proposed method. This is because RIDs have the ability to reduce the effects of the cross-terms while still providing a good resolution [6].

2.1.2 Need for image processing techniques to extract new features

The improvements provided by the method described in [14–16] are attributed to the T-F approach as this allows the use of extra information related to the non-stationarity of the signals, which is directly available in the T-F domain. Such information includes the distribution of the signal energy over T-F plane and the IF characterizing the variation of its frequency content over time. The impact of this new method is that such variations are the key critical information that describes the non-stationarity of the signals.

As all the T-F features discussed in [14–16] were extracted from TFDs of the EEG segments, there is still room for improving the performance of the proposed approach by exploiting the information provided by the full T-F representation viewed as an image.

The use of T-F image techniques can then improve the method performance by selecting and extracting new T-F features. Section 3.1 reviews the proposed approaches based on T-F image techniques in order to select a new methodology to define new features.

2.1.3 Array processing and multichannel analysis

The abnormality localization problem needs to consider the multiple EEG channels simultaneously since EEG data often come as correlated time series from multiple electrodes on the scalp.

In practice, each channel of the 1D time-domain signal is first transformed to reveal its 2D T-F representation. By stacking all the 2D T-F arrays from all the channels, we form a 3D array in the space "T-F domain". Then, this 3D array is processed in order to detect and locate with accuracy the abnormality source on the scalp, and also their propagation in the head. To do this, the techniques borrowed from radar and communications statistical signal processing can be applied on this 3D data by considering the correlation between 2D T-F arrays. This processing can be made in two ways: (1) correlation between the 2D T-F arrays corresponding to the TFDs of multiple EEG signals, or (2) correlation between the "actual features" extracted from these 2D T-F arrays.

2.1.4 Multichannel T-F approach for localization

The design of a novel approach for automatic abnormalities detection and localization using a multichannel EEG signal for newborn is necessary to help the specialists to detect the presence of different abnormalities by interpreting the EEG signal in terms of abnormality type and its source localization, and to decide on a suitable diagnosis.

The selected approach is to exploit the additional information provided by the signal variation in terms of non-stationarity observed in the TFD; and then, develop novel features extracted from the TFD of the EEG signals using the T-F image processing techniques, for the purpose of detecting the different neonatal abnormalities and classifying the EEG signals. Also, the localization of the abnormality sources on the scalp is dealt with by analyzing the multichannel EEG signal.

2.2 T-F formulation

Quadratic time-frequency distributions form a class of TFDs which distribute the signal energy in a joint T-F domain from which we can generate features for detection and classification. Related backgrounds regarding QTFDs and IF estimation are presented in the next sections.

2.2.1 Formulation of quadratic TFDs

Quadratic time-frequency distributions were shown to be the most appropriate methods for the analysis and processing of non-stationary signals in many practical applications. They can be formulated as [6, 39]:

$$\rho \left(t,f\right)=W_z\left(t,f\right)\underset{\left(t,f\right)}{**}\gamma \left(t,f\right)$$

(1)

where ρ(t, f) denotes the TFD, W _z (t, f) the Winger-Ville distribution (WVD), γ(t, f) the T-F kernel of the distribution, and $\underset{\left(t,f\right)}{**}$ the 2D convolution operation in time and frequency. According to Equation 1, QTFDs can be considered as smoothed versions of the WVD, with γ(t, f) being a 2D smoothing filter. Different kernels in Equation 1 define different distributions in the class, that are most specifically adapted to particular classes of signals [[6], pp. 71-76]. The above formulation can also be given in any of the other three joint domains (time-lag (t, τ), doppler-frequency (ν, f) and doppler-lag (ν, τ)) linked to the T-F domain via Fourier transform [6].

At the core of the class formulated by Equation 1 is the WVD which is characterized by a T-F kernel defined as γ(t, f) = δ(t)δ(f), where δ is the Dirac delta function. For a real-valued signal s(t), the WVD is defined as [6, 39]:

$$W_z\left(t,f\right)=\underset{-\infty }{\overset{+\infty }{\int }}z\left(t+\frac{\tau }{2}\right)z^{*}\left(t-\frac{\tau }{2}\right)e^{-j2\pi f\tau }d\tau$$

(2)

where $z\left(t\right)=s\left(t\right)+\mathcal{H}\left(s\left(t\right)\right)$ is the analytic associate of s(t), stands for the Hilbert transform, and z*(t) its complex conjugate [[6], pp. 13-15]. The WVD provides the best joint T-F concentration among all QTFDs for linear frequency modulation (LFM) signals [6]. As EEG seizures were shown to be mostly Piece-wise LFM (PWLFM), the WVD seems to be the ideal tool. However, being quadratic in nature, the WVD introduces artifacts, or cross-terms, in the case of multi-component signals and/or non-linear FM signals. The presence of these artifacts can mask the true signal components and make the interpretation of the TFD a difficult task [40]. For those signals, reduced interference TFDs and polynomial WVDs may be used [6].

Using Equation 2, ρ(t, f) in Equation 1 can be rewritten as [[6], pp. 67]:

$$\begin{array}{cc}\hfill \rho \left(t,f\right)& =\underset{\tau \to f}{\mathcal{F}}\left\{G\left(t,\tau \right)\underset{t}{*}z\left(t+\frac{\tau }{2}\right)z^{*}\left(t-\frac{\tau }{2}\right)\right\}\hfill \\ =\underset{-\infty }{\overset{\infty }{\int }}\underset{-\infty }{\overset{\infty }{\int }}G\left(u,\tau \right)z\left(t-u+\frac{\tau }{2}\right)z^{*}\left(t-u-\frac{\tau }{2}\right)du{e}^{-j2\pi f\tau }d\tau \hfill \end{array}$$

(3)

where $G\left(t,\tau \right)=\underset{f\to \tau }{{\mathcal{F}}^{-1}}\left\{\gamma \left(t,f\right)\right\}$ is the time-lag kernel of the TFD.

There is a special class of kernels, referred to as separable kernel [6], whose members are expressed as the product of two single-variable functions (i.e., γ(t, f) = g₁(t)G₂(f)). This formulation allows the smoothing in time direction and frequency direction to be quasi-independently performed. The length and shape of the smoothing window g₁(t) control outer cross-term reduction. And, G₂(f) is the Fourier transform of the analysis window g₂(τ) whose length and shape control inner-artifacts reduction [[6], Chapter 3]. The MBD is one such method that was shown to be the best to represent newborn EEG signals using an objective measure for comparison [6, 29]. Figure 5 shows an example of a joint T-F representation of newborn EEG signals using MBD which illustrates the non-stationary and multi-component characteristics of the EEG signal. Compared to time domain and frequency domain (given in Figure 2), the TFD provides a more informative description of the EEG seizure by revealing the temporal variation of its frequency content through the IF.

For a given analytic signal z[n] associated with the real discrete time signal x[n], n = 0, 1, ..., N - 1, the discrete version of Equation 3 is given by [[6], pp. 237]:

$$\rho \left[n,k\right]=2\underset{n\to k}{\mathsf{\text{DFT}}}\left\{G\left[n,m\right]\underset{n}{*}\left(z\left[n+m\right]z^{*}\left[n-m\right]\right)\right\}.$$

(4)

For an N-point real signal x[n], ρ[n, k] is represented by an N × M matrix ρ where M (M ≥ N) is the number of FFT points used in calculating the TFD. Note that n = t.f _s and $k=\frac{2M}{f_s}f$ where t and f are, respectively, the continuous time and frequency variables, and f _s is the sampling frequency of the signal [[6], pp. 236]. Table 1 lists the discrete time-lag kernels of the TFDs used in this study.

Table 1 The time-Lag kernels of TFDs used in this article [6]

2.2.2 Instantaneous frequency

The IF is a key parameter of a non-stationary signal which describes how its frequency content changes with time [37]. For a mono-component analytical signal z(t) = a(t)e^jφ(t)the IF is defined as [37]:

$$f_i\left(t\right)=\frac{1}{2\pi }\frac{d\phi \left(t\right)}{dt}$$

(5)

where a(t) and φ(t) are respectively the instantaneous amplitude and instantaneous phase of the signal. The concept of IF has been successfully used in many technical fields such as radar, sonar, seismic, biomedical applications and communications (e.g. [41]). Many IF estimating techniques have been developed and a detailed review can be found in [38].

For mono-component signals, the peaks of the constant-time cross-sections of the TFD as well as its first order moment give estimates of the IF law of the signal. The notion of a single IF becomes inappropriate for a multi-component signal. To characterize this type of signals, an IF law is assigned to each signal component. Various IF estimation approaches for multi-component signals have been proposed (see e.g., [[6], Chapter 10] and [42, 43]). Such methods first localize and extract signal components from the signal T-F representation and then apply an IF estimation procedure. The methods have the advantage of not requiring prior knowledge about the signal under analysis except that its components are separated in the T-F domain, a condition often satisfied by newborn EEG signals [6].

A typical implementation of a basic multi-component IF estimation method such as that proposed in [42] includes the following two steps:

• (a) T-F transformation: The given signal is first mapped to the T-F domain using a suitable TFD which is chosen based on the characteristics of the signal under analysis. For newborn EEG signals, due to their non-stationary and multi-component nature, the selected TFD should provide high spectral resolution and have good cross-term reduction-capability. Previous studies have shown that separable kernel TFDs, i.e. γ(t, f) = g ₁(t) G ₂(f) are the most suitable ones [[6], Section 5.7]. We can use typical spectral analysis windows such as a Hanning window and a Gaussian window for g ₁(t) and G ₂(f) respectively, or the windows used in MBD which was shown to be best for representing EEG signals.

• (b) TFD Local peaks detection & component linking: The IF estimation method proposed in [42], considers the resulted TFD (ρ(t, f)) as a 2D image with time and frequency as its row and column coordinates and identifies the Local maxima (with respect to frequency) using both the first and second derivative tests. Only those local maxima which are higher than a threshold (i.e. ρ(t, f) > 0.01 × max(ρ(t, f))) are considered as valid peaks. By assigning value 1 to the locations of valid peaks and value 0 to all others, the T-F image can then be transformed to a binary image B(t, f) [42].

  A component linking algorithm then detects linked components in B(t, f) by examining the pixel connectivity and the number of connected pixels. This is based on the fact that ideally, the IF of a signal component where signal energy concentrates will be observed in the TFD as a ridge describing the IF. Among the sets of connected pixels, only those with a number of pixels exceeding a preset threshold are identified as true linked components of seizure IFs. To remove the IFs of non-seizure components, this threshold is set to the minimum time duration of a seizure component in samples (equivalent to 10 s).

2.2.3 T-F analysis of multichannel EEG using spatial T-F distributions

The concept of spatial T-F distribution (STFD) was introduced to analyze an array of non-stationary signals and has been successfully used in the area of array signal processing for source separation, for example (see [[6], Chapter 8]). STFDs are a generalization of TFDs to a vector of multi-sensor signals.

Let us denote the EEG signal recorded at the i^th channel at the discrete time n as x _i [n]; i = 1, 2, ..., P; n = 1, 2, ..., N where P is the total number of EEG channels, and the vector Z[n] as the analytic associate of the vector X[n] as a set of recorded signals. The general form of the STFD of the signal vector Z[n] is defined as [[6], pp. 349-350]:

$$D_{xx}\left[n,k\right]=\sum _{m=-M_0}^{M_0}\sum _{p=-M_0}^{M_0}\mathbf{G}\left[n-p,m\right]\odot \mathbf{Z}\left[p+m\right]{\mathbf{Z}}^{*}\left[p-m\right]e^{-j4\pi \frac{mk}{n}}$$

(6)

where [G[n, m]] _ij = G _ij [n, m] is the time-lag kernel associated with the signals z _i [n] and z _j [n], ⊙ represents the Hadamard product, Z* is the conjugate transpose (Hermitian transpose) of the vector Z, and $M_0=\frac{N-1}{2}.$ Each element of the STFD is given by:

$${\left[D_{xx}\left[n,k\right]\right]}_{ij}=\sum _{m=-M_0}^{M_0}\sum _{p=-M_0}^{M_0}{G}_{ij}\left[n-p,m\right]z_i\left[p+m\right]z_j^{*}\left[p-m\right]e^{-j4\pi \frac{mk}{n}}$$

(7)

The off-diagonal elements of the STFD are the cross-TFDs and the diagonal entries are the classical auto-TFDs of the EEG signals. STFD-based methods exploit the non-stationary characteristics of the signals together with the spatial diversity provided by the multiple sensors. This property makes them suitable for analyzing multichannel EEGs.

2.2.4 Selecting a data adapted TFD

The selection of a suitable TFD for representing the EEG signals is the first step in forming the expressions (6) and (7) for any detection and classification scheme in the T-F domain. A suitable TFD is the one which is capable of highlighting the signal non-stationary features that best discriminate between different classes under consideration. For the analysis of multi-component non-stationary EEG signals [6, 32] and to generate suitable features for detection, a TFD needs to have the following properties [6]:

• (a) Realness: ρ[n, k] must be real.

• (b) IF estimation: The IF of the signal can be estimated using the peak of the TFD of the signal.

• (c) Local energy: The energy of the signal in a certain region in the T-F plane can be found by the integral of the TFD over that region.

• (d) Reduced interference: The TFD attenuates the unwanted cross-terms in the T-F domain relative to the signal terms.

• (e) High resolution: The reduced interference property is achieved while preserving a good T-F resolution.

Several measures that take into account the above properties were proposed in [29, 44, 45]. Comparative studies of these measures indicated that the selection of TFDs for a particular application is data-dependent [29].

Table 1 lists the expressions for the time-lag kernels G[n, m] of some of these QTFDs known as RIDs which are used in this study. These RIDs were designed to suppress or attenuate the cross-terms in the WVD [[6], Section 5.2]. The WVD is also considered in this study as it is a reference, for performance comparison. The exact formula for each method can be found in [[6], Section 6.1].

3 T-F image processing methodology for newborn EEG analysis

As mentioned earlier, T-F image processing techniques represent a set of image processing methods that process the TFD of a given signal considered as an image. Several studies applied image processing techniques to the signal TFD in order to select a methodology suitable for the problem considered, leading to an improvement in terms of precision, resolution, performance or robustness. The applications of these techniques include: segmentation [46–55], denoising and signal-to-noise ratio (SNR) enhancement [56–59], classification [60–63], coding and compression [64, 65], watermarking [66–70], and IF estimation and cross-term reduction [42, 52, 71, 72]. A review of proposed approaches is given in the following section.

3.1 T-F image processing techniques: overview

A comprehensive review of proposed T-F image processing approaches shows that they can be grouped in six classes (corresponding to the numbering of the next subsections). The intent is to evaluate these methods to determine their suitability in the main application considered in this article, i.e. feature selection and extraction, and classification.

3.1.1 Denoising or SNR enhancement of T-F images

The presence of noise and/or artifact (e.g. muscle activity and eye blinks) in the recorded EEG signals degrades the performance of any EEG based signal processing method. Image processing techniques can be used to reduce noise and artifact from the TFD of EEG signals while retaining key details such as edges and texture of the TFD. T-F image techniques can then be adapted and used to enhance the quality of T-F representations in order to better represent the T-F features which characterize different abnormalities.

There are several denoising T-F methods that use T-F image processing techniques for the removal of noise from a signal [56]. An adaptive block thresholding denoising algorithm based on Stein Unbiased Risk estimator is proposed to remove noise from audio signals in the T-F representation in [73]. Another approach uses thresholding denoising algorithms in conjunction with the S-transform [57]. A Bayesian image denoising method is also proposed in [58] which removes Gaussian noise from the T-F representation of EEG signals [59]. These methods can be adapted following the approach based on LPA-RICI.^a The idea consists of transforming a 2D image into 1D array using the row-wise or diagonal-wise transformation, and its denoising using the LPA-RICI method.

The above indicates that there is a choice of several algorithms for denoising methods which reduce noise and artifacts in the EEG signals from their TFDs considered as images. The power of noise and artifact in the signal resulting from reconstructing the enhanced TFDs are expected to be much less than those in the original signals. This in turn can significantly improve the performance of any newborn EEG classification/source localization method.

3.1.2 T-F image processing for IF estimation and cross-term reduction

Image processing techniques can be used to estimate the signal IF and attenuate cross-terms which appear in QTFDs due to their bilinear nature. One such image processing techniques estimates the IF laws of a multi-component signal based on component linking [42]. Another approach uses morphological image processing operators to reduce cross-terms in the WVD [71]. Alternative approaches are also proposed in [52, 72], all assuming a prior knowledge of the interference geometry.

The use of the above mentioned methods not only improves the performance of the extracted features in discriminating between different classes, but also facilitates the estimation of the IF laws of multi-component signals. The statistics of the estimated IF laws of the signal using image processing techniques can be used as T-F features to characterize the non-stationarity of the EEG signal.

3.1.3 Segmentation of T-F images

Image segmentation and edge detection methods can be applied to TFDs to localize significant structures in them and extract T-F signatures of different components of the signal under analysis [74]. The goal of a T-F image segmentation is to create homogeneous regions that correspond to separate components of the signal. This is performed by a complete partitioning of image C (the original TFD) into n disjoint sub-regions, C₁, C₂, ..., C _n . Morphological filtering was used in the analysis of TFDs in several applications, for example, musical transcription of audio recording [47] and analysis of submarine and seismic signals [48, 49]. Similar to the method presented in [49], authors in [46] apply the water-shed segmentation procedure on T-F images of non-stationary signals consisting of multiple closely-spaced components in the presence of noise. In the context of segmentation, it is not only important to extract desirable patterns from noisy background but also to separate the T-F image subsets corresponding to different signal components. This can be achieved by computing an over-segmented watershed image and re-merging the adjacent regions into pertinent structures, based on their statistical and geometry features [46]. An extension of the segmentation procedure based on morphological T-F image processing techniques is also proposed in [50] for speech spectrogram segmentation.

In addition, a new approach for blind separation of nonstationary sources using their TFDs, was proposed in [75]. This approach is based on the observation that a monocomponent FM signal is represented by a linear feature corresponding to the energy concentration points in the T-F image. The idea is based on a line detection algorithm, which is an adaptation of the road network extraction method [76], to extract separately all the components using a spatially averaged T-F image of their mixtures. The line detection is done at the pixel level by determining whether a pixel belongs to a line crossing it along a particular direction.

These techniques are especially useful for analyzing EEG signals as they have the ability to provide us with new features which capture the non-stationarity and multi-component nature of the EEG signals. These features can be grouped into the following four classes:

• Topological features such as the number of components of the signals,

• Morphometric features based on the geometric shapes of the T-F components of the signals such as: size, position, and orientation of the components,

• Intensity features based on the histograms of the TFDs over relevant zones (typically components) such as moments, median, and mode; and

• Texture features such as statistics of spatial grayscale variations of the TFDs such as contrast, entropy, and energy.

The use of the above-mentioned features is expected to improve the performance of any EEG abnormality detector.

3.1.4 Classification of T-F images

Electroencephalogram signals can be classified based on their TFDs using image processing techniques which allocate a particular T-F image belongs to a certain class. Several approaches have been proposed for different applications. In the context of ECG, three methods were proposed to classify the signals using the images obtained from their WVDs [60]. The first method uses features that are problem specific and highly sensitive to noise such as the frequency range and the multi-component characteristics of the signal. The second one uses correlation in the T-F domain. The third method is based on a decision theoretic pattern recognition which uses the SVD and the ambiguity function as features for the classification [60].

Another approach determines the modulation mode of the signal through a T-F method with autoregressive modeling to detect phase shifts, frequency shifts and amplitude shifts [61]. Then the classification is performed through a simple decision tree. Another approach is to use the regional correlation between TFDs in order to achieve a better performance such as automatically selecting regions of interest before performing the correlation [62]. For audio applications, an approach considers the T-F spectrograms related to music samples as textured images and extracts texture-based features to classify these music samples according to the predominant musical instrument [63]. Another method directly classifies each signal according to the best match of its TFD (such as Spectrogram, QTFDs or Gabor) with a set of ideal masks. The best fit is determined through Frobenius inner products as in [77].

Most of the classification methods proposed in the image processing literature can be adapted and applied to EEG classification. These classifiers may outperform conventional classifiers such as Neural Networks and SVMs in terms of classification accuracy, and can be useful in the context of EEG signal classification.

3.1.5 Coding and compression of T-F images

Time-frequency image compression aims at reducing irrelevant and redundant information in the T-F data in order to store or transmit non-stationary data in an efficient form. So, T-F image compression and coding techniques can be applied to TFD records as a solution to the transmission and storage of complex non-stationary signals. For example, one can use a T-F image coding technique by dividing the T-F image into a combination of time and frequency components and then code them separately [64]. The key of this algorithm is to locate the sharp edges in both horizontal and vertical directions, and to select a threshold size for which the sharp edges are to be coded. Another approach is based on the estimation of the sequence duration using the autocorrelation of the T-F image; it was used to detect and characterize a frequency hopping (FH) signal in the context of spectrum surveillance [65]. Since the obtained T-F image is often very large, a compression step reduces the image before processing it. Following the above approach, an audio data compression algorithm based on the T-F image of music signals could reproduce an audio signal which keeps only the main sense of music with smaller size [78]. The method can be used for tasks of identification of a signal, search of pieces of music, etc.

The above indicates that the T-F representation or T-F image can be used in general coding and compression algorithms of EEG signals. This implies that these T-F methods can be adapted and used in the case of large and complex biomedical data, including EEG, as a possible innovative solution to the transmission and storage of key T-F features describing specific abnormalities.

3.1.6 T-F watermarking

The main goal of T-F watermarking is to hide a message m (watermark) in a non-stationary signal using a T-F representation of data w to obtain new data w' practically indistinguishable from w, by people, in such a way that an eavesdropper cannot remove or replace m in w'. This allows to develop tools for data hiding and copyright protection, using the characteristics of non-stationarity signals.

Time-frequency image watermarking methods can be classified in two traditional approaches: spatial and spectral domain techniques. In the spatial domain, the watermark is embedded in selected regions chosen based on the texture of the given T-F image. In the spectral domain, the watermark is embedded in the transform domain using methods such as discrete cosine transform (DCT) and DWT, in the mid-frequency range to ensure transparency and robustness of the watermark. A joint TFD of images can give a more comprehensive representation of the image compared to looking at each domain individually [67, 70]. Image watermarking in the T-F domain can be done using the basic WVD [68]. In this method, the watermarked cells are selected based on their impact on image quality as well as resilience to JPEG.

Another watermarking algorithm based on the WVD [69] does not assume any particular structure for the embedded watermark and offers a theoretical framework for designing the minimum probability of error detector. The embedding algorithm in the joint T-F domain is based on two aspects: the first one embeds the watermark directly into the WVD of the image, while the second one embeds the WVD of the watermark. This result reduces the computational complexity of embedding and detecting the watermark. This method is shown to be transparent and robust under attack. A variant of the above method led to a robust and high-capacity image watermarking algorithm using the discrete evolutionary transform (DET) calculated by the Gabor expansion to represent an image in the joint T-F domain [70]. The watermark is then embedded onto selected coefficients of the TFD.

Another approach to watermarking of speech signals in the T-F domain [66] is based on the cross-terms free S-transform and the time-varying filtering used for watermark modeling. The main advantages and drawbacks of various TFDs applied to digital watermarking are discussed in [67].

The above indicates that image watermarking methods can be adapted and used for biomedical data. For example, watermarking methods are needed for EEG data if information such as acquisition measures and/or patient information, is confidential or registered. The idea is that this information can be embedded as a hidden message in the T-F image of the EEG signals, using the methods proposed in [66, 67, 69, 70].

3.2 Range of applicability of T-F image processing methods to EEG classification

The previous section indicates that there is a wide range of T-F image processing methods that can be developed once one has formed the T-F image using a QTFD that describes the characteristics of a non-stationary signal. These methods described above can be assessed in terms of performance against quality criteria for EEG abnormalities detection and classification. Table 2 shows some quality criteria for each T-F image processing category. The performance quality criterion indicates the applicability of a given T-F image processing technique in increasing the EEG classification accuracy. Performance and precision are the key quality criteria which can be enhanced by relevant T-F image processing techniques.

Table 2 T-F image processing techniques and corresponding quality criteria

Tables 3 and 4 show the applicability of these T-F image processing techniques to the EEG abnormalities detection and classification problem. T-F image processing techniques can therefore improve the analysis of multichannel EEG signals in key areas, namely: pre-processing, feature extraction, cross-term reduction, and classification. Section 4.1 discusses a number of features that can be extracted from the T-F image and used for classification, as a supplement to signal based features used in previous work [14–16].

Table 3 T-F image processing techniques and their applicability (or not) to the EEG abnormalities detection and classification problem

Table 4 Selected T-F image processing techniques and their applicability to the different steps of the EEG abnormalities detection and classification problem

4 EEG abnormality detection and localization methodology

The proposed method for EEG abnormalities detection and classification, illustrated in Figure 4, includes three stages: monochannel or multichannel T-F analysis, feature selection and extraction, and final classification.

As mentioned earlier, spectral analysis of non-stationary signals leads to T-F analysis. It is thus necessary to perform a T-F analysis on the EEG signals in the presence and the absence of abnormality in order to determine features such as the frequency bands in which the energy of the signals is concentrated for each case. The proposed T-F analysis aims at extracting new key features from the EEG signals which carry information related in particular to their non-stationary nature. These features include the T-F features proposed recently in [14–16], and also some features based on T-F image processing techniques as detailed in the following subsection.

4.1 Feature extraction methodology

4.1.1 Signal related features

The discrete-time EEG signal and its TFD are respectively denoted as x[n] and ρ[n, k]. Ten T-F features from the TFD ρ[n, k] corresponding to each signal are described below. These features are extracted for each EEG segment of length T seconds after calculating ρ[n, k]. Five other features based on T-F image techniques are then also extracted. Only the features denoted F 1 - F 6 take into account directly the non-stationary nature of the EEG signals, as discussed in [14–16].

• (1) IF-based features: As mentioned earlier, the IF can be estimated either as the peak of the TFD of the signal (Equation 8)

  $$f_i\left[n\right]=\frac{f_s}{2M}arg\left\{\underset{k}{max}\rho \left[n,k\right]\right\}$$

  (8)

  or the first order moment of the TFD (Equation 9).

  $$f_i\left[n\right]=\frac{f_s}{2M}\frac{{\sum }_{k=1}^{M}k\rho \left[n,k\right]}{{\sum }_{k=1}^M\rho \left[n,k\right]}$$

  (9)

  The first selected feature, denoted as F_{S 1}, is chosen as the mean of f _i [n] (MEIF), i.e. $F_{S1}=\frac{1}{N}{\sum }_{n=1}^N{f}_i\left[n\right].$ The second feature F_{S 2}represents the deviation of the IF (DEIF) of the EEG signals, i.e.

  $$F_{S2}=\mathrm{\Delta }{f}_i\left[n\right]=max\left(f_i\left[n\right]\right)-min\left(f_i\left[n\right]\right)$$

  (10)

• (2) SVD-based features: The SVD is performed on the N × M matrix ρ. This divides the TFD matrix into two subspaces, signal subspace and an orthogonal alternate subspace of the form given as follows [14–16]

  $$\rho =\mathbf{US}{\mathbf{V}}^{\mathbf{H}}$$

  (11)

  where U is an N × N matrix, S is an N × M diagonal matrix with non-negative real numbers (S _i , i = 1, 2, ..., N) on the diagonal, and V^H (the conjugate transpose of V) is an M × M real unitary matrix. The diagonal entries of S are known as the singular values of ρ. The singular values and vectors have proved useful in characterizing EEG abnormalities [13, 74]. Here, the maximum and variance of the singular values, denoted as MASV and VASV respectively, are selected as characteristics of the singular values of ρ and chosen as features F_{S 3}and F_{S 4}, following [14–16].

• (3) T-F complexity (TFCM): The TFCM feature denoted as F _{S 5}is an extension of the measure proposed in [12, 79] to the T-F plane as proposed in [14–16]. It uses both SVD and Shannon entropy and is given by

  $$F_{S5}=-\sum _{i=1}^N\overline{S_i}log\overline{S_i}$$

  (12)

  Where $\overline{S_i}$, i = 1, 2, ..., N are the normalized singular values,i.e.: $\overline{S_i}=\frac{S_i}{{\sum }_{i=1}^N{S}_i}$

  According to the basic theorem of information theory, F_{S 5}≤ log N, and the equality holds only when all the normalized singular values are equal, i.e. $\overline{S_i}=\frac{1}{N}$; i = 1, 2, ..., N (see for example [[80], pp. 88]. TFCM therefore represents the magnitude and the number of the non-zero singular values of the TFD of the EEG signals. TFCM is a useful feature as the number of non-zero singular values of ρ and their magnitudes have a strong relationship with the information content in the TFD [14–16, 81].

• (4) Energy concentration measure (ECOME): For a multi-component signal, the energy concentration measure, denoted as ECOME, determines the concentration of the dominant component at each location in the T-F domain [82]. We use the definition given in [14–16, 83] as it does not discriminate between low concentrated components and highly concentrated ones. It is given by:

  $$F_{S6}={\left(\sum _{n=1}^N\sum _{k=1}^M|\rho \left[n,k\right]{|}^{\frac{1}{2}}\right)}^2$$

  (13)

• (5) Sub-bands energies: Sub-band energy-based features represent the energy of the EEG signal x[n] in the sub-band where this energy is mostly concentrated. The sub-band interval is determined in the T-F representation by analyzing the EEG signal. The energy is computed by proper frequency windowing of ρ[n, k] and is derived using the formula representing the partial information contained in a certain region of the T-F domain [[6], pp. 46].

  In [14–16], the use of four sub-bands energies features F _Si , i = 7, 8, 9, 10 based respectively on 0-4 Hz (δ), 4-8 Hz (θ), 8-12 Hz (α), and 12-30 Hz (β) sub-bands, were proposed. Their sub-bands have been chosen by analyzing the EEG signals taken from the EEG database [84]. This database is described in [85], and will be used in our experimental simulation in the next section.

  These features are defined as follows [14–16]:

  $$F_{S7}=\sum _{n=1}^N\sum _{k=1}^{M_{\delta }}\rho \left[n,k\right]$$

  (14)
  $$F_{S8}=\sum _{n=1}^N\sum _{k=M_{\delta }}^{2M_{\delta }}\rho \left[n,k\right]$$

  (15)
  $$F_{S9}=\sum _{n=1}^N\sum _{k=2M_{\delta }}^{3M_{\delta }}\rho \left[n,k\right]$$

  (16)
  $$F_{S10}=\sum _{n=1}^N\sum _{k=3M_{\delta }}^{M_{\beta }}\rho \left[n,k\right]$$

  (17)

  where M _δ = [8M/f _s ], M _β = [60M/f _s ] and [.] stands for the floor operator. Note that k = M corresponds to the maximum frequency component in the signal (f _s / 2). It follows that M _δ and M _β correspond to the frequency components at 4 and 30 Hz, respectively. We note that the features F_{S 7}- F_{S 10}may also be estimated from the spectrum of the signal as they do not contain information about non-stationarity.

  For the newborn EEG database used in this study (as described in Section 5.2 and also in [15, 16]), the energies of the EEG signals in 0-4 Hz (δ) and 4-8 Hz (θ) sub-bands were chosen as features. Only the features F_{S 7}and F_{S 8}will be used for this database in our experimental simulation in the next section.

4.1.2 T-F image processing-based features

Further to the features presented earlier that followed [14–16], we are also interested in the use of the T-F image segmentation techniques described in Section 3.1 to extract other features so as to complement the signal related features. We therefore propose new T-F features to describe visually the normal and seizure patterns in the TFD. These features are based on visual descriptors, and are extracted from T-F representations considered as images using image processing techniques. The idea is, for example, to apply a segmentation algorithm on T-F images to detect regions where all EEG information appear (e.g., all components, normal/seizure pattern). The watershed segmentation technique in [46, 50, 86] is adapted to detect this homogeneous region and then to compute their statistical and geometrical features. In this context, T-F image data is interpreted as a topographic surface where watershed boundaries separate individual catchment basins. The boundaries of the groupings are placed along the crest lines of the gradient image and as a result the watershed transformation can partition the image into meaningful regions. Figure 6 shows an example of the T-F image of a newborn EEG signal obtained using the MBD and its binary segmented image, obtained using the watershed segmentation; this makes it possible to detect and select the normal/seizure pattern i.e., the region where the normal/seizure pattern appears. From the binary segmented image, we could get several shape descriptors that include bounding box (a rectangle that circumscribes the segmented object), convex hull (the smallest convex shape that contains the object), centroid and contour.

Figure 6

Illustration of the time-frequency image methodology: An example of the T-F image of newborn EEG signal using the MBD (a) with the corresponding grayscale (b) and binary-segmented image using the methodology presented (c). The white region in the binary-segmented image corresponds to the region where all components appear in the TFD.

We now describe five morphometric features based on the geometric shape of the segmented regions.

1. Image grayscale moments: The moments of the T-F binary-segmented image denoted ρ'[n, k] can be used to compute shape features, such as height, width, area, perimeter, compactness, centroid, etc. Using these shape descriptors we could further compute other ratios of geometric features, such as: $\mathsf{\text{Rectangularity}}\left(=\frac{\mathsf{\text{lengt}}{\mathsf{\text{h}}}_{\mathsf{\text{bounding}}}{}_{\mathsf{\text{box}}}}{\mathsf{\text{widt}}{\mathsf{\text{h}}}_{\mathsf{\text{bounding}}}{}_{\mathsf{\text{box}}}}\right)$, $\mathsf{\text{Aspect Ratio}}\left(=\frac{\mathsf{\text{Are}}{\mathsf{\text{a}}}_{\mathsf{\text{object}}}}{\mathsf{\text{Are}}{\mathsf{\text{a}}}_{\mathsf{\text{bounding}}}{}_{\mathsf{\text{box}}}}\right)$ and $\mathsf{\text{Form Factor}}\left(=\frac{4\mathrm{\pi }\times \mathsf{\text{Are}}{\mathsf{\text{a}}}_{\mathsf{\text{object}}}}{{\mathsf{\text{Perimeter}}}_{\mathsf{\text{object}}}^{\mathsf{\text{2}}}}\right)$.

The moment of order (p, q) for ρ'[n, k] is defined as [87]

$$m_{pq}=\sum _n\sum _k{n}^p{k}^q\rho \prime \left[n,k\right]$$

(18)

where p, q = 0, 1, 2, .... The central moments are expressed as

$${\mu }_{pq}=\sum _n\sum _k{\left(n-\overline{n}\right)}^p{\left(k-\overline{k}\right)}^q\rho \prime \left[n,k\right]$$

(19)

where $\overline{n}=\frac{m_{10}}{m_{00}}$ and $\overline{k}=\frac{m_{01}}{m_{00}}$ are centroids of the T-F image. The μ₀₀ is the area of the segmented region, $\overline{n}$ and $\overline{k}$ give the x and y coordinates of the centroid for the segmented region. Higher order moments may give even more detailed shape characteristic, but they are not considered in this study.

2. Convex Hull: The convex Hull H(ρ') of a T-F binary-segmented image subset ρ'[n, k] is defined as the smallest convex set containing a given subset [87]. The only parameter required in our study is the area of the convex Hull (i.e. the number of pixels, d(H(ρ')))). This can be calculated using the central moments of order (0, 0) defined in Equation 19.

In summary, the following five morphometric features based on the geometric shape of the segmented T-F regions are considered in this study as a supplement to the signal-based features listed in the previous subsection:

• Convex Hull/area: F_{I 1}= μ₀₀

• Perimeter: F_{I 2}= (m₃₀ + m₁₂)² + (m₀₃ + m₂₁)²

• Compactness^b: F_{I 3}= (F_{I 2})²/F_{I 1}

• Coordinates of the centroid for the segmented region: $F_{I4}=\frac{m_{10}}{m_{00}}$ and $F_{I5}=\frac{m_{01}}{m_{00}}$

The total number of features for this study was therefore 15 for the adult EEG data ({F_{S 1}, F_{S 2}, ..., F_{S 10}, F_{I 1}, ..., F_{I 5}}) and 13 for the newborn EEG data ({F_{S 1}, F_{S 2}, ..., F_{S 8}, F_{I 1}, ..., F_{I 5}}). These features include both signal related features and image related features.

4.2 Classification

During the classification step, an EEG signal is allocated a certain class based on the location of its feature vector. The T-F features extracted from EEG signals, and described above, are used to train a classifier. We opted for the classification algorithms which were recently used to classify the EEG signals in the T-F domain in [14–16, 88]. In particular, two classification algorithms are used in this study:

• The Multi-class SVM classifier used in [14–16].

• The Neural network-based classifier used in [88].

These methods extract features from the TFD of the EEG signals under analysis and then feed them to the classifier as a vector. These classification methods are then compared by evaluating their performance in classifying the EEG signals using the selected T-F features defined in the previous section, in terms of sensitivity, specificity and classification accuracy.

4.3 Localization

The classification step consists in detecting the presence or absence of the abnormality in the EEG signals and allocating it to a particular class. For the problem of localizing the spatial location of an abnormality on the scalp, it requires the simultaneous analysis of the multiple EEG signals which are provided from the different electrodes placed on the scalp.

In this study, the selected methodology to detect an abnormality and locate its source on the scalp is as follows:

• (1) For each electrode, the EEG signal is transformed to reveal its T-F representation, described by its TFD, using the MBD.

• (2) Construct the 3D array data in the space-T-F domain by stacking all the 2D T-F arrays corresponding to the TFDs (by concatenation of all 2D data). The stacking order corresponds to the location of electrodes on the scalp. This assigns the space direction in the space-T-F domain.

• (3) Calculate the correlation between the 2D T-F arrays in order to detect the presence of an abnormality. The correlation can be calculated between the 2D T-F arrays or the T-F features extracted from these 2D T-F arrays [31].

• (4) Determine the electrodes for which an abnormality is detected in their corresponding 2D T-F arrays.

We assume here that the problem of EEG synchronization does not arise or has been already dealt with in a pre-processing stage. Furthermore, fusion approaches can be applied on the 3D array data for detecting and localizing the abnormalities [89–91].

5 Experimental results and discussion

This section assesses the performance of the EEG abnormalities detection and localization schemes using the T-F image processing techniques, defined in the previous section in terms of features selection and extraction, localization and classification.

5.1 Adult EEG seizure detection and classification

The performance of T-F features, defined in Section 4.1, for EEG abnormalities detection and classification (Figure 4-(a)) is first assessed in the case of real adult EEG data.

The data set used is described in [85]; it consists of five sets denoted as A, B, C, D and E. Each set contains 100 single channel EEG segments with duration of 23.6 s. The signals have been recorded at f _s = 173.6 Hz sampling rate and therefore each EEG signal in each set has f _s × 23.6 = 4, 096 samples. Sets A and B have been taken from surface EEG recordings of five healthy volunteers with eye open and closed, respectively. EEG signals in sets C and D, respectively, were acquired in seizure-free intervals from five epileptic patients in the epileptogenic zone and from the hippocampal formation of the opposite hemisphere of the brain. Set E contains seizure activity intervals in the EEG signals acquired from the five patients. More details about the dataset can be found in [84]. Here, for practical reasons, the desired outcome of the classification stage is one of three different classes of the EEG signals in the dataset, namely: classes H, F, and S defined as follows:

• Class H includes sets A and B which contain the EEG signals acquired from H ealthy volunteers.

• Class F includes sets C and D which contain seizure-F ree intervals acquired from epileptic volunteers, and

• Class S includes set E which contains S eizure-activity intervals of the EEG signals collected from epileptic patients.

The T-F feature set {F_{S 1}, F_{S 2}, ..., F_{S 10}, F_{I 1}, ..., F_{I 5}} was extracted from the TFD of each EEG segment of length T seconds. Only five TFDs are chosen in this simulation: MBD, SPEC, SWVD, GKD and WVD. The parameters of the MBD and GKD were respectively chosen as β = 0.01 and σ = 0.9. These values are typical ones for which the MBD and GKD have shown good performances in analyzing EEG signals [[6], Sections 7.4 and 15.5] and [15, 16]. The window w[n] for SWVD and SPEC distributions, was chosen to be a Hanning window of length ⌊N/ 4⌋ samples. Note that we did not optimize the window functions and lengths used for SPEC and SWVD. The simulations were carried out in Matlab.

For performance evaluation, the multi-class SVM and neural network classifiers were trained using the T-F features extracted from the EEG signals. We compared the results of the classification for different TFDs. The original database was split in two parts, 60% of the data were used for training and 40% for testing the classifier.

Table 5 shows the confusion matrices for classification results using 15 features extracted from different T-F representations of EEG segments of length T = 11.8 s (N = 2,048 samples). The results between parentheses are the classification results using the 10 features {F_{S 1}, F_{S 2}, ..., F_{S 10}} (without the features based on T-F image processing techniques) and the multi-class SVM classifier [14–16]. Each row shows, for a particular TFD, the total number of EEG segments correctly classified as well as those misclassified as other classes. The total number of EEG segments in each class is 100 segments. The results indicate that class F signals are most often confused with class H and S signals; likewise class S with class F.

Table 5 Confusion matrices for EEG classification using T-F features set {F_{S 1}, F_{S 2}, ..., F_{S 10}, F_{I 1}, ..., F_{I 5}} extracted from the TFD with SVM classifier and Neural Network-based classifier.

Table 5 also includes the values of the statistical parameters of the classifier; namely: its sensitivity, specificity, and total classification accuracy. These parameters are used in order to evaluate the classification performance and are defined as follows:

$$\mathsf{\text{Sensitivity}}=\frac{\mathsf{\text{TP}}}{\mathsf{\text{TP}}+\mathsf{\text{FN}}}$$

(20a)

$$\mathsf{\text{Specificity}}=\frac{\mathsf{\text{TN}}}{\mathsf{\text{TN}}+\mathsf{\text{FP}}}$$

(20b)

$$\mathsf{\text{Total}}\mathsf{\text{accuracy}}=\frac{\mathsf{\text{TP}}+\mathsf{\text{TN}}}{\left(\mathsf{\text{TP}}+\mathsf{\text{FN}}\right)+\left(\mathsf{\text{TN}}+\mathsf{\text{FP}}\right)}$$

(20c)

where TP, TN, FN, and FP, respectively represent the numbers of true positive, true negative, false negative, and false positive.

We notice that the classification results using the feature vector {F_{S 1}, F_{S 2}, ..., F_{S 10}, F_{I 1}, ..., F_{I 5}} with the SVM-based classifier, are better compared with the results using the feature vector {F_{S 1}, F_{S 2}, ..., F_{S 10}}. This is confirmed by the total classification accuracy calculated for each TFD. Also, the results given by the neural network-based classifier are comparable with the SVM classification results. We note that the neural network-based classifier is sensitive to over-training compared to the SVM-based classifier. The total classification accuracy is in the range (93% to 95%) for 100 EEG segments. This can be improved by increasing the number of EEG segments in the training-step. These results indicate that T-F features based on image processing techniques are also able to characterize non-stationary EEG signals, and therefore, detect the different abnormalities with accuracy.

5.2 Newborn EEG seizure detection and classification

The performance of the proposed T-F features in newborn EEG seizure detection and classification (Figure 4-(a)) is now assessed using real newborn EEG data.

The newborn EEG data used here includes 14-channel EEG recordings of nine neonates using mono-polar montages. The recordings have been marked by a neurologist for seizures. The EEG signals have been band-pass filtered in [0.7-50] Hz band with a notch filter at 50 Hz and collected at 256 Hz sampling frequency. From the dataset, we extracted two sets of seizure and non-seizure EEG epochs referred to as sets S and N respectively. Each set contains 50 segments. Each segment was band-pass filtered in the range 0.5-10 Hz and down-sampled from 256 to 20 Hz. This was based on findings that neonatal EEG seizures have spectral activities mostly below 12 Hz [92]. The length of each segment is 12.8 s with 256 samples.

Table 6 shows the confusion matrices representing the classification results using the 15 features extracted from different T-F representations of newborn EEG segments. The result between parentheses is the classification result using only the signal-based features {F₁, F₂, ..., F₁₀} [15, 16]. This result is obtained using the multi-class SVM classifier. Each row shows, for a particular TFD, the total number of newborn EEG segments correctly classified as well as those misclassified as another class, and also, the three statistical parameters: sensitivity, specificity and total classification accuracy. Table 6 shows that the classification results using the combined T-F signal & image related feature vector {F_{S 1}, F_{S 2}, ..., F_{S 8}, F_{I 1}, ..., F_{I 5}} are better compared to results using only the T-F signal-related feature vector {F_{S 1}, F_{S 2}, ..., F_{S 8}}. This is confirmed by the total classification accuracy calculated for each TFD. The best total classification accuracy is obtained using the MBD and SPEC; and is in a range (96% to 97%) for 100 segments. This can be improved by increasing the number of EEG segments in the training-step. The results confirm that including T-F image features improves the classification performance.

Table 6 Confusion matrices for newborn EEG classification using T-F features set {F_{S 1}, F_{S 2}, ..., F_{S 8}, F_{I 1}, ..., F_{I 5}} extracted from the TFD

5.3 Seizure localization in newborn EEG signals

The performance of T-F features in EEG abnormalities detection and localization (Figure 4-(b)) is now assessed using real multichannel EEG signals recorded from newborns.

For the simulation, we have chosen only five newborn EEG segments for five channels (namely channel A, B, C, D and E) from the newborn EEG database described in Section 5.2. Figure 7 shows these newborn EEG segments in the time domain with their corresponding magnitude spectra.

Figure 7

An example of real newborn multichannel EEG. Only five channels are shown. These channels are labeled A, B, C, D and E. The channels A, D and E have newborn EEG seizure patterns. This was detected manually by a neurologist and also confirmed using the methodology described in Section 4.3.

For the problem of seizure detection and localization, we assume that the seizure was detected in the newborn EEG signal of channel A and then we want to locate other channels where the EEG signal have the same seizure pattern. We applied the methodology described in Section 4.3 which consists of calculating the correlation between signals in the T-F domain.

Table 7 shows the correlation resulting between channel A and channels {B, C, D, E} in the spatial domain and T-F domain. We have used the correlation coefficient to measure the correlation between sample vectors between newborn EEG signals in the time domain, and also between their corresponding T-F representations using the MBD. The results indicate that a strong correlation exists in the T-F domain between channels A ↦ D and A ↦ E. This implies that the seizure pattern observed in the newborn EEG signal from channel A is also detected in the EEG signals from channels D and E. The results in the time domain does not provide enough significant information for determining the existence of significant correlations between EEG signals from all channels.

Table 7 Correlation between channel A and channels B, C, D and E in the spatial domain and T-F domain

In addition, we notice that only the T-F features vector is able to detect and locate the seizure pattern in different channels. This is confirmed by the Mahalanobis distance[93] for which results are given in Table 8. This distance calculates the correlation/similarity between feature vectors, and is defined as follows:

$$d\left(\overrightarrow{x},\overrightarrow{y}\right)=\sqrt{{\left(\overrightarrow{x}-\overrightarrow{y}\right)}^T{S}^{-1}\left(\overrightarrow{x}-\overrightarrow{y}\right)}$$

(21)

Table 8 Mahalanobis distance between channel A and channels B, C, D and E in the T-F domain using the feature vector

where $\overrightarrow{x}$ and $\overrightarrow{y}$ are two feature vectors and S is the covariance matrix.

The results show also that the correlation is better detected using the complete 15 T-F features vector. This suggests that features based on T-F image processing techniques are able to help characterize better the abnormality patterns in newborn EEG signals, and also to detect and locate them on the scalp.

6 Conclusion and perspectives

This article concludes that it is possible to improve the analysis, classification and localization of non-stationary signals by using a general methodology that combines techniques from three complementary fields: T-F signal analysis, image processing and multichannel analysis. The approach is illustrated on the key problem of newborn EEG abnormality diagnostic and localization for which a solution would help improve health outcomes for newborns. In addition, the same problem was also considered for the case of adults. More precisely, this article reports the results of a study on EEG abnormalities detection and classification in the case of mono-channel T-F analysis first, and then, the EEG abnormalities sources localization in the case of multichannel T-F analysis. The novelties of this article include: (1) a comprehensive review of T-F image processing techniques, and (2) a review of methods to process the T-F representation of EEG signal considered as an image, as well as (3) a novel approach that combines the above with T-F features to characterize the non-stationary newborn EEG signal, following [14–16].

In particular, this article has adapted and applied a segmentation algorithm on the T-F image in order to detect the region where all EEG information appear. Then, new morphometric features were defined based on the geometric shape of the segmented region. Two T-F image classification methods were selected to evaluate the performance of the EEG classification using the selected T-F features. These T-F features were also able to evaluate the correlation between newborn EEG signals in order to locate the abnormality source on the scalp. The experimental results show that the features based on T-F image processing techniques are able to improve the performance of both EEG classification and EEG abnormality source localization.

Finally, the above results allow further progress with a new focus on the adaptation of T-F image processing techniques in order to better use them in the problem considered. This should result in a more advanced design for automatic abnormality diagnosis and localization systems based on a complete multichannel T-F image processing approach. In addition, this work can be further refined by considering the TFD which is most appropriate for this T-F image processing approach following the algorithmic perspective developed in [6, 94–96] and the selection criteria presented in [29].

Endnotes

^aLocal polynomial approximation-relative intersection of confidence intervals (for more detail see [97]). ^bThe compactness is the most popular shape feature, and is obtained by perimeter²/area.