Multicamera multiobject voxelbased Monte Carlo 3D tracking strategies
 Cristian CantonFerrer^{1}Email author,
 Josep R Casas^{1},
 Montse Pardàs^{1} and
 Enric Monte^{1}
DOI: 10.1186/168761802011114
© CantonFerrer et al; licensee Springer. 2011
Received: 15 May 2011
Accepted: 23 November 2011
Published: 23 November 2011
Abstract
This article presents a new approach to the problem of simultaneous tracking of several people in lowresolution sequences from multiple calibrated cameras. Redundancy among cameras is exploited to generate a discrete 3D colored representation of the scene, being the starting point of the processing chain. We review how the initiation and termination of tracks influences the overall tracker performance, and present a Bayesian approach to efficiently create and destroy tracks. Two Monte Carlobased schemes adapted to the incoming 3D discrete data are introduced. First, a particle filtering technique is proposed relying on a volume likelihood function taking into account both occupancy and color information. Sparse sampling is presented as an alternative based on a sampling of the surface voxels in order to estimate the centroid of the tracked people. In this case, the likelihood function is based on local neighborhoods computations thus dramatically decreasing the computational load of the algorithm. A discrete 3D resampling procedure is introduced to drive these samples along time. Multiple targets are tracked by means of multiple filters, and interaction among them is modeled through a 3D blocking scheme. Tests over CLEARannotated database yield quantitative results showing the effectiveness of the proposed algorithms in indoor scenarios, and a fair comparison with other stateoftheart algorithms is presented. We also consider the realtime performance of the proposed algorithm.
1 Introduction
Tracking multiple objects and keeping record of their identities along time in a cluttered dynamic scene is a major research topic in computer vision, basically fostered by the number of applications that benefit from the retrieved information. For instance, multiperson tracking has been found useful for automatic scene analysis [1], humancomputer interfaces [2], and detection of unusual behaviors in security applications [3].
A number of methods for camerabased multiperson 3D tracking have been proposed in the literature [4–7]. A common goal in these systems is robustness under occlusions created by the multiple objects cluttering the scene when estimating the position of a target. Singlecamera approaches [8] have been widely employed, but they are vulnerable to occlusions, rotation, and scale changes of the target. In order to avoid these drawbacks, multicamera tracking techniques exploit spatial redundancy among different views and provide 3D information at the actual scale of the objects in the real world. Integration of data extracted from multiple cameras has been proposed in terms of a fusion at feature level as image correspondences [9] or multiview histograms [10] among others. Information fusion at data or raw level has been achieved by means of voxel reconstructions [11], polygon meshes [12], etc.
Most multicamera approaches rely on a separate analysis of each camera view, followed by a feature fusion process to finally generate an output. Exploiting the underlying epipolar geometry of a multicamera setup toward finding the most coherent feature correspondence among views was first tackled by Mikič et al. [13] using algebraic methods together with a Kalman filter, and further developed by Focken et al. [14]. Exploiting epipolar consistency within a robust Bayesian framework was also presented by CantonFerrer et al. [9]. Other systems rely on detecting semantically relevant patterns among multiple cameras to feed the tracking algorithm as done in [15] by detecting faces. Particle filtering (PF) [16] has been a commonly employed algorithm because of its ability to deal with problems involving multimodal distributions and nonlinearities. Lanz et al. [10] proposed a multicamera PF tracker exploiting foreground and color information, and several contributions have also followed this path: [4, 7]. Occlusions, being a common problem in feature fusion methods, have been addressed in [17] using HMM to model the temporal evolution of occlusions within a PF algorithm. Information about the tracking scenario can also be exploited toward detecting and managing occlusions as done in [18] by modeling the occluding elements, such as furniture, in a training phase before tracking. It must be noted that, in this article, we assume that all cameras will be covering the area under study. Other approaches to multicamera/multiperson tracking do not require maximizing the overlap of the field of view of multiple cameras, leading to the nonoverlapped multicamera tracking algorithms [19].
Multicamera/multiperson tracking algorithms based on a data fusion before doing any analysis was pioneered by Lopez et al. [20] by using a voxel^{a} reconstruction of the scene. This idea was further developed by the authors in [5, 21] finally leading to the present article. Up to our knowledge, this is the first approach to multiperson tracking exploiting data fusion from multiple cameras as the input of the algorithms. In this article, we first introduce a methodology to multiperson tracking based on a colored voxel representation of the scene as the start of the processing chain. The contribution of this article is twofold. First, we emphasize the importance of the initiation and termination of tracks, usually neglected in most tracking algorithms, that has indeed an impact on the performance of the overall system. A general technique for the initiation/termination of tracks is presented. The second contribution is the filtering step where two techniques are introduced. The first technique applies PF to input voxels to estimate the centroid of the tracked targets. However, this process is far from realtime performance and an alternative, that we call Sparse Sampling (SS). SS aims at decreasing computation time by means of a novel tracking technique based on the seminal PF principle. Particles no longer sample the state space but instead a magnitude whose expectancy produces the centroid of the tracked person: the surface voxels. The likelihood evaluation relying on occupancy and color information is computed on local neighborhoods, thus dramatically decreasing the computation load of the overall algorithm. Finally, effectiveness of the proposed techniques is assessed by means of objective metrics defined in the framework of the CLEAR [22] multitarget tracking database. Computational performance is reviewed toward proving the realtime operation of the SS algorithms. Fair comparisons with stateoftheart methods evaluated using the same database are also presented and discussed.
2 Tracker design methodology
2.1 Input and output data
When addressing the problem of multiperson tracking within a multicamera environment, a strategy about how to process this information is needed. Many approaches perform an analysis of the images separately, and then combine the results using some geometric constraints [10]. This approach is denoted as an information combination by fusion of decisions. However, a major issue in this procedure is dealing with occlusion and perspective effects. A more efficient way to combine information is data fusion [23]. In our case, data fusion leads to a combination of information from all images to build up a new data representation, and to apply the algorithms directly on these data. Several data representations aggregating the information of multiple views have been proposed in the literature such as voxel reconstructions [11, 24], level sets [25], polygon meshes [12], conexels [26], depth maps [27], etc. In our research, we opted for a colored voxel representation due to both its fast computation and accuracy.
Redundancy among cameras is exploited by means of a ShapefromSilhouette (SfS) technique [11]. This process generates a discrete occupancy representation of the 3D space (voxels). A voxel is labeled as foreground or background by checking the spatial consistency of its projection on the N_{C} segmented silhouettes, and finally obtaining the 3D binary reconstruction shown in Figure 2(c). We will denote this raw voxel reconstruction as $\mathcal{V}$. The visibility of a surface voxel onto a given camera is assessed by computing the discrete ray originating from its optical center to the center of this voxel using Bresenham's algorithm and testing whether this ray intersects with any other foreground voxel. The most saturated color among pixels of the set of cameras that see a surface voxel is assigned to it. A colored representation of surface voxels of the scene is obtained, denoted as ${\mathcal{V}}^{C}$. An example of this process is depicted in Figure 2(d). It should be taken into account that, without loss of generality, other background/foreground and 3D reconstruction algorithms may be used to generate the input data to the tracking algorithm presented in this article.
The resulting colored 3D scene reconstruction is fed to the proposed system that assigns a tracker to each target and the obtained tracks are processed by a higher semantic analysis module. Information about the environment (dimensions of the room, furniture, etc.) allows assessing the validity of tracked volumes and discarding false volume detections.
Finally, the output of the overall tracking algorithm will be a number of hypotheses for the centroid position of each of the targets present in the scene.
2.2 Tracker state and filtering
One of the major challenges in multitarget tracking is the estimation of the number of targets and their positions in the scene, based on a set of uncertain observations. This issue can be addressed from two perspectives. First, extending the theory of singletarget algorithms to multiple targets. This approach defines the working state space $\mathcal{X}$ as the concatenation of the positions of all N_{T} targets as $\mathcal{X}=\left[{x}_{1},{x}_{2}...{x}_{{N}_{T}}\right]$. The difficulty here is the time variant dimensionality of this space. Monte Carlo approaches, and specifically PF approaches, to this problem have to face the exponential dependency between the number of particles required by the filter and the dimension of $\mathcal{X}$, turning out to be computationally infeasible. Recently, a solution based on random finite sets achieving linear complexity has been presented [29].
Multitarget tracking can also be tackled by tracking each target independently, that is to maintain N_{T} trackers with a state space ${\mathcal{X}}_{i}={x}_{i}$. In this case, the system attains a linear complexity with the number of targets, thus allowing feasible implementations. However, interactions among targets must be modeled in order to ensure the most independent set of tracks. This approach to multiperson tracking will be adopted in our research.
2.3 Track initiation and termination
A crucial factor in the performance of a tracking system is the module that addresses the initiation and termination of tracks. The initiation of a new tracker is independent of the employed filtering technique and only relies on the input data and the current state (position) of the tracks in the scene. On the other hand, the termination of a new tracking filter is driven by the performance of the tracker.
The initialization of a new filter is determined by the correct detection of a person in the analyzed scene. This process is crucial when tracking, and its correct operation will drive the overall system's accuracy. However, despite the importance of this step, little attention is paid to it in the design of multiobject trackers in the literature. Only few articles explicitly mention this process such as [30] that employs a face detector to detect a person or [31] that uses scout particle filters to explore the 3D space for new targets. Moreover, it is assumed that all targets in the scene are of interest, i.e., people, not accounting for spurious objects, i.e., furniture, shadows, etc. In this section, we introduce a method to properly handle the initiation and termination of filters from a Bayesian perspective.
2.3.1 Track initiation criteria
The 3D input data $\mathcal{V}$ fed to the tracking system is usually corrupted and presents a number of inaccuracies such as objects not reconstructed, mergings among adjacent blobs, spurious blobs, etc. Hence, defining a track initialization criterium based solely on the presence of a blob might lead to poor performance of the system. For instance, objects such as furniture might be wrongly detected as foreground, reconstructed and tracked. Instead, a classification of the blobs based on a probabilistic criteria can be applied during this initialization process aiming at a more robust operation. Training of this classifier is based on the development set of the used database, together with the available ground truth describing the position of the tracked objects.
We will consider the region of influence of a target with centroid x as the ellipsoid $\mathcal{E}\left(x,y\right)$ with axis size s = (s_{ x }, s_{ y }, s_{ z }) centered at c.
that is to assign x_{ j }to the component with the largest volume enclosed in the region of influence. It must be noted that some x_{ j }might not have any ${\mathcal{C}}_{i}$ associated due to a wrong segmentation or faulty reconstruction of the target. Moreover, the set of components not associated to any ground truth position can be identified as spurious objects, reconstructed shadows, etc.
Features employed by the person/noperson classifier where magnitude ${\left[\mathcal{V}\right]}_{\left\{x,y,z\right\}}$ denotes the x, y, or z coordinates of voxel $\mathcal{V}$
Feature  Expression 

Weight  $\left{\mathcal{C}}_{i}\right{s}_{v}^{3}\rho $ ρ = 1.1 [gr/cm^{3}] 
Centroid (zaxis)  ${\left{\mathcal{C}}_{i}\right}^{1}\sum _{v\in {\mathcal{C}}_{i}}{\left[\mathcal{V}\right]}_{z}$ 
Top  $\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{z}$ 
Height  $\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{z}\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{z}$ 
Bounding box  $\begin{array}{c}max\left\{\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{x}\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{x},\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{y}\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{y}\right\}\\ min\left\{\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{x}\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{x},\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{y}\underset{\mathcal{V}\in {\mathcal{C}}_{i}}{max}{\left[\mathcal{V}\right]}_{y}\right\}\end{array}$ 
A number of standard binary classifiers has been tested and their performances have been evaluated, namely Gaussian, Mixture of Gaussians, Neural Networks, KMeans, PCA, Parzen and Decision Trees [33, 34]. Due to the aforementioned properties of the statistic distributions of the features, some classifiers are unable to obtain a good performance, i.e., Gaussian, PCA, etc. Other classifiers require a large number of characterizing elements, such as KMeans, MoG, or Parzen. Decision trees [33] have reported the best results. Separable variables such as height, weight, and bounding box size are automatically selected to build up a decision tree that yields a high recognition rate with a precision of 0.98 and a recall of 0.99 in our test database.
Another complementary criterium employed in the initiation of new tracks is based on the current state of the tracker. It will not be allowed to create a new track if its distance to the closest target is below a threshold.
2.3.2 Track termination criteria

If two or more tracks fall too close to one another, this indicates that they might be tracking the same target, hence only one will be kept alive while the rest will be removed.

If tracker's efficiency becomes very low it might indicate that the target has disappeared and should be removed.

The person/noperson classifier is applied to the set of features extracted from the voxels assigned to a target. If the classifier outputs a noperson verdict for a number of frames, the target will be considered as lost.
3 Voxelbased solutions
The filtering block shown in Figure 1 addresses the problem of keeping consistent trajectories of the tracked objects, resolving crossings among targets, mergings with spurious objects (i.e., shadows) and producing an accurate estimation of the centroid of the target based on the input voxel information. Although there is a number of papers addressing the problem of multicamera/multiperson tracking, very few contributions have been based on voxel analysis [20, 21].
3.1 PF tracking
Hence, the weights are proportional to the likelihood function that will be computed over the incoming volume z_{ t }.
Basically, in the PF operation loop two steps must be defined: likelihood evaluation and particles propagation. In the following, we present our proposal for the PF implementation.
3.1.1 Likelihood evaluation
Factor λ controls the influence of each term (foreground and color information) in the overall likelihood function. Empirical tests have shown that λ = 0.8 provides satisfactory results. A more detailed review of the impact of color information in the overall performance of the algorithm is addressed in Section5.1.
Likelihood associated to raw data is defined as the ratio of overlap between the input data ${\mathcal{V}}_{t}$ and the ellipsoid ${\mathcal{E}}_{t}^{j}$ defined by particle ${x}_{t}^{j}$ (see Section 2.3.1)
where ${\mathcal{E}}_{t}^{\stackrel{\u0303}{x}}$ stands for the ellipsoid placed in the centroid estimation ${\stackrel{\u0303}{x}}_{t}$ and α is the adaptation coefficient. In our experiments, α = 0.9 provided satisfactory results.
3.1.2 Particle propagation
The propagation model has been chosen to be a Gaussian noise added to the state of the particles after the resampling step: ${x}_{t+1}^{j}={x}_{t}^{j}+N$. The covariance matrix P corresponding to N is proportional to the maximum variation of the centroid of the target and this information is obtained from the development part of the testing dataset. More sophisticated schemes employ previously learnt motion priors to drive the particles more efficiently [6]. However, this would penalize the efficiency of the system when tracking unmodeled motions patterns and, since our algorithm is intended for any motion tracking, no dynamical model is adopted.
3.1.3 Interaction model
where $k\propto {s}_{x}^{2}$ is the parameter that drives the sensibility of the exclusion zone.
3.2 SS tracking
In the presented PF tracking algorithm, likelihood evaluation can be computationally expensive, thus rendering this approach unsuitable for realtime systems. Moreover, data are usually noisy and may contain merged blobs corresponding to different targets. A new technique, SS, is proposed as an efficient and flexible alternative to PF.
3.2.1 Degree of mass and degree of surfaceness
where $\rho s\left(\mathcal{W}\right)\in \left[0,1\right]$ measures the "degree of surfaceness" of voxel $\mathcal{W}$. Within this context, functions ρ(·) and ρ_{S}(·) might be understood as pseudolikelihood functions and Equations 16 and 15 as a samplebased representation of an estimation problem.
3.2.2 Difference with particle filters
There is an obvious similarity between these representation and the formulation of particle filters but there is a significant difference. While particles in PF represent an instance of the whole body, our samples $\left(\mathcal{W}\in {\mathcal{W}}_{t}\right)$ are points in the 3D space. Moreover, particle likelihoods are computed over all data while sample pseudolikelihoods will be computed in a local domain.
where N_{s} is the number of sampling points. When using SS we are no longer sampling the state space since ${y}_{t}^{i}$ cannot be considered an instance of the centroid of the target as happened with particles, ${x}_{t}^{j}$, in PF. Hence, we will talk about samples instead of particles and we will refer to ${\left\{\left({y}_{t}^{i},{\omega}_{t}^{i}\right)\right\}}_{i=1}^{{N}_{s}}$ as the sampling set. This set will approximate the surface of the k th target, ${\mathcal{V}}^{\mathsf{\text{S}},k}$, and will fulfill the sparsity condition ${N}_{s}\ll \left{\mathcal{V}}^{\mathsf{\text{S}},k}\right.$.
4 SS implementation
In order to define a method to recursively estimate ${\stackrel{\u0303}{x}}_{t}$ from the sampling set ${\left\{\left({y}_{t}^{i},{\omega}_{t}^{i}\right)\right\}}_{i=1}^{{N}_{s}}$, a filtering strategy has to be set. Essentially, the proposal is to follow the PF analysis loop (resampling, propagation, evaluation, and estimation) with some opportune modifications to ensure the convergence of the algorithm.
4.1 Pseudolikelihood evaluation
Partial likelihoods will be computed on a local domain centered in the position ${y}_{t}^{i}$. Let $\mathcal{C}\left({y}_{t}^{i},q,r\right)$ be a neighborhood of radius r over a connectivity q domain on the 3D orthogonal grid around a sample place in a voxel position ${y}_{t}^{i}$. Then, we define the occupancy and color neighborhoods around ${y}_{t}^{i}$ as ${O}_{t}^{i}={\mathcal{V}}_{t}\cap \mathcal{C}\left({y}_{t}^{i},q,r\right)$ and ${C}_{t}^{i}={\mathcal{V}}_{t}^{C}\cap \mathcal{C}\left({y}_{t}^{i},q,r\right)$, respectively.
Ideally, when the sample ${y}_{t}^{i}$ is placed in a surface, half of its associated occupancy neighborhood will be occupied and the other half empty. The proposed expression attains its maximum when this condition is fulfilled.
Since ${C}_{t}^{j}$ contains only local color information with reference of the global histogram ${H}_{t}^{k}$, the distance D(·) is constructed toward giving a measure of the likelihood between this local colored region and ${H}_{t}^{k}$. For every voxel in ${C}_{t}^{j}$, it is decided whether it is similar to ${H}_{t}^{m}$ by selecting the histogram value for the tested color and checking whether it is above a threshold γ or not. Finally, the ratio between the number of similar color and total voxels in the neighborhood gives the color similarity score. Since reference histogram is updated and changes over time, a variable threshold γ is computed, so that the 80% of the values of ${H}_{t}^{m}$ are taken into account.
One of the advantages of the SS algorithm is its computational efficiency. The complexity to compute $p\left({z}_{t}{y}_{t}^{i}\right)$ is quite reduced since it only evaluates a local neighborhood around the sample in comparison with the computational load required to evaluate the likelihood of a particle in the PF algorithm. This point will be quantitatively addressed in Section5.2.
The parameters defining the neighborhood were set to q = 26 and r = 2 yielding to satisfactory results. Larger values of the radius r did not significantly improve the overall algorithm performance but increased its computational complexity.
4.2 Sample propagation and 3D discrete resampling
A sample ${y}_{t}^{i}$ placed near a surface will have an associated weight ${\omega}_{t}^{j}$ with a high value. It is a valid assumption to consider that some surrounding positions might also be part of this surface. Hence, placing a number of new particles in the vicinity of ${x}_{t}^{j}$ would contribute to progressively explore the surface of a voxel set. This idea leads to the spatial resampling and propagation scheme that will drive samples along time in the surface of the tracked target.
4.2.1 Interaction model
The flexibility of a samplebased analysis may, sometimes, lead to situations where particles spread out too much from the computed centroid. In order to cope with this problem, a intratarget samples' interaction model is devised. If a sample is placed in a position such that $\u2225{\left[{y}_{t}^{i}\right]}_{x,y}\left[{\stackrel{\u0303}{x}}_{t1}\right]x,y\u2225>\delta $ it will be removed (that is to assign ${\omega}_{t}^{i}=0$) and we set the threshold as δ = αs_{ x }, with s_{ x }= 30 cm. Factor α = 1.5 produced accurate results in our experiments.
The interaction among targets is modeled in similar way as in the PF approach. Formulas in Equations 11 and 12 are applied to samples with the appropriate scaling parameter k.
5 Results and evaluation
Metrics proposed in [4] for multiperson tracking evaluation have been adopted, namely the Multiple Object Tracking Precision (MOTP), which shows tracker's ability to estimate precise object positions, and the Multiple Object Tracking Accuracy (MOTA), which expresses its performance at estimating the number of objects, and at keeping consistent trajectories. MOTP scores the average metric error when estimating multiple target 3D centroids, while MOTA evaluates the percentage of frames where targets have been missed, wrongly detected or mismatched.
The aim of a tracking system would be to produce high values of MOTA and low values of MOTP thus indicating its ability to correctly track all targets and estimate their positions accurately. When comparing two algorithms, there will be a preference to choose the one outputting the highest MOTA score.
5.1 Results
To demonstrate the effectiveness of the proposed multiperson tracking approaches, a set of experiments were conducted over the CLEAR 2007 database. The development part of the dataset was used to train the initiation/termination of tracks modules as described in Section 2.3 and the remaining test part was used for our experiments.
First, the multicamera data are preprocessed performing the foreground and background segmentations and 3D voxel reconstruction algorithm. In order to analyze the dependency of the tracker's performance with the resolution of the 3D reconstruction, several voxel sizes were employed ${s}_{v}=\left\{2,5,10,15\right\}$ cm. A colored version of these voxel reconstructions was also generated, according to the technique introduced in Section 2.1. Then, these data were the input fed to the PF and SS proposed approaches.

Number of samples/particles: There is a dependency between the MOTP score and the number of particles/samples, especially for the SS algorithm. The contribution of a new sample to the estimation of the centroid in the SS has less impact than the addition of a new particle in the PF, hence the slower decay of the MOTP curves for the SS than for the PF. Regarding the MOTA score, there is not a significant dependency with N_{s} or N_{p}. Two factors drive the MOTA of an algorithm: the track initiation/termination modules, that mainly contributes to the ratio of misses and false positives, and the filtering step that has an impact to the mismatches ratio. The low dependency of MOTA with N_{s} or N_{p} shows that most of the impact of the algorithm in this score is due to the particle/sample propagation and interaction strategies rather than the quantity of particles/samples itself. Moreover, the influence in the MOTA score is tightly correlated with the track initiation/termination policy. This assumption was experimentally validated by testing several classification methods (mixture of Gaussians, PCA, Parzen, and KMeans) in the initiation/termination modules yielding to a drop in the MOTA score proportional to their ability to correctly classify a blob as person/noperson.

Voxel size: Scenes reconstructed with a large voxel size do not capture well all spatial details and may miss some objects thus decreasing the performance of the system (both in SS and PF). It can be observed that MOTP and MOTA scores improve as the voxel size decrease.

Color features: Color information improves the performance of SS and PF in both MOTP and MOTA scores. First, there is an improvement when using color information for a given voxel size, specially for the SS algorithm. Moreover, the smaller the voxel size the most noticeable difference between the experiments using raw and color features. This effect is supported by the fact that color characteristics are better captured when using small voxel sizes. The performance improvement when using color in the SS algorithm is more noticeable since samples are placed in the regions with a high likelihood to be part of the target. For instance, this effect is more evident in cases where the subject is sitting and the particles concentrate in the upper body part, disregarding the part of the chair. In the SS algorithm, MOTP score benefits from this efficient sample placement. PF algorithm is constrained to evaluate the color likelihood in the ellipsoid defined in Equation 9 thus not being able to differentiate between parts of the blob that do not belong to the tracked target. Color information used within the filtering loop leads to a better distinguishability among blobs, thus reducing the mismatch ratio and slightly improving the MOTA score. Merging of adjacent blobs or complex crossing among targets is also correctly resolved. An example of the impact of color information is shown in Figure 10 where the usage of color avoids the mismatch between two targets. This effect is more noticeable when targets in the scene are dressed in different colors.
Results presented at the CLEAR 2007 [22] by several partners
Method  MOTP  MOTA  FP  Miss  MM 

(mm)  (%)  (%)  (%)  (cases)  
Face detection+Kalman filtering [15]  91  59.66  06.99  30.89  2.46 
Appearance models+PF [7]  141  59.62  18.58  20.66  1.14 
Upper body detection+PF [31]  155  69.58  14.50  15.09  0.83 
Zenithal camera analysis+PF [31]  222  54.94  20.24  23.74  1.08 
Voxel analysis+Heuristic tracker [5]  168  30.49  40.19  27.74  1.58 
Voxel analysis+PF (best case)  147  74.56  14.03  10.48  0.91 
Voxel analysis+SS (best case)  144  81.50  09.34  08.70  0.46 
In order to visually show the performance of the SS algorithm, some videos corresponding to the most challenging tracking scenarios have been made available at http://www.cristiancanton.org.
5.2 Computational performance
Comparing obtained metrics among different algorithms can give an idea about their performance in a scenario where computational complexity is not taken into account. An analysis of the operation time of several algorithms under the same conditions and the produced MOTP/MOTA metrics might give a more informative and fairer comparison tool. Although there is not a standard procedure to measure the computational performance of a tracking process, we devised a method to assess the computational efficiency of our algorithms to present a comparative study.
The first noticeable characteristic of these charts is that, due to the computational complexity of each algorithm, when comparing SS and PF algorithms under the same operation conditions, the RTF associated with SS is always higher than the associated with PF. Similarly, the computational load is higher when analyzing colored than raw inputs. All the plotted curves attain lower RTF performance values as the size of the voxel ${s}_{v}$ decreases since the amount of data to process increases (note the different RTF scale ranges for each voxel size in Figure 11). Regarding the MOTP/MOTA metrics, there is a common tendency to a decrease in the MOTP and an increase in the MOTA as the RTF decreases. The separation between the SS and PF curves is bigger as the voxel size decreases since the PF algorithm has to evaluate a larger amount of data.
The observation of these results yields the conclusion that the SS algorithm is able to produce a similar and, in some cases, better results than the PF algorithm with a lower computational cost. For example, using ${s}_{v}=5$ cm, a MOTP score of around 165 mm can be obtained using SS with a RTF ten times larger than when using PF and similarly with the MOTA score.
6 Conclusions
In this article, we have presented a number of contributions to the multiperson tracking task in a multicamera environment. A block representation of the whole tracking process allowed to identify the performance bottlenecks of the system and address efficient solutions to each of them. Realtime performance of the system was a major goal hence efficient tracking algorithms have been produced as well as an analysis of their performance.
The performance of these systems has thoroughly been tested over the CLEAR database and quantitatively compared through two scores: MOTP and MOTA. A number of experiments have been conducted toward exploring the influence of the resolution of the 3D reconstruction and the color information. Results have been compared with other stateoftheart algorithms evaluated with the same metrics using the same testing data.
The relevance of the initiation and termination of filters have been proved, since these modules have a major impact on the MOTA score. However, most articles in the literature do not specifically address the operation of these modules. We proposed a statistical classifier based on classification trees as a way to discriminate blobs between the person/noperson classes. Training of this classifier was done using data available in the development part of the employed database and a number of features (namely weight, height, top in zaxis, bounding box size) were extracted and provided as the input to the classifier. Another criterium such as a proximity to other already existing tracks was employed to create or destroy a track. Performance scores in Table 2 for the PF and SS systems present the lowest values for the false positives (FP) and missed targets (Miss) ratios hence supporting the relevance of the initiation and termination of tracks modules.
Two proposals for the filtering step of the tracking system have been presented: PF and SS. An independent tracker was assigned to every target and an interaction model was defined. PF technique proved to be robust and leaded to stateoftheart results but its computational load was unaffordable for small voxel sizes. As an alternative, SS algorithm has been presented achieving a similar and, in some occasions, better performance than PF at a smaller computational cost. Its samplebased estimation of the centroid allowed a better adaptation to noisy data and distinguishability among merged blobs. In both PF and SS, color information provided a useful cue to increase the robustness of the system against track mismatches thus increasing the MOTA score. In the SS, color information also allowed a better placement of the samples allowing to distinguish among parts belonging to the tracked object and parts of a merging with a spurious object, leading to a better MOTP score.
Future research within this topic involves multimodal data fusion with audio data toward improving the precision of the tracker, MOTP, and avoid mismatches among targets, thus improving the MOTA score.
End notes
^{a}Analogously to the pixel definition (picture element) as the minimum information unit in a discrete image, the voxel (volume element) is defined as the minimum information unit in a 3D discrete representation of a volume.
^{b}For the sake of simplicity in the notation, pseudolikelihood functions will be denoted as p(·) instead of defining a specific notation for it.
^{c}When selecting the best system, the MOTA score is regarded as the most significant value.
The authors declare that they have no competing interests
Declarations
Authors’ Affiliations
References
 Park S, Trivedi MM: Understanding human interactions with track and body synergies captured from multiple views. Comput Vis Image Understand 2008,111(1):220. 10.1016/j.cviu.2007.10.005View ArticleGoogle Scholar
 Project CHILComputers in the Human Interaction Loop2004. [http://chil.server.de]
 Haritaoglu I, Harwood D, Davis LS: W^{ 4 }: realtime surveillance of people and their activities. IEEE Trans Pattern Anal Mach Intell 2000,22(8):809830. 10.1109/34.868683View ArticleGoogle Scholar
 Bernardin K, Elbs A, Stiefelhagen R: Multiple object tracking performance metrics and evaluation in a smart Room environment. Proceedings of IEEE International Workshop on Visual Surveillance 2006.Google Scholar
 CantonFerrer C, Salvador J, Casas JR: Multiperson tracking strategies based on voxel analysis. In Proceedings of Classification of Events, Activities and Relationships Evaluation and Workshop. Volume 4625. Lecture Notes on Computer Science; 2007:91103.Google Scholar
 Khan Z, Balch T, Dellaert F: Efficient particle filterbased tracking of multiple interacting targets using an MRFbased motion model. Proceedings of International Conference on Intelligent Robots and Systems 2003,1(1):254259.Google Scholar
 Lanz O, Chippendale P, Brunelli R: An appearancebased particle filter for visual tracking in smart rooms. In Proceedings of Classification of Events, Activities and Relationships Evaluation and Workshop. Volume 4625. Lecture Notes on Computer Science; 2007:5769.Google Scholar
 Yilmaz A, Javed O, Shah M: Object tracking: a survey. ACM Comput Surv 2006,38(4):145.View ArticleGoogle Scholar
 CantonFerrer C, Casas JR, Pardàs M: Towards a Bayesian approach to robust finding correspondences in multiple view geometry environments. In Proceedings of 4th International Workshop on Computer Graphics and Geometric Modelling. Volume 3515. Lecture Notes on Computer Science; 2005:281289.Google Scholar
 Lanz O: Approximate Bayesian multibody tracking. IEEE Trans Pattern Anal Mach Intell 2006,28(9):14361449.View ArticleGoogle Scholar
 Cheung GKM, Kanade T, Bouguet JY, Holler M: A real time system for robust 3D voxel reconstruction of human motions. IEEE Conference on Computer Vision and Pattern Recognition 2000, 2: 714720.Google Scholar
 Isidoro J, Sclaroff S: Stochastic refinement of the visual hull to satisfy photometric and silhouette consistency constraints. Proceedings of IEEE International Conference on Computer Vision 2003, 2: 13351342.View ArticleGoogle Scholar
 Mikič I, Santini S, Jain R: Tracking objects in 3D using multiple camera views, in. Proceedings of Asian Conference on Computer Vision 2000.Google Scholar
 Focken D, Stiefelhagen R: Towards visionbased 3D people tracking in a Smart Room. Proceedings of IEEE International Conference on Multimodal Interfaces 2002, 400405.View ArticleGoogle Scholar
 Katsarakis N, Talantzis F, Pnevmatikakis A, Polymenakos L: The AIT 3D audiovisual person tracker for CLEAR 2007. In Proceedings of Classification of Events, Activities and Relationships Evaluation and Workshop. Volume 4625. Lecture Notes on Computer Science; 2007:3546.Google Scholar
 Arulampalam MS, Maskell S, Gordon N, Clapp T: A tutorial on particle filters for online nonlinear/nonGaussian Bayesian tracking. IEEE Trans Signal Process 2002,50(2):174188. 10.1109/78.978374View ArticleGoogle Scholar
 Lien K, Huang C: Multiviewbased cooperative tracking of multiple human objects. EURASIP J. Image Video Process 2008,8(2):113.View ArticleGoogle Scholar
 Osawa T, Wu X, Sudo K, Wakabayashi K, Arai H: MCMC based multibody tracking using full 3D model of both target and environment. Proceedings of IEEE Conference on Advanced Video and Signal Based Surveillance 2007, 224229.Google Scholar
 Black J, Ellis T, Rosin P: Multi view image surveillance and tracking. Proceedings of Workshop on Motion and Video Computing 2002, 169174.Google Scholar
 López A, CantonFerrer C, Casas JR: Multiperson 3D tracking with particle filters on voxels. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing 2007, 1: 913916.Google Scholar
 CantonFerrer C, Sblendido R, Casas JR, Pardàs M: Particle Filtering and sparse sampling for multiperson 3D tracking. Proceedings of IEEE International Conference on Image Processing 2008, 26442647.Google Scholar
 CLEARClassification of Events, Activities and Relationships Evaluation and Workshop2007. [http://www.clearevaluation.org]
 Hall DL, McMullen SAH: Mathematical Techniques in Multisense Data Fusion. Artech House 2004.Google Scholar
 Kutulakos KN, Seitz SM: A theory of shape by space carving. Int J Comput Vis 2000,38(3):199218. 10.1023/A:1008191222954View ArticleMATHGoogle Scholar
 Faugeras O, Keriven R: Variational principles, surface evolution, PDE's, level set methods and the stereo problem. Proceedings of 5nd IEEE EMBS International Summer School on Biomedical Imaging 2002.Google Scholar
 Casas JR, Salvador J: Imagebased multiview scene analysis using conexels. Proceedings of HCSNet Workshop on Use of Vision in HumanComputer Interaction 2006, 1928.Google Scholar
 Kolmogorov V, Zabin R: What energy functions can be minimized via graph cuts? IEEE Trans Pattern Anal Mach Intell 2004,26(2):147159. 10.1109/TPAMI.2004.1262177View ArticleGoogle Scholar
 Stauffer C, Grimson W: Adaptive background mixture models for realtime tracking, in. Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition 1999, 252259.Google Scholar
 Maggio E, Piccardo E, Regazzoni C, Cavallaro A: Particle PHD filtering for multitarget visual tracking. Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing 2007, 1: 11011104.Google Scholar
 Talantzis F, Pnevmatikakis A, Constantinides AG: Audiovisual active speaker tracking in cluttered indoors environments. IEEE Trans Syst Man Cybern B 2008,38(3):799807.View ArticleGoogle Scholar
 Bernardin K, Gehrig T, Stiefelhagen R: Multilevel particle filter fusion of features and cues for audiovisual person tracking. In Proceedings of Classification of Events, Activities and Relationships Evaluation and Workshop. Volume 4625. Lecture Notes on Computer Science; 2007:7081.Google Scholar
 Tuckey JW: Exploratory Data Analysis. AddisonWesley 1977.Google Scholar
 Breiman L, Friedman JH, Olshen RA, Stone CJ: Classification and Regression Trees. Chapman and Hall 1993.Google Scholar
 Duda RO, Hart PE, Stork DG: Pattern Classification. WileyInterscience 2000.Google Scholar
 Crisco JJ, McGovern RD: Efficient calculation of mass moments of intertia for segmented homogeneous threedimensional objects. J Biomech 1998,31(1):97101.View ArticleGoogle Scholar
 Leu JG: Computing a shape's moments from its boundary. Pattern Recogn 1991,24(10):116122.MathSciNetView ArticleGoogle Scholar
 Yang L, Albregtsen F: Fast and exact computation of Cartesian geometric moments using discrete Green's theorem. Pattern Recogn 1996,29(7):10611073. 10.1016/00313203(95)001476View ArticleGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.