Open Access

Shadow detection in color aerial images based on HSI space and color attenuation relationship

EURASIP Journal on Advances in Signal Processing20122012:141

DOI: 10.1186/1687-6180-2012-141

Received: 25 October 2011

Accepted: 11 June 2012

Published: 12 July 2012

Abstract

Many problems in image processing and computer vision arise from shadows in a single color aerial image. This article presents a new algorithm by which shadows are extracted from a single color aerial image. Apart from using the ratio value of the hue over the intensity in some state-of-the-art algorithms, this article introduces another ratio map, which is obtained by applying the saturation over the intensity. Candidate shadow and nonshadow regions are separated by applying Otus’s thresholding method. The color attenuation relationship that describes the relationship between the attenuation of each color channel is derived from the Planck’s blackbody irradiance law. For each region, the color attenuation relationship and other determination conditions are performed iteratively to segment it into smaller sub-regions and to identify whether each sub-region is a true shadow region. Compared with previous methods, the proposed algorithm presents better shadow detection accuracy in the images that contain some dark green lawn, river, or low brightness shadow regions. The experimental results demonstrate the advantage of the proposed algorithm.

Keywords

Color aerial image Color attenuation relationship HSI space Shadow detection

1 Introduction

Shadows in digital images are either helpful or troublesome in image processing and pattern recognition. The shadows in an aerial image provide visible evidence of the existence of objects. The shadows can be used to recognize and track object in video surveillance and estimate the height and/or position of buildings. However, the existence of shadows also causes some undesirable problems. For example, the shadows may cause objects to merge or shapes to distort, thus resulting in information loss or distortion of objects [1, 2]. On one hand, the shadows attached to some detected objects will misclassify the objects and their shadows as a totally erroneous object in the image. On the other hand, the shape distortion makes the segmentation method less reliable. Therefore, it is of great practical significance to detect shadows in an aerial image.

Shadow detection methods are classified into two types: the method based on features and the method based on models [3]. The first type of shadow detection method, feature-based method, uses the intensity values, chromaticity information, or geometric characteristics to detect the shadows. For example, if the intensity value of a region is lower than that of the pixels around, the region is detected as a shadow region in gray aerial images. Some algorithms have been presented for this kind of images [46]. However, here arises the problem, for some nonshadow regions with low-intensity surface features may be identified as shadows, such as black cars or buildings. Color images are thus introduced in the shadow detection. The chromaticity information in color aerial images is used to improve the shadow detection accuracy. Finlayson et al. [710] have proposed a method to locate the shadows in a single RGB image by using an invariant color model. In this method, the scene is assumed to be lit by Planckian light, the surface of the objects in the image is assumed to be Lambertian, and the image is regarded to be captured by a narrow-band camera. Thus, it may not work well for color aerial images because the condition may not be satisfied. Tsai [11] analyzed the intensity and color properties of shadows in color aerial images, based on color space such as HSI, HSV, YCbCr, HCV, and YIQ, and brought forward a method by calculating the ratio of hue over the intensity for each pixel to build a ratio map. Then, the ratio map is used with thresholding method to extract shadow regions. But, this method tends to misclassify the dark blue and dark green color surfaces as shadow regions. Chung et al. [1] proposed a successive thresholding scheme for detecting shadows in color aerial images. This algorithm has better shadow detection accuracy compared with that of Tsai’s, because the image segmentation is performed locally using different thresholds. The second type of shadow detection method, model-based method, needs some prior knowledge about the scene, such as sun altitude [12], sun angle [13], or NDVI index [14]. However, it is difficult to obtain the measurement for an arbitrary scene since the sunlight changes at different hours and at different places. Makarau [15] and Tian [16] use a blackbody radiator model for shadow detection. Their algorithms are fully motivated by the physical process of shadow formation. They need the prior measurement of the color temperature or the center wavelength for the sensors of each channel.

This article is inspired by Tsai [11], Chung et al. [1], and Tian [16]. Instead of using only one ratio map proposed in their articles, we propose a novel method for shadow detection in color aerial images, which uses two ratio maps plus color attenuation relationship derived from blackbody radiator model. The new ratio map is obtained by applying the saturation over the intensity. The color attenuation relationship describes the relationship between the value attenuation in R and B channels. And the attenuation is compared with shadow sub-region pixels in the proposed method, rather than compared with nonshadow sub-region pixels in Tian’s study [16], although the proposed and the Tian’s studies are both derived from blackbody radiator model. This is because, in this study, we suppose a tested sub-region to be a shadow sub-region first, and we can judge whether this guess holds true or not by other properties. In Tian’s study, the pixels whose values are larger than the mean value are taken as the nonshadow pixels. However, their study sometimes meets problems: although their algorithm is automatic and simple, it depends more or less upon the accuracy of the priori segmentation result and the global thresholds selection. The experiment results show subjective and objective evaluations of six tested images, and prove that our proposed method has better shadow detection performance than the algorithms proposed by Tsai and Chung et al.

In general, the result of shadow detection is followed by the shadow region removal process. Methods for shadow removal in a single image can be found in [2, 1719]. In this article, we focus on the detection of shadows.

The rest of the article is organized as follows: In the following section, the shadow detection algorithm proposed by Chung et al. is reviewed. In the section “Our proposed shadow detection algorithm”, our proposed HSI space and color attenuation relationship-based algorithm is offered. In the section “Experiment results”, the performance comparison among our proposed algorithm, Chung et al.’s algorithm, and the algorithm of Tsai is presented, and finally the article is ended with conclusion section.

2 Previous shadow detection work by Chung et al

To detect shadows in a color aerial image, Chung et al. transform the input image from RGB color model into HSI color model, because the HSI color model has the best shadow detection performance among five invariant color models [11]. Chung et al. make some changes in the calculation of hue (H) in traditional HSI color model, and form the following equation:
I V 1 V 2 = 1 3 1 3 1 3 6 6 6 6 6 3 1 6 2 6 0 R G B S = V 1 2 + V 2 2 H = tan 1 V 2 V 1 + π × 255 2 π
(1)
Hence, the values of I and H are bounded in the range [0, 255]. The ratio map RM is defined by:
R M ( x , y ) = r o u n d H ( x , y ) I ( x , y ) + 1
(2)
where the denominator can avoid being dividing by zero. In order to enlarge the RM value gap between shadow and nonshadow pixels near a threshold T s , the modified ratio map RM' is defined by
R M ( x , y ) = { e ( R M ( x , y ) T S ) 2 4 σ 2 × 255 , i f R M ( x , y ) < T S 255 , o t h e r w i s e .
(3)

In their implementation, T s is determined when the condition i = 0 Ts P ( i ) = P S is held, where P(i) denotes the probability of the ratio value i in RM, and σ is calculated by i = 0 T S 1 P ( i ) ( i T S ) 2 . The value of P s is set to 0.95, empirically.

Based on the modified ratio map, a global thresholding process is performed to separate the input image into candidate shadow and nonshadow pixels. The connected component process [20] is performed to identify the candidate shadow regions. Then, the local thresholding process is performed iteratively to each candidate shadow region to detect true shadow pixels. After that, a fine-shadow determination process is presented to extract true shadows from candidate shadows. Furthermore, the remaining candidate shadows are enforced to be the nonshadows.

However, such detection is not accurate in the images that contain some dark green areas or low brightness shadow regions.

3 Our proposed shadow detection algorithm

In this section, we propose a novel algorithm to detect shadows from a single color aerial image. We use two ratio maps, which are the ratio value of the hue over the intensity and the ratio value of saturation over the intensity, to obtain candidate shadow and nonshadow regions. The two different regions are constructed by applying Otus’s thresholding method. Color attenuation relationship that describes the relationship of the attenuation between each color channel is derived based on Planck’s blackbody irradiance law. The color attenuation relationship and other determination conditions are performed iteratively to segment each region into smaller sub-regions. Subsequently, whether each sub-region is a true shadow region is identified.

A. Hue singularity and two ratio maps

In the algorithm by Chung et al., the hue (H) is defined by (1). However, when the values of R, G, and B in a pixel are the same or the value of R + G + B is near zero, the value of V1 in (1) is zero or near zero. The hue singularity will occur [21]. H will change dramatically if it is affected by noise. This may result in undesirable shadow region detection. For instance, in Figure 1a–c, an input color aerial image, its modified ratio map obtained by Chung et al.’s algorithm, and its shadow detection result are shown, respectively. Note that, in Figure 1c, the pure white pixels are the detected shadow pixels. However, the shadow pixels on the up-left part of the image are not detected, because the pixels in this region have similar R, G, B values. Therefore, only a small part of the pixels are detected as shadow pixels. Most pixels in this region are identified as nonshadow pixels.
https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig1_HTML.jpg
Figure 1

Shadow detection results of Chung et al.’s algorithm. (a) Input color aerial image. (b) Chung et al.’s modified ratio map of (a). (c) Shadow detection result of (b).

In order to overcome the problem mentioned above, we use the HSI color model proposed by Gonzalez et al. [20] to make sure that the hue singularity only occurs when R = G = B. The HSI space transformation of our proposed method is defined in formula (4). We scale H and S to the range [0, 255], and obtain H e  = H × 255/360, and S e  = S × 255.
H = { θ i f B G 360 θ i f B > G θ = arccos 1 2 ( R G ) + ( R B ) ( R G ) 2 + ( R B ) ( G B ) 1 / 2 S = 1 3 R + G + B min ( R , G , B ) I = 1 3 R + G + B
(4)

We extract the singular pixels with R = G = B or R + G + B < Tsum from the original image. They will be classified into shadow regions or nonshadow regions when the other pixels in the image are identified. Tsum is set to 3, empirically.

The saturation in the shadow areas is higher than that in the nonshadow areas due to the Rayleigh scattering phenomenon in the atmosphere [22]. For this reason, we propose another ratio map to identify shadow regions. The value of each pixel in the new ratio map is defined as its saturation over intensity. The two ratio maps applied in our method are given by (5).
R M H ( x , y ) = r o u n d H e ( x , y ) I ( x , y ) + 1 R M S ( x , y ) = r o u n d S e ( x , y ) I ( x , y ) + 1
(5)

We also adopt the modified ratio map defined in (3) to calculate modified ratio maps R M H ( x , y ) and R M S ( x , y ) in our proposed method. A denoising filter [23] is applied to both modified ratio maps to alleviate the noise effect and at the same time protect the edges between shadow and nonshadow regions.

From the modified ratio map R M H ( x , y ) and R M S ( x , y ) , Otsu’s thresholding method [24] is applied to determine a threshold T(T RH or T RS ), which separates the pixels in R M H ( x , y ) and R M S ( x , y ) into two classes: candidate shadow and nonshadow pixels. The threshold T is defined by:
T = arg min T i = 0 T P ( i ) ( i μ 1 ) 2 + i = T + 1 255 P ( i ) ( i μ 2 ) 2
(6)

where μ 1 = i = 0 T i P ( i ) / W 1 , W 1 = i = 0 T P ( i ) , and μ 2 = i = T + 1 255 i P ( i ) / W 2 , W 2 = i = T + 1 255 P ( i ) , and P(i) is the probability of pixels with gray level i in the modified ratio map R a t i o H ( x , y ) or R a t i o S ( x , y ) .

T RH and T RS are the thresholds for the modified ratio maps R M H ( x , y ) and R M S ( x , y ) respectively. Based on the thresholds, candidate shadow pixels C P = C P H C P S are obtained, where CP H and CP S are the candidate shadow pixels determined by
C P H ( x , y ) = { 1 , R M H ( x , y ) > T RH 0 , o t h e r w i s e C P S ( x , y ) = { 1 , R M S ( x , y ) > T RS 0 , o t h e r w i s e
(7)

CP(x, y) = 1 indicates that the pixel at position (x, y) is a candidate shadow pixel, while CP(x, y) = 0 means that the pixel at that position is a candidate nonshadow pixel. Based on the candidate shadow and nonshadow pixels, the candidate shadow and nonshadow regions are identified by using the connecting component analysis.

B. Color attenuation relationship

The simplest way to judge a candidate region comes from the following two basic features of shadow region: the intensity is lower than that of the neighboring pixels and the chromaticity in shadow region is similar to that of its neighboring pixels [1]. However, mere consideration of these two features will sometimes lead to the misclassification of shadow regions. In this article, the relationship between the attenuation of R component and that of B component in shadow regions is applied to distinguish shadow and nonshadow regions.

When the sunlight passes through the atmosphere, it undergoes a process of being absorbed by the atmosphere and scattering in the air. Rayleigh scattering occurs when the diameter of the particle is much smaller than the wavelength of the light. Therefore, we can infer that the blue light, whose wavelength is shorter, will scatter in a much greater angle than red and green lights. The main illumination of shadow regions is the scattered blue light from the atmosphere.

The phenomenon is considered in detecting shadow pixels whose blue component is larger than red component and whose intensity in a color aerial image is low [25]. However, with this consideration, only spectral irradiance factor is considered, while surface reflecting factor is neglected.

In this article, we derive, under some assumptions, the relationship between the attenuation of R(ΔR) and the attenuation of B(ΔB) in shadow regions, which satisfies formula (8) from Planck’s blackbody irradiance law. Here, ΔR represents the difference between the intensity of the R channel in a candidate region when it is covered by shadow and when it is not covered by shadow. While ΔB represents the difference between the intensity of the B channel in the same candidate region when it is covered by shadow and when it is not covered by shadow.
Δ R Δ B = 1.70 × ρ S R ρ S B
(8)

where ρ S R is the R-channel intensity value and ρ S B is B-channel intensity value, in a given pixel in shadow. The relationship is derived later in the Appendix.

Since the color temperature of the sunlight and the skylight is changing from time-to-time, we set a range to the relationship through the subsequent experiments:
1.30 × ρ S R ρ S B Δ R Δ B 2.10 × ρ S R ρ S B
(9)

We can also derive the relationship between ΔR and ΔG or ΔG and ΔB, but we only use the relationship between ΔR and ΔB in the following steps for simplicity. Now, we do not know whether the candidate region is a shadow region or not. We can regard the tested candidate region as a shadow region, and consider its neighboring pixels as nonshadow region. We define the neighboring pixels as 5 pixels wide around the candidate region exclusive of the pixels that have been detected as shadow pixels. Therefore, we will be able to judge whether the tested candidate region is a shadow region by using formula (9).

C. Our proposed method

Although we have separated the original image into candidate shadow and nonshadow regions, there are also some shadow sub-regions in the candidate nonshadow regions, and some nonshadow sub-regions in the candidate shadow regions. Therefore, we should distinguish true shadow regions from these two candidate regions.

After extracting the hue singularity pixels from the original image and classifying the pixels in the image as candidate shadow and nonshadow regions, the following two properties are useful to distinguish true shadow regions from candidate shadow and nonshadow regions.

Property 1: A shadow region usually has lower mean intensity values than its neighboring pixels.

Property 2: A shadow region usually has similar chromaticity values to its neighboring pixels.

These two properties together with the color attenuation relationship can be performed iteratively to identify whether each sub-region is a true shadow region or not.

Assume we have m candidate shadow regions and n candidate nonshadow regions. The mean intensity value of i th candidate shadow region is μ P CS i I , and the mean intensity value of j th candidate nonshadow region is μ P CNS j I . The mean intensity values of their neighboring pixels are μ P CS i , N I and μ P CNS j , N I , respectively. In order to satisfy Property 1, the relationship of the mean intensity values can be illustrated in Figure 2.
https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig2_HTML.jpg
Figure 2

Relationship of the mean intensity values.

Since the candidate shadow regions satisfy the shadow detection condition of the modified ratio maps, the threshold T I CS can be set to be a small value. However, the threshold value for nonshadow regions should be larger, because the pixels in candidate nonshadow regions are more likely to be nonshadow pixels. In order to get optimal threshold for shadow detection, we have tried different thresholds for the images in the section “Experiment results”. Figure 3 presents the difference of the detection accuracy of the six testing images under different thresholds for T I CS and T I CNS . As illustrated in Figure 3, T I CS = 25 and T I CNS = 40 will be the optimal thresholding values for the testing images. If the thresholds T I CS and T I CNS are too small, many nonshadow regions will be regards as shadow regions, because the intensity difference between them is not obvious. However, if the values of the thresholds are too large, some true shadow regions will not be detected from candidate nonshadows regions.
https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig3_HTML.jpg
Figure 3

Optimal thresholds selection. (a) Detection accuracy with different thresholds for T I CS . (b) Detection accuracy with different thresholds for T I CNS .

In order to satisfy Property 2, the relationship of the chromaticity values of the candidate regions can be illustrated in Figure 4. where μ P CS i H e and μ P CS i , N H e denote the mean H e value of i th candidate shadow region and that of its neighboring pixels, μ P CNS j H e and μ P CNS j , N H e denote the mean H e value of j th candidate nonshadow region and that of its neighboring pixels; σ P CS i H e and σ P CS i , N H e denote the standard deviation of H e value of i th candidate shadow region and that of its neighboring pixels, σ P CNS j H e and σ P CNS j , N H e denote the standard deviation of H e value of j th candidate nonshadow region and that of its neighboring pixels.
https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig4_HTML.jpg
Figure 4

Relationship of the chromaticity values.

The thresholds T μ C S H e and T σ C S H e are set as 12 and 8, T μ C N S H e and T σ C N S H e are set as 9 and 6, empirically. The selection of thresholds is based on similar experiments as is shown in Figure 3. The application of these thresholds is to determine whether the mean chromaticity value and standard deviation of each candidate region are similar to those of the neighboring pixels. As for smaller thresholds, more candidate regions can be identified as nonshadow regions, but some true shadow regions may be identified as nonshadow regions. However, if the values of the thresholds are too great, some true shadow regions will not be detected from candidate nonshadows regions.

If the mean intensity value and chromaticity value of a candidate region satisfy the relationship illustrated above and if the color attenuation relationship satisfies formula (9), the candidate region can be classified as a shadow region. Otherwise, the region should be separated into smaller sub-regions by Otsu’s thresholding method, and then the sub-regions should be further tested under the conditions mentioned above. The iteration is executed, until the standard deviation of the sub-region is smaller than a threshold T σ , which is set to 5, empirically.

Our multi-step algorithm for detecting shadow in a color aerial image is as follows:

Step 1 Extract the hue singularity pixels that R = G = B or R + G + B near zero from an image, and Q is a set to store these pixels.

Step 2 Transfer the color space from RGB color model to HSI color model by formula (4), and construct two ratio maps by formula (5).

Step 3 Modify the ratio maps by formula (3), from which two ratio maps are obtained: R M H ( x , y ) and R M S ( x , y ) .

Step 4 Apply a denoising filter [23] to both R M H ( x , y ) and R M S ( x , y ) so as to alleviate the noise effect.

Step 5 Segment the image into candidate shadow and nonshadow pixels by formula (6). Apply the well-known Otsu’s thresholding method and formula (7) to both the modified ratio maps.

Step 6 Conduct a connected component analysis to classify the candidate shadow and nonshadow pixels as candidate shadow regions and candidate nonshadow regions.

Step 7 Verify each candidate region to make it clear whether it satisfies the color attenuation relationship (9) and the relationship described by Figures 2 and 4. If it satisfies these conditions, the region is identified as a shadow region; otherwise, the standard deviation of the region should be calculated. If the standard deviation of the region is small enough, say, smaller than T σ , it is identified as a nonshadow region. Otherwise, the region classification by iteration manner from Step 5 should be implemented, and the region should be segmented into smaller sub-regions, until all the regions are classified as shadow regions or nonshadow regions.

Step 8 Calculate the number of shadow and nonshadow pixels in the neighborhood of each hue singularity pixel in set Q. If the number of shadow pixels is greater than that of the nonshadow pixels, identify the hue singularity pixel as a shadow pixel, and vice versa.

4 Experiment results

In this section, some image results are selected from the experiment to compare the concerned shadow detection algorithms. Our proposed method is compared with the algorithms of Tsai’s and Chung et al.’s methods in the shadow detection accuracy. And subjective and objective evaluations of the three algorithms are demonstrated as follows.

A. Subjective evaluation

Figures 5a, 6a, 7a, 8a, 9a, and 10a show six original color aerial images containing shadow areas. Manual interpretations of shadowed regions are shown in Figures 5b, 6b, 7b, 8b, 9b, and 10b as the ideal shadow detection images, so that ground truth of shadow regions is created to evaluate the shadow detection performance. The shadow detection results of algorithms of Tsai’s, Chung et al.’s, and our proposed method are demonstrated in Figures 5c, 6c, 7c, 8c, 9c, and 10c, Figures 5d, 6d, 7d, 8d, 9d, and 10d, and Figures 5e, 6e, 7e, 8e, 9e, and 10e, respectively. Note that the pure white pixels in each image represent the detected shadow pixels.
https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig5_HTML.jpg
Figure 5

Results of the shadow detection comparison among the concerned three algorithms for the first testing image. (a) Original image. (b) Ideal shadow detection result of (a). (c) Shadow detection result by the algorithm of Tsai. (d) Shadow detection result by the algorithm of Chung et al. (e) Shadow detection result by the proposed algorithm.

https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig6_HTML.jpg
Figure 6

Results of the shadow detection comparison among the concerned three algorithms for the second testing image. (a) Original image. (b) Ideal shadow detection result of (a). (c) Shadow detection result by the algorithm of Tsai. (d) Shadow detection result by the algorithm of Chung et al. (e) Shadow detection result by the proposed algorithm.

https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig7_HTML.jpg
Figure 7

Results of the shadow detection comparison among the concerned three algorithms for the third testing image. (a) Original image. (b) Ideal shadow detection result of (a). (c) Shadow detection result by the algorithm of Tsai. (d) Shadow detection result by the algorithm of Chung et al. (e) Shadow detection result by the proposed algorithm.

https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig8_HTML.jpg
Figure 8

Results of the shadow detection comparison among the concerned three algorithms for the fourth testing image. (a) Original image. (b) Ideal shadow detection result of (a). (c) Shadow detection result by the algorithm of Tsai. (d) Shadow detection result by the algorithm of Chung et al. (e) Shadow detection result by the proposed algorithm.

https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig9_HTML.jpg
Figure 9

Results of the shadow detection comparison among the concerned three algorithms for the fifth testing image. (a) Original image. (b) Ideal shadow detection result of (a). (c) Shadow detection result by the algorithm of Tsai. (d) Shadow detection result by the algorithm of Chung et al. (e) Shadow detection result by the proposed algorithm.

https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig10_HTML.jpg
Figure 10

Results of the shadow detection comparison among the concerned three algorithms for the sixth testing image. (a) Original image. (b) Ideal shadow detection result of (a). (c) Shadow detection result by the algorithm of Tsai. (d) Shadow detection result by the algorithm of Chung et al. (e) Shadow detection result by the proposed algorithm.

For the first four image groups as shown in Figures 5, 6, 7 and 8, the shadow detection accuracy is quite similar when either Chung et al.’s or our proposed algorithm are used, and the both algorithms are much better than the algorithm of Tsai’s. With Tsai’s algorithm, the greensward in Figure 5c is identified as shadow region and the river and the green lawn in Figure 6c are regarded as shadows. With our proposed algorithm, some error shadow detection in the green lawn in the up-right corner of the image can be eliminated when compared with Figure 6d and 6e. In the same way, in Figures 7 and 8, some vegetation regions are mistakenly detected as shadows by the algorithms of Tsai’s and Chung et al.’s. While by using our proposed method, true shadow areas are retained and the false detected regions are eliminated effectively.

The other two testing images in Figures 9 and 10 show obviously that, of the three concerned algorithms, ours has the best shadow detection accuracy. Its detection results are close to the ideal shadow detection images, as shown in Figures 9b and 10b. In Figure 9c, the shadows are over detected when compared with the ideal shadow images, while in Figure 9d, some low brightness shadow areas are not detected. In Figures 10c and 10d, the river is regarded as shadows by the algorithms of Tsai’s and Chung et al.’s.

The subjective evaluation demonstrates that our proposed algorithm has the best accuracy among the three concerned algorithms.

B. Objective evaluation

The performance of shadow detection algorithms is usually assessed by several metrics objectively. In this section, we adopt the three commonly used metrics introduced in [11] to evaluate the three concerned shadow detection algorithms. The three types of evaluation, namely the producer’s accuracy, the user’s accuracy, and the overall accuracy, are used here in the objective evaluation.

The producer’s accuracy contains two parameters η s and η n . They are defined as follows:
η s = T P T P + F N η n = T N T N + F P
(10)

where the true positive (TP) denotes the number of true shadow pixels correctly identified; the false negative (FN) denotes the number of true shadow pixels identified as nonshadow pixels; the false positive (FP) denotes the number of nonshadow pixels identified as shadow pixels; and the true negative (TN) denotes the number of nonshadow pixels correctly identified. The parameters η s and η n stand for the ratio of the correctly detected true shadow and nonshadow over the total true shadow and nonshadow, respectively.

The user’s accuracy also contains two parameters p s and p n . They are defined as follows
p s = T P T P + F P p n = T N T N + F N
(11)

where the parameters p s and p n stand for the ratio of the correctly detected true shadow and nonshadow over the totally detected true shadow and nonshadow, respectively.

If both producer’s accuracy and user’s accuracy are combined, the overall accuracy τ is defined as follows
τ = T P + T N T P + T N + F P + F N
(12)

where TP + TN stands for the number of the correctly detected true shadow and nonshadow pixels, and TP + TN + FP + FN stands for the total number of pixels in the image.

Tables 1, 2, 3, 4, 5 and 6 demonstrate the detection accuracy comparison of the algorithms of Tsai’s, Chung et al.’s and our proposed algorithms in Figures 5, 6, 7, 8, 9 and 10, respectively. In Tables 1, 2, and 3, the detection accuracy of our proposed algorithm is very close to that of Chung et al.’s algorithm, and the performance of the both algorithms are better than that of Tsai’s algorithm. Tables 4, 5, and 6 show that our proposed algorithm has the best performance among the three concerned algorithms.
Table 1

Detection accuracy of three methods for Figure 5

Method

Producer’s accuracy

User’s accuracy

Overall accuracy

η s (%)

η n (%)

p s (%)

p n (%)

τ (%)

Proposed

74.63

97.84

36.32

99.57

97.46

Chung et al.’s

79.18

96.86

29.67

99.64

96.57

Tsai’s

70.03

73.87

4.27

99.33

73.81

Table 2

Detection accuracy of three methods for Figure 6

Method

Producer’s accuracy

User’s accuracy

Overall accuracy

η s (%)

η n (%)

p s (%)

p n (%)

τ (%)

Proposed

94.52

95.38

94.79

95.14

94.97

Chung et al.’s

93.36

98.93

92.21

99.10

98.27

Tsai’s

99.66

27.00

15.56

98.83

35.64

Table 3

Detection accuracy of three methods for Figure 7

Method

Producer’s accuracy

User’s accuracy

Overall accuracy

η s (%)

η n (%)

p s (%)

p n (%)

τ (%)

Proposed

77.22

94.86

78.62

94.45

91.39

Chung et al.’s

82.90

88.86

64.56

95.50

87.69

Tsai’s

80.86

75.26

44.35

94.16

76.36

Table 4

Detection accuracy of three methods for Figure 8

Method

Producer’s accuracy

User’s accuracy

Overall accuracy

η s (%)

η n (%)

p s (%)

p n (%)

τ (%)

Proposed

60.91

98.66

75.54

97.38

96.25

Chung et al.’s

76.35

93.81

45.61

98.31

92.70

Tsai’s

71.41

91.38

35.70

97.95

90.13

Table 5

Detection accuracy of three methods for Figure 9

Method

Producer’s accuracy

User’s accuracy

Overall accuracy

η s (%)

η n (%)

p s (%)

p n (%)

τ (%)

Proposed

62.79

92.88

78.54

85.74

84.05

Chung et al.’s

8.94

99.71

92.74

72.56

73.08

Tsai’s

71.86

47.07

36.05

80.12

54.35

Table 6

Detection accuracy of three methods for Figure 10

Method

Producer’s accuracy

User’s accuracy

Overall accuracy

η s (%)

η n (%)

p s (%)

p n (%)

τ(%)

Proposed

43.06

99.37

59.13

98.80

98.19

Chung et al.’s

87.15

8.19

1.98

96.77

9.84

Tsai’s

83.81

8.50

1.91

96.11

10.07

The reason for the worst performance by Tsai’s algorithm lies in that Tsai’s algorithm uses only one threshold to separate the shadow and nonshadow regions. However, in the ratio maps, the value of shadow pixels in one place may equal to the value of nonshadow pixels in another place. Therefore, the local threshold strategy proposed in this article and Chung et al.’s study makes more sense.

The shadow pixels in the up-left of Figure 9d are not detected, because of the existence of the hue singularity pixels in this regions. When the hue singularity pixels are separated and classified independently by the proposed algorithm, better detection performance is achieved in Figure 9e. In Figure 10d, only one ratio map is used for shadow detection. The detection result in Figure 10e performs better than that in Figure 10d, because an additional constraint, the new ratio map of saturation over the intensity, is combined with hue over the intensity. The color attenuation relationship also improves the shadow detection performance in many detail regions when compared with Tsai’s and Chung et al.’s algorithm in Figures 5, 6, 7, 8, 9 and 10.

C. Limitation of the proposed algorithm

In the above experiments, the scenes are shown in the daytime. However, when we use the thresholds and color attenuation relationship for the images taken in the evening, the shadow detection accuracy will drop obviously. Figure 11 is such an example. Figure 11a is the original image, and the pure white regions in Figure 11b are the detected shadows. Most shadow regions in Figure 11b are not detected. This problem is caused by the following: First, the contrast in Figure 11a is lower than those in the images taken in the daytime. Therefore, the intensity value difference between the shadow and nonshadow regions is smaller than that in the daytime. The threshold for T I CS and T I CNS should be adjusted to a value smaller than those in the daytime. Second, the color temperature T in the evening is lower than that in the daytime. Hence, the color attenuation relationship should be changed accordingly in this case. Finally, the R, G, and B values of many shadow pixels in Figure 11a are equal to zero. These pixels are the hue singularity pixels. However, the pixels in the neighborhood of these hue singularity pixels are classified as nonshadow pixels. Therefore, these hue singularity pixels are misclassified as nonshadow regions.
https://static-content.springer.com/image/art%3A10.1186%2F1687-6180-2012-141/MediaObjects/13634_2011_Article_384_Fig11_HTML.jpg
Figure 11

Limitation of the proposed algorithm. (a) Original image. (b) Shadow detection result by the proposed algorithm.

5 Conclusions

This article is devoted to the problem of shadow detection in color aerial images. Hue singularity pixels are extracted. The candidate shadow and nonshadow regions are constructed on the base of the modified ratio maps by using the Otsu’s thresholding method and the connected component analysis. The intensity property, chromaticity property of the shadow areas, and the color attenuation relationship derived from Planck’s blackbody irradiance law are used iteratively to segment each candidate region into smaller sub-regions, so that whether each sub-region is true shadow region is identified. The extracted hue singularity pixels are classified on the base of its neighboring pixels. From the above experimental results, it could be concluded that our proposed shadow detection algorithm presents best shadow detection accuracy when compared with Tsai’s and Chung et al.’s algorithms. Future work need to be done to solve the auto thresholds selection problem.

Appendix

Suppose the RGB color at a pixel (x, y) in a color aerial image is formed from illumination with spectral power distribution E(λ, x, y) impinging on a surface with surface spectral reflectance S(λ, x, y), and if the Q k (λ) is the relative spectral response of the imaging device’s k th sensor, then the RGB color ρ(x, y) can be deduced from an integral over the visible wavelengths [9]:
ρ k ( x , y ) = σ ( x , y ) E ( λ , x , y ) S ( λ , x , y ) Q k ( λ ) d λ , k = R , G , B
(13)

where σ is exposure time, and λ is wavelength.

If the camera sensitivities Q k (λ) are exactly Dirac delta functions, Q k ( λ ) = q k δ ( λ λ k ) , then formula (13) becomes simpler:
ρ k = σ E ( λ k ) S ( λ k ) q k , k = R , G , B
(14)
Denoting the spectral power distribution of the illumination on nonshadow region as E1(λ k ) and that on shadow region as E2(λ k ), we will obtain the attenuation of each color channel:
Δ R Δ G Δ B = ρ N S R ρ S R ρ N S G ρ S G ρ N S B ρ S B = σ E 1 ( λ R ) E 2 ( λ R ) S ( λ R ) q R E 1 ( λ G ) E 2 ( λ G ) S ( λ G ) q G E 1 ( λ B ) E 2 ( λ B ) S ( λ B ) q B
(15)

where ρ N S k and ρ S k (k = R, G, B) are the k-channel intensity values of a given pixel in nonshadow and shadow regions, respectively.

We analyze the relationship between ΔR and ΔB, because they have the longest and shortest wavelengths in the three channels. From (15), we have
Δ R Δ B = E 1 ( λ R ) E 2 ( λ R ) E 1 ( λ B ) E 2 ( λ B ) · S ( λ R ) q R S ( λ B ) q B
(16)
From (16), we have:
ρ S R = σ E 2 ( λ R ) S ( λ R ) q R ρ S B = σ E 2 ( λ B ) S ( λ B ) q B
(17)
So
S ( λ R ) q R S ( λ B ) q B = E 2 ( λ B ) E 2 ( λ R ) · ρ S R ρ S B
(18)
Substituting (17) into (16), we get:
Δ R Δ B = E 1 ( λ R ) E 2 ( λ R ) E 1 ( λ B ) E 2 ( λ B ) · E 2 ( λ B ) E 2 ( λ R ) · ρ S R ρ S B
(19)
We do not take the penumbra pixels into consideration, because the number of penumbra pixels is usually very small in an aerial image during daytime. If Esun is denoted as the spectral power distribution of direct sunlight illumination, and Esky as that of the scattering light from the atmosphere, then the nonshadow region is lighted by Esun and Esky, and the shadow region is only lighted by Esky. Therefore, E 1 ( λ k ) = E sun ( λ k ) + E sky ( λ k ) , E 2 ( λ k ) = E sky ( λ k ) . We get
Δ R Δ B = E sun ( λ R ) E sun ( λ B ) · E 1 ( λ B ) E sun ( λ B ) E 1 ( λ R ) E sun ( λ R ) · ρ S R ρ S B
(20)
If the illumination is restricted to be Planckian, an illuminant spectral power distribution can be parameterized by its color temperature T when the blackbody radiator model is used
E ( λ , T ) = c 1 λ 5 e c 2 T λ 1 1
(21)

Where c 1 = 2 π h c 2 and c 2 = ( h c ) / k , in which

c is the velocity of light: 3.0 × 10 8 m / s ; h is the Plank constant: 6.63 × 10 34 J · s ; k is the Boltzmann constant: 1.38 × 10 23 J / K . Hence, c 1 = 3.74 × 10 16 W · m 2 and c 2 = 1.43 × 10 2 m · K . Substituting formula (20) into (19), we can easily get:
Δ R Δ B = ( e c 2 T sun λ B 1 ) / ( e c 2 T 1 λ B 1 ) 1 ( e c 2 T sun λ R 1 ) / ( e c 2 T 1 λ R 1 ) 1 · ρ S R ρ S B
(22)
where Tsun is the color temperature of direct sunlight, T1 is the color temperature of daylight. Given T1 = 6500 K, which is the standard daylight D65[25], given T sun = 5500 K , which represents the direct sunlight, and given the wavelengths λ R = 650 n m , and λ B = 440 n m , respectively, then we get
Δ R Δ B = 1.70 × ρ S R ρ S B
(23)

Declarations

Acknowledgments

We thank the referees for their comments that have significantly improved the manuscript of this work.

Authors’ Affiliations

(1)
DSP and SOC Laboratory, Department of Electrical Engineering, School of Electronic Information, Wuhan University

References

  1. Chung K-L, Lin Y-R, Huang Y-H: Efficient shadow detection of color aerial images based on successive thresholding scheme. IEEE Trans Geosci Remote Sens 2009, 47: 671-682.View ArticleGoogle Scholar
  2. Su Y-F, Chen HH: A three-stage approach to shadow field estimation from partial boundary information. IEEE Trans. Image Process. 2010, 19: 2749-2760.MathSciNetView ArticleGoogle Scholar
  3. Yang J, Zhao Z, Yang J: A shadow removal method for high resolution remote sensing image. Geomatics Inf Sci Wuhan Univ 2008, 33: 17-20.Google Scholar
  4. Chao X, Yanjun L, Ke Z: Shadow detecting using PSO and Kolmogorov test, in Sixth International Conference on Natural Computation (ICNC), Yantai, Shandong. China August 2010, 2: 572-576.Google Scholar
  5. Yu H-Y, Sun J-G, Liu L-N: MSER based shadow detection in high resolution remote sensing image, in International Conference on Machine Learning and Cybernetics (ICMLC), Qingdao. China July 2010, 6: 780-783.Google Scholar
  6. Zhu J, Samuel KGG, Masood SZ, Tappen MF: Learning to recognize shadows in monochromatic natural images. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco; 2010:223-230.Google Scholar
  7. Finlayson GD, Fredembach C, Drew MS: Detecting illumination in images. in IEEE 11th International Conference on Computer Vision (ICCV), Rio de Janeiro, Brazil; 2007:1-8.Google Scholar
  8. Finlayson GD, Drew MS, Lu C: Intrinsic images by entropy minimization. in Europeon Conference on Computer Vision (ECCV), Prague, Czech Republic; 2004:582-595. vol: 3023/2004Google Scholar
  9. Finlayson GD, Hordley SD, Lu C, Drew MS: On the removal of shadows from images. IEEE Trans. Pattern Anal. Mach. Intell. 2006, 28: 59-68.View ArticleGoogle Scholar
  10. Finlayson GD, Hordley SD, Drew MS: Removing shadows from images. in Europeon Conference on Computer Vision (ECCV), Copenhagen, Denmark; 2002:129-132. vol: 2353/2006Google Scholar
  11. Tsai VJD: A comparative study on shadow compensation of color aerial images in invariant color models. IEEE Trans. Geosci. Remote Sens 2006, 44: 1661-1671.View ArticleGoogle Scholar
  12. Li Y, et al.: Integrated shadow removal based on photogrammetry and image analysis. Int J Remote Sens 2005, 26: 3911-3929. 10.1080/01431160500159347View ArticleGoogle Scholar
  13. Liu W, Yamazaki F: Shadow extraction and correction from quickbird images. in IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Honolulu, Hawaii, USA; 2010:2206-2209.Google Scholar
  14. Cai D, Li M, Bao Z, Chen Z, Wei W, Zhang H: Study on shadow detection method on high resolution remote sensing image based on HIS space transformation and NDVI index. in 18th International Conference on Geoinformatics, Beijing, China; 2010:1-4.Google Scholar
  15. Makarau A, Richter R, Muller R, Reinartz P: Adaptive shadow detection using a blackbody radiator model. IEEE Trans. Geosci. Remote Sens. 2011, 49: 2049-2059.View ArticleGoogle Scholar
  16. Tian J, Sun J, Tang Y: Tricolor attenuation model for shadow detection. IEEE Trans. Image Process. 2009, 18: 2355-2363.MathSciNetView ArticleGoogle Scholar
  17. Arbel E, Hel-Or H: Shadow removal using intensity surfaces and texture anchor points. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 33: 1202-1216.View ArticleGoogle Scholar
  18. McFeely R, Hughes C, Jones E, Glavin M: Removal of nonuniform complex and compound shadows from textured surfaces using adaptive directional smoothing and the thin plate model. IET Image Process 2011, 5: 233-248. 10.1049/iet-ipr.2009.0198View ArticleGoogle Scholar
  19. Liu Q, Cao X, Deng C, Guo X: Identifying image composites through shadow matte consistency. IEEE Trans. Inf. Forensics Security 2011, 6(3):1111-1122.View ArticleGoogle Scholar
  20. Gonzalez RC, Eddins SL: Digital Image Processing Using MATLAB. Pearson Education, India; 2004.Google Scholar
  21. Gevers T, Smeulders AWM: PicToSeek: combining color and shape invariant features for image retrieval. IEEE Trans. Image Process. 2000, 9(1):102-119. 10.1109/83.817602View ArticleGoogle Scholar
  22. Polidorio AM, Flores FC, Imai NN, Tommaselli AMG, Franco C: Automatic shadow segmentation in aerial color images. XVI Brazilian Symposium on Computer Graphics and Image Processing, SIBGRAPI 2003, Sao Carlos, Brazil; 2003:270-277.Google Scholar
  23. Rudin LI, Osher S, Fatemi E: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 1992, 60: 259-268. 10.1016/0167-2789(92)90242-FView ArticleGoogle Scholar
  24. Otsu N: A threshold selection method from gray-level histograms. IEEE Trans Systems Man Cybern 1979, 9(1):62-66.MathSciNetView ArticleGoogle Scholar
  25. Wyszecki G, Stiles WS: Color Science: Concepts and Methods, Quantitative Data and Formulas. Wiley, New York; 1967.Google Scholar

Copyright

© Shi and Li; licensee Springer. 2012

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.