Open Access

A Total Variation Regularization Based Super-Resolution Reconstruction Algorithm for Digital Video

  • Michael K. Ng1Email author,
  • Huanfeng Shen1, 2,
  • Edmund Y. Lam3 and
  • Liangpei Zhang2
EURASIP Journal on Advances in Signal Processing20072007:074585

DOI: 10.1155/2007/74585

Received: 13 September 2006

Accepted: 21 April 2007

Published: 27 June 2007


Super-resolution (SR) reconstruction technique is capable of producing a high-resolution image from a sequence of low-resolution images. In this paper, we study an efficient SR algorithm for digital video. To effectively deal with the intractable problems in SR video reconstruction, such as inevitable motion estimation errors, noise, blurring, missing regions, and compression artifacts, the total variation (TV) regularization is employed in the reconstruction model. We use the fixed-point iteration method and preconditioning techniques to efficiently solve the associated nonlinear Euler-Lagrange equations of the corresponding variational problem in SR. The proposed algorithm has been tested in several cases of motion and degradation. It is also compared with the Laplacian regularization-based SR algorithm and other TV-based SR algorithms. Experimental results are presented to illustrate the effectiveness of the proposed algorithm.


Authors’ Affiliations

Department of Mathematics, Hong Kong Baptist University
The State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University
Department of Electrical and Electronic Engineering, The University of Hong Kong


  1. Tsai RY, Huang TS: Multi-frame image restoration and registration. Advances in Computer Vision and Image Processing 1984,1(2):317-339.Google Scholar
  2. Kim SP, Bose NK, Valenzuela HM: Recursive reconstruction of high resolution image from noisy undersampled multiframes. IEEE Transactions on Acoustics, Speech, and Signal Processing 1990,38(6):1013-1027. 10.1109/29.56062View ArticleGoogle Scholar
  3. Kim SP, Su W-Y: Recursive high-resolution reconstruction of blurred multiframe images. IEEE Transactions on Image Processing 1993,2(4):534-539. 10.1109/83.242363View ArticleGoogle Scholar
  4. Rhee S, Kang MG: Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Optical Engineering 1999,38(8):1348-1356. 10.1117/1.602177View ArticleGoogle Scholar
  5. Chan RH, Chan TF, Shen L, Shen Z: Wavelet algorithms for high-resolution image reconstruction. SIAM Journal of Scientific Computing 2003,24(4):1408-1432. 10.1137/S1064827500383123MathSciNetView ArticleMATHGoogle Scholar
  6. Ng MK, Sze CK, Yung SP: Wavelet algorithms for deblurring models. International Journal of Imaging Systems and Technology 2004,14(3):113-121. 10.1002/ima.20014View ArticleGoogle Scholar
  7. Nguyen N, Milanfar P: A wavelet-based interpolation-restoration method for superresolution (wavelet superresolution). Circuits, Systems, and Signal Processing 2000,19(4):321-338. 10.1007/BF01200891View ArticleMATHGoogle Scholar
  8. Ur H, Gross D: Improved resolution from subpixel shifted pictures. CVGIP: Graphical Models and Image Processing 1992,54(2):181-186. 10.1016/1049-9652(92)90065-6Google Scholar
  9. Irani M, Peleg S: Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 1991,53(3):231-239. 10.1016/1049-9652(91)90045-LGoogle Scholar
  10. Stark H, Oskoui P: High-resolution image recovery from image-plane arrays, using convex projections. Journal of the Optical Society of America A: Optics and Image Science, and Vision 1989,6(11):1715-1726. 10.1364/JOSAA.6.001715View ArticleGoogle Scholar
  11. Tekalp AM, Ozkan MK, Sezan MI: High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration. Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '92), March 1992, San Francisco, Calif, USA 3: 169-172.Google Scholar
  12. Patti AJ, Sezan MI, Tekalp AM: High-resolution image reconstruction from a low-resolution image sequence in the presence of time-varying motion blur. Proceedings of IEEE International Conference Image Processing (ICIP '94), November 1994, Austin, Tex, USA 1: 343-347.Google Scholar
  13. Patti AJ, Sezan MI, Tekalp AM: Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Transactions on Image Processing 1997,6(8):1064-1076. 10.1109/83.605404View ArticleGoogle Scholar
  14. Tom BC, Katsaggelos AK: Reconstruction of a high-resolution image from multiple-degraded misregistered low-resolution images. Visual Communications and Image Processing, September 1994, Chicago, Ill, USA, Proceedings of SPIE 2308: 971-981.View ArticleGoogle Scholar
  15. Schultz RR, Stevenson RL: Extraction of high-resolution frames from video sequences. IEEE Transactions on Image Processing 1996,5(6):996-1011. 10.1109/83.503915View ArticleGoogle Scholar
  16. Hardie RC, Tuinstra TR, Bognar J, Barnard KJ, Armstrong EE: High resolution image reconstruction from digital video with global and non-global scene motion. Proceedings of IEEE International Conference on Image Processing (ICIP '97), October 1997, Santa Barbara, Calif, USA 1: 153-156.View ArticleGoogle Scholar
  17. Elad M, Feuer A: Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Transactions on Image Processing 1997,6(12):1646-1658. 10.1109/83.650118View ArticleGoogle Scholar
  18. Elad M, Feuer A: Superresolution restoration of an image sequence: adaptive filtering approach. IEEE Transactions on Image Processing 1999,8(3):387-395. 10.1109/83.748893View ArticleGoogle Scholar
  19. Chung J, Haber E, Nagy J: Numerical methods for coupled super-resolution. Inverse Problems 2006,22(4):1261-1272. 10.1088/0266-5611/22/4/009MathSciNetView ArticleMATHGoogle Scholar
  20. Hardie RC, Barnard KJ, Armstrong EE: Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Transactions on Image Processing 1997,6(12):1621-1633. 10.1109/83.650116View ArticleGoogle Scholar
  21. Woods NA, Galatsanos NP, Katsaggelos AK: Stochastic methods for joint registration, restoration, and interpolation of multiple undersampled images. IEEE Transactions on Image Processing 2006,15(1):201-213.MathSciNetView ArticleGoogle Scholar
  22. Shen H, Zhang L, Huang B, Li P: A MAP approach for joint motion estimation, segmentation, and super resolution. IEEE Transactions on Image Processing 2007,16(2):479-490.MathSciNetView ArticleGoogle Scholar
  23. Sasahara R, Hasegawa H, Yamada I, Sakaniwa K: A color super-resolution with multiple nonsmooth constraints by hybrid steepest descent method. Proceedings of IEEE International Conference on Image Processing (ICIP '05), September 2005, Genova, Italy 1: 857-860.Google Scholar
  24. Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing 2006,15(1):141-159.View ArticleGoogle Scholar
  25. Akgun T, Altunbasak Y, Mersereau RM: Super-resolution reconstruction of hyperspectral images. IEEE Transactions on Image Processing 2005,14(11):1860-1875.View ArticleGoogle Scholar
  26. Segall CA, Katsaggelos AK, Molina R, Mateos J: Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing 2004,13(7):898-910. 10.1109/TIP.2004.827230View ArticleGoogle Scholar
  27. Segall CA, Molina R, Katsaggelos AK: High-resolution images from low-resolution compressed video. IEEE Signal Processing Magazine 2003,20(3):37-48. 10.1109/MSP.2003.1203208View ArticleGoogle Scholar
  28. Capel D, Zisserman A: Super-resolution enhancement of text image sequences. Proceedings of the 15th International Conference on Pattern Recognition (ICPR '00), September 2000, Barcelona, Spain 1: 600-605.View ArticleGoogle Scholar
  29. Han YB, Wu LN: Super resolution reconstruction of video sequence based on total variation. Proceedings of International Symposium on Intelligent Multimedia, Video and Speech Processing (ISIMP '04), October 2004, Hong Kong 575-578.Google Scholar
  30. Vazquez C, Aly H, Dubois E, Mitiche A: Motion compensated super-resolution of video by level sets evolution. Proceedings of IEEE International Conference on Image Processing (ICIP '04), October 2004, Singapore 3: 1767-1770.Google Scholar
  31. Farsiu S, Robinson MD, Elad M, Milanfar P: Fast and robust multiframe super resolution. IEEE Transactions on Image Processing 2004,13(10):1327-1344. 10.1109/TIP.2004.834669View ArticleGoogle Scholar
  32. Chan TF, Shen J: Mathematical models for local nontexture inpaintings. SIAM Journal on Applied Mathematics 2002,62(3):1019-1043. 10.1137/S0036139900368844MathSciNetView ArticleMATHGoogle Scholar
  33. Borman S, Stevenson RL: Spatial resolution enhancement of low-resolution image sequences: a comprehensive review with directions for future research. Laboratory for Image and Signal Analysis (LISA), University of Notre Dame, Notre Dame, Ind, USA; 1998.Google Scholar
  34. Park SC, Park MK, Kang MG: Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 2003,20(3):21-36. 10.1109/MSP.2003.1203207View ArticleGoogle Scholar
  35. Nguyen N, Milanfar P, Golub G: Efficient generalized cross-validation with applications to parametric image restoration and resolution enhancement. IEEE Transactions on Image Processing 2001,10(9):1299-1308. 10.1109/83.941854MathSciNetView ArticleMATHGoogle Scholar
  36. Capel D, Zisserman A: Computer vision applied to super resolution. IEEE Signal Processing Magazine 2003,20(3):75-86. 10.1109/MSP.2003.1203211View ArticleGoogle Scholar
  37. Schultz RR, Meng L, Stevenson RL: Subpixel motion estimation for super-resolution image sequence enhancement. Journal of Visual Communication and Image Representation 1998,9(1):38-50. 10.1006/jvci.1997.0370View ArticleGoogle Scholar
  38. Tekalp AM: Digital Video Processing. Prentice-Hall, Englewood Clliffs, NJ, USA; 1995.Google Scholar
  39. Farsiu S, Robinson D, Elad M, Milanfar P: Advances and challenges in super-resolution. International Journal of Imaging Systems and Technology 2004,14(2):47-57. 10.1002/ima.20007View ArticleGoogle Scholar
  40. Rudin L, Osher S, Fatemi E: Nonlinear total variation based noise removal algorithms. Physica D 1992,60(1–4):259-268.View ArticleMathSciNetMATHGoogle Scholar
  41. Vogel CR, Oman ME: Fast, robust total variation-based reconstruction of noisy, blurred images. IEEE Transactions on Image Processing 1998,7(6):813-824. 10.1109/83.679423MathSciNetView ArticleMATHGoogle Scholar
  42. Chambolle A: An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 2004,20(1-2):89-97.MathSciNetGoogle Scholar
  43. Li Y, Santosa F: A computational algorithm for minimizing total variation in image restoration. IEEE Transactions on Image Processing 1996,5(6):987-995. 10.1109/83.503914View ArticleGoogle Scholar
  44. Bioucas-Dias JM, Figueiredo MAT, Oliveira JP: Total variation-based image deconvolution: a majorization-minimization approach. Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '06), May 2006, Toulouse, France 2: 861-864.Google Scholar
  45. Vogel CR, Oman ME: Iterative methods for total variation denoising. SIAM Journal of Scientific Computing 1996,17(1):227-238. 10.1137/0917016MathSciNetView ArticleMATHGoogle Scholar
  46. Vogel CR: Computational Methods for Inverse Problems, Frontiers in Applied Mathematics. SIAM, Philadelphia, Pa, USA; 2002.View ArticleGoogle Scholar
  47. Lin F-R, Ng MK, Ching W-K: Factorized banded inverse preconditioners for matrices with Toeplitz structure. SIAM Journal of Scientific Computing 2005,26(6):1852-1870. 10.1137/030601272MathSciNetView ArticleMATHGoogle Scholar
  48. Chan RH, Chan TF, Wong C-K: Cosine transform based preconditioners for total variation deblurring. IEEE Transactions on Image Processing 1999,8(10):1472-1478. 10.1109/83.791976View ArticleGoogle Scholar
  49. Ng MK, Chan RH, Chan TF, Yip AM: Cosine transform preconditioners for high resolution image reconstruction. Linear Algebra and Its Applications 2000,316(1–3):89-104.MathSciNetView ArticleMATHGoogle Scholar
  50. Kolotilina LY, Yeremin AY: Factorized sparse approximate inverse preconditionings I: theory. SIAM Journal on Matrix Analysis and Applications 1993,14(1):45-58. 10.1137/0614004MathSciNetView ArticleMATHGoogle Scholar
  51. Nguyen N, Milanfar P, Golub G: A computationally efficient superresolution image reconstruction algorithm. IEEE Transactions on Image Processing 2001,10(4):573-583. 10.1109/83.913592MathSciNetView ArticleMATHGoogle Scholar


© Michael K. Ng et al. 2007

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.