Learning More About NokiaCV A Mobile Based Computer Vision Algorithm Suite

1
Learning More About NokiaCVA Mobile Based
Computer Vision Algorithm Suite
2
Current Scenario

• Mobile cameras are widespread
• Main uses for camera capturing images/videos
• Still significant barriers to create new
  applications
• There is not enough support for camera/imaging in
  S60 SDK
• Only a handful of small companies making camera
  apps. Most of these are experts computer vision
  expertise, image/signal processing experience

3
Solution Nokia Computer Vision Library (NokiaCV)

• Developers do not have to start imaging/camera
  applications from scratch
• Shared platform will encourage further and more
  complex work in imaging domain
• Nokia Computer Vision library built on top of
  Symbian
• Fundamental operations for imaging and camera
  processing

4
Prior work

• A few libraries available for mobile devices,
  including S60
• Symbian mainly supports bitmap drawing,
  image/video capture.
• No processing support
• Difficult/slow to access individual pixels
  common operation in processing
• Other S60 imaging libraries are research-quality
  code, no support

5
Goal

• Internally to be used in imaging products, as a
  research test bed, etc.
• Externally to provide developers in binary form
  to facilitate third-party imaging
• Intels OpenCV is the desktop version of what NCV
  will become for mobiles
• Full functionality
• Wide range of imaging operations, however some
  not possible on mobile (yet)
• NokiaCVs main difference supports operations
  that are useful and fast right now.

6
Use Cases
7
Capture Enhancement

• Image color conversions, imaging wrapper to
  modify images easily and manipulate pixels
  directly and easily
• Building blocks present for developers to make
  high-quality stitching algorithms for panoramas
• Building blocks present for developers to
  implement complex algorithms without requiring
  the building and debugging of a platform

8
Gaming/UI

• Provides robust motion estimation as a software
  component, ready for games and applications

9
Post-capture editing

• Image warping operation provided to create
  standard warping or in other entertainment uses
• Image compositing provided to add items to
  captured images

10
NokiaCV Library Overview

• Library builds on OS, extending imaging
  capabilities
• Image object provided standardizing OS image
  internals
• Standard image operations provided
• Linear algebra used in many advanced imaging
  applications, present in NCV
• Building block for future advanced libraries,
  both externally and internally

11
Functional Blocks
12

• CCamus
• CEgoMovement
• CEigenvalues
• CFixed
• CImageOp
• CArithmeticOp
• CColorConversionOp
• CComparisonOp
• CConvolutionOp
• CCornerDetectionOp
• CEdgeDetectionOp
• CGaussianSmoothOp
• CMedianFilterOp
• CMorphologicalOp
• CResizeOp
• CRotationOp
• CShearOp
• CTemplateMatchOp
• CThresholdOp

13
API Overview

• This library extends the image processing and
  math capabilities of the Symbian Series 60
  platform, targeted for applications using
  computer vision techniques on camera phones.
• The library provides an image class,
  CNokiaCVImage for handling RGB, grayscale and
  black and white images. Provides functions to
  perform arithmetic operations, transformations,
  feature extraction, color conversions. In
  addition, operations to compute image statistics
  such as histograms and pixel color means are
  provided.
• The library has two algorithms (CCamus and
  CMotionHistoryImage) to determine motion
  estimates and history.
• Matrix and vector classes CNokiaCVMatrix and
  CNokiaCVVector allow creation of matrices and
  vectors of arbitrary dimensionality. Math
  operations and some linear algebra techniques
  that work with these classes are available in the
  library.

14
Library Features
15
CNokiaCVImage Class

• CNokiaCVImage is a wrapper for CFbsBitmap that is
  provided by the platform. This class provides
  DisplayMode independent access to the pixel data.
• CFbsBitmap iInternalBitmap
• Usage examples
• // load from file
• CNokiaCVImage incvBitmap CNokiaCVImageNewL()
  
• TRAPD(err1, incvBitmap-gtCreateL( aImageFullName
  )) ...
• // draw bitmap
• CWindowGc gc SystemGc()
• gc.BitBlt( TPoint( 0,0 ), incvBitmap-gt Bitmap()
  )

16
Functions

• static IMPORT_C CNokiaCVImage CNokiaCVImageNewL
  C ( )  Implements two-phase construction
  functions (NewLC(), ConstructL()) to create a new
  instance of the object.
• Returns CNokiaCVImage
• IMPORT_C void CNokiaCVImageCreateL (const TDesC
   aFileName  )  Create image from a file.
• static IMPORT_C void CNokiaCVImageCopy
  (CNokiaCVImage  aSource, CNokiaCVImage
   aTarget, TInt aBlendingRatio 100 ) Copy a
  source image to the target image.

17
CPixelAccess Class

• CPixelAccess class provides access to
  CNokiaCVImage on a pixel-level. Derived from
  TBitmapUtil.
• Example
• CPixelAccess out CPixelAccessNewL(aTarget)
  
• CPixelAccess in CPixelAccessNewL(iImage,
  out)
• ...
• in-gtSetPos( TPoint( rotX, rotY ) )
• out-gtSetPos(TPoint( x, y ) )
• out-gtSetRGB( in-gtGetRGB() ) // set pixel value
  from in to out
• ...
• delete out
• delete in

18
Functions

• static IMPORT_C CPixelAccess CPixelAccessNewLC
  (CNokiaCVImage  aImage  )  Constructor. Sets
  the current pixel position to (0,0) position.
• IMPORT_C TUint32 CPixelAccessGetRGB ( ) 
  Return pixel from the current position as a RGB.
• IMPORT_C void CPixelAccessSetRGB
  (TUint32 aPixel  ) Set pixel to the current
  position.
• Parameters
• aPixel RGB value as TUint32 in format used by
  TRgb 0x00BBGGRR

19
CPixelColorModel Class

• IMPORT_C TUint8 CPixelColorModelRed ( )  Get
  the red component
• Returns TUint8 Red-component
• IMPORT_C CYuvColorModel CPixelColorModelToYUV
  ( )  Convert to YUV.
• Returns CYuvColorModel.
• IMPORT_C CHsvColorModel CPixelColorModelToHSV
  ( ) 
• Convert to HSV
• return CHsvColorModel

20
TColorModel Class

• class CRgbColorModel public TColorModel
• public
• TInt iR
• TInt iG
• TInt iB
• class CYuvColorModel public TColorModel
• public
• TInt iY
• TInt iU
• TInt iV

21
CNokiaCVMath Class

• This class gathers some utility functions in one
  place.
• static IMPORT_C TInt SinLUT_Rad (TReal aTrg,
  const TReal aSrc) Computes Sin using lookup
  table.
• static IMPORT_C TInt TanLUT_Rad (TReal aTrg,
  const TReal aSrc) Computes Tan using lookup
  table.

22
CNokiaCVMatrix Class

• Fixed point matrix representation. Usage
• CNokiaCVMatrix m CNokiaCVMatrix-gtNewLC(5,5)
• (m)(3,3) 2 (m)(2,3) 4.23
• CNokiaCVMatrix n CNokiaCVMatrix-gtOnes(5,5)
  CleanupStackPushL(n)
• CNokiaCVMatrix res (m)(n)5
• CleanupStackPushL(res)
• CFixed det res-gtDet()
• ...
• CleanupStackPop(m) CleanupStackPop(n)
  CleanupStackPop(res)
• delete m delete n delete res

23
Functions

• static IMPORT_C CNokiaCVMatrix  NewL (const TInt
  aRows, const TInt aCols) Creates a new matrix
  according to size given with initial value zero.
  
• IMPORT_C CNokiaCVMatrix  Inverse ()
  const Calculates the inverse matrix of this
  matrix.
• IMPORT_C CFixed  Eigenvalues () Calculates the
  eigenvalues of this matrix.
• IMPORT_C CFixed  Singularvalues () Calculates
  the singular values of this matrix using ncvSVD.

24
CSVD Class

• Singular value decomposition of a given real
  matrix.
• Householder bidiagonalization and a variant of
  the QR algorithm are used.
• The SVD algorithm is derived from JAMA/TNT, the
  Java Matrix Package.
• Usage
• CNokiaCVMatrix a CNokiaCVMatrixNewL(rows,col
  s)
• CSVD svd
• TRAPD(err, svd CSVDNewL(a))
• ... //Handle possible errors
• CNokiaCVMatrix v svd-gtV()
• ...
• delete a delete svd delete v

25
CNokiaCVVector Class

• Fixed point representation of a vector.
• class CNokiaCVVector public CNokiaCVMatrix
• IMPORT_C TInt Length () Getter of the vector
  size/length.
• IMPORT_C CNokiaCVVector  Normalize () Creates
  the normalization of this vector.
• IMPORT_C CNokiaCVVector  CrossProduct
  (CNokiaCVVector aVector) Calculates the cross
  product of this and the given vector.
• IMPORT_C CFixed DotProduct (CNokiaCVVector
  aVector) Calculates the dot product of to same
  size vectors.

26
CMotionHistoryImage Class

• This class computes a motion history image (MHI)
  for a series of input frames. The MHI is a
  grayscale bitmap in which recently moved pixels
  are represent with bright values and "older" with
  darker values. Frame-by-frame the values from
  previous frames are decayed linearly.

27
Functions

• static IMPORT_C CMotionHistoryImage  NewL
  (CNokiaCVImage aImage, const TInt
  aHistorySize) Construct CMotionHistoryImage from
  the first image of motion history.
• Parameters
• aImageSize Size of the images.
  aHistorySize Maximum number of images in the
  history.
• IMPORT_C void AddImage (CNokiaCVImage
  aImage) Add an image to the motion history.
• IMPORT_C void GetImage (CNokiaCVImage
  aImage) Get a CNokiaCVImage containing the
  motion history image.

28
COpticalFlow Class

• The optical flow class motion estimations for
  each pixel. This class also implements methods
  for calculating higher level information from the
  optical flow.
• Friends class CCamus Camus algorithm for
  estimation of optical flow from a series of
  bitmaps.
• Member CEgoMovement Class class for pixel
  motion estimation values.
• CFixed iX Movement along the x-axis.
  CFixed iY Movement along the y-axis.

29
Functions

• static IMPORT_C COpticalFlow  NewL (const TSize
  aImageSize) Factory method for creating an
  instance. Parameters
• aImageSize Size of the optical flow map
• IMPORT_C CEgoMovement Mean () const Calculate the
  mean value of the optical flow. Represents
  tilting and panning of the camera.
• IMPORT_C CFixed Rotation () Calculate the mean
  rotation of the optical flow in the image plane.
  

30
Functions

• void SetCamusOpts (const TInt aCamusOpts) Set
  runtime parameters of the Camus motion estimation
  algorithm.
• IMPORT_C CEgoMovement (const CFixed aX, const
  CFixed aY) Constructor.
• IMPORT_C CFixed Norm () Calculates the euclidian
  norm of the movement.

31
CImagePyramid Class

• Creates a Laplacian or Gaussian image pyramid
  from a given bitmap.
• IMPORT_C CNokiaCVImage  LevelImage (TInt
  aLevel) Getter for an image of certain level.
• Returns
• CNokiaCVImage Pointer to the level image
• static IMPORT_C CImagePyramid  NewLC
  (CNokiaCVImage aImage, TInt aLevels,
  TImagePyramidType aTypeEGaussianPyramid) Construc
  tor.
• Parameters
• aImage CNokiaCVImage to construct the pyramid
  from aLevels How many levels in the pyramid. If
  the number is too large it will be truncated.
  aType Which type of pyramid to construct. Default
  Gaussian.

32
CImageStatistics Class

• Calculating various statistics from
  CNokiaCVImages. CImageStatistics provides the
  following statistical functions for
  CNokiaCVImages
• Histogram
• Mean
• Standard deviation
• Image moments
• m00
• m01
• m10
• m11
• m20
• M02
• Usage
• TReal mean CImageStatisticsMean(iSourceBitmap
  , mean, (TChannel)aChannel)

33
Functions

• static IMPORT_C void Histogram (CNokiaCVImage
  aImage, THistogram aArray, TChannel
  aChannelEGrayChannel) Calculate the histogram
  from a CNokiaCVImage.
• static IMPORT_C void Mean (CNokiaCVImage aImage,
  TReal aMean, TChannel aChannelEAllChannels) Calc
  ulate the mean of CNokiaCVImage. static IMPORT_C
  void StDev (CNokiaCVImage aImage, TReal aStDev,
  TReal aMean, TChannel aChannel) Calculate the
  standard deviation of a CNokiaCVImage.

34
CImageOperations Class

• Provides an simplified access to image
  operations. Each operation is a static method
  that creates and returns the result bitmap.
• Usage
• CNokiaCVImage iTarget CImageOperationsRotate
  (iSource, 45)
• ...
• delete iTarget

35
Functions

• static IMPORT_C CNokiaCVImage  Resize
  (CNokiaCVImage aSource, TSize aSize, TBool
  aBilinearEFalse) Resizes the source image. See
  CResizeOp for more information.
• static IMPORT_C CNokiaCVImage  Convolution
  (CNokiaCVImage aSource, CNokiaCVMatrix aKernel,
  TInt aNormFactor) Performs convolution of
  CNokiaCVImage and a kernel (CNokiaCVMatrix).
• static IMPORT_C CNokiaCVImage  Gaussian
  (CNokiaCVImage aSource, TUint aSize) Gaussian
  smoothing filter.
• static IMPORT_C CNokiaCVImage  EdgeDetectLaplacia
  n (CNokiaCVImage aSource) Laplacian edge
  detection filter. See CEdgeDetectionOp for more
  info.

36
CCamus Class

• Implements the Camus algorithm for estimation of
  optical flow from a series of bitmaps. This class
  creates an COpticalFlow object that contains an
  estimation of the motion vector (x and y motion
  in pixels) for each pixel of the input frame.
• Can use the methods of COpticalFlow to extract
  higher level information such as estimated camera
  rotation/movement.
• The algorithm is highly parameterizable.
• Features of the algorithm and some hints
• Maximum estimated motion is 1 or 2 pixels/frame,
  so it is typically reasonable to use small
  bitmaps (e.g. 16x12 pixels).
• History of several frames is used to achieve sub
  pixel accuracy
• Based on template matching
• Uses grayscale images
• Large motions are hard so having a good frame
  rate is essential since the per-frame motion is
  inversely proportional to the frame rate
• Small bitmaps are preferred (especially when
  estimation of quick movements is required) as its
  faster to compute higher frame rate.

37
Camus Algorithm

• Basic idea
• Simple technique to perform optical flow
  computations using correlation or patch-making
  methods. The idea here is to match features
  between images and computing the SD
  (sum-difference) of patches.
• Reference T. Camus, Real-Time Optical Flow, PhD
  Thesis, Brown University Technical Report
  CS-94-36, 1994.

38
Functions

• static IMPORT_C CCamus CCamusNewL
• (CNokiaCVImage  aImage,
• const TInt aHistorySize KDefaultCamusHistorySiz
  e )
• IMPORT_C void CCamusGetOpticalFlow
  (COpticalFlow  aOpticalFlow, CNokiaCVImage
   aImage, const TInt aCamusOpts
  KDefaultCamusOpts )

39
How to Use CCamus

• COpticalFlow iOpticalFlow COpticalFlowNewL(iF
  irstFrame-gtSize())
• CNokiaCVImage iCamus CCamusNewL(iFirstFrame,
  5)
• iCamus-gtGetOpticalFlow(iOpticalFlow,
  iSecondFrame, CAMUS_SETCAMUSOPTS(CCamusEMedium,
  CCamusEMedium, 3))
• CEgoMovement mean iOpticalFlow-gtMean()

40
CEgoMovement Class

• Simple vector-like wrapper class for pixel motion
  estimation values.

41
Functions

• IMPORT_C CEgoMovementCEgoMovement (const
  CFixed aX, const CFixed aY )Constructor.
• Parameters
• aX Movement along the x-axis.
• aY Movement along the y-axis.
• IMPORT_C CFixed CEgoMovementNorm ( ) Calculates
  the Euclidian norm of the movement.
• Returns
• Norm (the length of the movement in cartesian
  coordinate system).
• IMPORT_C CFixed CEgoMovementDotProduct (const
  CEgoMovement  aMovement  ) Calculates a dot
  product.
• Parameters
• aMovement Vector movement.
• Returns
• Dot product value.

42
CEigenValues Class

• Eigenvalue decomposition of a real matrix. Usage
  
• CNokiaCVMatrix matrix ... CEigenvalues
  eigs
• TRAPD(err, eigs CEigenvalues-gtNewL(matrix))
• if(err ! KErrNone)
• //eigs not found do something
• else
• TReal real eigs-gtRealEigenvalues()
• delete eigs
• ...

43
Functions

• IMPORT_C TReal CEigenvaluesRealEigenvalues
  ( ) 
• Getter for the real part of the eigenvalues
  (iD).
• Returns
• the real part of the eigenvalues in an array
• IMPORT_C TReal CEigenvaluesImaginaryEigenvalues
  ( ) 
• Getter for the imaginary part of the
  eigenvalues (iE).
• Returns
• the imaginary part of the eigenvalues in an array
  

44
CFixed Class

• Fixed point class.
• The class implements basic arithmetic for fixed
  point numbers.
• CFixed can be used instead of TReal when
  performance more critical than accuracy.

45
CImageOp Class

• Image operations interface.
• This class represents an interface that all image
  operation classes must implement. Most of the
  operations implementing this interface are also
  included in CImageOperations.
• There are various image operations that implement
  this interface, e.g. for rotating, scaling and
  filtering bitmap.
• Usage example (CRotationOp)
• iSource ....
• iTargetMode EGray256
• iTarget1 CNokiaCVImageNewL()
• iTarget1-gtCreateL(iCurrSource-gtSize(),
  iTargetMode)
• CRotationOp op(iCurrSource, 90)
  op.DoOperation(iTarget1) ...

46
Functions

• virtual IMPORT_C void CImageOpDoOperation
  (CNokiaCVImage  aTarget  ) pure virtual
• All image operations must implement this method.
  This method does the actual operation.
• Typically you need to allocate and create the
  target bitmap yourself before doing the operation
  
• Parameters
• aTarget target bitmap
• Note
• the function can leave
• Implemented in CArithmeticOp, CColorConversionOp,
  CComparisonOp, CConvolutionOp, CCornerDetectionOp,
  CEdgeDetectionOp, CGaussianSmoothOp,
  CMedianFilterOp, CMorphologicalOp, CResizeOp,
  CRotationOp, CShearOp, CTemplateMatchOp,
  CThresholdOp, and CWarpOp.

47
CArithmeticOp

• CArithmeticOp provides basic arithmetic
  operations for CNokiaCVImages.
• Target and source bitmaps must be of same size.
• Usage
• IMPORT_C void CArithmeticOpDoOperation
  (CNokiaCVImage  aTarget  ) 
• Does the actual operation.
• Parameters
• aTarget CNokiaCVImage to store the result in.
  Must be allocated and created beforehand.
• Implements CImageOp.

48
Image Arithmetic Examples
49
CColorConversionOp Class

• Provides CNokiaCVImage with color conversions.

50
Functions

• IMPORT_C CColorConversionOpCColorConversionOp
  (CNokiaCVImage  aImage, TDisplayMode aTargetMode 
  )
• Constructor. Converts given image to given
  display mode.
• Parameters
• aImage Image to convert aTargetMode Target
  display mode Valid TDisplayModes for aTargetMode
  are
• EGray2
• EGray4
• EGray16
• EGray256
• EColor4K
• EColor64K
• EColor16M
• IMPORT_C void CColorConversionOpDoOperation
  (CNokiaCVImage  aTarget  ) 
• Parameters
• aTarget CNokiaCVImage to store the result in.

51
CComparisonOp Class

• Provides CNokiaCVImage with pixel-wise image
  comparison operations "greater than", "less
  than" and "equal".
• Matching pixels are set white. Non-matching and
  pixels outside the comparison region are set
  black.
• Valid values are
• EGreater
• ELess
• EEqual

52
Functions

• IMPORT_C CComparisonOpCComparisonOp
  (CNokiaCVImage  aImage1, CNokiaCVImage
   aImage2, TComparisonType aComparison )Constructo
  r.
• Parameters
• aImage1 images to compare aImage2 another images
  to compare. aComparison Comparison operation
  .EGreater, ELess or EEqual
• IMPORT_C CComparisonOpCComparisonOp
  (CNokiaCVImage  aImage1, CNokiaCVImage
   aImage2, TComparisonType aComparison,
  TRect aSubregion )Constructor.
• Parameters
• aImage1 images to compare aImage2 another images
  to compare. These two images must be of same bit
  depth and same size
• aComparison Comparison operation .EGreater,
  ELess or EEqual
• aSubregion subregion of images 1 and 2 that are
  compared
• IMPORT_C void CComparisonOpDoOperation
  (CNokiaCVImage  aTarget  ) 
• Parameters
• aTarget CNokiaCVImage to store the result in. The
  target bitmap must be black and white.

53
CConvolutionOp Class

• Convolution operation.
• Performs convolution of CNokiaCVImage and a
  kernel (CNokiaCVMatrix).

O57 I56K11 I58K12 I59K13 I67K21 I68K22
I69K23
54
Functions

• IMPORT_C CConvolutionOpCConvolutionOp
  (CNokiaCVImage  aImage, CNokiaCVMatrix
   aKernel ) Constructor.
• Parameters
• aImage CNokiaCVImage to use as source
  aKernel CNokiaCVMatrix Kernel to use for
  convolution. Only 3x3 and 5x5 matrices are
  supported.
• IMPORT_C void CConvolutionOpDoOperation
  (CNokiaCVImage  aTarget  ) 
• Parameters
• aTarget CNokiaCVImage to store the result in.

55
CCornerDetectionOp

• Corner detection for CNokiaCVImage.
• Moravec corner detector is implemented.
• Defines interest points as points where there is
  a large intensity variation in every direction
  i.e. corners.

56
Functions

• IMPORT_C CCornerDetectionOpCCornerDetectionOp
  (CNokiaCVImage  aImage, TCornerDetectionMethod aM
  ethod EMoravec, TInt aNeighborhoodSize 20,
  TInt aConstant 25 ) Constructor.
• Parameters
• aImage CNokiaCVImage to detect from
• aMethod Which corner detection method to use
• aNeighborhoodSize Size of the neighborhood to use
  for adaptive thresholding
• aConstant a constant value to subtract from the
  mean threshold
• IMPORT_C void CCornerDetectionOpDoOperation
  (CNokiaCVImage  aTarget  ) 
• Parameters
• aTarget CNokiaCVImage to store the result

57
CEdgeDetectionOp

• Edge detection filters using convolution.
• Possible values are
• ESobel
• EPrewitt
• ELaplacian
• ECanny

Canny edge detection with different convolution
sizes
58
Functions

• IMPORT_C CEdgeDetectionOpCEdgeDetectionOp
  (CNokiaCVImage  aImage, TEdgeDetectionMethod aMet
  hod ) Constructor.
• Parameters
• aImage CNokiaCVImage to detect from aMethod Which
  edge detection method to use
• IMPORT_C void CEdgeDetectionOpDoOperation
  (CNokiaCVImage  aTarget  ) 
• Parameters
• aTarget CNokiaCVImage to store the result

59
CGaussianSmoothOp

• Gaussian smoothing operator.
• The Gaussian smoothing operator is a 2-D
  convolution operator used to blur' images and
  remove detail and noise.
• The shape of the kernel represents the shape of a
  Gaussian (bell-shaped') point-spread function.

5 X 5 Gaussian Kernel
60
Functions

• IMPORT_C CGaussianSmoothOpCGaussianSmoothOp
  (CNokiaCVImage  aImage, TInt aKernelSize )
  Constructor.
• Parameters
• aImage CNokiaCVImage a source image
  aKernelSize The number of rows and columns in
  Gaussian kernel Possible values are. 3 and 5.
• IMPORT_C void CGaussianSmoothOpDoOperation
  (CNokiaCVImage  aTarget  ) 
• Parameters
• aTarget CNokiaCVImage to store the result

61
CMedianFilterOp

• Median filter operation.
• The median filter is normally used to reduce
  noise in an image.
• Does a reasonable job of preserving useful detail
  in the image.
• Replaces every pixel with the median of its
  neighboring pixels.

Median Filter 3X3
62
Functions

• IMPORT_C CMedianFilterOpCMedianFilterOp
  (CNokiaCVImage  aImage, TInt aKernelRows )
  Constructor.
• Parameters
• aImage CNokiaCVImage to use as source
  aKernelRows How big a kernel to use. Possible
  values are
• 3 for 3x3
• 5 for 5x5
• 7 for 7x7
• IMPORT_C void CMedianFilterOpDoOperation
  (CNokiaCVImage  aTarget  )
• Parameters
• aTarget CNokiaCVImage to store the result in.

63
CMorphologicalOp Class

• Dilation/Erosion filter.
• Here, the state of any given pixel in the output
  image is determined by applying a rule to the
  corresponding pixel and its neighbors in the
  input image. The rule used to process the pixels
  defines the operation as a dilation or an
  erosion.
• Dilation The value of the output pixel is the
  maximum value of all the pixels in the input
  pixel's neighborhood. In a binary image, if any
  of the pixels is set to the value 1, the output
  pixel is set to 1.
• Erosion The value of the output pixel is the
  minimum value of all the pixels in the input
  pixel's neighborhood. In a binary image, if any
  of the pixels is set to 0, the output pixel is
  set to 0.

64
Functions

• IMPORT_C CMorphologicalOpCMorphologicalOp
  (CNokiaCVImage  aImage, CNokiaCVMatrix
   aStruct, TMorphologicalOperation aOperation )
  Constructor.
• Parameters
• aImage CNokiaCVImage to use as source
• aStruct CNokiaCVMatrix a structuring element to
  use for morphing
• aOperation Which operation to do
• IMPORT_C void CMorphologicalOpDoOperation
  (CNokiaCVImage  aTarget  ) 
• Parameters
• aTarget CNokiaCVImage to store the result in.

65
CResizeOp Class

• Resizes the CNokiaCVImage.
• IMPORT_C CResizeOp (CNokiaCVImage aImage, TBool
  aBilinearEFalse) Constructor.
• IMPORT_C void DoOperation (CNokiaCVImage
  aTarget) 
• Do the actual resize. Take the new size from
  target image and store the result in it.

66
CRotationOp Class

• Rotates a CNokiaCVImage.
• Implements an arbitrary angle rotation.
• IMPORT_C CRotationOp (CNokiaCVImage aImage, TInt
  aAngle) Constructor.
• IMPORT_C void DoOperation (CNokiaCVImage
  aTarget)
•  Do the actual operation.

67
CShearOp Class

• Shear operation for CNokiaCVImage.
• Shears bitmap Horizontally or Vertically (creates
  a parallelogram).
• IMPORT_C CShearOp (CNokiaCVImage aImage, TInt
  aAngle, TDirection aDirectionEHorizontal) Constru
  ctor.
• IMPORT_C void DoOperation (CNokiaCVImage
  aTarget) 
• Do the actual operation.

68
CTemplateMatch Class

• Template matching operation for two
  CNokiaCVImages.
• Used for finding small parts of an image that
  match a template image.
• Algorithm
• This method is implemented by first creating a
  subimage (the template).
• The center of the subimage is simply moved over
  each (x,y) point in the a candidate image.
• The sum of products between the coefficients in
  the candidate image and the corresponding
  neighborhood pixels in the area spanned by the
  template is calculated to determine best match.
• Also applicable for color images.

69
Functions

• IMPORT_C CTemplateMatchOpCTemplateMatchOp
  (CNokiaCVImage  aImage, CNokiaCVImage
   aTemplate )
• Constructor for template matching operation.
• Parameters
• aImage CNokiaCVImage to match against aPattern 
• CNokiaCVImage containing the template to match
• IMPORT_C void CTemplateMatchOpDoOperation
  (CNokiaCVImage  aTarget NULL  )
• Parameters
• aTarget CNokiaCVImage to store the resulting
  correlation values
• Implements CImageOp.
• IMPORT_C void CTemplateMatchOpDoOperation
  (TPoint  aPoint, CNokiaCVMatrix  aCorrMatrix
  NULL ) Parameters
• aPoint resulting matching poin
• aCorrMatrix correlation matrix

70
CThresholdOp

• Provides CNokiaCVImage with a threshold
  operation.
• Thresholding is the most common method of
  segmenting images into particle regions and
  background regions.
• Creates an black and white bitmap from
  CNokiaCVImage by thresholding.
• Implements two thresholding methods
• simple with static threshold value
• Local adaptive Selects an individual threshold
  for each pixel based on the range of intensity
  values in its local neighborhood. This allows for
  thresholding of an image whose global intensity
  histogram doesn't contain distinctive peaks.

71
Functions

• IMPORT_C CThresholdOpCThresholdOp
  (CNokiaCVImage  aImage, TUint8 aThresholdValue )
• Constructor for thresholding with given
  value.
• Parameters
• aImage CNokiaCVImage to apply the thresholding
  to.
• aThresholdValue an integer value (0-255) to use
  as a threshold.
• IMPORT_C CThresholdOpCThresholdOp
  (CNokiaCVImage  aImage, TInt aSize,
  TInt aConstant, TThresholdType  EMean )
• Constructor for adaptive thresholding.
• Parameters
• aImage CNokiaCVImage to apply the thresholding
  to.
• aSize aSize x aSize neighborhood to use for
  adaptation
• aConstant Constant value that is subtracted from
  the mean.
• IMPORT_C void CThresholdOpDoOperation
  (CNokiaCVImage  aTarget  )
• Parameters
• aTarget CNokiaCVImage to store the result in.

72
CWarpOp Class

• Provides CNokiaCVImage with a warp operation The
  result of the operation is like dragging an
  elastic image from one point to a given
  direction.
• IMPORT_C CWarpOpCWarpOp (CNokiaCVImage
   aImage, TPoint aSource, TPoint aDestination )
• Constructor.
• Parameters
• aImage Source image. aSource Source point to
  start the warp from. aDestination Destination
  point to warp towards.
• IMPORT_C void CWarpOpDoOperation (CNokiaCVImage
   aTarget  ) 
• Parameters
• aTarget CNokiaCVImage to store the result in.

73
Demos
74
Outline

• Cube
• Motion Estimation
• Image Processing
• Game

75
Cube

• Using a cube to represent camera movement
• Obtaining optical flow using Camus algorithm
• Mean movement of optical flow ? Cubes movement
  along x and y
• Mean rotation of optical flow ? Cubes rotation
• Depth info movement along z axis ? Cubes size
• NokiaCV classes used
• CCamus
• CEgoMovement
• COpticalFlow
• CNokiaCVImage
• CPixelAccess
• CFixed

76
Motion Estimation

• Calculate and display motion history image
• Calculate optical flow, indicate horizontal and
  vertical motion
• NokiaCV classes used
• CMotionHistoryImage
• CNokiaCVImage
• CCamus
• COpticalFlow

77
Image Processing

• Loads two images
• Perform image processing based on user selection
• NokiaCV classes used
• All image processing classes