CMOS Image Sensor Processing with FPGAs

An overview of the image sensor processing pipeline, including camera front-end IP implementation on a low-cost FPGA fabric, Bayer to RGB conversion and efficient use of memory.

By Suhel Dhanani, Senior Manager, DSP, Altera Corporation
and Dr. Andy Robertson, CEO, Bitec

Page 1 of 6
Video/Imaging DesignWire
(11/2/2009 2:00:19 AM)

CMOS sensors are rapidly becoming the de-facto standard when it comes to digital image capture. However, the output of an image sensor is not in a standard video format that can be processed – resized (scaled), de-interlaced, or composited for example.

In fact, the sensor image must first go through a processing pipeline that may include a range of functions such as pixel correction, noise reduction, Bayer-to-RGB conversion, color correction, gamma correction, and other functions, as shown in Figure 1.

View full size

Figure 1. A Processing Pipeline for a typical CMOS Image Sensor

Given the wide variety of image sensors used and the range of application requirements, each processing pipeline will be unique — requiring in many cases hardware customization to produce an optimum image.

This article discusses some of the common functions implemented in such a sensor processing pipeline, and then delves into detailing a fully-verified sensor processing pipeline and the implementation of this design on a low-cost FPGA fabric.

CMOS Sensor Basics
Generally a CMOS sensor will consist of a grid of pixel sensors, each containing a photo-detector and an active amplifier.

CMOS sensors will record the image data in grayscale. However color information is gathered by applying a color filter over the pixel grid. Such a color filter array (CFA) allows only light of a given primary color (R, G, or B) to pass through – everything else is absorbed, as shown in Figure 2.


Figure 2. Color Filter Array allows R, G, or B colors to pass through

This means each pixel sensor collects information about any one color. This is important to note since the data for each pixel is not composed of three colors (RGB), rather it is composed of any one color. The other colors need to be ‘guessed’ or interpolated by the electronics behind the CMOS sensor.

NEXT: Color Filter Arrays, Image Signal Processing

Page 1: next page

Pages: 1 2 3 4 5 6