A Camera Front-End IP
Implementing the de-mosaic techniques described earlier in an FPGA is a non-trivial task. The efficient use of FPGA resources along with pipelined operation ensures a reliable low power design.
Parameterization is necessary in order to accommodate different camera image geometries and pixel depths. With the growing interest in FPGA technology and camera sensor processing offering a fully-parameterized, device-optimized solution for the integration of front end processing on a FPGA device is a key differentiator. The, “Camera Front-End IP core” (shown in Figure 10) enables fast time to market and seamless integration in to the Altera Video Processing IP Suite.
The structure of the Camera Front-End IP shows how the Bayer pattern camera data is fed into a line buffer that gives vertical access to the mosaic data. The de-mosaic algorithm is then executed in a pipelined manner yielding an RGB triple for each pixel location at each pixel clock cycle. The de-mosaic algorithm block implements both the simple Nearest Neighbor interpolation and the advanced, adaptive algorithm employing proprietary adaptive metrics for enhanced image quality.
Figure 10. Camera Front-End IP Block Diagram
Figure 11 is a block diagram showing the line buffer. Each line is implemented as an n-length shift register where n is the camera horizontal resolution.
Figure 11. Line Buffer Block Diagram
FPGAs such as Altera’s Cyclone III family provide dedicated embedded memory blocks which are ideal for image line buffering (ranging from 46 blocks to 432 9K). Although abundant, care must be taken to reduce the number of memory blocks required to implement the line buffers. For example, a 1024 image line with 10-bit pixel depth would require 10.42Kbits of memory which would map into two 9K embedded memory blocks. If three lines are required then six memory blocks would typically be used. However, a quick calculation shows that three, 1024 lines requires only 30.72kBits or four 9K RAM blocks. Clearly two memory blocks would be consumed unnecessarily. The line buffer is able to combine lines into a continuous memory region thus consuming the least amount of memory required for line buffering.
The de-mosaic algorithm is performed in a pipelined manner. Pipelining allows high-speed operation using minimal logic. Figure 12 shows the implementation of the simpler Nearest Neighbor algorithm.
Figure 12. Nearest Neighbour Bi-linear Interpolation Pipeline Operation
The pipeline consists of three stages. The Bayer stream enters into the algorithm at pipeline stage 0. At this point the intermediate values X0 and X1 are calculated and passed to Pipeline stage 1. In pipeline stage 1 intermediate values are calculated and passed to the next stage. At pipeline stage 2 the calculated RGB triple is available.
The pipeline consists of three stages. The Bayer stream enters into the algorithm at pipeline stage 0. At this point the intermediate values X0 and X1 are calculated and passed to Pipeline stage 1. In pipeline stage 1 intermediate values are calculated and passed to the next stage. At pipeline stage 2 the calculated RGB triple is available.
Finally, the resulting RGB triples are then available for post-processing the FPGA fabric. Typically, the pixel stream undergoes some sort of color correction and perhaps dead-pixel removal. Image enhancement via 2 or 3-D filtering is another common requirement. The FPGA then normally outputs the image either to a flat panel LCD or converts it to some video transport standard such as DVI, HDMI or SDI. Alternatively, an H.264 encoder can be implemented in the FPGA fabric to achieve a low-power, reconfigurable network IP camera.
NEXT:A CMOS Sensor Based Camera Design
Page 5: next page
