DEPTH IMAGING AND APPLICATION IN POTATO QUALITY GRADING
DEPTH IMAGING AND APPLICATION IN POTATO QUALITY GRADING
Qinghua Su
Beijing Information Science & Technology University
Copyright © 2023 by Cayley Nielson Press, Inc.
ISBN: 978-1-957274-15-7
Cayley Nielson Press Scholarly Monograph Series Book Code No.: 213-11-2
US$168.50
Preface
As one of the world's most important crops, potatoes play an important role in maintaining the stability of the global food supply. Many countries, including China, believe that food supply security is a basic condition for maintaining national stability and development. Therefore, potatoes can not only solve the problem of international food shortage, but also promote the development of international trade. In recent years, with the continuous improvement of planting technology, the global production and trade volume of potatoes have also been continuously increasing. However, the development of traditional potato quality grading technology is relatively slow. Currently, it still relies on manual sorting in many countries and regions. Because workers can not keep their attention for a long time under huge work pressure and their understanding of grading standards is inconsistent, large amount of wrong potato grading often occurs. This result not only affects farmers' income, but also causes serious waste in the potato processing due to unqualified raw potatoes. In addition, with the continuous increase of manual wages, the cost of manual grading of potatoes has under challenge. Therefore, achieving automation of potato quality grading is imperative.
Traditional grading system mainly uses cameras to capture potato color images, and achieves potato quality grading through color information analysis. This method can reach high success rate for certain defects detection, such as green skin, surface rot and mechanical damage. Due to the variety of shapes of potatoes growing underground, the appearance defects, such as bending, bump and hollow, are widely existing. These abnormal samples may fail to be detected and grade to wrong quality groups, the 3D appearance information cannot be fully perceived in 2D color images.
In response to such issues, we have decided to build a machine vision system based on depth cameras, which can obtain depth images of potatoes with 3D shape information. Unlike each pixel in a color image that stores color information, each pixel in a depth image stores the distance from the target to the camera. Therefore, the potato 3D surface features can be sensed and used for bump and hollow defects detection.
To capture high-quality depth images, we have constructed a specialized depth imaging system, and developed the image acquisition software based on OpenCV and OpenNI framework. Then, each potato surface features are analyzed and extracted for shape analysis, defect detection, and overall quality grading.
In recent years, machine learning technology has developed rapidly and has been widely applied in fields such as object recognition and feature detection. Hence, we also apply machine learning technology to the field of potato quality grading. By developing a machine learning model based on convolutional neural networks, we can directly input potato depth images and get the corresponding quality level of the samples. The experiment achieved good grading results.
Since color and depth images of potatoes are actually collected simultaneously in data collection step, a novel algorithm is developed for potato 3D model rebuilding. The method is based on Point Cloud Library and OpenGL technology, and it shows the advantage in solving the problem of data traceability, especially when users have objections to automatic quality classification results. This model not only displays 3D potato shape model, but also supports scaling and 360-degree rotation operations.
Overall, we believe that with the development of machine learning and depth sensing, potato quality grading systems will become more intelligent, efficient and low-cost.
Qinghua Su
Beijing Information Science & Technology University
Beijing, China
November 16, 2023
Contents
1 Introduction 1
1.1 Motivation 1
1.1.1 Potato market analysis in worldwide 1
1.1.2 Potato market analysis in China 2
1.1.3 Potato grading status and problem 4
1.1.4 Achievements in agricultural product grading by machine vision technology 6
1.1.5 Achievements in potato grading by machine vision technology 12
1.2 Objective 14
1.3 Research contribution 15
2 Depth imaging system 17
2.1 Depth camera 17
2.1.1 Structured light 19
2.1.2 Stereo depth 21
2.1.3 Time of Flight (TOF) 23
2.1.4 Overview 25
2.1.5 Primesense Carmine 1.09 camera 26
2.2 Illumination system 27
2.2.1 Illumination sources 28
2.2.2 Key factors of vision illumination 30
2.2.3 Techniques in illumination 31
2.3 Depth imagining system building 33
3 Image processing with OpenCV 36
3.1 Install OpenCV libraries 37
3.1.1 Prebuilt version 38
3.1.2 Build from sources 38
3.2 The core functionality 42
3.2.1 Mat: the basic image container 42
3.2.2 Image blurring 45
3.2.3 Thresholding operation 50
3.2.4 Morphological operations 57
3.2.5 Contour detection 61
4 Depth camera controlling via OpenNI framework 85
4.1 Install OpenNI framework 85
4.2 Creating new project with OpenNI 86
4.3 Develop an application 87
4.4 Related Classes and APIs for camera control 88
4.4.1 Class OpenNI 88
4.4.2 Class Device 89
4.4.3 Class VideoStream 91
4.4.4 Class VideoMode 92
4.4.5 Class SensorInfo 93
4.4.6 Class CoordinateConverter 94
4.5 Demo code for controlling Primesense camera 96
4.6 Improvements on depth imaging system 103
4.6.1 Adjustment for camera height 104
4.6.2 Area loss by light incident angles 105
4.6.3 Improvement of depth image quality 106
5 Potato quality grading via image processing 112
5.1 Chinese official grades and specifications of potatoes 112
5.1.1 Specification of the official grades of potatoes 112
5.1.2 Specification of mass level 114
5.1.3 Other specification 114
5.2 Features acquisition via depth image 115
5.2.1 Morphological features 115
5.2.2 Demo code for morphological features calculation 120
5.3 Surface deformities detection 132
5.3.1 Contour shape detection 132
5.3.2 Bump detection 133
5.3.3 Hollow detection 134
5.3.4 Demo code for deformity detection 135
6 Potato quality grading via machine learning 151
6.1 Python 151
6.1.1 Introduction 151
6.1.2 Install Python 154
6.2 TensorFlow 156
6.2.1 Introduction 156
6.2.2 TensorFlow Core 158
6.2.3 High level API 163
6.3 Keras 163
6.3.1 Introduction 163
6.3.2 Set up the environment 166
6.3.3 Install Keras 167
6.3.4 Data loading & preprocessing 167
6.3.5 Building models with the functional APIs 172
6.3.6 Training and validating models 174
6.3.7 Evaluating test performance & generating predictions 176
6.4 Convolutional Neural Network (CNN) 177
6.4.1 Artificial Neural Networks (ANN) 177
6.4.2 CNN architecture 178
6.4.3 Working principle 178
6.4.4 Layers in CNN 180
6.4.5 Features of CNN 181
6.5 Potato quality grading based on depth image and CNN 182
6.5.1 Potato samples 182
6.5.2 CNN model for potato quality grading 184
6.5.3 Demo code of the CNN model 185
6.5.4 Grading result 192
7 Potato 3D shape model rebuilding 193
7.1 Point Cloud Library (PCL) 193
7.1.1 Install PCL 193
7.1.2 Related Classes and Functions 194
7.2 Demo code for 3D model building 197
Appendix A: Programming with Python 204
A.1 Variable and data structures 204
A.2 Input 209
A.3 Selection 210
A.4 Functions 211
A.5 Iteration (Looping) 213
A.6 Modules 218
A.7 NumPy 222
References 232
Readership
This book should be useful for students, scientists, engineers and professionals working in the areas of optoelectronic packaging, photonic devices, semiconductor technology, materials science, polymer science, electrical and electronics engineering. This book could be used for one semester course on adhesives for photonics packaging designed for both undergraduate and graduate engineering students.
