A REAL-TIME DEFECT DETECTION IN PRINTED CIRCUIT BOARDS APPLYING DEEP LEARNING

Inspection of defects in the printed circuit boards (PCBs) has both safety and economic significance in the 4.0 industrial manufacturing. Nevertheless, it is still a challenging problem to be studied in-depth due to the complexity of the PCB layouts and the shrinking down tendency of the electronic component size. In this paper, a real-time automated supervision algorithm is proposed to test the PCBs quality among different scenarios. The density of the PCBs layout and the complexity on the surface are analyzed based on deep learning and image feature extraction algorithms. To be more detailed, the ORB feature and the Brute-force matching method are utilized to match perfectly the input images with the PCB templates. After transferring images by aiding the RANSAC algorithm, a hybrid method using modern computer vision algorithms is developed to segment defective areas on the PCBs surface. Then, by applying the enhanced Residual Network-50, the proposed algorithm can classify the groove defects on the surface mount technology electronic components which minimum size up to 1×3 mm. After the training process, the proposed system is capable to categorize various types of overproduced, recycled, and cloned PCBs. The speed of the quality testing operation maintains at a high level with an average precision rate up to 96.29 % in case of good brightness conditions. Finally, the computational experiments demonstrate that the proposed system based on deep learning can obtain superior results and it outperforms several existing works in terms of speed, precision, and robustness.


Introduction
Printed circuit boards (PCBs) [1][2][3][4][5] are a mandatory part of most modern electronic devices. Furthermore, the upward trend of consumer electronics products and the increasing demands for industrial electronic equipment have promoted applications of the accuracy of PCBs assembly operation. The circuit layout of a PCB presents a unique and single pattern in the image. Since the electronic device parts are shrinking down to minimize sizes, PCB is becoming more and more sophisticated and dedicated. Hence, detecting minor defects on the PCBs surface plays a significant role in the new era of intelligent industry. During manufacturing processes, numerous PCB defects can be engendered such as mistaken open circuits, missing components, causing the yield to drop, and component shifting.
The past decade has witnessed a huge growth in this PCB inspection field. One of the most widely used methods is the reference inspection method. In [6], the reference-free path-walking method is designed to detect component errors on high-density packages. However, it is only suitable for those with a ball grid array. The PCB reverse engineering through X-ray computed tomography (CT) is also proposed in [7]. Nevertheless, the X-ray CT framework needs a complicated configuration process to estimate exactly the separate sides of the PCB structure. Besides, performing the alignment of the reference images and the testing data precisely is the remarkable difficulty of this solution. In general, the image alignment could be solved by two well-known approaches: region-based match [8,9] and feature-based match [10,11]. The region-based approach directly uses the pixel values for the match. The similarity parameters are measured in detail Computer Sciences by aiding the simple image subtraction, such as Sum of Squared Differences (SSD) [12], Sum of Absolute Differences (SAD) [13], and Normalized Cross Correlation (NCC) [14,15]. Unfortunately, these methods are sensitive to transition and rotation changes. In addition, they require the repeat evaluation to test all possible geometric transformations of the input images, the computation load could be over-intensive. Meanwhile, the feature-based approach applied the edge points to calibrate the images continuously. It is called a straightforward feature representation. The Scale Invariant Feature Transform (SIFT) [16] and the RANdom SAmple Consensus (RANSAC) [17] are commonly used to search for the best edge points. Since these algorithms need a series of complicated processes to fit the matching rates, they are computationally very time constraints. To overcome these above drawbacks, Tsai and Chieh [18] offer an edge-based Expectation-Maximization algorithm to optimize the electronic part positions on the PCB surface. Even though the positioning precision could reach to a higher level, but the detected objects fail to indicate a minor geometric deviation with respect to the template inquiry.
With the development of deep-learning techniques, convolutional neural networks (CNN) [19][20][21][22][23] are also applied in the 4.0 manufacturing systems. Compared with traditional detection solutions, it has a plethora of merits such as high-speed detection, cost reduction, and good precision. In the literature review, the CNN-based models are incorporated with Support Vector Regression in [24]. Their study provides a powerful methodology to obtain the geometric transformation measurements of a test image without applying any manual search. Moreover, the subpixel accuracy of the images is boosted up rapidly within a few milliseconds, so this deep-learning model is appropriate for real-time implementation. The major weakness in their study is the intensity of the illumination. If the illumination condition is not well-controlled and changed terribly, the quality of the inspection procedure is seriously affected. In [25], an Improved Faster-RCNN and Feature Pyramid Network in run-time based on the Surface-mount technology (SMT) generation line, which belongs to the non-reference method is applied to deal with the illumination issues. In [26], the CNN based end-to-end network is developed in order to expand the size of the inputs. Additionally, they design two modules (the module of threshold estimation and the module of multi-label scoring) to enable multi-label classification in dysfunctional screen-printer. In [27], the architect ALexNet based on the CNN is discovered to locate six types of defects on the PCB surface (correct soldering, incorrect soldering, missing soldering, excess soldering, short circuit, and undefined object). The modified and compact tiny-YOLO-v2 network [28] is also integrated to classify various objects in a single picture of PCBs. Nonetheless, these approaches failed to control all characteristics of the same type of defects due to different materials and processes in PCB production. Hence, the detection performance was declined significantly in some negative cases. In addition, these techniques are highly complex and require several billion floating-point operations per second (FLOP) for a single pass over to achieve an image resolution of 224×224 in case of mimic components. As a result, the robustness against the noise of the system is declined rapidly.
In light of the remarkable importance and advantages previously mentioned, a real-time defect detection system is explored to ensure the quality of the PCBs surface. The proposed method is designed by coupling the ORB feature, the Brute-force matching method [29,30] with the RANSAC algorithm. In particular, the PCB features are extracted and collated with the template data based on the ORB feature. The matching rates are adjusted by utilizing the Bruteforce matching approach. Next, the geometric transformation of the input images is operated by aiding the RANSAC algorithm to locate defective areas on the PCBs surface. Then, the enhanced Residual Network-50 [31,32] is developed and integrated with the control system to classify the groove defects. Compared to the existing works, the proposed approach has several advantages as follows: 1. By sorting in order a large number of layers and the Focal Loss function inside the Resnet-50 network, the proposed method is adaptable to point out the errors of the electronic components and check the component orientation. Besides, the proposed approach is consistent with classifying the mimic components which have the size of 1×3 mm.
2. The accumulation of the PCB defect database requires a short time for obtaining and updating. Hence, the self-calibrated ability of the proposed method is improved.

Computer Sciences
3. The proposed approach is applied on a real-time inspection system to ensure the PCBs quality. Besides, the display of the PCBs detailed information is built automatically based on the Graphical User Interface using PyQt. Thus, the workforce and the manual resources are limited steadily.

Materials and methods
Before operating the system, the users select the types of PCBs that need to be inspected. This pre-processing is complete automatically. After that, the core system operation is indicated as in Fig. 1. Overall, there are two main stages included in this process. The first stage is finding and creating ROI errors in PCBs. To be more detailed, this stage is performed over five sub-steps. To begin with, ORB features of the input images are extracted carefully. Then, these features are matched with the similar ORB features in the template data by aiding Bruce -Force matching method. As a result, the best edge key points will be found. By integrating those best key points with the RANSAC algorithm, the transform matrix is figured out. This matrix is used to calibrate the input images into the same size and orientation as the PCBs template. Since then, both the input images and the template data are converted to YUV color space and performed image subtraction. Next, the resulting images are filtered noise by using the opening method and threshold in order to create ROI errors of the PCBs. At the second stage, all types of ROI errors are classified based on the RESNET-50 models to output the names of the missing components or check the orientation of the components.

Fig. 1. Flowchart of the proposed algorithm
After receiving from the camera, the image data is calibrated to eliminate distortion which could affect the accuracy of the proposed algorithm. The difficult problem is matching perfectly the input one with the PCBs template images. It is dealt with this issue by establishing the most suitable transform matrix between two images. Assume that [u i , v i ] is the coordinate of a point in the input image and ′′ is the coordinate of a point in PCB template image, H is the homography transformation matrix. The 2D -form of the matrix H is defined as: (1) The equation (1) is solved by applying the singular-value Decomposition algorithm [33]. It is clear that at least four pairs of the points are utilized to calculate the matrix H. To find four pairs of points which describe the image features, the ORB feature and Brute-force matching method [29,30] is used. In particular, the ORB feature is adopted to search extracted key points from the two images. Brute-Force matcher is used to calculate the distance value between each pair of key points in two images.
After the calculation process, the matrix H is created. However, one major shortcoming of the distance value is treating the feature vectors equally. Hence, some pairs of points could be matched wrong and become outliers. To handle this drawback, the RANSAC algorithm [17] is applied. Basically, the RANSAC algorithm chooses random pairs of feature points to calculate the matrix H. The RANSAC starts with a simple model based on a small subsample of the data, then uses the remaining data points to identify consensus and outlier points. The outliers are removed, and the model is recomputed. Finally, the most suitable matrix H is established based on the result pairs of points. The input image is transformed into the template image as illustrated in Fig. 2.
The optimized samples of the RANSAC algorithm K are chosen by the formula below: where ∂ is the probability that a point is an outlier; n is the number of points in a single sample; K is the number of optimized samples; ρ is desired probability that a good sample is created.
After transforming the input images successfully, both the result images and the PCB template are converted into YUV color space to decline the impact of the luminance. Since then, they are called the YUV dataset. At the next step, the absolute difference of each channel in this YUV dataset is computed. Let's denote that I is a pixel value of the input images, T is a pixel value of the template images, c is one of three Y, U, V channels, E is absolute difference pixel values. The absolute difference is expressed as: The result images are transformed into binary images to segment error areas. The error areas are called a blob. Since the noise phenomena that appear in the image could decline the resolution and the quality of the results, the OPENING method [34] is applied. This method is designed based on two consecutive methods: Dilation and Erosion [35]. Therefore, the ratios of the distorted noise are eliminated by utilizing three kinds of thresholds with the blob detailed parameters as illustrated in the group of inequalities as follow: where w is the blob width, h is the blob height, s is the blob area.

Computer Sciences
The process of finding error areas is shown in Fig. 3.
The PCB defects can be generated in various production processes, such as missing values, lacking components, mistaken open circuits, and short circuits, causing the yield to drop. In this paper, our system concentrates on two popular types of errors on the PCBs surface: component identification and mimic components are reversed. Thus, two deep learning models are used to complete these tasks. Specifically, RESNET-50 is the main backbone of the proposed system. RESNET model has the advantage of building a CNN model with hundreds or thousands of layers without being affected by the Vanishing Gradient problem. Vanishing Gradient problem occurs encountered when training deep learning model with gradient-based learning and backpropagation. In such a case, weights of the neural network are updated proportionally to the partial derivative of the loss function with respect to the current weight in each loop of the training process. However, if the gradient value vanishing is small, it may prevent the weight from changing its value. In some cases, this problem can completely end the deep learning model from deeper training. In order to solve this problem, the RESNET model uses Residual block. It includes two 3×3 convolutional layers with the same number of output channels and skips connections, or shortcuts to jump over layers. In this paper, the enhanced RESNET-50 network is applied with 50 layers. As mentioned above, there are two RESNET-50 models integrated in the proposed system. The first model is used for classifying the name of missing components and the second one is used for checking the orientation of the components. The input of the first model is the error areas detected in the previous step which is resized into 300×300 pixels. The output of the model is the name of the electronic components. In the second model, the input is pre-defined components which need to be detected the orientation such as a microchip, a capacitor, etc. The output of this model is the respective orientation of the components. If these values are similar to the predefined orientation of the components, the components are correct. In contrast, the orientation checking process is conducted continually until reaching the best results.
The loss function plays a vital role in the retention of precision rate in a deep learning network. In a normal classification network, Cross-entropy loss is usually utilized in semantic segmentation tasks. The equation of the Cross-entropy loss is expressed as: Ep dp tn kwh j where as N, K and P are the mini-batch size, the number of classes and number of pixels, respectively; d nkwh is a one-hot vector and d k = 1 when k is a true label; p t is the model estimated probability for the class.
However, there has been a huge bias in the number of each electronic component image, so the normal loss function such as Binary cross-entropy or categorical cross-entropy could not be adaptable to maintain the high accuracy of the models. For instance, the dataset might have 2000 images of resistors but there are only 100 push button images. Thus, the Focal loss [36] is applied in our mentioned models. This solution improves on addressing class imbalance during the training process. To be more detailed, the Focal Loss calculates a modulating term to cross-entropy with γ is the negative focus parameter. If γ = 0, the Focal loss equals the Cross-entropy loss. Fig. 4 presents the hardware configuration of the proposed system. To begin with, a 5 Mpx webcam is installed to take the PCBs pictures. The images are then inserted into the Jetson Nano Developer Kit. After that, the NVIDIA Jetson Nano Developer Kit B01 is applied as the main core to perform the image processing algorithms. This kit allows running multiple Neural Networks in parallel mode for applications such as image classification, object detection, segmentation, and speech processing. Moreover, it processes several high-resolution sensors simultaneously. All platforms are the friendly user that operates at as little as 5 Watt. The Jetson Nano offers 472 GFLOPS with a quad-core 64-bit ARM CPU, an integrated 128-core NVIDIA GPU, and 4 GB LPDDR4 memory. At the next step, the input images are converted into the analog signal and transmitted to the Arduino Mega. The Arduino Mega 2560R3 is the center controller for this system.

Results and discussion
The electromechanical system includes five main parts: the control box, the convey belts, the images processing box, the picking manipulator, and the HMI display monitor. The conveying system is divided into two separate convey belts. The first conveyor belt is used to move PCBs into the images processing box. Whereas the mission of the second conveyor belt is conducted the PCBs classification. The images processing box includes a camera and embedded computer JETSON NANO. Besides, the lighting sub-system is set up in the images processing box to optimize the classification accuracy. The picking manipulator is operated by a stepper motor, a pneumatic cylindrical, and vacuum equipment. If a PCB is detected as a failure, the manipulator picks it out of the convey belt. Finally, the HMI observation screen integrated with the Graphical user interface (GUI) displays detailed information for users to control the quality of the PCBs. The layout of the electromechanical system is shown in Fig. 5. The designed system contains two mentioned RESNET-50 models. The first model is utilized to classify the component errors. In the first model, the dataset is trained with 7800 images of more than 20 different types of electronic components such as resistor, ceramic capacitor, main chips, etc. Meanwhile, the second model is applied to inspect the orientation of the PCBs components. The second model dataset includes 15000 component images with different orientations. The input dataset is shown in Fig. 6. The configuration of the local computer is as follows: Windows 10 Pro 64-bit, Intel® Core™ i5-8500 CPU @ 3 GHz (6 CPUs), and 8 GB RAM. After finishing the training process, the highest precision rate of the first model is about 99.12 % on the validation dataset while the proportion for the second model could peak at 98.34 %.
To present the detailed information on the PCBs, let's develop a display application based on PyQT Open Library. After capturing by the camera system, the initial PCBs images are shown on the designed monitor. Then, the processing results are obtained and added to the data fields of the main window for the purpose of evaluating the PCBs quality. The main display interface is demonstrated in Fig. 7.   Fig. 6. The data set of training models: a -the first training model; b -the second training model Fig. 7. The display interface of the proposed system The designed system is evaluated among the different scenarios: poor brightness condition, full brightness condition. In order to show the improvement in performance, the proposed approach is compared to the tiny YOLOv2 algorithm [28]. The evaluation criteria are divided into three main elements: the lack of the electronic components, the component orientation, and the defect detection of the PCBs surface.

Computer Sciences
The experiments are conducted with 3 types of PCBs: Adruino mega, Adruino Uno, and Adruino Uno SMD. To be more specific, the number of test boards is 1000 units per kind. Each board is tested 200 times in total. The results are demonstrated in the sub-sections below. Fig. 8-10 present the comparisons of the output results between the two mentioned approaches. In particular, when operating in good illumination, the possibility to inspect the passed PCBs is similar between the two methods as indicated in Fig. 9. Nevertheless, it is remarkable that the proposed method always maintains better performance compared with the tiny YOLOv2 in the case of detecting failed PCBs. For instance, in Fig. 10, the proposed approach could precisely detect all kinds of errors on the PCB surface such as two scratched 1x3 mm capacitors, one missing button, one missing main chip, and one reversed capacitor. In contrast, the tiny YOLOv2 not only shows the insufficiency of failed components but also displays wrong results.    .08 %, and 77.53 %, respectively. Especially, the tiny YOLOv2 method could not identify the Adruino Mega surface because the size of some electronic components is smaller than 1×3 mm. As a result, it can't inspect the quality of mimic components. On the other hand, it is noticeable that the proposed system still detects the Adruino Mega errors with the precision rate peak at 83.48 % even when the light intensity is declined significantly. In conclusion, the most remarkable characteristic of the proposed system to emerge from the experimental results is that it is well-adjusted to be embedded in the 4.0 industrial manufacturing. Because of the limited local computer configuration, the proposed system neglects some tiny cracks or scratches on the PCB surfaces. In future work, other AI inspection algorithms will be conducted to further improve this inspection.

Conclusions
In this paper, a real-time defect detection approach is proposed to test the PCBs quality among different scenarios. The detailed structure of the designed method is calibrated based on deep learning image feature extraction algorithm. An innovative strategy, which is combined bet ween ORB feature, Brute-force matching, and RANSAC algorithm, is devised. Moreover, the enhanced Resnet-50 network is embedded inside the system to identify the electronic components on the PCBs surface. Besides, the PCB information display is also developed by aiding the PyQT Library. In summary, our method has several advantages. To begin with, the technique is robust against noise. Secondly, an obvious advantage is applying the Resnet-50 network to speed up the computations. This holds true even after implementing the two mentioned models. Finally, the method can be applied to detect defects in automatic inspection in real-time.