Abstract

CNNs enables end-to-end learning with- out feature extraction and in-situ estimation of the process outputs.

cnn使端到端学习没有特征提取和现场估计的过程输出。

The papers was classified into 5 groups: four for supervised learning models and one for unsupervised learning models.

论文分为5组:4组为监督学习模型，1组为无监督学习模型。

The classification of supervised learning groups was based on the application of transfer learning and data augmentation.

监督学习组的分类是基于迁移学习和数据增强的应用。

Supervised learning with training from scratch

Original data utilization

Z. Guo et al.16proposed a model for classifying normal and defective welds by applying CNN to electric resist- ance welding in the line pipe manufacturing process, and achieved an accuracy of 99.01%.

Z. Guo等人16)将CNN应用于线管制造过程中的电阻焊，提出了一种正常焊缝与缺陷焊缝的分类模型，准确率达到99.01%。

A. Kjumaidi et al.17proposed a CNN-based defect de- tection model using photo images of arc welding bead surface as input, and using only 3 hidden CNN layers and 2 hidden FC layer, test accuracy of 95.83% was achieved for 4-class classification of defects (good, po- rosity, undercut, and spatter).

a . Kjumaidi等17提出了一种基于CNN的缺陷检测模型，以弧焊道表面的照片图像为输入，仅使用3个隐藏CNN层和2个隐藏FC层，对4类缺陷(良、po- sity、咬边、飞溅)进行分类，测试准确率达到95.83%。

D. Bacioiu et al.20) constructed various DNN models for welding defects classification in GTA welding of aluminum 5083 alloy and compared the model per- formances.

D. Bacioiu等。20)建立了5083铝合金GTA焊接缺陷分类的DNN模型，并对模型性能进行了比较。

Z. Zhang et al.22) used a CNN model to determine the penetration state of tailor-rolled blank (TRB) in-situ la- ser welding. The applied penetration state classification was 2-class (incomplete penetration, complete penetration), 3-class (incomplete penetration, desirable-complete penetration, overpenetration), and 4-class (incomplete penetration, desirable penetration, complete penetration, overpenetration). Six CNN models with different number of kernels and convolutional layers were trained and evaluated in comparison.

张振宇等22)利用CNN模型确定了拼焊毛坯(TRB)原位la- ser焊的熔深状态。应用渗透状态分为2级(不完全渗透、完全渗透)、3级(不完全渗透、理想-完全渗透、过渗透)和4级(不完全渗透、理想渗透、完全渗透、过渗透)。对6个不同核数和卷积层的CNN模型进行训练和比较。

Data augmentation

W. Hou et al.23) investigated a defect classification model (good, incomplete penetration, pore, slag inclusion, crack) from X-ray images of various welds in an open database GDXray. First, 3503 of 32×32-pixel images were extracted from 88 X-ray images by subsampling. Since the distribution of each type of defect is unbalanced, resampling was performed using ROS (Random Over Sampling), RUS (Random UnderSampling), and SMOTE (Synthetic Minority Oversampling TEchnique) methods.

W. Hou等人23)研究了一个开放数据库GDXray中各种焊缝的x射线图像的缺陷分类模型(良好、不完全穿透、孔隙、夹渣、裂纹)。首先，对88张x射线图像进行分采样，提取32×32-pixel图像3503张。由于每种类型缺陷的分布是不平衡的，我们使用ROS (Random Over Sampling)、RUS (Random UnderSampling)和SMOTE (Synthetic Minority Oversampling TEchnique)方法进行重采样。

J. Park et al.24) proposed two-step CNN models for de- fect detection in engine transmission welds. In the first CNN model, the representation of collected images of the circular welding area of the engine transmission was converted from Polar coordinates to Cartesian co- ordinates and the center point was predicted. Assuming that the weld width was fixed, the background except the welds was removed, thereby optimizing the input data to the second defect detection CNN model.

J. Park等人24)提出了用于发动机传动焊缝缺陷检测的两步CNN模型。在第一个CNN模型中，将采集到的发动机传动圈焊接区域图像的表示从极坐标转换为笛卡尔坐标，并对中心点进行预测。假设焊缝宽度固定，去除除焊缝以外的背景，从而优化输入数据到第二个缺陷检测CNN模型。

Z. Zhang et al.26) developed a CNN model for in-situ weld defects prediction in pulsed gas tungsten arc (GTA) welding of aluminum alloys.

张振宇等26)建立了一种用于铝合金脉冲气体钨极氩弧焊(GTA)原位焊缝缺陷预测的CNN模型。

Supervised learning with transfer learning

Research using original data

H. Zhu et al.27) applied a CNN model for defects (normal, overlap, spatter, porosity) classification on weld surfaces of Gas Metal Arc Welding (GMAW). Transfer learning was performed using LeNet-5, and since the softmax function, which is frequently used in the last layer of the final fully connected layer of the CNN model, has a disadvantage of performance degradation when the number of training datasets is insufficient, Random Forest and SVM classifier were used for comparison of the performance.

H. Zhu等人27)应用CNN模型对气体金属电弧焊(GMAW)焊缝表面缺陷(正态、重叠、飞溅、气孔)分类。迁移学习使用LeNet-5进行，由于在CNN模型最终全连通层的最后一层经常使用的softmax函数在训练数据集数量不足时性能下降的缺点，所以使用Random Forest和SVM分类器进行性能比较。

Y. Yang et. al28) used photographs of laser welding area taken by CMOS digital camera and transfer learning of a CNN model was performed for classification of weld quality into 3 classes (normal, porosity, level misalignment) and 2 classes (qualified, defect).

Y. Yang等28)利用CMOS数码相机拍摄的激光焊接区域照片，对CNN模型进行迁移学习，将焊接质量分为3类(正常、气孔、水平不对中)和2类(合格、缺陷)。

W. Jiao et al.29) developed a weld penetration prediction model using CNN in the GTA spot welding process.

W. Jiao等人利用CNN建立了GTA点焊熔深预测模型。

Data augmentation

N. Yang et al.31) proposed a LeNet-5-based transfer learning CNN model for defect classification using X-ray weld images.

N. Yang等31)提出了一种基于lenet-5的转移学习CNN模型，用于x射线焊缝图像的缺陷分类。

C. V. Dung et al.32) developed a CNN model for detection of fatigue cracks from photographic images of welded joints in a structure.

c.v. Dung等人32)开发了一种CNN模型，用于从结构焊接接头的摄影图像中检测疲劳裂纹。

Unsupervised learning with autoencoder

Autoencoder is unsupervised learning using CNN and reconstructs the original images through encoding and decoding processes.

Autoencoder是利用CNN进行无监督学习，通过编码和解码过程重建原始图像。

J. Guenther et al.11) applied autoencoder method with CNN for feature extraction in laser welding and ex- tracted 16 features from the input image. The extracted 16 features were used for reinforcement learning, and the weld quality was achieved by controlling the output in laser welding.

J. Guenther等11)在激光焊接中应用了自编码器方法和CNN进行特征提取，从输入图像中提取了16个特征。利用提取的16个特征进行强化学习，通过控制激光焊接的输出来实现焊接质量的提高。

Summary and Outlook

The in-situ measurement of waveform-based time series signals have been actively used in determination of weld quality in previous studies, and in the future, there will be increased investigation and adoption of mul- ti-sensor-based deep learning techniques where con- tinuous waveform sensors and image sensors are ap- plied simultaneously.

在以往的研究中，基于波形的时间序列信号的原位测量已积极应用于焊缝质量的确定，在未来，将有更多的研究和采用基于多传感器的深度学习技术，即连续波形传感器和图像传感器同时应用。

the images at the time of measurement are used for quality classification or regression, but in the future, hybrid models of combining RNN and CNN will be applied, leading to more intelligent models in which the in- formation extracted from images in the past will be transferred to the current state prediction.

测量时的图像用于质量分类或回归，但未来将采用RNN和CNN相结合的混合模型，导致更智能的模型，过去从图像中提取的信息将转移到当前的状态预测。