In order to solve the problems of YOLOv5 on the problems of too many small targets on metal surface defects and the detection results are easy to be interfered by background, an improved metal surface defect detection algorithm was proposed. By introducing the coordinate attention mechanism in the backbone network, the model pays attention to defects, and some CBS and C3 modules in the backbone network are replaced with GhostNetV2 structure to build a lightweight network to optimize the performance and efficiency of the model. A bidirectional feature fusion network (BiFPN) was used to enhance the neck layer to generate rich representations, deepen the whole network and reuse low-level features. Finally, extensive experimental results show that the accuracy of CGB-YOLO on NEU-DET reaches 75.0% mAP, which is 3.8% higher than that before the improvement. The model has good comprehensive performance in metal surface defect detection.