Object detection occupies an important place in computer vision and is gradually becoming the technical basis for many applications.The application of object detection in fruit recognition can improve the efficiency of fruit picking, but the efficiency of fruit object recognition based on traditional methods is relatively low due to the problems of background complexity,large model similarity, serious texture interference, and local occlusion of fruits. In order to solve this problem,this paper proposes a fruit detection and classification algorithm MSP-YOLO based on YOLOv5. Firstly,the backbone network of YOLOv5 was replaced with another more lightweight backbone network, MobileNetV3, which could reduce the size of the model and improve the detection rate of fruits. Secondly, this paper introduces the SE attention mechanism into the improved network, and adds the SE attention mechanism to the neck middle layer network of the YOLOv5 base model. The advantage of the SE attention mechanism is that it can help the model pay more attention to the feature channels with the most channel information, so as to suppress those channel features that are not important to the whole, so as to improve the accuracy. Finally, by changing the loss function CIoU to MPDIoU, the similarity comparison between the bounding boxes is simplified, which can better optimize the dataset and improve the recognition accuracy of fruit detection. Experimental results show that the accuracy of MSP-YOLO on the dataset reaches 92.7% mAP, which is 3.3% higher than that of the unimproved YOLOv5, and the improved algorithm is superior to the main object detection models Faster R-CNN, SSD, YOLOv7-tiny, YOLOv3-tiny, YOLOv4 and the original model YOLOv5 in terms of detection accuracy and rate.