In order to realize the rapid recognition of walnut shell defects and improve the walnut sorting efficiency based on machine vision, a walnut shell defect detection method based on improved yolov5s is proposed. In yolov5s network, convolution with convolution kernel size 3 is widely used for feature extraction. In order to reduce the amount of network calculation, this paper proposes to use depth wise separable convolution instead of convolution with convolution kernel size 3 in residual network, so as to improve the speed of walnut shell detection. In addition, in order to ensure that the accuracy can meet the requirements, this paper also uses the improved mean clustering to initialize the detection frame, improve the quality of the generated detection frame, and then improve the detection accuracy of walnut shell defects. Because the clustering method has less computation than the whole network structure, it has little impact on the speed of peach shell detection. Through experimental comparison and analysis, the improved yolov5s can quickly identify walnut shell defects, and the recognition accuracy remains basically unchanged.