Clouds play an important role in weather forecasting. Accurate identification and segmentation of ground-based cloud images can effectively guide weather forecasting. Now most of the existing datasets are only suitable for single task learning, and ground-based cloud image recognition and segmentation technologies are mostly implemented by single task, thus identification and detection efficiency are low and the robustness of the algorithm is poor. Considering these problems, we construct the ground-based cloud image datasets GBCD and GBCD-GT with labels and suitable for multi-task learning, and then on this basis, a ground-based cloud image recognition and segmentation joint network model GCRSegNet based on multi-task learning is designed. The model firstly extracts shared features through convolutional neural network, then a special network is designed for each task to extract more recognizable features. The segmentation network learns shared features to achieve ground-based cloud image segmentation, and the recognition network combines sharing features and segmentation features to achieve ground-based cloud image recognition. Multiple groups of comparative experiments indicate that the network in this paper can accurately represent the features of ground-based cloud image. Meanwhile, the accuracy of the recognition task can reach 94.28%, the pixel accuracy of segmentation task can reach 93.85%, and mean intersection over union reach 71.58%. These results can provide a possibility for practical application.