This paper proposed a method which combines color space with an improved frame differential method to remove the raindrops in color images, based on the study of the existing rain removal image restoration method of using Color space and frame difference method. First analyzes the characteristics of HSV color space, comparing with the RGB space only V channel is affected by rain on a rainy day shooting video in HSV space, thus translate color image from RGB space into HSV space. Then deal the rain with the improved five frame difference method on the V channel. Because of the same pixel on the two consecutive frames may not be covered by the same rain .Then introduce the difference between average and median in five frame pixel as constraint condition to detect the rain. Experiments show that this algorithm can well remove the raindrops on the image. And the processing speed is greatly improved compared with the frame differential method in RGB space and k-means in YCbCr