The small and medium-sized public building energy consumption accounted for 60% of the total energy consumption of buildings, energy consumption analysis can effectively regulate energy consumption, improve energy efficiency, and can provide the technical foundation for formulating energy consumption quota and energy saving reconstruction of future public building. Through the monitoring of energy consumption of Anhui province’s 29 target buildings, the energy consumption data is in detail. Through statistical analysis of architectural features, construction scale, construction time, in conjunction with other research results, we find out the factors influencing energy consumption of small and medium-sized public buildings in the hot summer and cold winter area. By taking methods to analyze the factors of building energy consumption and the using multiple linear regression analysis, through the gradual optimization, we set up energy consumption prediction model. Through comparison with the measured data, interpretation level of small and medium sized public buildings energy consumption model is higher, and get better prediction results. Compared with similar research, the method has higher accuracy, is more simple and has proposed for predicting the specific model of energy consumption in hot summer and cold winter.