Food security has been and will continue to be a major challenge in Ethiopia. The country's smallholder, rainfed agriculture renders its food production system extremely vulnerable to climate variability and extremes. In this study, we investigate the impact of past climate variability and change on the yields of five major cereal crops in Ethiopia—barley, maize, millet, sorghum, and wheat—during the period 1979–2014 using the Decision Support System for Agrotechnology Transfer (DSSAT) crop model. The model is calibrated at both the site and agroecological-zone scales. At the sites studied, the model results suggest that climate in the past four decades may have contributed to an increasing trend in maize yield, a decreasing trend in wheat yield, and no clear trend in the yields of barley and millet; cereal crop yield is positively correlated with growing season solar radiation and temperature, but negatively correlated with growing season precipitation. For modeled cereal crops across the nation during the study period, yield in western Ethiopia is positively correlated with solar radiation and day time temperature; in the eastern and southeastern Ethiopia where water is a limiting factor for growth, yield is positively correlated with precipitation but negatively correlated with solar radiation and both day time and night time temperature. The national average of simulated yields of most crops (except maize) showed an overall decreasing (although not statistically significant) trend induced by past climate variability and changes. Over a large portion of the highly productive areas where there is a negative correlation between yield and temperature, yield is simulated to have significantly decreased over the past four decades, an indication of adverse climate impact in the past and potential food security concern in the future.