Large-scale modes of climate variability can force widespread crop yield anomalies and are therefore often presented as a risk to food security. We quantify how modes of climate variability contribute to crop production variance. We find that the El Niño Southern Oscillation (ENSO), the Indian Ocean Dipole (IOD), tropical Atlantic variability (TAV), and the North Atlantic Oscillation (NAO) together account for 18, 7, and 6% of globally aggregated maize, soybean, and wheat production variability, respectively. The lower fractions of global-scale soybean and wheat production variability result from substantial but offsetting climate-forced production anomalies. All climate modes are important in at least one region studied. In 1983, ENSO, the only mode capable of forcing globally synchronous crop failures, was responsible for the largest synchronous crop failure in the modern historical record. Our results provide the basis for monitoring, and potentially predicting, simultaneous crop failures.