This is a misconception; hot air balloons can indeed fly in cold air. The concept behind hot air ballooning relies on the density difference between hot air inside the balloon and the cooler air outside. As long as the temperature inside the balloon remains higher than the ambient air temperature, the balloon will continue to rise and float. Cold air is denser than warm air so a balloon must be heated more in cold weather but it remains possible.