The answer is pretty simple: the helium leaks out. Helium is a very small atom and latex is pretty porous at the scale of a helium atom (if you ever go looking to buy balloons, you will see that there are "helium-grade" balloons, which try to be thicker and less porous).
The weight balance that keeps a balloon afloat does not leave a lot of room for leakage, so once a little leaks out the balloon falls.
See How Helium Balloons Work for complete details on the weight that a helium balloon can lift!
Here are some interesting links:
