They are formed by the slow dripping of mineral-rich water, which leaves behind deposits of calcium carbonate (usually calcite). Over time, these deposits build up to form the distinctive icicle-like formations.
It's easy to remember the difference between stalactites and stalagmites by remembering that stalactites hang tight to the ceiling.