Here's what we can say:
* No definitive origin: There's no single documented source for the phrase.
* Long history: It's been used in English for centuries, appearing in writings from the 1700s and possibly even earlier.
* Natural evolution: The expression likely arose from the simple fact that dirt is very old and represents the passage of time.
If you're looking for a specific historical reference, it's unlikely to be found. The phrase is more of a reflection of common human understanding and use of language over time.