Fish don't look down when they swim. They have eyes on the sides of their heads, giving them a wide field of vision but poor depth perception.

If it appears a fish is looking down, it was most likely looking forward.