To lift the object, we must push up on it with a force greater than or equal to the force of gravity acting on the object. To lower the object, we must push down on it with a force greater than or equal to the force of gravity. When we carry the object, we must overcome the force of friction acting between the object and the ground.
In all three cases, we are applying a force over a distance, which means that we are doing physical work. The amount of work done is equal to the force applied multiplied by the distance moved in the direction of the force.