There is no sulfur in copper sulfate. Copper sulfate has the formula CuSO4, which means it is composed of copper, sulfur, and oxygen atoms. In the electrolysis of copper sulfate, the copper ions (Cu2+) in the solution are reduced to copper atoms, which then deposit on the cathode. The oxygen ions (SO42-) are oxidized to oxygen gas, which bubbles off at the anode. The chemical reaction for this process is:

CuSO4(aq) + H2O(l) → Cu(s) + H2SO4(aq) + O2(g)

As you can see, the sulfur in the copper sulfate molecule remains in the solution as part of the sulfate ion.