The reactivity of metals is generally determined by their position in the periodic table. Metals that are lower in the periodic table are more reactive than metals that are higher in the periodic table. This is because the metals that are lower in the periodic table have a lower ionization energy, which means that they lose their electrons more easily.
Ag is below K in the periodic table, so it has a higher ionization energy and is less reactive than K.