Expended yet unexploded mortar bombs are dangerous because they have been through the firing process and therefore a delicate detonating mechanism has likely failed. Any small shock could therefore cause the mechanism to complete its function and explode the main charge.
Mortar bombs are dropped into the launcher pipe base first. The impact on the base detonates the propellant charge, expelling the device from the launcher. It also ignites the delayed action fuse which eventually explodes the main charge once the weapon reaches its intended target area. Therefore, if an unused mortar were to be dropped and its bottom hit the ground in such a way, this could cause the bomb to explode.
A WW1 mortar found today is likely to be over 100 years old. Many decades of exposure to the elements will have degraded the item, leaving it fragile and vulnerable to impact. Furthermore, submerging high explosive in water does not render it harmless.