Early telephones had a detachable speaker one held to one’s ear, and an microphone integrated into the stand, which one spoke into. These were eventually fused into one piece, a U-shaped microphone-speaker combo or “handset,” wired to the dialing mechanism (first rotary, later touch tone) on a stationary body.
When one picked up the phone, it was held at an angle with the earpiece back and up and the microphone down and forward. To hang up meant to actually hang the speaker on a prong (giving rise to the phrase) or laid on the cradle, in either case activating a mechanical switch of some kind that disconnected the line.
Despite no one having used this style of phone in decades, it has remained unchallenged as the icon for making (or declining) a phone call. An upright phone handset, often at an angle (and colored green for further clarity) represents answering, and a phone sitting at rest with the microphone and speaker face down represents hanging up.