Data storage has gone through many phases: early on, punchcards represented a certain standard quantum of data, and tape could be used as a more freeform output or input. The tape too was first paper, but later magnetic tape was found to allow denser storage. This thin tape was rolled up tightly into spools, and the spools prevented from unraveling with covers like vinyl records have. These covers could not be removed, however, and so had a missing or movable piece that provided access to the tape, as well as a way to spin the tape at the correct speed so data could be read by the head.
The first common standard for these disks was the 5.25-inch floppy disk or diskette (but not disc), due to its thin case being fairly flexible. It was later replaced by a 3.5-inch disk, which was more compact, held more data, and had a more rigid case. These also often had a slight chip out of one lower corner to indicate orientation, a metal shield that protected the disk inside but could be moved out of the way by a mechanism, and a place for a paper label to be attached.
That form of storage was widely used during the time early computer UIs were being developed, a time when computers may not have had internal storage at all. So when you wanted to save something for later, you saved it to a disk.
That’s no longer the case, but the gesture of saving to a storage medium is similar.
(Icon by Susan Kare)