The woman went into the nearby Ice Casino and reported what she saw to a county employee. The county police were notified and responded.
Kieran O'Leary, a county police spokesman, said, "The man was still on the boardwalk and he told our officer that he had a rifle and a BB gun/pistol that he wanted to get rid of, but did not know how to do so. So he decided to throw them into the Sound."
The 70-year-old Rye resident has not been charged at this time, O'Leary said.
The county's Marine Unit responded and was able to retrieve the two weapons, O'Leary said.
"If anyone has a weapon they want to dispose of, they should contact their local police department for guidance on how to do so safely and legally," O'Leary added.
Click here to follow Daily Voice Rye and receive free news updates.