I believe BAT was reasonably effective, but not much used. An active-radar anti-ship missile also used against bridges. IIRC the first combat launch resulted in the sinking of a Japanese destroyer, from 20 miles away. One, shot, one hit, one kill and after the war the employment of ASMs died off for a decade or so.
Re: 20mm or .50 cal It certainly could, and did, destroy cargo ships, usually by setting off an explosion amongst the cargo when this was possible.