An aging, respected commander Paul Blanchard, played by Charlton Heston, is on his final submarine tour before promotion to command of a submarine squadron (COMSUBRON). Surfaced and returning to port, the submarine USS Neptune, is struck by a freighter in heavy fog, and sinks to a depth of 1,450 feet (442 meters) on a canyon ledge above the ocean floor. A United States Navy rescue force, commanded by Captain Bennett (Stacy Keach), arrives on the scene, but Neptune is subsequently rolled by a landslide to a greater angle that does not allow the Navy's Deep Submergence Rescue Vehicle (DSRV) rescue submarine to complete its work.

A small experimental submersible, Snark, is brought in to assist with the rescue. Snark is very capable, but run by a nonconformist U.S. Navy officer misfit, Captain Gates (David Carradine). The tiny submersible is the only hope for a rescue.

Ronny Cox plays Neptune's executive officer, scheduled to take over command from Charlton Heston's character; Ned Beatty plays Snark's second crewman; and Christopher Reeve appears in his first film role in a bit part that a sharp-eyed viewer might notice.

In the opening credits, footage of the real-life submarine USS Trout (SS-566), filmed specifically for Gray Lady Down, depicts the fictional USS Neptune. Gray Lady Down also re-used submarine special-effect footage and the large-scale submarine model originally used to portray the fictional submarine USS Tigerfish in the 1968 movie Ice Station Zebra to depict USS Neptune.