On 1 December of the same year, Fraser led an Australian medal sweep of the event at the Olympic Games in Melbourne, setting another world record in the process. She gained a second gold medal in leading the Australian team to a world record in the 4x100m freestyle relay and earned a silver medal in the 400m freestyle.
Four years later, at the Olympic Games in Rome in 1960, Fraser repeated her victory at 100m to become the first woman to defend an Olympic swimming title. She also placed fifth in the 400m and picked up silver medals in the medley relay (the first time it was included in the Olympic programme) and in the freestyle relay. On 27 October 1962, Fraser, swimming in Melbourne, became the first woman to break the one-minute barrier for 100m when she covered the longer distance of 110 yards in 59.9 seconds.
At the 1964 Olympic Games in Tokyo, Fraser fought off a stiff challenge from Sharon Stouder of the United States to win the 100m again, becoming the first Olympic swimmer of either sex to win the same event three times. She also gained a silver medal in the freestyle relay and placed fourth in the 400m. While in Tokyo, Fraser was arrested for stealing a flag from the entrance to the Emperor's Palace, but after she apologized, the charges were dropped and the emperor gave her the flag as a gift. In 1988, Fraser was elected to the parliament of New South Wales and served until 1991.