Gerevich made his first Olympic appearance at the 1932 Games in Los Angeles when he was 22 years old, earning his first gold medal as a member of Hungary's sabre team. At the 1936 Olympic Games, Gerevich scored 17 wins against only 2 losses to lead the Hungarians to victory in the team sabre event. He then earned a bronze medal in the individual sabre. He also competed in the team foil event, placing seventh.
After the war, Gerevich returned to the 1948 Olympic Games stronger than ever. Not only did he win another gold medal in the team sabre and place fifth in the team foil, he earned the gold medal in the individual sabre with a record of 19 wins and only one loss. At the Olympic Games in Helsinki in 1952, Gerevich won a complete set of medals: bronze in team foil, gold in team sabre and silver in individual sabre. In 1956, Gerevich placed fifth in the individual sabre and again helped extend the Hungarian winning streak in the team sabre. He made his final Olympic appearance at the 1960 Olympic Games - at the age of 50 - earning yet another gold medal in the team sabre. Gerevich's wife, Erna Bogen, his son, Pal Gerevich, and his father-in-law, Albert Bogen, were also Olympic medalists.