Why not have a permanent home for the Olympics, say in Greece, near Olympia? Other countries could still act as hosts for specific games (and receive significant revenue thereby); but the permanent facility itself would be improved each time. Ongoing maintenance would be much cheaper than build-from-scratch every four yearas. Our current alternative leaves former Olympic sites without any clear purpose. See, for instance, "What Abandoned Olympic Venues from Around the World Look Like Today".
Greece might not be enough; Winter Olympics, for instance, might also want a permanent home in a colder climate. There might be other specific sports that need their own unique sites; but the principle of establishing a long-term venue still seems a sound one.