If the Republicans launch a third-party candidate, it will be just as disastrous as if Trump made a third-party run. There just aren't enough "mainstream" voters to go around. Their best bet is to just grit their teeth and support Trump and hope that it leads to a stronger majority in Congress.
The GOP should have kicked Trump out of the first debate when they had the chance and rolled the dice.
Now, Trump has access to GOP resources, money, brand, party loyalty, state campaign staffs on the ground. If they had kicked him out he would have had to build that out in all 50 states, essentially he would have been a more potent version of Buchanan 2000. Trump ran on the Reform Party ticket in 2000, he got tired of it and he quit early in the process, they should have made him do that again.
Annnnddd you have the media, which has had the advantage of saying 'hey let's look at what the Republican Party leading candidate said today and what x, y and z said about that.... [insert ratings #'s here]'. The man has turned the presidential race and whole party into a reality show, he really has.
Now they are paying a ##### of a price.
As for Congress - take a look at Angle in NV, McConnell in DE, the guy in Missouri, etc. The GOP had gotten through the whole Tea Party thing, but they probably lost the Senate in 1-2 terms because of it. This will be like putting one of those tea partiers at the top of the ticket. It will have disastrous effects IMO.
And if the GOP does win with Trump, what have they won? Their party is led by a guy who hasn't even been a Republican a good part of his recent life. Congrats to the Elephant.