How does Smiley find out Merlin is fake?

This won't answer your question, but there's one I have of my own, that the novel doesn't seem to really answer, and it poses what, IMO, is a bit of a plot hole.

Why does Irina refuse to reveal to Tarr the name of the mole? (Apart from his code name of 'Gerald', of course.) Does she not know? Is she simply refusing to give Tarr the name in an attempt to persuade him to take her back to England with him? It seems to make no sense--if she knows who the mole is, why wait to be taken to Alleline (who would probably turn away her information anyway)? If she doesn't know, what more could she tell Alleline than what she's already told Tarr?

I suspect I'm missing something here, but I've never quite been able to figure this particular plot point out satisfactorily.

The business with Tarr and Merlin is complicated, and it’s easy to miss its significance.

• After Tarr flees Lisbon and hides out in Marseilles, Karla is desperate to shut him up. After using Witchcraft to discredit him at the Circus, Karla sends people to look for Tarr.

• Tarr, realizing that he’s being looked for, decides it’s time to come in from the cold. In order to protect his daughter and his daughter’s mother, he sets up a diversion: He secures travel bookings to London for them in the name of “Poole,” to match his own false identity, which he knows is blown. He then gives them the false Swiss papers that he had on him when he went to Lisbon and sends them to an entirely different destination. At this point, the only people in the world who associate the “Poole” travel bookings with Tarr are the Russians.

• In Guillam’s meeting with Alleline et al, he learns that Merlin is relating the “Poole/Tarr family to London” story to the Circus as a way of further discrediting him before he gets to anyone at the Circus. This draws a direct line between Merlin and Karla.

The rest of it is a result of Smiley’s reading of the history of Witchcraft, and his realization from that of how Witchcraft serves the needs of Moscow Centre. While this is the subject of an entire chapter of the novel, the whole thing is explained in a single brief scene in the TV series, and not at all in the recent film. It’s not really surprising that it wasn’t really clear.

Actually, the novel gives several reasons for George Smiley's suspicions, and the story is the journey to determine the identity of the mole Gerald and the reasons for his (George's) retirement, which are rooted in Control's final days, Percy Alleline's schemes to reach the "high table", the inner workings of the Department (aka "the Circus") and of course, the inter-relationships of a small group of fellow warrior/spies fighting the Cold War.

Smiley knows, or thinks he knows, but he was kept away from these interactions by Control himself, sent off and overseas on various small missions during those final days before his own forced retirement and before Control's death. Now he must revisit his past. He is not surprised by any of his discoveries, not even the betrayal of his wife, Ann.

LeCarre/Cornwall writes an introduction sixteen years after publication of the novel about the novel, the original setting, the theme of betrayal, of Philby, Blake and Angleston, and the art of the double-cross system. He identifies with Blake, dislikes Philby, and recognizes the effect Angleston had on CIA; he uses these prejudices in the novel and incorporates them into various character

As LeCarre writes, "the balancing act of the double agent ... doing more harm to the country that employs it than good, or as Smiley has it, where it has been turned inside out". The final act of betrayal isn't just against Smiley's English service, the last knot is the betrayal of allies and friends.

Further, in this introduction, he also reveals his use and creation of jargon terms: 'mole (frequently credited to LeCarre, but in use by Francis Beacon in 1641 centuries before this writing), lamplighters, scalphunter, babysitter, honey-trap, shoemaker, neighbours, pianists', et. el. As he states, "other bits of jargon ... were all invented, but they too, I am told, have at least in part since been adopted by the professionals".

How does Smiley know: in part, through Connie Sack's research, partly because Smiley knows that Karla (his life-long opponent) has dozens of penetration agents (deep cover moles) and at least one of these 'has' to be in England, just as one was previously caught in Japan and one known to be alive and active in New York. Also, partly, because in his final days at the Circus, Smiley was caught in the struggle between Control and Alleline, who is now Control's replacement. Because, the case of Jim Prideaux (Ellis) was never really finalized. Finally, as the novel brings out, because "George was the best of all those lovely (pre-war) boys", smarter too, and better than Haydon, the latter-day Lawrence of Arabia and constant star of the Circus, who now acts as Alleline's cup-bearer.

While I prefer this movie version to the newer one, the book is beautifully written, rich in character and plot, and makes generous use of the flashback technique to follow Smiley's journey into his past.

I had that question too. After having watched this for the third time, it seems that Smiley is operating on Controls suspicions of "the magic circle". The whole case against the mole started with Control's suspicions and his digging into past cases to investigate missions blown. That suspicion of a mole transfers to the too-good-to-be-true Merlin source. Control intuits that the mole would want to discredit him and take over his job by producing better results.