Adams unwelcome?

That's what's great about that scene, no dialogue was required. From what I know about Washington and Adams' relationship is that there was definitely tension between the two. Adams was an incredibly revered lawyer that had strong opinions and voiced them often, and this was well known among those who knew him personally. If you look back on that series, Ben even Franklin tells him to keep his mouth shut after an altercation in the state house.

Adams proves to be right, however, and therein lies the influence and persuasion he had with politics, and to Washington, this was a danger to his presidency. If you consider how Washington and Adams were merely the first and second place winners of the election, there was an unfortunate insecurity that Washington held against Adams. This is made especially so when Adams attempts to "administer" an argument between Jefferson and Hamilton. Hamilton was Washington's right-hand man, and Washington usually always sided with him, so Washington also had to fear that Adams' friendship with Jefferson could prove costly to his administration. It could lead to a "coup," a political divide, or anything that proved dangerous to the FIRST presidency. Think about it, Washington had nothing to rely upon as the first of the nation's leaders other than what was already known about previous heads of state, most notably Caesar and the Senate; we all know how that ended.

Bottom line: Adams was not a part of the cabinet, and Washington was determined to let him know this.