Problem
You need to add an additional RD Session Host Server to an existing RDS deployment then create a new RDS Collection with the server so you proceed to add the server:
Once you’ve successfully added the server, you proceed to add the server as a RD Session Host Servers in the Remote Desktop Services management console:
Once successfully added as a RD Session Host server, you proceed to create a new collection with the server:
The collection is successfully created by the add server operation fails with the following error message:
<ServerName>.FQDN Unable to configure the RD Session Host server <ServerName>.FQDN. Invalid operation
Solution
As per the following KB article:
You cannot create a session collection and an error occurs in Windows Server 2012
https://support.microsoft.com/en-us/help/3014614/you-cannot-create-a-session-collection-and-an-error-occurs-in-windows
… this can be caused by GPOs configuring one of the following two settings:
Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Security
Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Licensing
In this example, the environment had a GPO that configured the RDS Licensing server:
To correct the issue, move the computer object to another OU to prevent the GPO from being applied thus unconfiguring the licensing server:
Delete the collection and recreate it: