Yes, you have to install the terminal server agent on all of these servers where you have more than one user logged in at the same time. You could deploy the agent by gpos or sccm for example. As long as you do not write your own terminal server agent software which then also needs to be installed on all server, there is no other way.
PS: installing global protect like proposed by @laurence64 will not work. Ok, you can install it of course, but you won't get the user-ip-port-mappings you need.