I am also facing the same issue occasionally. so we have to disconnect and reconnect.
The input mouse action from Viewer to Server is fine, mouse movement from Server to Client is slower. The Arrow mouse pointer is trying to catch the dot pointer in viewer is very slow.
Please let me know if we have solution for this issue.