The request came from @cixelsyd:
@lusis random #nagios plugin suggestion: check_rdp verifies port, handshake, authOkay, I thought. Sounds interesting. Let's give it a shot.
So I did the first thing which was to see if I could find an existing module for any of the scripting languages I know. It was a long shot and it came up empty.
Not a biggie. Let's see what we can find out about the RDP protocol. Maybe I can knock something together....
About 2 hours later I was done reading various Technet entries. RDP is pretty convoluted and only gotten more so as Microsoft iterates over the various versions.
So I decide to find some code I could attempt to read through. Of course rdesktop was my destination.
After spending the last hour or so navigating the rdesktop source, I'm not quite sure if it's possible to even do a headless RDP client. Mind you my C is very limited.
My first attempt was to simply shortcircuit the rdesktop client after handling the authentication. Each attempt kept leading me to various X-related code. That's a whole other beast that I'm just not remotely competent enough to learn. I compiled it with debugging and used a Vista machine at home as the test server. Each and everytime, it wanted to do some sort of screen rendering.
I'm going to spend some more time on it tonight including reading the source for a few other client implementations. Unfortunately, rdesktop is the only one I know of that supports RDP5. At least one that I have access to the source code for.
My thinking is that, if I can't totally remove the need for client rendering capabilities, that I can somehow fool rdesktop into using a null framebuffer of some kind or faking the capabilities of the client side display. All I need to do is authenticate but I can't really tell if the client window has to be available even if all the credentials are present in the PDU since there's a Basic Settings Exchange before the Security state is even reached.
Another option that I can't test at work but can at home, is to see how the rdp2vnc code works. It might be possible to bring that session up only to tear it down.
The really annoying part is that this would still not be a very efficient plugin unless I proceeded to somehow implement JUST the process up to the Security Exchange.