Using Hardware Acceleration in VNC

‘Any sufficiently advanced technology is indistinguishable from magic.’

I’d like to slightly modify that statement to ‘Any solution to a technology problem that works for an inexplicable reason is indistinguishable from magic’.

Here’s why:

The S.A.R.T. recently acquired a pair of stereoscopic cameras for use in creating a 3D model of the environment, complete with high-resolution textures. The cameras themselves are the ‘ZED Mini’ model from StereoLabs and, with some initial testing they seem to be exactly what we need.

We attached one of them to the robot and installed the software on our Jetson. In theory, a 3D model and texture map of the environment could be generated by simply driving the robot around.

However, this only works when the Jetson is plugged into a monitor, because that’s what the GPU is connected to. Obviously, we don’t want to follow the robot around with a monitor so we can use the ZED Mini, so I started looking at ways we can do this remotely.

I thought that it might be possible to do this over RDP, but traditionally that doesn’t support 3D applications – at least on Windows. In Linux, it throws you into a different shell (as opposed to Gnome or Unity) because it’s delivered over the network – the best I can describe is like ‘SSH with a GUI’. We can’t use RDP as the client crashes when launching a 3D accelerated application such as ZEDfu (the software that generates and textures the environment).

What I wanted is a program like Parsec, but that only works on Windows hosts, and works without a display connected.

I started looking at VNC, or Virtual Network Computing, but that experience was much like RDP – Visual SSH, and crashes immediately when a 3D application is launched.

I then stumbled across this forum post (https://serverfault.com/questions/174003/how-can-opengl-graphics-be-displayed-remotely-using-vnc) and the first solution seemed to work rather well – for a bit. After a while, the VNC client stops accepting input and behaves as though a display is continuously being disconnected and reconnected.

I then discovered that connecting an unpowered HDMI display completely resolved those issues, so instead I tried connecting an HDMI to VGA adapter – which worked for a bit, but eventually exhibited the same issues as when no display was connected.

The solution was to use an unpowered 800×480 HDMI Rasbperry Pi display attached to the robot. The resolution isn’t brilliant, however it’s still possible to see the status of the environment modelling in a VNC client.

To do this, you’ll want to

1. (skip to 2 if you’ve already installed a VNC server)

Enter your password.
No to view only.

2.

Connect to your desktop using its IP address and the corresponding port displayed by the command.

And now we come back to why ‘Any solution to a technology problem that works for an inexplicable reason is indistinguishable from magic‘. I have absolutely no idea why plugging in an unpowered display causes all the problems to disappear – for now, I suppose it can be magic.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.