I finally found out how to leave X apps running on my server detached, and come back to them later.
To do this you need Xephyr, xmove and xvfb. In addition it is useful to have GNU Screen and of course ssh with X forwarding.
xvfb provides a null x server (i.e. one without a real display) for applications to run on when you are not sshed in to the target machine. xmove performs the actual screen switching between the null server and one that you can actually look at. This may have to be Xephyr as various X things must match the null server.
So, log in remotely with or without X forwarding and launch GNU screen as usual. Then run (xauth add :99 . $(mcookie); Xvfb -screen 0 640x480x24 :99 &); env DISPLAY=:99 xmove. This start a fake X holding place on localhost:99, and xmove. You may need to use another resolution and depth if the defaults are changed in Xephyr.
In a second screen do env DISPLAY=localhost:1 xclock -update 1
Now you could detach screen and logout, leaving xclock running on the xvfb
You now want to ssh in with X forwarding to check the time. Run xauth add :3 . $(mcookie); Xephyr :3 &
Run xmovectrl :1 -list and you can see xclock is on localhost:99
Move the clock into xephyr with xmovectrl :1 -moveall localhost:3 and now you can interact with it within the Xephyr window. When you have finished, move it back to the headless X server with xmovectrl :1 -moveall localhost:99 and wait for the clock to disappear before closing the window. This is necessary otherwise the clock will die when the Xephyr window is closed.
It is worth going through this a few times before trying it with something more important than the xclock ☺. It may be necessary to start xmove initially pointing at Xephyr rather than xvfb if some things like the keyboard do not work.