XFree86 allows virtual displays so if it does run, I'm guessing that yes, it would be possible.
I'm guessing that once XFree86, all the QT, GTK+, Gnome and KDE libs as well as other necessary libs are installed, porting is the easy part; only a recompile is necessary, which (notice this is pure BS speculation and I have no idea what I'm talking about) usually would only involve a change in the Makefile. HOWEVER, most apps aren't designed to work with a 640x480 screen and/or a 400mhz XScale, not to mention only 64mb RAM.
After all, KDE itself was ported...