The reason for the increase in file size is simple: PNG support (libpng and zlib)... Removing those two the file size could easily be down under 200kb again..
A "GapiDraw Lite" is planned that will do exactly what you suggests... Unfortunately I cannot give any release date on this.