Thanks for your informative answer. I wonder if you can
point me at any docs on VM86 mode.
Anyway, the idea was to create a device which represents the video memory that an application can just mmap and write to. This won't need to go through the BIOS so no performance problems there. VBE is supposed to provide a linear frame buffer without the need to do banking, and returns the physical address of the frame buffer in one of the query functions that you call when you want to set the video mode.
In order to change video modes, the application would (probably) do an ioctl on the device file. This would need to go through the BIOS, but it will only be called a few times per application anyway. There may be a separate device file where one can read and write the palette registers. VBE2 can give you a protected-mode address to call to do this instead of going via the interrupt, so performance should be acceptable. You also get a protected-mode interface for changing the viewport.
The application will of course require r/w permissions on the devices involved. The best way of doing this might be to arrange that the devices get chowned when one logs in on the console (although this will hinder silly tricks like switching from an X session to another virtual console and letting someone else log in and also run an X session).