Bidirectional communications with (RGB) LEDs

I have an RGB LED on each device to convey status information
to the user (there are many "states" of interest).  I also
use this to communicate with a bit of test/configuration/debug
equipment instead of exposing an "electrical" connection to
the user (it's easy to mate something physically when you just
need LIGHT to bridge the gap).
As I don't want this to COST anything (even a two-pin electrical
connector has cost), I've implemented a really crude protocol
that seems reasonably reliable (again, this is tightly coupled).
But, I only get a few hundred bits per second out of the interface.
For the configuration activities, this is sufficient as I can
take "many seconds" to install "secrets", etc.  I don't really care
about incident ambient light triggering any unintentional
transactions, etc.
But, as a diagnostic/debug hook, it is sorely limited -- even if
I tokenize the interface.
Using more than one of the LEDs in the package would likely
make matters worse (as I don't see any easy way to create different
"channels" between emitters/detectors).
Currently, I use an extra pin to allow me to drive the LED or
reverse bias it to (indirectly) sense photocurrent.  No other actives
involved and the software is pretty trivial.
Any other approaches I can explore to increase the thickness of the pipe
without adding to hardware costs?  (even 9600 bps would be a big step up!)