The reason the normal headphone jacks turns off the speaker is because the jack contains a switch that detects if a plug is inserted. A prosound jack (a secondary one) typically doesn't have such a switch. An RCA jack certainly won't. I don't know if RCA jacks that can even have such a switch have ever been made. If it really bothers you, cut an old 3.5mm plug from a broken pair of headphones or something and use it as a dummy plug.

Well I don't have the gameboy with me right now. Open it up and look.

Test with headphones that do not have a mic-port.
Test with different cables i.e. 2 different 3.5mm stereo jacks, 2 different RCA jacks.

i tried it out at my friends house and it didn't do this... did the soldering come loose?

It wasn't soldered properly.

i just bought a modded dmg from my friend, and when i plug into the prosound jack (bottom or RCA) sound still comes out of the speaker? anyone know whats going on?