One of my Gameboys has had some weird issues and just tonight burnt out (possibly literally as I smelled a little burning). It's a DMG-01 with an RCA prosound, GetLoFi pitch bend kit, and a Nonfinite RGB backlight. The problems didn't start out tonight. It started after I put the backlight in. It was fine for a while, but one time I turned it on and it randomly had a horizontal dead line. Then it developed a group of dead pixels out of nowhere. Tonight, I was going to use it, plugged in the RCAs, turned on the Gameboy (at this point everything was fine), then plugged in the link cable. After that, the power shut off, but the backlight was still partly lit up due to the link cable being plugged in. That's when I smelled the burning and shut everything off/unplugged everything. The darn thing won't turn back on now, and I can't figure out what happened. It may be worth adding that the gameboy it was linked to was plugged into an A/C adapter and the one that burned out was using batteries. I can't see how that would matter, but I'm no expert. Does anyone have any idea why something like this would happen?
(Warning, extensive speculation below)
Here's my thoughts on the whole thing, although I really can't be sure:
If your RGB backlight is wired to the transistor on the back of the screen board (for the +5V supply) then if there is a short there that could explain why the screen went all screwy and you smelled burning.
I don't imagine there would have been an overload anywhere, but that is a possiblity.
If you just burned out the transistor and nothing else, you should be able to replace it (but if that happened its a pretty good bet that something else got burned out).
kitsch, nitro2k01, or apeshit would probably have a better explanation.
i'm not really familiar enough with the backlight (its installation/construction/whatnot) to know if it had anything to do with it. but,
-- did you use this set-up before installing the backlight?
-- if you can smell the burn through a closed case, you can probably see the aftermath on the PCB. what parts look fudged?
-- do you have a way to test the RGB colors independently? one of them could have burned out causing the smell, if your voltage was too high...
which gets to my speculation that voltages were summed somehow. again, don't know enough about the installation or how the kit works to say, but if something burned out its more than likely a voltage issue from where the units were connected via the link cable. regulator failed, voltages were summed, something along those lines.
i'm guessing this was an official Nintendo link cable? if not, it probably doesn't have the 5V pin wired (if it does, please share your source 
so forget what i just said.
The dead rows/pixels were probably broken/fragile before the mod was installed, and then finally broke from the extra stress.
I have a hard time seeing where the smell would come from unless something is shorted or incorrectly connected inside the unit. A short circuit could come from some form of debris (pieces of wire for example) tossing around inside the unit. I don't see how teh hell you actually manage to do it in this case, but if you've connected the batteries directly to a power supply unit, that will likely cause them to catch fire. I suggest you open up this 'boy and tro to located the offending component. Also take pictures while you're at it. You never know how they'll come in handy.
I've made some progress with this. I thought it might be the ribbon cable because after I opened it and extended the ribbon cable it came on. When I pushed it down, though, it would turn off. I unhooked one of my first mods, and it seems to be working better. I don't know if that was the problem or if it truly is the ribbon cable. Thanks for all the feedback guys. I'm getting closer. I'd be really glad if I could get it working again.
Hey I had a similar issue with one of my gameboys. It turned out, with mine atleast, the back light pushed a certain way it made my RCA prosounds wires touch just barely with I want to say my pitch shift if I remember right. It just took a slight moving of wires and it starting working just fine. Mine made a popping noise then a kind of hiss noise. You might also want to check you battery compartment for acid. I noticed after it happened some acid was in mine. I wasn't using a Rgb backlight just a normal white one but our issues seem similar ad it was fixable. Just play with it a little and check your soldering points..and shrink tubing is your friend or very small strips of electrical tape, which is what I used to ensure no crossing happened again. Good luck man and pics of the wiring would help.
Oh also I noticed when the gameboy was closed of you push down on the screen protector and case near the right side it would short out and turn off or reset itself. If you can close your gameboy and press around the case a little firmly and you have something similar remember the spot it happens at and pop it open and look for possible crossing wires in that area. May sound like a very stupid thing to do, and I'm sure it probably is, but it helwped me locate the issue.
It sounds kind of like what's happening to me, but I haven't found out exactly what's doing it yet. It does turn on and off when I move the front PCB around.
I had something similar happen to one of my backlit LCD's, the ribbon cable just quit on me all of a sudden. Here is a video of it, is this what you are experiencing?