The cheapest starting place I can think of is circuit-bending.
There are tons of tutorials online about circuit-bending.
Circuit-bending is mostly lots of its trial and error so you don't need to know much about electronics other than how to avoid electrocuting yourself to death.
I would recommend going to your local thrift store for battery operated toys.
Toys that make noise like toy keyboards, megaphones, furbys, and baby toys are a good place to start.
Then all you need is a few tools and basic electronic components like wire, alligator clips, solder, and a soldering iron.
You can salvage most of the other electronic components like potentiometer, resistors, capacitors, LEDs, and switches from broken electronics.
You might also want to learn how to use a multimeter, you can learn this online as well.
If you get good a circuit bending you can progress onto guitar effects pedal modding.
But you can also modify other electronics as your skills improve.
I've added LEDs to all my PS3 controller's PS buttons and I've modified pretty much every guitar effects pedal and guitar amplifier that I've ever owned.
Tube amp modding, building, or restoration would probably be the next step or you can just get creative and do whatever project you feel most passionately about.
Guitar amps are dangerous and they can electrocute you to death even when there not plugged, dangerous amounts of current can stay stored in their large capacitors so I would stay away from guitar amps until you learn more about them.
I took a digital electronics class in high school and I've always been fascinated with electronics since I was kid.
Never stop researching, learning, improving your skills, and always be creative.
Good Luck!