Power board problem - ?
Reset circuit components - ?
...
Crystal oscillator - ?

I suspect that the power board is causing problems. I would start by replacing the caps found on it first.
depending on the board revision you could need:
1 x 33uF @ 25v
or
1 x 33uF @ 25v, 1 x 22uF @ 10v and 1 x 100uF @ 10v

Yeah, it does that with any batteries. There's some oxidation in the contacts, but the fault is definitely not there. At first I thought it was completely broken and accidentally left it on my desk with the power switch on 'cause nothing happened when I toggled it. Five minutes or so pass and... "bling!"

Once I get it to start up, the power stays on and everything's normal. If I don't use it for some time, it takes long for it to start up for the first time. If I put the power off immediately after the first start, it takes a long time to restart. If I let the power be on for some time and then restart it again, it restarts like it should to.

Haven't tried dc power 'cause I don't have the wall wart, but I'm not even planning to use one.

This may be a totally stupid suggestion, but you have tried fresh batteries, correct?

Bought a used (d'uh) DMG-001 recently. It takes almost five minutes for it to start up. Once it has been on for some time, it can be shut down and it will start up right after putting the switch to "on"-position. So it has to "warm up" or something.

I'm not an electrician by any means, but could the fault be in a capacitor? Is this something common and is there a known fix to this? Tried to google it, but found nothing.