The long and short of it is that you're tapping into the circuit between the audio output from the CPU chip and the amplifier, so anything you plug into can affect the signal. The internal speaker on the other hand doesn't affect the prosound output.
Split wrote:Line-out + Speaker Max: Full signal, good recording. Internal speaker audible, has hum
Line-out + Speaker Min: Full signal, good recording. Internal speaker muted, has hum
This is the easiest case to explain. Nothing is affecting the circuit, so the recording is good. You hear hum in the speaker because the speaker always has hum that comes from the power supply line. The amp can be completely turned off when headphones are connected, and in this case the hum goes away because the amp is not driving the speaker.
Split wrote:Line-out + Headphones Max: Low signal, poor recording.
In this case, the headphones are connected directly to the top (signal) part of the pot. This is the same as if you would have a pre-pot prosound, and then a passive splitter connected to both line-in recording and headphones. The headphones are loading down the line, and the signal becomes distorted. In addition, the headphones work as microphones in this case, so if you repeat the experiment and tap the headphones, this should come through in the recording. Another way to look at this is that the pot, which is maybe 100 kohm and the headphones which is maybe 8 ohms, are connected in parallel, both going to ground. The headphones are much lower impedance and effectively bypasses the pot.
Split wrote:Line-out + Headphones Min: Full signal, good recording.
In this case, the headphones are connected directly to the bottom part of the pot (ground). The headphones are effectively separated from the circuit and make no difference to the audio quality.