I think this is not only about skydiving - any sort of activity which gives you (and spectators) emotional burst... like skiing, kayaking etc... does not have to be extreme. Almost all people are hungry for action once in a while. When you talk to your non-skydiving friends about it and they say "they would do it", they really mean it and not look for an excuse right at the moment. They surely picture themselves doing it (and it must be very sweet feeling, almost like real action), all roadblocks look insignificant comparing to rewards... But later, when sweet dream fades off a bit, and realization of price yo have to pay, time you have to dedicate, fear factor, anticipation of self-performance anxiety and frustration (which everybody is familiar with having real life experience in any field) comes to the other side of the "scale" - the excuse to back off comes in one or another form very naturally...
And here comes the difference between them and us - they can easily fulfill the hunger just dreaming about it once in a while. we can not...