These answers are interesting...
On your first skydive, you're facing freefall and potential death for the very first time, and doing it with unfamiliar equipment. That's understandably scary as hell.
On your first BASE jump, presumably, you're already an experienced skydiver who is used to falling through the sky, accepts the risk of death, and is familiar and comfortable with the gear. So it would seem that these factors would mitigate the fear on a 1st BASE jump. The only new aspect is the reduced altitude, proximity to objects, and zero launch speed.
So for BASE to still be scarier, despite all those advantages of exposure to previous skydiving experience, that really seems to say something about how comfortable we get with wide open skies and plenty of altitude!