You don't say if you passed the levels. What did your instructors say about how you did? Many jumpers experienced setbacks, less than perfect jumps during their AFF progression. Skydiving doesn't just come naturally to many people, it's something you have to work at to learn how to control yourself in the air. Just a guess, but your experience on your second jump may indicate you tensed up and were fighting the air, which isn't at all unusual. I think I was told I had to relax more after each of my first 3 or 4 jumps. Did you meet the goals of the jump? Were you aware of your altitude, and did you pull on time? If so you did fine. If not, it's common to have to re-do a level, or even multiple levels. You will progress faster, and have more fun, if you are in good shape so that is always a worthwhile goal. However you should be aware that if you take a long time off to get in shape you will have to go back a bit in your training. The point of having several levels to AFF is to build survival skills, and too big of a gap will interfere with that. If you are having trouble relaxing, maybe a tandem jump where you don't have the stress of performing can help you to relax and experience more of the fun, and help you get in the mindset to carry on with AFF. Also as has been suggested a better fitting jumpsuit could help you to fall faster and have better control without fighting a lot of flapping fabric. If you are very light, though, some added weight may still be needed so your instructors can fly with you.
If you can't stop thinking about skydiving, and your instructors didn't give you the "take up bowling instead" speech, you can work past these very common early self-doubts.