I agree, it can definitely make it harder, especially at lower jump numbers. But there are always exceptions. A friend of ours is quite large, at least 250, always has been. He has many thousands of jumps, 30 plus years in sport and he holds multiple RW and CRW world records. He dresses for success; suits with lots of drag help a lot. I'm a feather butt and we have done some jumps together, including him officiating my wedding jump. The only fall rate issues were me if I wasn't wearing a snug suit (ie I didn't dress for success).
But yeah, most of the larger jumpers I've known haven't stayed in the sport very long, at least partly because their learning curve is so steep due to their size and the fact that not every jumper, especially ones at about the same jump numbers, can keep up with them in freefall. But even if they don't do a lot of jumps, they got to experience skydiving as a skydiver not as a tandem passenger. That's not a bad thing :-)