If we're assigning rights based on potential, don't 500 million sperm cells have a whole lot of potential?
It's not so much about potential, but rather that it is literally just a very young human who will grow up to be a not so young human if cared for properly (and this isn't unique to fetuses, all very young and very old people require care). If you leave a sperm cell in your testicles it won't do much, but a fetus will age and experience the changes that every human does with aging.
Sure, someday it could've grown up to be a human
No, it won't "grow up to be a human", it is already a human, just a very young one. Denying a human their universal rights based on age is pretty arbitrary in itself, not to mention that there is literally an entire section in the UDHR about how every human is entitled to their rights "without distinction of any kind".