Should you have to ask people their permission if you wanted to take a photo or video of them?
Good Manners, which are a thing of the past
I have a cute kid. People will take his picture while we are out, and it irks me. Not like he happens to be in the frame but they actually point the lens at him. One asked me to move out of the shot. I would rather people didn't do that.
There is no expectation of privacy while out in public. Your image can be taken by anyone at all. However, it cannot be used for commercial purposes without your consent.
If they are a stranger
Only if you want to out it on the Internet
Have to? As in enforced by some means of punishment? I'm not sure...
But it's a common courtesy. Don't be a creeper.