Do you think it is unfair to move to better seats in a sporting event or concert, etc. even if they are open?
If you didn't pay for those seats stay out of then.
When I was a kid we used to move to better seats in Yankee Stadium, but that was when the game was half over at least.
Thats how I have done it too. Just to make sure no one is coming cause that would be awkward.
If you paid for a specific seat and took a more expensive one, it is wrong.