Show of Hands

If you're in public (a coffee shop for ex.) and someone starts fiddling with their smartphone in your field of vision (though perhaps afar), do you look away or do you consider it "public domain" and take no issue with not averting your gaze?