Show of Hands

I work in an office where I am forbidden to wear a tie. Given this, is the office even allowed to HAVE casual Friday? (I was told that my slacks and vest over a Tshirt was "too dressed up.")