Is this a common grad school anxiety? I think it is. If you’ve come straight out of your undergrad, you probably want to look older so that you don’t get hit on by some guy in the dining hall line who asks, “so, are you a freshman?” (#truestory). Some ideas:
-Messenger bag or tote>backpack. Some suggestions.
-Mix one business casual piece in with your casual wear (in this case, oxfords).
-Sweat pants/workout clothes are awesome and you have every right to wear them but they may increase the risk of the aforementioned junior-guy-hits-on-you scenario.
-Glasses have been shown to boost your perceived credibility.
You could also just decide you don’t care if someone mistakes you for an undergrad! But on days when you do care, the above can be helpful.