SAP Installs on RHEL in Larger enviroment

Is it common practice that every server has X and a desktop Manager installed so that SAP tools that are GUI can be used on each? We would like to reduce the footprint of each server as much as possible for obvious reasons and installing X with a Desktop Manager adds about 800 packages. It also appears managing tigerVNC for multiple Basis administrators to remotely access the X desktop manager will be somewhat of a nightmare. I've seen multiple how-to's out there showing the install using a GUI - it just seems somewhat shocking to me that this is a requirement on each server and common place in a larger environment. Our Basis team is saying the only way they will be able to reliable manage our SAP environment is with X and a desktop manager on every server to use the SAP management tools.