CXF is adding MTOM to SOAP XML even in cases when it is not actually used

Issue

We are migrating to JBoss EAP 6.4 with CXF for webservices and MTOM is being added to all responses from our webservices even though it only sometimes has an actual MTOM attachment. Is there a way to disable this so it will only enable MTOM on the individual responses that actually use it?