Is Keeping Your Head Still Necessary?