I have looked through the admin panel and the forums and haven't found if there is a way to do this.

Is there anyway, in agent settings and/or roles, to restrict who can and cannot adjust due dates?  We need to be able to restrict this, especially since the ticket history doesn't show you who/when a due date was changed, and to what it was changed.

Completely agree with this. Regular Service Desk agents should not be able to change their own due dates and override the SLAs.