When you create a workspace, account-wide admins are added by default and given admin permissions in that workspace. In order to remove such admins and control workspace membership manually, you can convert the workspace to a restricted workspace from workspace settings. The option to convert it is present under the three dots menu on the top-right.
When converted to a restricted workspace, all the admins with account-wide permissions will be removed. The workspace admin will have to manually re-add such users if required.
You can also create restricted groups within a restricted workspace to restrict ticket access only to agents who are part of the restricted group.
