How can I make sure that an agent goes off queue when they deselect their phone?

To prevent an agent from remaining on queue if they deselect their phone, remove the Telephony > Phone > Disassociate Self permission from the agent. Similarly, if you remove the permission, an agent cannot go on queue without a selected phone.