I am currently tracking agents based on Requester Wait Time. However, this includes on-hold time, which is with our development team or others.
Agent Work Time is a great metric for me, as I can track just what is in the control of the agents. However, this appears to not be available in explore reports.
Am I missing something? If it is not available out of the box, how can I report on this?

In order for Agent Work time to appear in your SLA metric you'll need to set it up first on your SLA policy.
Once you have tickets on which this SLA target has been applied, you can create a report similar to the one below.