We would like to make the chat widget available during office hours, and hide the widget when it's outside of business hours. We can use either the classic widget or messaging.
Is this possible? From what I can tell, it looks like the only option is to have the fallback be a form or the standard 'leave a message'.

You can prevent your customers from initiating chats when you do not have staff available to help them.The process for setting this up differ between live chat and messaging.
For live chat, If you have not enabled any offline or contact forms then the widget will not appear on your website when your staff is offline.
Messaging behaves more like conversations through social media apps where the conversation can be picked up at any time. As a result, there isn't an ability to turn off Messaging or to prevent customers from picking up where they left off. Instead, you can choose to notify your customers when you don't have staff available and set expectations about when they can expect to hear back. To do this, set up business hours and then create a business hours condition in your bot to send a message to your customers.
More information can found here Can I turn off chat when no staff is online?