When should an AI feature be automated vs. human-in-the-loop?

This piece is being written. The question is one I'm actively working through in practice, and I'd rather publish it late than publish it thin.