Out-of-time-order correlators in finite open systems