Can I override the plugin templates?

Yes, you can. Select the plugin template you want to override in the WooCommerce plugin folder, copy the concerned file and paste it into the “WooCommerce” folder in your theme on the same path (create the folder if it does not exist yet).