Hybrid electric powertrain systems present as effective alternatives to traditional vehicle and marine propulsion means with improved fuel efficiency, as well as reduced greenhouse gas (GHG) emissions and air pollutants. In this study, a new integrated, model-based design and optimization method for hybrid electric propulsion system of a marine vessel (harbor tugboat) has been introduced. The sizes of key hybrid powertrain components, especially the Li-ion battery energy storage system (ESS), which can greatly affect the ship’s life-cycle cost (LCC), have been optimized using the fuel efficiency, emission and lifecycle cost model of the hybrid powertrain system. Moreover, the control strategies for the hybrid system, which is essential for achieving the minimum fuel consumption and extending battery life, are optimized. For a given powertrain architecture, the optimal design of a hybrid marine propulsion system involves two critical aspects: the optimal sizing of key powertrain components, and the optimal power control and energy management. In this work, a bi-level, nested optimization framework was proposed to address these two intricate problems jointly. The upper level optimization aims at component size optimization, while the lower level optimization carries out optimal operation control through dynamic programming (DP) to achieve the globally minimum fuel consumption and battery degradation for a given vessel load profile. The optimized Latin hypercube sampling (OLHS), Kriging and the widely used Expected Improvement (EI) online sampling criterion are used to carry out “small data” driven global optimization to solve this nested optimization problem. The obtained results showed significant reduction of the vessel LCC with the optimized hybrid electric powertrain system design and controls. Reduced engine size and operation time, as well as improved operation efficiency of the hybrid system also greatly decreased the GHG emissions compared to traditional mechanical propulsion.