Ferromagnetic soft materials can generate flexible mobility and changeable configurations under an external magnetic field. They are used in a wide variety of applications, such as soft robots, compliant actuators, flexible electronics, and bionic medical devices. The magnetic field enables fast and biologically safe remote control of the ferromagnetic soft material. The shape changes of ferromagnetic soft elastomers are driven by the ferromagnetic particles embedded in the matrix of a soft elastomer. The external magnetic field induces a magnetic torque on the magnetized soft material, causing it to deform. To achieve the desired motion, the soft active structure can be designed by tailoring the layouts of the ferromagnetic soft elastomers. This paper aims to optimize multi-material ferromagnetic actuators. Multi-material ferromagnetic flexible actuators are optimized for the desired kinematic performance using the reconciled level set method. This type of magnetically driven actuator can carry out more complex shape transformations by introducing ferromagnetic soft materials with more than one magnetization direction. Whereas many soft active actuators exist in the form of thin shells, the newly proposed extended level set method (X-LSM) is employed to perform conformal topology optimization of ferromagnetic soft actuators on the manifolds. The objective function comprises two sub-objective functions, one for the kinematic requirement and the other for minimal compliance. Shape sensitivity analysis is derived using the material time derivative and the adjoint variable method. Three examples are provided to demonstrate the effectiveness of the proposed framework.