The heat-treated nanoparticle heat-treated magnesium silicate hydroxide (MSHH) was obtained based on the synthesis of lamellar nanoparticle magnesium silicate hydroxide (MSH) and analysis of thermal stability, and the morphology, phase composition, and chemical groups of nanoparticles were subsequently characterized. The heat treatment process induces partial dehydroxylation of MSHH, while preserving the layered structure. Compared with MSH, the tribological performances of MSHH as a lubricant additive have been greatly improved. The mechanical properties of MSH and MSHH are analyzed by calculation of elastic constants using density functional theory (DFT). The interactions among dispersant oleic acid (OA), nanoparticles (MSH and MSHH), and Fe tribopairs were investigated by simulations of classical molecular dynamics (CMD) from the views of adsorption energy and confined shear. The tribological mechanism of MSHH as a lubricant additive is proposed based on the decreased shear strength and weakened agglomeration.