The effect of fluid viscosity on the developing process of the instability in an over-hung flexible rotor partially filled with fluid, the dynamical behavior of the rotor system while the instability occurs, the unstable region of rotational speeds, and the whirl frequency of the rotor system in the unstable region of rotational speeds, are experimentally investigated in this paper. It is shown that when the rotational speed is just over the reduced first critical speed of the fluid-filled rotor system that is less than the first critical speed of empty rotor system, the unstable motion occurs. The rotor system in the unstable speed region does not whirl at either a constant rotational speed or the first critical speed of the empty rotor system, the whirl frequency of the rotor system in the unstable speed region, dominated by the fluid-filled ratio and weakly depending on the viscosity of fluid, linearly increases with the rotational speed. There exists a hysteresis range of rotational speeds at the upper bound of the unstable speed region, which is not caused by the rotor transient motion when passing through the unstable speed region. As the fluid viscosity increases, both the unstable speed region and the hysteresis region narrow. The influence of the fluid viscosity on the unstable speed region of a rotor filled with a high viscosity fluid must be considered.