The macroscopic viscoelastic behavior of collagen gel was studied through relaxation time distribution spectrum obtained from stress relaxation tests and viscoelastic constitutive modeling. Biaxial stress relaxation tests were performed to characterize the viscoelastic behavior of collagen gel crosslinked with Genipin solution. Relaxation time distribution spectrum was obtained from the stress relaxation data by inverse Laplace transform. Peaks at the short (0.3 s–1 s), medium (3 s–90 s), and long relaxation time (>200 s) were observed in the continuous spectrum, which likely correspond to relaxation mechanisms involve fiber, inter-fibril, and fibril sliding. The intensity of the long-term peaks increases with higher initial stress levels indicating the engagement of collagen fibrils at higher levels of tissue strain. We have shown that the stress relaxation behavior can be well simulated using a viscoelastic model with viscous material parameters obtained directly from the relaxation time spectrum. Results from the current study suggest that the relaxation time distribution spectrum is useful in connecting the macro-level viscoelastic behavior of collagen matrices with micro-level structure changes.