https://orcid.org/0000-0002-9088-7404
Abstract
Model reproducibility is a point of emphasis for the National Institutes of Health (NIH) and in science, broadly. As the use of computational modeling in biomechanics and orthopedics grows, so does the need to assess the reproducibility of modeling workflows and simulation predictions. The long-term goal of the KneeHub project is to understand the influence of potentially subjective decisions, thus the modeler's “art”, on the reproducibility and predictive uncertainty of computational knee joint models. In this paper, we report on the model calibration phase of this project, during which five teams calibrated computational knee joint models of the same specimens from the same specimen-specific joint mechanics dataset. We investigated model calibration approaches and decisions, and compared calibration workflows and model outcomes among the teams. The selection of the calibration targets used in the calibration workflow differed greatly between the teams and was influenced by modeling decisions related to the representation of structures, and considerations for computational cost and implementation of optimization. While calibration improved model performance, differences in the postcalibration ligament properties and predicted kinematics were quantified and discussed in the context of modeling decisions. Even for teams with demonstrated expertise, model calibration is difficult to foresee and plan in detail, and the results of this study underscore the importance of identification and standardization of best practices for data sharing and calibration.
Issue Section:
Research Papers
References
1.
Collins
,
F. S.
, and
Tabak
,
L. A.
, 2014
, “
NIH Plans to Enhance Reproducibility
,” Nature
,
505
(7485
), pp. 612
–613
.10.1038/505612a2.
Peng
,
R. D.
,
Dominici
,
F.
, and
Zeger
,
S. L.
, 2006
, “
Reproducible Epidemiologic Research
,” Am. J. Epidemiol.
,
163
(9
), pp. 783
–789
.10.1093/aje/kwj0933.
Open Science Collaboration
, 2015
, “
Psychology. Estimating the Reproducibility of Psychological Science
,” Science
,
349
(6251
), p. aac4716
.10.1126/science.aac47164.
National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Committee on Science, Engineering, Medicine, and Public Policy; Board on Research Data and Information; Division on Engineering and Physical Sciences; Committee on Appli, and S. S. C. on R. and R. in S.,
2019
, “
Understanding Reproducibility and Replicability
,” Reproducibility and Replicability in Science
, National Academies Press (U.S.)
, Washington, DC.https://www.ncbi.nlm.nih.gov/books/NBK547546/#:̃:text=B2%3A%20%E2%80%9CReproducibility%E2%80%9D%20refers%20to,using%20the%20original%20author’s%20artifacts5.
Erdemir
,
A.
,
Besier
,
T. F.
,
Halloran
,
J. P.
,
Imhauser
,
C. W.
,
Laz
,
P. J.
,
Morrison
,
T. M.
, and
Shelburne
,
K. B.
, 2019
, “
Deciphering the ‘Art’ in Modeling and Simulation of the Knee Joint: Overall Strategy
,” ASME J. Biomech. Eng.
,
141
(7
), p. 071002
.10.1115/1.40433466.
Rooks
,
N. B.
,
Schneider
,
M. T. Y.
,
Erdemir
,
A.
,
Halloran
,
J. P.
,
Laz
,
P. J.
,
Shelburne
,
K. B.
,
Hume
,
D. R.
,
Imhauser
,
C. W.
,
Zaylor
,
W.
,
Elmasry
,
S.
,
Schwartz
,
A.
,
Chokhandre
,
S. K.
,
Abdollahi Nohouji
,
N.
, and
Besier
,
T. F.
, 2021
, “
Deciphering the ‘Art’ in Modeling and Simulation of the Knee Joint: Variations in Model Development
,” ASME J. Biomech. Eng.
,
143
(6
), p. 061002
.10.1115/1.40500287.
Mulugeta
,
L.
,
Drach
,
A.
,
Erdemir
,
A.
,
Hunt
,
C. A.
,
Horner
,
M.
,
Ku
,
J. P.
,
Myers
,
J. G.
,
Vadigepalli
,
R.
, and
Lytton
,
W. W.
, 2018
, “
Credibility, Replicability, and Reproducibility in Simulation for Biomedicine and Clinical Applications in Neuroscience
,” Front. Neuroinform.
,
12
(April
), pp. 1
–16
.10.3389/fninf.2018.000188.
Peters
,
A. E.
,
Akhtar
,
R.
,
Comerford
,
E. J.
, and
Bates
,
K. T.
, 2018
, “
Tissue Material Properties and Computational Modelling of the Human Tibiofemoral Joint: A Critical Review
,” PeerJ
,
2018
(1
), pp. 1
–48
.10.7717/peerj.42989.
Taylor
,
M.
, and
Prendergast
,
P. J.
, 2015
, “
Four Decades of Finite Element Analysis of Orthopaedic Devices: Where Are We Now and What Are the Opportunities?
,” J. Biomech.
,
48
(5
), pp. 767
–778
.10.1016/j.jbiomech.2014.12.01910.
Erdemir
,
A.
,
Guess
,
T. M.
,
Halloran
,
J.
,
Tadepalli
,
S. C.
, and
Morrison
,
T. M.
, 2012
, “
Considerations for Reporting Finite Element Analysis Studies in Biomechanics
,” J. Biomech.
,
45
(4
), pp. 625
–633
.10.1016/j.jbiomech.2011.11.03811.
Balci
,
O.
, 2012
, “
A Life Cycle for Modeling and Simulation
,” Simulation
,
88
(7
), pp. 870
–883
.10.1177/003754971243846912.
Henak
,
C. R.
,
Anderson
,
A. E.
, and
Weiss
,
J. A.
, 2013
, “
Subject-Specific Analysis of Joint Contact Mechanics: Application to the Study of Osteoarthritis and Surgical Planning
,” ASME J. Biomech. Eng.
,
135
(2
), p. 021003
.10.1115/1.402338613.
Guan
,
S.
,
Gray
,
H. A.
,
Keynejad
,
F.
, and
Pandy
,
M. G.
, 2016
, “
Mobile Biplane X-Ray Imaging System for Measuring 3D Dynamic Joint Motion During Overground Gait
,” IEEE Trans. Med. Imaging
,
35
(1
), pp. 326
–336
.10.1109/TMI.2015.247316814.
Liukkonen
,
M. K.
,
Mononen
,
M. E.
,
Tanska
,
P.
,
Saarakkala
,
S.
,
Nieminen
,
M. T.
, and
Korhonen
,
R. K.
, 2017
, “
Application of a Semi-Automatic Cartilage Segmentation Method for Biomechanical Modeling of the Knee Joint
,” Comput. Methods Biomech. Biomed. Eng.
,
20
(13
), pp. 1453
–1463
.10.1080/10255842.2017.137547715.
Maletsky
,
L.
,
Shalhoub
,
S.
,
Fitzwater
,
F.
,
Eboch
,
W.
,
Dickinson
,
M.
,
Akhbari
,
B.
, and
Louie
,
E.
, 2015
, “
In Vitro Experimental Testing of the Human Knee: A Concise Review
,” J. Knee Surg.
,
29
(2
), pp. 138
–148
.10.1055/s-0035-156673916.
Trad
,
Z.
,
Barkaoui
,
A.
,
Chafra
,
M.
, and
Tavares
,
J. M. R. S.
, 2018
, FEM Analysis of the Human Knee Joint
,
Springer
,
Cham
.17.
Blankevoort
,
L.
, and
Huiskes
,
R.
, 1996
, “
Validation of a Three-Dimentional Model of the Knee
,” J. Biomech.
,
29
(7
), pp. 955
–961
.10.1016/0021-9290(95)00149-218.
Andreassen
,
T. E.
,
Hamilton
,
L. D.
,
Hume
,
D.
,
Higinbotham
,
S. E.
,
Behnam
,
Y.
,
Clary
,
C.
, and
Shelburne
,
K. B.
, 2021
, “
Apparatus for In Vivo Laxity Assessment Using High-Speed Stereo Radiography
,” ASME J. Med. Devices
,
15
(4
), p. 041004
.10.1115/1.405183419.
Kupper
,
J. C.
,
Westover
,
L.
,
Frayne
,
R.
, and
Ronsky
,
J. L.
, 2016
, “
Application of a Novel Measure of In Vivo Knee Joint Laxity
,” ASME J. Biomech. Eng.
,
138
(10
), p. 104501
.10.1115/1.403416920.
Markolf
,
K. L.
,
Mensch
,
J. S.
, and
Amstutz
,
H. C.
, 1976
, “
Stiffness and Laxity of the Knee - The Contributions of the Supporting Structures
,” J. Bone Jt. Surg.
,
58
(5
), pp. 583
–594
.10.2106/00004623-197658050-0000121.
Emery
,
M.
,
Moffroid
,
M.
,
Boerman
,
J.
,
Fleming
,
B.
,
Howe
,
J.
, and
Pope
,
M.
, 1989
, “
Reliability of Force/Displacement Measures in a Clinical Device Designed to Measure Ligamentous Laxity at the Knee
,” J. Orthop. Sports Phys. Ther.
,
10
(11
), pp. 441
–447
.10.2519/jospt.1989.10.11.44122.
Daniel
,
D. M.
,
Stone
,
M. L.
,
Sachs
,
R.
, and
Malcom
,
L.
, 1985
, “
Instrumented Measurement of Anterior Knee Laxity in Patients With Acute Anterior Cruciate Ligament Disruption
,” Am. J. Sports Med.
,
13
(6
), pp. 401
–407
.10.1177/03635465850130060723.
Blankevoort
,
L.
,
Huiskes
,
R.
, and
de Lange
,
A.
, 1988
, “
The Envelope of Passive Knee Joint Motion
,” J. Biomech.
,
21
(9
), pp. 705
–720
.10.1016/0021-9290(88)90280-124.
Harris
,
M. D.
,
Cyr
,
A. J.
,
Ali
,
A. A.
,
Fitzpatrick
,
C. K.
,
Rullkoetter
,
P. J.
,
Maletsky
,
L. P.
, and
Shelburne
,
K. B.
, 2016
, “
A Combined Experimental and Computational Approach to Subject-Specific Analysis of Knee Joint Laxity
,” ASME J. Biomech. Eng.
,
138
(8
), p. 081004
.10.1115/1.403388225.
Chokhandre
,
S.
,
Schwartz
,
A.
,
Klonowski
,
E.
,
Landis
,
B.
, and
Erdemir
,
A.
, 2023
, “
Open Knee(s): a Free and Open Source Library of Specimen-Specific Models and Related Digital Assets for Finite Element Analysis of the Knee Joint
,” Ann. Biomed. Eng.
,
51
(1
), pp. 10
–23
.10.1007/s10439-022-03074-026.
SimTK
, 2018
, “
Reproducibility in Simulation-Based Prediction of Natural Knee Mechanics: Project Home
,” SimTK, accessed Dec. 1, 2018, https://simtk.org/projects/kneehub27.
University of Denver
, 2018
, “Natural Knee Data
,” Center for Orthopaedic Biomechanics, University of Denver, Denver, CO, accessed Dec. 1, 2018, https://digitalcommons.du.edu/natural_knee_data/28.
SimTK
, 2018
, “Open Knee(s): Virtual Biomechanical Representations of the Knee Joint: Specifications
,” accessed Jan. 12, 2018, https://simtk.org/plugins/moinmoin/openknee/Specifications/ExperimentationAnatomicalImaging29.
SimTK
, 2018
, “Open Knee(s): Virtual Biomechanical Representations of the Knee Joint: Specifications
,” accessed Jan. 12, 2018, https://simtk.org/plugins/moinmoin/openknee/Specifications/ExperimentationJointMechanics30.
Halloran
,
J. P.
,
Abdollahi Nohouji
,
N.
,
Hafez
,
M. A.
,
Besier
,
T. F.
,
Chokhandre
,
S. K.
,
Elmasry
,
S.
,
Hume
,
D. R.
,
Imhauser
,
C. W.
,
Rooks
,
N. B.
,
Schneider
,
M. T. Y. Y.
,
Schwartz
,
A.
,
Shelburne
,
K. B.
,
Zaylor
,
W.
, and
Erdemir
,
A.
, 2023
, “
Assessment of Reporting Practices and Reproducibility Potential of a Cohort of Published Studies in Computational Knee Biomechanics
,” J. Orthop. Res.
,
41
(2
), pp. 325
–334
.10.1002/jor.2535831.
Rooks
,
N. B.
,
Schneider
,
M. T. Y.
,
Erdemir
,
A.
,
Halloran
,
J. P.
,
Laz
,
P. J.
,
Shelburne
,
K. B.
,
Hume
,
D. R.
,
Imhauser
,
C. W.
,
Zaylor
,
W.
,
Elmasry
,
S.
,
Schwartz
,
A.
,
Chokhandre
,
S. K.
,
Abdollahi Nohouji
,
N.
, and
Besier
,
T. F.
, 2021
, “
A Method to Compare Heterogeneous Types of Bone and Cartilage Meshes
,” ASME J. Biomech. Eng.
,
143
(11
), p. 111002
.10.1115/1.405128132.
Grood
,
E. S.
, and
Suntay
,
W. J.
, 1983
, “
A Joint Coordinate System for the Clinical Description of Three-Dimensional Motions: Application to the Knee
,” ASME J. Biomech. Eng.
,
105
(2
), pp. 136
–144
.10.1115/1.313839733.
Halloran
,
J. P.
,
Petrella
,
A. J.
, and
Rullkoetter
,
P. J.
, 2005
, “
Explicit Finite Element Modeling of Total Knee Replacement Mechanics
,” J. Biomech.
,
38
(2
), pp. 323
–331
.10.1016/j.jbiomech.2004.02.04634.
Galbusera
,
F.
,
Freutel
,
M.
,
Durselen
,
L.
,
D'Aiuto
,
M.
,
Croce
,
D.
,
Villa
,
T.
,
Sansone
,
V.
, and
Innocenti
,
B.
, 2014
, “
Material Models and Properties in the Finite Element Analysis of Knee Ligaments: A Literature Review
,” Front. Bioeng. Biotechnol.
,
2
(November
), pp. 1
–11
.10.3389/fbioe.2014.0005435.
Orozco
,
G. A.
,
Tanska
,
P.
,
Mononen
,
M. E.
,
Halonen
,
K. S.
, and
Korhonen
,
R. K.
, 2018
, “
The Effect of Constitutive Representations and Structural Constituents of Ligaments on Knee Joint Mechanics
,” Sci. Rep.
,
8
(1
), pp. 1
–15
.10.1038/s41598-018-20739-w36.
Naghibi Beidokhti
,
H.
,
Janssen
,
D.
,
van de Groes
,
S.
,
Hazrati
,
J.
,
Van den Boogaard
,
T.
, and
Verdonschot
,
N.
, 2017
, “
The Influence of Ligament Modelling Strategies on the Predictive Capability of Finite Element Models of the Human Knee Joint
,” J. Biomech.
,
65
, pp. 1
–11
.10.1016/j.jbiomech.2017.08.03037.
Weiss
,
J. A.
,
Gardiner
,
J. C.
,
Ellis
,
B. J.
,
Lujan
,
T. J.
, and
Phatak
,
N. S.
, 2005
, “
Three-Dimensional Finite Element Modeling of Ligaments: Technical Aspects
,” Med. Eng. Phys.
,
27
(10
), pp. 845
–861
.10.1016/j.medengphy.2005.05.00638.
Baldwin
,
M. A.
,
Laz
,
P. J.
,
Stowe
,
J. Q.
, and
Rullkoetter
,
P. J.
, 2009
, “
Efficient Probabilistic Representation of Tibiofemoral Soft Tissue Constraint
,” Comput. Methods Biomech. Biomed. Eng.
,
12
(6
), pp. 651
–659
.10.1080/1025584090282255039.
Rooks
,
N. B.
,
Besier
,
T. F.
, and
Schneider
,
M. T. Y.
, 2022
, “
A Parameter Sensitivity Analysis on Multiple Finite Element Knee Joint Models
,” Front. Bioeng. Biotechnol.
,
10
(May
), pp. 1
–11
.10.3389/fbioe.2022.84188240.
Hamilton
,
L. D.
,
Andreassen
,
T. E.
,
Myers
,
C.
,
Shelburne
,
K. B.
,
Clary
,
C.
, and
Rullkoetter
,
P. J.
, 2022
, “
Supine Leg Press as an Alternative to Standing Lunge in High-Speed Stereo Radiography
,” J. Biomech.
,
138
(April
), p. 111118
.10.1016/j.jbiomech.2022.11111841.
Kim
,
H. Y.
,
Kim
,
K. J.
,
Yang
,
D. S.
,
Jeung
,
S. W.
,
Choi
,
H. G.
, and
Choy
,
W. S.
, 2015
, “
Screw-Home Movement of the Tibiofemoral Joint During Normal Gait: Three-Dimensional Analysis
,” CiOS Clin. Orthop. Surg.
,
7
(3
), pp. 303
–309
.10.4055/cios.2015.7.3.30342.
Asano
,
T.
,
Akagi
,
M.
,
Tanaka
,
K.
,
Tamura
,
J.
, and
Nakamura
,
T.
, 2001
, “
In Vivo Three-Dimensional Knee Kinematics Using a Biplanar Image-Matching Technique
,” Clin. Orthop. Relat. Res.
, 388, pp. 157
–166
.10.1097/00003086-200107000-0002343.
Hallén
,
L. G.
, and
Lindahl
,
O.
, 1966
, “
The ‘Screw-Home’ Movement in the Knee-Joint
,” Acta Orthop. Scand.
,
37
(1
), pp. 97
–106
.10.3109/1745367660898940744.
Maas
,
S. A.
,
Ellis
,
B. J.
,
Ateshian
,
G. A.
, and
Weiss
,
J. A.
, 2012
, “
FEBio: Finite Elements for Biomechanics
,” ASME J. Biomech. Eng.
,
134
(1
), p. 011005
.10.1115/1.400569445.
Fedorov
,
A.
,
Beichel
,
R.
,
Kalpathy-Cramer
,
J.
,
Finet
,
J.
,
Fillion-Robin
,
J.-C.
,
Pujol
,
S.
,
Bauer
,
C.
,
Jennings
,
D.
,
Fennessy
,
F.
,
Sonka
,
M.
,
Buatti
,
J.
,
Aylward
,
S.
,
Miller
,
J. V.
,
Pieper
,
S.
, and
Kikinis
,
R.
, 2012
, “
3D Slicer as an Image Computing Platform for the Quantitative Imaging Network
,” Magn. Reson. Imag.
,
30
(9
), pp. 1323
–1341
.10.1016/j.mri.2012.05.00146.
ASME
, 2018
, Assessing Credibility of Computational Modeling Through Verification and Validation: Application to Medical Devices
,
ASME
, New York.47.
Begley
,
C. G.
, and
Ioannidis
,
J. P. A.
, 2015
, “
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
,” Circ. Res.
,
116
(1
), pp. 116
–126
.10.1161/CIRCRESAHA.114.30381948.
Imhauser
,
C. W.
,
Baumann
,
A. P.
,
Liu
,
X.(cheryl)
,
Bischoff
,
J. E.
,
Verdonschot
,
N.
,
Fregly
,
B. J.
,
Elmasry
,
S. S.
,
Abdollahi
,
N. N.
,
Hume
,
D. R.
,
Rooks
,
N. B.
,
Schneider
,
M. T.
,
Zaylor
,
W.
,
Besier
,
T. F.
,
Halloran
,
J. P.
,
Shelburne
,
K. B.
, and
Erdemir
,
A.
, 2023
, “
Reproducibility in Modeling and Simulation of the Knee: Academic, Industry, and Regulatory Perspectives
,” J. Orthop. Res.
, epub.10.1002/jor.25652Copyright © 2023 by ASME
You do not currently have access to this content.
