Robust Resource Allocation for Calibration and Validation Tests

In engineering applications, it is often required to estimate the system response under untested conditions using available computational models and test data at different conditions. The computational model aims to describe the physics of the system and can be denoted as $Y=F(X;θ)$⁠, where $Y$ is the system response, $X$ is the set of model inputs, and $θ$ is the set of model parameters. The uncertainty in an input $X$ can be described by a probability distribution $πX(x)$⁠. In the actual test, in some cases, we can control an input at a nominal value but the control is not perfect; thus, $πX(x)$ characterizes this imperfect control. In some other cases, an input such as outdoor temperature cannot be controlled but measured; thus, $πX(x)$ characterizes the natural variability in $X$⁠. The model parameters $θ$ have fixed but unknown values in all tests on the same specimen. The uncertainty regarding the values of $θ$ is epistemic uncertainty due to lack of information, which can be reduced using test data. (In some problems, the model parameters $θ$ may not be physical quantities but simply artifacts of modeling, in which case the concept of true value may not be applicable; such cases are not considered in this paper. Also, in some problems, the model parameters could be input-dependent; this paper does not consider such cases.)

Two important questions in the system response prediction are: (1) how to quantify and reduce the uncertainty in $θ$ and (2) how to validate the agreement of the computational model to the true physics or quantify their difference. These two questions are resolved by model calibration and model validation, respectively. Usually, model calibration is conducted first to quantify the values of $θ$ or reduce the uncertainty about their values, and then model validation follows. Various approaches to model calibration and validation have been studied in the literature. Consider an example of model calibration using Bayesian inference. While some researchers directly use the computational model $Y=F(X;θ)$ and calibrate $θ$⁠, others [1] use a model discrepancy term $δ(X)$ to correct the computational model and calibrate both $θ$ and $δ(X)$⁠. Consider another example regarding the use of test data. Some researches treat all the data as calibration data and use the calibrated model parameters in predicting the system response [2,3]; others integrate the results of model calibration and model validation (each done with different sets of data) in predicting the system response [4–6].

No matter what approaches are pursued, model calibration and validation always require test data. Due to the variability in test outcomes, two sets of test data of the same size may lead to two distinct system response predictions (after calibration and/or validation) even if the same computational model and the same framework of model calibration/validation are used. Here, “test outcome” is defined as the value of test data, i.e., the measurements of test inputs and outputs. The variability in the test outcome is due to the following reasons: (1) the input is controlled at a nominal value but the control is imperfect; (2) the input has natural variability, which means that the input cannot be controlled; and (3) there is measurement error in the input and output data.

If a single data point is used in model calibration/validation, the calibration/validation result will be affected by the value of this data point significantly. However, as more data points are applied, the calibration/validation result will converge; thus, the consequent system response prediction will also converge. Thus, as the number of tests increases, the model prediction uncertainty becomes less and less sensitive to variability in the test outcomes. This raises the following questions, when test budget is limited: (1) is it possible to organize the test campaign to make the system response prediction robust to variability in test outcomes and (2) how many tests of each type are necessary to achieve the robustness objective. Note that in this paper, the term “test type” refers to two attributes: (1) whether the test data are for calibration or validation and (2) the physical quantity measured in the test. For example, if three quantities are measured in tests and all data are used for calibration, we have $3×1=3$ types of test; but if part of the data is used for calibration and the remaining data are used for validation, then we have 3 $×$ 2 = 6 types of tests. The focus of this paper is to develop an optimization approach to answer this question, assuming the computational model and the framework of model calibration/validation are given. The design variables of this optimization are the numbers of each type of test, denoted as $N∈ℕq$ if $q$ types of tests are available; the objective function and constraints will be discussed later. Note that (1) this optimization needs to be solved before any actual test is conducted [4] and (2) this optimization needs to consider test outcome uncertainty due to which the subsequent system response prediction is also uncertain.

Several approaches for test resource allocation have been studied in the literature [4,7–11], and the main difference among these approaches is the choice of the objective function. Note that model calibration aims to reduce the uncertainty in model parameters, and thus reduce the uncertainty in the subsequent system response prediction. Thus, in the case that only model calibration is considered in system response prediction, generally the objective of test resource allocation optimization is to minimize the system response prediction uncertainty subject to limited budget. Several quantities have been used to represent system response prediction uncertainty, and the first one is variance. Sankararaman et al. [4] minimized $E(V(Y))$ where $V(Y)$ is the variance of the system response prediction $Y$ at given numbers of each type of test, and $E(⋅)$ denotes the average of $V(Y)$ over different synthetic data sets. Similarly, Vanlier et al. [8] defined the variance reduction of $Y$ via model calibration as $1−E(σnew2/σold2)$ and maximized it, where $σnew2$ is the variance of the system response prediction using the posterior distribution and $σold2$ is the variance of the system response prediction using the prior distribution. Entropy measures have also been used to represent system response prediction uncertainty. In Ref. [9], the authors maximized the relative entropy (Kullback–Leibler divergence) from the system response prediction $π′(y)$ using the prior distribution and the system response prediction $π″(y)$ using the posterior distribution; while in Refs. [10] and [11], the authors maximized the mutual information, i.e., the change of entropy from $π′(y)$ to $π″(y)$⁠.

The previously mentioned approaches that select only calibration tests to minimize the uncertainty in the system response prediction are not applicable when model validation is also incorporated in the system response prediction. The reason is that model validation may indicate that the calibrated model is not exactly valid; accounting for this result increases the uncertainty in the system response prediction. Thus, the earlier optimization formulations would lead to the conclusion that model validation is not necessary. Mullins et al. [12] proposed a method considering both model calibration and model validation, in which model calibration is via Bayesian inference, and model validation is via a stochastic model reliability metric, i.e., describing model validity through a probability distribution. In this method, the objective regarding model validation tests was to minimize the spread in the family of system response predictions that results from the uncertainty in model validity, denoted as $E{V[E(Y)]}$ where the inner $E(Y)$ is the system response prediction mean at given synthetic data set and given value of model validity, and $V[⋅]$ is the average over the distribution of model validity, and the outer $E{⋅}$ is the average over the different data sets. The objective regarding model calibration tests is still to minimize the variance of the system response prediction, denoted as $E{E[V(Y)]}$ where $V(Y)$ is the system response prediction variance based on a given synthetic data set and given value of model validity; the inner $E[⋅]$ is the average over the distribution of model validity, and the outer $E{⋅}$ is the average over different synthetic data sets.

In this paper, the proposed concept of “test resource allocation for system response prediction robustness” means that the system response prediction becomes insensitive to the variability in test outcomes; thus, at the optimal value of the design variables (i.e., number of tests $N∈ℕq)$⁠, different test outcomes result in consistent system response predictions. This concept and the required objective function will be explained in Sec. 2. The approach is suitable in different situations when only model calibration tests are considered or when both calibration and validation tests are considered. Note that the proposed methodology only selects the number of each type of test; it does not design the actual tests, i.e., select the input for the test. Experimental design is a subsequent step to test selection; we only focus on test selection.

The constraint in the optimization of test resource allocation is generally the budget. Note that the constraint and objective are interchangeable, i.e., the optimization may have two alternative formats: (1) subject to the budget constraint, optimize the design variable $N∈ℕq$ (the number of each type of test) to reach the most robust system response prediction; or (2) subject to the robustness requirement in the system response prediction, find $N$ to minimize the budget. The proposed concept can be realized with either formulation.

In addition, it is important to note that the data considered in test resource allocation analysis has to be synthetic since it is done before any actual test. The actual physical test data from a test are obtained by: (1) selecting the values of inputs $X$⁠; (2) applying $X$ to the physical test configuration where the model parameters $θ$ are at their true but unknown values; and (3) recording the input–output data, where both the input and output measurements may be subject to measurement errors. In actual tests where the values of $X$ have been decided, the test outcome uncertainty arises only from experimental variability, including measurement errors. The generation of synthetic data is a simulation of the three steps mentioned earlier, with the physical test configuration replaced by a computational model and the model parameters being unknown. Thus, two additional uncertainty sources are introduced in the synthetic data: (1) uncertainty regarding the value of $θ$ and (2) model discrepancy, i.e., the difference between the computational model and the actual physics. In a Bayesian framework, the first one can be represented by the prior distribution of $θ$ based on available knowledge. But no information on the model discrepancy is available before any testing.

In addition, compared to the actual test, the physical meaning of the input distribution $πX(x)$ may be changed in generating the synthetic data. As explained at the beginning of Sec. 1, for an actual test, the uncertainty characterized by $πX(x)$ is due to the following sources: (1) imperfect control over the true value, (2) natural variability of the input, and (3) measurement errors. In generating the synthetic data, $πX(x)$ accounts for the same uncertainty sources in the case that test conditions are known (for example, the nominal values of the inputs are known). But in the case of unknown test condition, $πX(x)$ mainly accounts the uncertainty about which experimental conditions will be subsequently selected. For example, if the tester only mentions that the possible nominal value of an input is between 5 and 10, then we may have a uniform distribution $πX(x)∼U(5,10)$ to represent this uncertainty in the nominal value. In this case, the uncertainty in $πX(x)$ is epistemic. In this paper, the proposed method is versatile and able to handle both cases. It is possible for the decision-maker to apply the proposed method before and after knowing the test conditions, and different answers can be obtained due to changed availability of knowledge.

In summary, the objectives of this paper are to: (1) find the optimal number of each type of test such that different data sets result in consistent system response predictions; (2) develop solutions for both formats of the optimization problem; and (3) adapt to different cases when only model calibration tests are considered or when both calibration and validation tests are considered. The rest of this paper is organized as follows: Section 2 proposes the objective in the optimization of robust test allocation. Section 3 analyzes the uncertainty sources in the synthetic data and the use of Sobol’ indices to assess their contributions toward the uncertainty in the system response prediction. Section 4 develops a flexible approach for test resource allocation optimization. Section 5 uses two numerical examples to illustrate the proposed approach.

The objective of the proposed test resource allocation optimization can be visually represented as in Fig. 1, which shows the families of the system response prediction probability density functions (PDFs) at different values of the design variables $N$⁠. Within a sub-figure, the variation between the PDFs is caused by the test outcome variability among different data sets. From Figs. 1(a)–1(c), this variation becomes smaller and the system response predictions reveal stronger consistency due to: (1) the decreased variability of mean values $E(Y)$ across the PDFs, meaning that the centroids of the family members are closer; and (2) the decreased variability of the variance $V(Y)$ across the PDFs, meaning that the ranges of values covered by the PDF are similar. In other words, at the value of optimal $N$ in Fig. 1(c), the effects of test outcome uncertainty on $E(Y)$ and $V(Y)$ are small so that consistent system response predictions can be obtained with different sets of test data. Note that this paper is only concerned about the mean value $E(Y)$ and variance $V(Y)$ in the system response prediction, not the exact shape of the PDF in Fig. 1. Here, the “variability” of $E(Y)$ and $V(Y)$ is captured by their variance, i.e., $V(E(Y))$ and $V(V(Y))$ across different data sets.

Therefore, this paper defines the objective for robust test resource allocation as: minimize the contribution of test outcome uncertainty toward the variability (i.e., variance) in the system response prediction mean value $E(Y)$ and the system response prediction variance $V(Y)$⁠.

Global sensitivity analysis (GSA) using Sobol’ indices is a prominent approach [13–15] to quantify the contributions of input uncertainty toward the variance in the output. A brief introduction to Sobol’ indices is given in Sec. 2.2. One challenge is to establish a deterministic function required by the Sobol’ indices computation, in mapping the test outcome uncertainty to the system response prediction uncertainty. This challenge will be analyzed and overcome in Sec. 3.

$SXP$ is a combined measure of the individual contributions of the components of $Xp$ and of the interactions among them.

where $X−p$ is the complementary subset of $Xp$⁠. $SXPT$ is a combined measure of the individual contributions of the components of $Xp$⁠, the interactions among them, and the interactions between $Xp$ and $X−p$⁠.

The direct computation of Sobol’ indices requires double-loop Monte Carlo simulation and thus is expensive. Taking $Sl$ in Eq. (1) as an example, we need: (1) an inner loop $E(Y|Xl)$ to compute the mean value of $Y$ using $n$ random samples of $X−l$ and (2) an outer loop to compute $V(E(Y|Xl))$ by iterating the inner loop $n2$ times at different values of $Xl$⁠. In addition, other $n3$ Monte Carlo simulation iterations are required to compute $V(Y)$⁠. Various algorithms have been developed in the literature to reduce the computational cost [21–25]. Any one of them can be used to compute the Sobol’ indices in this paper. Several illustrative examples on computing Sobol’ indices can be found in Ref. [19].

The Sobol’ index computation requires (1) a deterministic input–output function and (2) the representation of all the inputs by uncorrelated continuous probabilistic distributions. These two requirements need to be achieved before applying Sobol’ indices in the proposed approach for test resource allocation. Section 3 analyzes the uncertainty sources in test outcomes and develops an approach to achieve both requirements.

Recall that all the data considered in test resource allocation analysis have to be synthetic since the analysis is done before any actual test. The uncertainty in the synthetic data depends on specific test conditions, including: (1) the possible values of inputs $X$⁠; (2) the number of test types; and (3) whether a single test specimen or multiple specimens are used for each type of test.

Regarding the first condition, this paper assumes that a distribution of $X$ is provided by the testing personnel or assumed based on some information. For example, for a single model input $X∈X$⁠, we may have $X∼U(LX,UX)$ where $LX$ is the lower bound and $UX$ is the upper bound. We can also use other types of distribution such as Gaussian distribution to capture the uncertainty in $X$ if additional information is available.

This section will analyze the uncertainty sources in the synthetic data regarding the second and third conditions; the corresponding deterministic function required by the Sobol’ indices also varies correspondingly. The rest of this section starts from the simplest case of one type of test and single specimen, and subsequently extends it to multiple types of tests and multiple test specimens.

If only one type of test is available and all tests are conducted on a single specimen, the actual test data is a set of $N$ data points obtained from the same specimen. Figure 2 shows the generation and usage of the synthetic data in this case. As shown in the left part of Fig. 2, to generate a data set of $N$ synthetic data points, four steps should be followed: (1) select and fix the values of $θ∈ℝdθ$⁠, where $dθ$ is the dimension of model parameters; (2) generate $N$ samples of model inputs $xj∈ℝdX(j=1 to N)$⁠, where $dX$ is the dimension of model inputs; and (3) propagate $xj(j=1 to N)$ and $θ$ through the computational model $F(⋅)$⁠; and (4) record the model input and output with measurement errors added. The resultant data set contains pairwise data points ${ωj,zj}(j=1 to N)$ as

where $ej∈ℝdX$ is the model input measurement error and $ϵj∈ℝ$ is the model output measurement error. If the model input measurement error is ignored, then $ωj=xj$⁠.

A crucial point in the generation of synthetic data is regarding the model parameters $θ$⁠. For a single specimen, $θ$ have true but unknown values, meaning that the uncertainty in $θ$ is epistemic. Thus, the uncertainty caused by $θ$ is the uncertainty in selecting the values of $θ$before generating a synthetic data set; once selected, the values of $θ$ are fixed within the synthetic data set. This uncertainty in $θ$ only exists in the synthetic data; actual tests will fix the value of $θ$ at their true values.

The four steps mentioned earlier indicate three uncertainty sources in generating a pairwise synthetic data point ${ωj,zj}$⁠, including:

1. (1)

   Uncertainty regarding the values of model parameters $θ$ can be represented by their prior distribution $π′(θ)$ based on available knowledge before conducting any physical test. This uncertainty is epistemic since $θ$ have unknown but fixed true values.

2. (2)

   Uncertainty regarding the possible values of inputs $xj$ to be used in the tests. As mentioned earlier, a distribution of $X$ has been provided or assumed. This uncertainty is also epistemic if the values of $X$ are unknown during test selection analysis, but will be decided by the test personnel in actual tests.

3. (3)

   Uncertainty regarding input measurement errors $ej$ and output measurement errors $ϵj$⁠. Usually, measurement error is assumed to have a zero mean Gaussian distribution; thus, $ej∼N(0,ΣX)$ and $ϵj∼N(0,σ2)$⁠. The uncertainty in $ej$ and $ϵj$ is aleatory if the values of $ΣX$ and $σ$ are known; but additional epistemic uncertainty regarding $ΣX$ and $σ$ will be introduced if their values are unknown.

where $αj={xj,ej,ϵj}∈ℝ2dX+1$ for $j=1 to N$ representing the uncertainty sources in generating a single pairwise data point ${ωj,zj}$⁠, and $N$ is the number of pairwise data points; $G(⋅)$ represents the entire process shown in Fig. 2, including both synthetic data generation and model calibration/validation analyses before predicting the system response.

In Eq. (6), the uncertainty in ${α1,…,αN}$ represents the variability in the actual test outcomes; while the epistemic uncertainty in $θ$ only exists in the synthetic data, not in actual test data. To minimize the sensitivity of the system response prediction to the variability in the test outcomes, we need to minimize the sensitivity index of ${α1,…,αN}$ in Eq. (6) so that $E(Y)$ and $V(Y)$ are insensitive to the variability in test outcomes and consistent system response prediction distributions can be achieved under different actual test outcomes. However, this minimization requires the sensitivity index closer to zero while numerical accuracy is always a challenge for small sensitivity indices.

Instead, this paper chooses to maximize the sensitivity index of $θ$⁠. If that is achieved, the epistemic uncertainty in $θ$ will be dominant toward the uncertainty in the system response prediction mean $E(Y)$ and the system response prediction variance $V(Y)$ (based on synthetic data). In the system response prediction using actual test data where $θ$ are fixed at their true values, the most dominant uncertainty contribution to $E(Y)$ and $V(Y)$ will be removed. Therefore, the uncertainty in $E(Y)$ and $V(Y)$ caused by test outcome uncertainty will reduce significantly and consistent system response prediction distributions can be achieved under different actual test outcomes. In sum, the basic idea of the proposed approach is to maximize the contribution of epistemic uncertainty regarding model parameters in the synthetic data.

Note that the proposed approach guarantees consistent system response predictions regardless of what the true values of $θ$ are, since the Sobol’ index is a global sensitivity analysis method and considers the entire distribution of $θ$⁠.

For a single type of test, multiple test specimens are required if the test is destructive so that each specimen can be used only once. Two examples of destructive tests are fatigue test and tensile strength test. The true value of a model parameter $θl∈θ$ for $l=1 to dθ$ is fixed for a single specimen, but varies across different specimens. This variability of $θ$ may be represented by a probability distribution $π(θl|Pθl)$ where $Pθl$ are the distribution parameters of $θl$⁠. For example, $Pθl={μ,σ}$ if $θl$ has a Gaussian distribution $N(μ,σ2)$ where $μ$ is the mean value and $σ$ is the standard deviation. In addition, the entire set of distribution parameters for all components of $θ$ are denoted as $Pθ$ where $Pθl∈Pθ$ for $l=1 to dθ$⁠. In this case, $Pθ$ have unknown true values; thus, the uncertainty in $Pθ$ is epistemic; and this uncertainty can be represented by a prior distribution $π(Pθ)$ based on available knowledge. Thus, model calibration aims to quantify the uncertainty in $Pθ$⁠, instead of $θ$⁠. (Note that $θ$ have both aleatory and epistemic uncertainty, whereas the uncertainty in $Pθ$ is epistemic.)

In the case of single type of test and multiple test specimens, the steps in generation and usage of the synthetic data set of $N$ data points are similar to those in Fig. 2, but the box “model parameters $θ$ ” should be replaced by “ $Pθ→θj$⁠,” where $θj$ is the value of $θ$ generated for the $j$ th specimen (i.e., the $j$ th test). Compared to Fig. 2, the values of $Pθ$ are now selected before generating a synthetic data set; once selected, the values of $Pθ$ are fixed within the synthetic data set. The values of model parameters $θj(j=1 to N)$ for each of the $N$ specimens are generated from the conditional distribution $π(θl|Pθl)$ for $l=1 to dθ$⁠.

It seems natural to replace $θ$ in Eq. (6) with $Pθ$ and build new functions for the Sobol’ indices computation. However, the new functions will not be deterministic functions as required by the Sobol’ indices. A specific realization of $Pθ$ does not determine the values of $θ$ but only the distribution $π(θl|Pθl)$ for $l=1 to dθ$⁠; thus, $θ$ are still stochastic at given $Pθ$⁠. Only deterministic values of $θ$ and $αi={xj,ej,ϵj}$ (⁠$j=1 to N$⁠) can decide the subsequent system response prediction distribution $πY(y)$ and its mean value $E(Y)$ and variance $V(Y)$⁠. In sum, an approach to establish a deterministic relationship from $Pθ$ to $θ$ is needed.

where $Fθl|Pθl−1(⋅)$ is the inverse CDF of $θl$ at given $Pθl$⁠. Note that $Uθl$ has the standard uniform distribution $U(0,1)$⁠. Equation (7) indicates three steps: (1) generate the values of $Pθl$ from their prior distribution to produce the conditional distribution $π(θl|Pθl)$⁠; (2) generate the value of $Uθl$ from $U(0,1)$⁠; and (3) substitute $Uθl$ into the inverse CDF $Fθl|Pθl−1(⋅)$ to obtain a unique value of $θl$⁠.

The uncertainty in model parameter $θl$ consists of two components: (1) the epistemic uncertainty in distribution parameters $Pθl$⁠, represented by the prior distribution $π(Pθl)$⁠; and (2) the aleatory uncertainty in $θl$ at given $Pθl$⁠, represented by the conditional distribution $π(θl|Pθl)$⁠. These two parts are coupled since $π(θl|Pθl)$ depends on the value of $Pθl$⁠. The introduced auxiliary variable $Uθl$ captures the aleatory uncertainty, and also helps to decouple the aleatory and epistemic uncertainties [26] since the distribution of $Uθl∼U(0,1)$ does not depend on $Pθl$⁠.

As explained earlier, the basic idea of the proposed approach is to maximize the contribution of epistemic uncertainty of $θ$ in the synthetic data, in the case of a single specimen. In the case of multiple specimens, we need the contribution of $Pθ$ to be dominant in the context of Eq. (8). If that is achieved, in the system response prediction using actual test data where $Pθ$ are fixed at their true values, the most dominant uncertainty contribution to $E(Y)$ and $V(Y)$ will be removed. Therefore the uncertainty in $E(Y)$ and $V(Y)$ caused by test outcome uncertainty will be reduced significantly, and different actual test outcomes will lead to consistent system response predictions.

In the case that $q$ different types of tests are to be considered and each type utilizes only one specimen (nondestructive test), Fig. 2 expands to Fig. 3, and Eq. (6) expands to

Equation (9) gives the required deterministic functions for Sobol’ indices computation. In Eq. (9), $Ai={α1i,…,αNii}$ for $i=1 to q$ represents the uncertainty regarding inputs and measurement errors in generating the synthetic data for the $i$ th type of test, where $αji={xji,eji,ϵji}$ for $i=1 to q$ and $j=1 to Ni$⁠; $j$ represents the test number and $Ni$ is the total number of the $i$ th type of test. Note that here $θ$ is the vector of the model parameters in all types of tests, and test type refers to calibration test versus validation test, and the output quantities measured, as explained in Sec. 1.

Similar to the earlier discussion, in the test resource allocation optimization regarding Eq. (9), we need the contribution of the epistemic uncertainty in $θ$ toward the uncertainty in $E(Y)$ and $V(Y)$ to be dominant. For the case of multiple types of tests and single test specimen, an example with a framework considering only model calibration is considered in Sec. 5.1; another example of a framework incorporating both model calibration and model validation is considered in Sec. 5.2.

Similarly, in the test resource allocation optimization regarding Eq. (10), we need the contribution of the epistemic uncertainty in $Pθ$ toward the uncertainty in $E(Y)$ and $V(Y)$ to be dominant.

Thus far, deterministic functions for Sobol’ indices computation in different test conditions have been established. Robust design of resource allocation can be achieved by maximizing the contribution of the epistemic uncertainty regarding either $θ$ (single specimen) or $Pθ$ (multiple specimen). This epistemic uncertainty is represented by a set of random variables (⁠$θ$ in Eqs. (6) and (9); $Pθ$ in Eqs. (8) and (10)). The total effect sensitivity index considers the interactions between the subset of random variables and its complement; thus, to be more comprehensive, the optimization in this paper uses Eq. (4) to compute the total effect index for the subset of epistemic uncertainty (either $θ$ or $Pθ$⁠). In the rest of the paper, Sobol’ index indicates the total effect index in Eq. (4). The computed Sobol’ indices are denoted as $SmE(Y)$ for $E(Y)$ and $SmV(Y)$ for $V(Y)$⁠. In the case of single specimen, $m=θ$ so that $SmE(Y)$ and $SmV(Y)$ are the Sobol’ indices of $θ$⁠; in the case of multiple specimen, $m=Pθ$ so that $SmE(Y)$ and $SmV(Y)$ are the Sobol’ indices of $Pθ$⁠.

where $Ci>0$ is the unit cost of the $i$ th $(i=1 to q)$ type of test and $Ni$ is the number of tests of the $i$ th type; and $C0$ is the budget constraint; and $p1$ and $p2$ are user-defined positive constant weight coefficients.

where $λE(Y)$ and $λV(Y)$ are the desired lower bounds of the Sobol’ index for $E(Y)$ and $V(Y)$⁠, respectively.

Equations (11) and (12) are both integer optimization problems since the decision variables $Ni(i=1 to q)$ are integers. Sometimes, integer optimization is solved using a relaxation approach [28], where the integer constraint is first relaxed, and the integers nearest to the resultant optimal solution are used as the solution of the original (unrelaxed) problem. Unfortunately, this approach is not applicable here because the synthetic data to be used in model calibration/validation can be generated only if $Ni(i=1 to q)$ are integers. It is not possible to generate test data for a noninteger number of tests.

A simulated annealing algorithm [29] is used for the solution of Eqs. (11) and (12) because it can handle stochastic discrete optimization problems without requiring relaxation. For discrete optimization problems such as in Eqs. (11) and (12), this algorithm aims to minimize an objective function $f(s)$ where $s={s1,…,sL}$ is a vector of integers and its feasible region is $Ω$⁠. If the objective is to maximize $f(s)$ as shown in Eq. (11), $−f(s)$ ought to be minimized.

As shown in Fig. 4, the simulated annealing algorithm starts from an initial value $s0∈Ω$⁠. If $s$ is the optimal solution in an iteration, a new value $s′$ will be randomly selected within the neighborhood of $s$⁠. This neighborhood, denoted as $ℵ(s)$⁠, can be defined by different proposal density functions; and this paper defines $ℵ(s)=[s1±d1,…,sl±dL]∩Ω$ where $dl$ is a user-defined positive integer for $l=1 to L$⁠. In one iteration, if $f(s′)<f(s)$⁠, the new value $s′$ is accepted as the new optimal solution; otherwise, the probability to accept $s′$ is

where $T0$ is the user-defined starting value of $T$⁠, $k$ is the current iteration number, $K$ is the total number of iterations allowed, and $α$ is a user-defined exponent that determines the rate of decrease of $T$⁠. This iteration proceeds until the total allowed number of iterations $K$ is expended.

This section proposed formulations for test resource allocation optimization, considering two formats: (1) maximizing the Sobol’ index of the epistemic uncertainty in $θ$ or $Pθ$ subject to budget constraint and (2) minimizing the cost subject to the Sobol’ index threshold. Both formats are applicable to the cases of single or multiple specimens and single or multiple types of tests. As a result, the system response predictions become insensitive to the variability in test outcomes. The decision variables (numbers of tests) are discrete variables, and a simulated annealing algorithm is used to solve this discrete optimization. In this optimization, the Sobol’ index of the epistemic uncertainty in $θ$ or $Pθ$ is computed by the method discussed in Sec. 2.

This section uses two examples to illustrate the proposed method. The first example is a mathematical problem and the second example is a structural dynamics problem. Regarding the types of tests, specimen, and calibration/validation, the first example considers: (1) multiple types of tests, (2) model calibration only, and (3) both the cases of single and multiple specimens. The second example considers: (1) multiple types of tests, (2) both model calibration and validation, and (3) single specimen only.

The inputs $X1$ and $X2$ are assumed to be independent random variables; the uncertainty regarding their values in tests is represented by uniform distributions $X1∼U(90,110),X2∼U(40,60)$⁠, based on ranges obtained from the test personnel.

Two types of tests are available. Test type I measures $W1$ with measurement error $ϵ1∼N(0,502)$⁠; and test type II measures $W2$ with measurement error $ϵ2∼N(0,402)$⁠. The resultant synthetic data are pairwise data ${X1,W1}$ and ${X2,W2}$⁠, respectively. Assume that the unit cost of type I test is 4 and the unit cost of type II test is 1.

Two cases are considered in this example: single test specimen versus multiple test specimens. In case 1 of single specimen, model parameter $θ={θ1,θ2}$ has true but unknown values to be calibrated. In case 2 of multiple specimens, ${θ1,θ2}$ follow normal distributions $N(μθ1,σθ12)$ and $N(μθ1,σθ12)$ across specimens, and the parameters to be calibrated are $Pθ={μθ1,σθ1,μ2,σθ2}$⁠.

The process to realize the system response prediction $Y$⁠, i.e., the framework of model calibration/validation with the synthetic data is shown in Fig. 5, where the posterior distributions of calibration parameters together with the known distributions of $X1$ and $X2$ are propagated through the computational model in Eq. (15) to obtain the distribution of $Y$⁠. Note that model validation is not considered in this example; only calibration is considered. The proposed test resource allocation approach can also handle model validation, as shown in the next numerical example.

In this case, model parameters $θ={θ1,θ2}$ have unknown deterministic values and prior distributions, $θ1∼N(5,0.52)$⁠, $θ2∼N(10,12)$⁠, are assumed for them. This case is applied to the two optimization formulations in Eqs. (11) and (12).

where $N1$ is the number of type I tests and $N2$ is the number of type II tests. $N1$ and $N2$ are the decision variables, i.e., we need to decide the number of replications of each type of test.

The simulated annealing algorithm is used to solve Eq. (16), and Fig. 6 records the process of optimization. Figure 6(a) shows that the optimization starts at an initial design point $(N1,N2)=(1,1)$ and terminates at the optimal solution $(N1,N2)=(2,8)$⁠. Figure 6(b) shows that only some of the random walks are accepted and the maximized Sobol’ index sum $SθE(Y)+SθV(Y)$ is 1.89. The feasible region in Fig. 6(a) covers the combinations of $N1$ and $N2$ such that $4N1+N2≤16$⁠. Note that (1) this feasible region is obtained by extra computation and (2) this feasible region is shown only to help in visualizing the result but is not needed in the optimization.

As discussed in Sec. 3.1, since the robustness objective $SθE(Y)+SθV(Y)$ is maximized, the optimal solution $(N1,N2)=(2,8)$ for Eq. (16) should lead to consistent system response prediction regardless of the true values of $θ$⁠. Three steps are pursued to verify it: (1) assume “true” values of $θ$⁠; (2) generate multiple sets of data with the size of $(N1,N2)=(2,8)$ based on the assumed value of $θ$ from step 1; and (3) plot the family of system response prediction PDFs using the data sets in step 2 and observe whether they are consistent. Although the data are still synthetic, this is a simulation of the system response prediction using the actual test data since the model parameters $θ$ are fixed at the same value across different data sets; while in the synthetic data generation for test resource allocation shown in Fig. 2, the model parameters are fixed within a single data set but vary across different data sets. The results of this verification are shown in Fig. 7. Figure 7(a) indicates that $(N1,N2)=(2,8)$ leads to consistent system response predictions if the true values of model parameters are ${θ1,θ2}={4.9,9.5}$⁠; similarly, Figs. 7(b) and 7(c) show that consistent system response predictions are also obtained if ${θ1,θ2}={5.4,9.8}$ or ${θ1,θ2}={5.0,10.5}$⁠.

As a comparison, Fig. 8 shows the same results as Fig. 7 but at a suboptimal solution of $(N1,N2)=(1,12)$⁠. This suboptimal solution spends the same cost as the optimal solution, but the enlarged variation across different PDFs in Fig. 8 indicates that this suboptimal solution cannot guarantee consistent predictions as the optimal solution. To quantify this conclusion, Table 1 compares the “variance of the variance of the prediction” $V(V(Y))$ at the optimal and suboptimal solution. This table clearly shows that the optimal solution always has smaller values of $V(V(Y))$ at different values of $θ$⁠, which proves that the optimal solution gives more consistent predictions.

The simulated annealing algorithm is used to solve Eq. (17), and Fig. 9 records the process of optimization. Figure 9(a) shows that the optimization starts at an initial design point $(N1,N2)=(8,8)$ and terminates at the optimal solution $(N1,N2)=(3,7)$⁠. Figure 9(b) shows that only some of the random walks are accepted and the minimized cost is 19. The feasible region in Fig. 9(a) covers the combinations of $N1$ and $N2$ such that $SθE(Y)≥0.95$ and $SθV(Y)≥0.95$⁠. Similar to Fig. 6, note that (1) this feasible region is obtained by extra computation and (2) this feasible region is shown only to help in visualizing the result but is NOT needed in the optimization.

As discussed in Sec. 3.1, since the robustness constraints $SθE(Y)≥0.95,SθV(Y)≥0.95$ are satisfied, the optimal solution $(N1,N2)=(3,7)$ for Eq. (17) should lead to consistent system response prediction regardless of the true values of $θ$⁠. The same three steps for Fig. 7 are pursued to verify it. The results of this verification are shown in Fig. 10. Figure 10(a) indicates that $(N1,N2)=(3,7)$ leads to consistent system response predictions if the true values of model parameters are ${θ1,θ2}={5.7,10.5}$⁠; similarly, Figs. 10(b) and 10(c) show that consistent system response predictions are also obtained if ${θ1,θ2}={5.2,9.1}$ or ${θ1,θ2}={4.6,10.8}$⁠.

In this case, model parameters $Pθ={μθ1,σθ1,μ2,σθ2}$ have unknown deterministic values and uniform prior distributions $μθ1∼U(4,6)$⁠, $σθ1∼U(0.2,1)$⁠, $μθ2∼U(8,10)$⁠, $σθ2∼U(0.8,1.5)$ are assumed for them. This case is also applied to the two optimizations in Eqs. (11) and (12). The unit cost of type I test is 4 and the unit cost of type II test is 1.

The simulated annealing algorithm is used to solve Eq. (18), and Fig. 11 records the process of optimization. Figure 11(a) shows that the optimization starts at an initial design point $(N1,N2)=(5,5)$ and terminates at the optimal solution $(N1,N2)=(5,13)$⁠. Figure 11(b) shows that only some of the random walks are accepted and the maximized Sobol’ index sum $SPθE(Y)+SPθV(Y)$ is 1.92.

As discussed in Sec. 3.1, since the robustness objective $SPθE(Y)+SPθV(Y)$ is maximized, the optimal solution $(N1,N2)=(5,13)$ for Eq. (18) should lead to consistent system response prediction regardless of the true values of $Pθ$⁠. The results of this verification are shown in Fig. 12.