Human Digital Twin, the Development and Impact on Design

The idea of the digital twin (DT) was born at NASA in the 1960s as a “living model” of the Apollo mission [1]. A visual representation of the concept was proposed by Michael Grieves in 2002 [2] and has evolved over time [3]. Over the past few decades, it has garnered considerable attention with the widespread availability of the internet of things (IoT) infrastructure and advancements in physics, electronics, mathematical modeling, and computing.

Though there are various definitions of the DT, a meaningful and updatable temporal model, which represents a digital replica of the physical object, along with the associated enabling technologies always form the key elements of a DT. Wright and Davidson [4] have emphasized that a DT model possess the following attributes: (1) “sufficiently physics-based that updating parameters within the model based on measurement data is a meaningful thing to do,” (2) “sufficiently accurate that the updated parameter values will be useful for the application of interest,” and (3) “sufficiently quick to run those decisions about the application can be made within the required timescale.” Regarding the data flow between the physical and digital components, Fuller et al. [5] introduced the terms “digital model,” “digital shadow,” and “digital twin” to address the real-time attribute of the DT.

In the past decades, DT applications have been reported in different fields, e.g., smart city [6,7] and Industry 4.0 [8,9], mainly due to their strong ability in providing the up-to-date status of the object and meaningful predictions as the famous quote “I don’t care what anything was designed to do. I care about what it can do” by the chief flight director Gene Kranz of NASA during the Apollo 13 mission. Advantages of using DTs, e.g., improved efficiency [10], reduced (operational) cost [11], facilitated more sustainable processes [12], and enabled predictive maintenance [13], were frequently reported by researchers and practitioners.

Human factors play an important role in complex DT applications, as humans are always part of the system, acting as designers, supervisors, operators, and/or users. However, the unique characteristics of humans were not fully addressed and embedded in the concept of the DT. For instance, humans often facilitate the sensing and actuation functions of the digital model and digital shadow, even implicitly supervising the digital twin applications [5]. However, the physical, mental, and social abilities of humans in the process were not always discussed, e.g., occupational fatigue may lower the performance of the workers over time [14].

Researchers proposed different methods to address this issue. Geselschap et al. [15] indicated that “DT was the key-enabler to take people along in this development, explain and discuss the risks.” However, they did not elaborate on how human factors can be incorporated into DT applications. A digital human-in-the-loop framework was proposed by Onan Demirel et al. [16] for meeting sustainability objectives, where digital human models on comfort, biomechanics, reach envelope, metabolic energy envelope, etc. were embedded in the framework next to conventional DT elements, such as object models, prototypes, and structural analysis tools. In the context of Industry 4.0, researchers also proposed the Operator 4.0 concept to address the roles of people, e.g., on human–robot cooperation [17] and on visual inputs of the operators [18]. More recently, Nguyen [19] argued that human digital twins (HDTs) should coexist with other DTs in the Metaverse for agent-based modeling and simulation.

Though researchers made considerable progress in addressing the needs of human factors in DT applications [20] and proposed the concept of HDT, there is little consensus on the framework of HDT, the roles of HDT, nor the uses of HDT in different applications. In this paper, based on a range of literature, we aim to clarify the concept of HDT, propose an open architecture for HDT, outline the relations between HDT and associated elements in the context of usages, highlight key enabling technologies, and explore the potentials of using HDT in design.

The structure of this paper is as follows: Sec. 2 discusses the definitions and architecture of HDT. Section 3 focuses on the key enabling technologies. Section 4 outlines various application areas of HDT. Section 5 summarizes the limitations, followed by a short conclusion in Sec. 6.

Literature study has indicated that the ideas of HDT have been present since 2019, and researchers have attempted to definite HDT based on the concept of DT, considering its unique characteristics. Some of the definitions proposed are as follows:

• Interagency Modeling and Analysis Group in the National Institutes of Health indicated that “A digital twin is a digital replica of a living or non-living physical entity, such as a manufacturing process, medical device, piece of medical equipment, and even a person” [21];

• Barricelli et al. defined the HDT as “computer models of humans tailored to any patient to allow researchers and clinicians to monitor the patient's health, for providing and test treatment protocols” [22,23];

• Toshima et al. indicated that an HDT is “A human model that reproduces the individuality and characteristics of humans” [24];

• Chmiel describes the HDT as the “digital representations of humans as a very complex physical, biochemical, and electrical creature” [25];

• Miller and Spatz defined the HDT as “an integrated model which facilitates the description, prediction, or visualization of one or more characteristics of a human or class of humans as they perform within a real-world environment” [26].

These definitions highlight various perspectives of HDT. While there may be variations in the details, the consensus is that HDT represents a (temporal) digital replica of an individual, encompassing his/her characteristics across different aspects and for various tasks within different contexts. The characteristics of an HDT can be summarized as

• Personal: HDT is personalized, tailored to each individual, and reflects their unique characteristics.

• Private: HDT includes personal data and information, emphasizing the need for privacy and data protection.

• Multidimensional: HDT captures various aspects of an individual, including physiological, physical, mental, and social attributes.

• Updateable: HDT is dynamic and can be updated to reflect changes in an individual’s characteristics over time.

• Context-dependent: at current stage, HDT’s use and applicability depend on the specific context and tasks.

• Descriptive and predictive: HDT enables the description and prediction of an individual’s characteristics, behavior, and performance, although often with (significant) uncertainties.

• Mobile: The mobility of individuals in a specific context and for a specific task is crucial in the development of an HDT.

• Uncertainty: HDT involves uncertainties due to data and model limitations and variability.

• Robust: Despite the uncertainty in human modeling, HDT is designed to be reliable and resilient, minimizing the risk of failures or errors, and ensuring a meaningful representation while prioritizing the safety of an individual.

• Interactible and integrable: HDT is capable of interacting with other physical and digital objects, e.g., for visualization, and can be integrated into a system.

In defining a framework to support the implementation of HDT, Sparrow et al. indicated that for workers operating in the context of Industry 4.0, an HDT should address the communication, data aggregation, simulation, and scheduling requirements that support the responsibility of information provision, maintaining and managing a local schedule, virtual execution, and controlling the resource [27]. Zibuschka et al. considered an HDT with four units: virtual sensors, observations, functional units, and derived knowledge [28]. Sahal et al. proposed a personal digital twin concept on four aspects: (1) mental activities; (2) physical activities; (3) social activities; and (4) biological scales, with a focus on personalized healthcare [29]. Löcklin et al. considered that the unique ID, data of the represented individual or role, and models are the key components of an HDT [30]. Based on the DT concept, Lauer-Schmaltz et al. discussed a series of add-ons for HDT, including behavior mechanisms such as trust and motivation [31]. Nguyen also presented a list of psychology theories for modeling human behaviors in HDT regarding cyber-securities [19].

In summary, while the use of terminologies may vary in the literature due to different tasks and contexts, these elements can be categorized into personal data, models, interfaces between the HDT and the outside world, and data and existing models that support them. Utilizing this knowledge, we propose the architecture of the HDT framework with three modules: personal data, model, and interface, supported by a data lake of human data, as well as a library of models and interfaces as shown in Fig. 1.

Personal data module serves as a repository for storing multimodal (temporal) personal data in HDT. Different from other applications, data in HDT are personal and private, and is regulated by ethical principles and laws, e.g., by General Data Protection Regulation (GDPR) in Europe. Each data record encompasses the documentation of physiological, physical, mental, and/or social aspects of the individual related to specific properties. The data module features (nearly) static components, housing data record such as the person’s identity in the social part and his/her stature in the physical part [30]. Additionally, the data module stores temporal data related to the specific task within the designated context, capturing different timestamps to represent the “history” and “current” status of the HDT. This includes details such as the movement of (part of) the human body.

Within each data record, a “properties” section is embedded to document the ID, context, task, and other pertinent details. For example, when integrating the HDT with the DT of a robot manufacturing system, the recorded context within the HDT consists of the specific robot manufacturing system itself.

The complexity of the human, the heterogeneous nature of the data, and the limited data acquisition tools made it difficult to capture the full spectrum of human data regarding both quality and quantity. For instance, it might be a challenge for people to report their comfort experience every minute through questionnaires [32]; on the other hand, 4D scans of human physical shapes can achieve the speed of 30 frames per second (fps) [33,34]. It is essential to note that in most cases, each category of data only offers an “incomplete” representation, providing a reduced-dimensional objective and/or subjective view of the person at a particular moment specified by the context and tasks described in the “properties.”

The model module within the HDT framework encompasses three distinct types of (personal) models: static, dynamic, and updateable models. Static models represent fixed information about an individual in the context and time frame of the usage scenario, such as a 3D shape of the subject. On the other hand, dynamic models utilize “history” data and/or Multiphysics principles to adapt and evolve over time, following specific time-dependent principles, rules, or functions. An example would be using a personalized gait model to predict the leg positions of an individual [35]. Updateable models, often built upon dynamic models, have the capability to modify their structures and/or parameters based on both the “history” and “current” inputs, encompassing time and extending beyond it. For instance, a personalized long short-term memory (LSTM) model can be utilized to predict human decisions based on inputs over time during the use [36].

Depending on the application’s requirements, a model can be either personal or populational. For instance, a personalized gait model, as demonstrated by Millard and Mombaur [35], tends to offer higher accuracy compared to a population model. However, a population model can still be utilized within an HDT for gait prediction, though with larger uncertainties.

The interface module serves as the bridge between the HDT and the outside world, as shown in Fig. 2. It encompasses two primary functions: data acquisition and data sharing. Data acquisition tools are responsible for updating the HDT’s “current” status by incorporating newly sensed information, e.g., updating human postures based on real-time captured images [37,38]. Meanwhile, the previous “current” status is utilized to enrich the “history” of the HDT.

The interface module establishes connections between the HDT and physical objects/services, and in most cases, their associated DTs for acquiring specified context and environmental information. For instance, Scheifele et al. proposed a real-time co-simulation for virtual commissioning through a loosely coupled interface between DTs [39]. Data sharing tools enable the HDT to interact with services (e.g., visualization tools), DTs, other HDTs, and digital objects (e.g., digital service, databases, Metaverse). For instance, leveraging extended reality (XR) technology [40], information from various HDTs can be synchronized and presented in the Metaverse, where human and virtual agents participate in co-creation sessions [41,42].

Objective and subjective (population) data support the HDT, and the model and interface library can be stored in the data lake of human data. Data stored in the data lake can be open datasets, e.g., the CelebFaces Attributes (CelebA) dataset of human faces [43]. Personal data stored in an HDT can also be donated to the data lake via ethical approval and anonymization. Data can be categorized based on different criteria, e.g., human body shapes can be recorded in 3D/4D scans [44], and heart rate variability, which is often used in studies where the emotional stimulation is relatively strong [45,46], can be documented as time series data. Meanwhile, the subjective feeling of emotion can be recorded as time series data as well, but with asynchronous timestamps.

In an HDT, models are often sourced or adapted from the model and interface library, and if necessary, further personalized using personal data as shown in Fig. 3. A simple example is to adapt a statistical shape model (SSM) of human shapes to the 3D shape of an individual by adjusting the coefficients of principal components to fit the inputs [47]. For complex models such as deep learning models, Schneider and Vlachos summarized three techniques: early shaping, sample weighing, and transfer learning, which utilize collected personal data in the HDT [48]. As an example, Saeedi et al. applied transfer learning to create a personalized human activity recognition (HAR) model using data obtained from inertia measurement units [49].

While models play a crucial role in forecasting the future status of the HDT, leveraging the stored historical and current data [50], it is important to note that models are highly context and task-dependent. Each model represents a partial truth within its defined boundaries, often accompanied by (high) uncertainties. Safety, which is one of the top concerns in different contexts, should always be prioritized in developing models for an HDT, taking into account these uncertainties.

While HDT and DT share significant overlaps in terms of key enabling technologies, HDT possesses unique challenges related to sensing, modeling, and acting. For example, each individual’s body shape is distinctive, and in addition to physics-based modeling, data-driven modeling techniques are widely used to capture physiological, physical, mental, and social aspects of humans. In this section, starting with the IoT architecture, we present our vision of the key enabling technologies that facilitate the development and utilization of HDT with the focus on personalization, privacy, multimodality, context-dependency, mobility, robustness, and uncertainty. Furthermore, we also highlight the opportunities and potential barriers associated with it.

The IoT infrastructure serves as the foundation for sensing and actuating in the context of human digital twin applications. However, the (near) real-time requirements of sensing multimodality data of (moving) humans in asynchronous timestamps present challenges for sensing, storage capacity, computing power, and bandwidth limitations. For example, transmitting 4 K 60 Hz video for HAR can require a high bandwidth of up to 32 Gbps, and multiple cameras are needed as the subject is often moving. HDT designers and engineers need to adopt a holistic view of the system and carefully allocate the necessary data, models, and interfaces to different parts of the IoT infrastructure, e.g., using the edge, fog, and cloud architecture [51]. Example applications indicated that real-time response data and models can be deployed at the edge and fog, while background tasks like continual learning of user behaviors can be handled at the cloud side [52]. Researchers have also invested significant efforts in downsizing advanced machine learning models for their utilization in edge computing [53]. Furthermore, the HDT should have the ability to tolerate possible loss of real-time data or compensate for it by utilizing data from other modalities, ensuring a more robust and safe performance of the HDT.

Data security is one of the top concerns in developing and using HDT, encompassing ethical considerations on privacy, regulatory compliance, and business requirements. Technically, principles and implementations of HDT data security are not different from security and privacy research in general [54]. A promising technology is using blockchain, which improves security utilizing computing capacity, bandwidth, and storage resources. The major advantages that blockchain has brought to security can be summarized as tamper-proofing, disaster recovery, and privacy protection [55]. Götz et al. [56] explored the usage of blockchain technology in DT regarding the applicability, interoperability, and integrability. Liu et al. [57] also introduced blockchain into the internet of vehicles to improve the accuracy and efficiency of access control. An example of using blockchain in medical HDT applications is the decentralized epidemic alerting system proposed by Sahal et al. [58]. However, the immutability and transparency of the personal data documented by the blockchain in the HDT might lead to ethical issues [59], as Articles 17 and 19 of GDPR [60] assure the “right to be forgotten” of personal data of individuals. New flexible security protocols tailored to the requirements of the HDT might be helpful regarding users’ sensitive information.

Sensors deployed in the environment, such as cameras and microphones, facilitated by HAR and speech recognition algorithms [61], have often been used to sense humans in relatively fixed environments. However, wearables carried by individuals offer a more flexible manner for updating personal data in the HDT. In the past decade, the prevalence of wearable devices and services has enabled the potential for wide uses of the HDT. It was reported that till 2022, there will be up to 1.1 billion connected wearable devices worldwide [62]. This is especially true with the development of the pandemic [63].

Wearables can provide essential information for the HDT via the data acquisition interface, e.g., human activities can be recognized based on data collected by wearables [64], human heart rates can be monitored by a wrist watch [65], and hand gestures can be recognized by a smart glove [66]. Professional wearables are often used in HAR, e.g., Skals et al. [67] using Xsens [68] in the study of physical demands of work in material handling in supermarkets. Though accurate, the cost of these wearables is high, the size and/or the weight are large, and professional training is often needed.

Consumer-level wearables did encourage users to participate in physical activities [69]. On the other side, there is skepticism about the quality of the data and the effectiveness of some human activity tracking applications [70]. For instance, in a study using electronic assistive technology to prevent falls for solo-living adults, researchers found little evidence of the effectiveness [71], which is affirmed by Montero-Odasso et al. [72]. Besides, the wearability of body-worn devices and technologies is often a concern in the use of wearables [73]. Wearables that prioritize technical requirements over usability often result in an unsatisfactory user experience, leading to low customer loyalty [74]. More work is needed for wearable developers to develop comfort wearables that are able to provide reliable and meaningful information, to support the functions of HDT for the user as well as the stakeholders [75] with clear evidence.

Modeling humans is a highly complex task. Despite the abundance of prior knowledge about humans, they are inherently dynamic, and their feelings and behaviors exhibit various levels of uncertainty in different contexts. These complexities present significant challenges when constructing context-sensitive temporal models of humans, especially for individuals. The literature indicates that our current understanding of humans only scratches the surface. Typically, a model can only capture certain aspects of the intricate nature of humanity from a specific perspective, usually within a particular context and for a specific task, while still having substantial uncertainties. For instance, a 3D scan of a human hand can achieve a mean absolute error of 0.62 mm [76]. However, when predicting changes in comfort feeling, researchers reported 11% and 21% errors in comfort and discomfort prediction, respectively [77].

Physiological models for HDT have attracted much attention in the past decades. A series of models were created at the cell level, e.g., cancer [78,79], the organ level, e.g., the heart [80,81] and the liver [82], and the system level, e.g., the musculoskeletal system [83,84]. For different interventions, surgeons also benefited from the use of the (H)DT in training or in operation [85]. As part of the revolution in the healthcare industry, the use of the HDT enables personalized medicine [86], i.e., individuals are able to make personalized health and wellbeing management (including prevention) with clinicians and/or AI models [75]. For clinicians, HDTs also enable them to deliver more personalized, intelligent, and proactive prevention, cure, and care [29].

While the inside structures and the contour of humans, e.g., the musculoskeletal system of a subject, can be revealed by different types of medical images, e.g., ultrasound and magnetic resonance imaging [87], the cost is high, and the procedure is often time-consuming. And some medical imaging methods are intrusive for humans as well, e.g., computed tomography. A more accessible method of acquiring physical shapes of humans is 3D/4D scanning. Recent developments in technology made it possible to track human body deformation in real-time using different sensors, e.g., Azure Kinect [88]. However, the cost of 3D/4D scanning is still high, and post-processing is often needed to make the scanned data useful. SSMs have been widely used in the past decades to approach the physical shape of humans [47] based on acquired anthropometric measures, e.g., the stature. An example of the 3D human SSM is the DINED database [44]. For 4D shapes of humans, rigging a skeleton (with fixed degree-of-freedoms) in a 3D human shape (avatar) and deforming it accordingly are widely adopted techniques in animations [89], though the authenticity of the deformation strongly depends on the requirements of the applications. For instance, an error in several millimeters might be acceptable for visualizing the shape of a virtual human agent in the Metaverse [90], but it might lead to fitting problems for a personalized splint [47,91]. Recent development of 4D scanning technique enables more precise data capture of 4D shapes where the skinned multi-person linear (SMPL) model is a typical example [92]. However, accuracy on details with different populations still can be improved.

Mental models are essential for many HDT applications, e.g., in the manufacturing context where human emotion might influence the effectiveness and efficiency of the work [93]. Nguyen [19] indicated that many qualitative mental models can be beneficial for the development of the HDT; however, developing a quantitative mental model is often a challenge [94]. Researchers have explored the possibility of using objective measures to interpret human mental states, e.g., using electroencephalogram [95] and using facial features [96]. Existing quantitative models on mental models mainly focus on cognition and behaviors [97], where data-driven methods, e.g., deep learning, were widely adopted in modeling. For instance, recent developments in large language model demonstrate the feasibility and the potential in modeling cognition of humans [98]. Meanwhile, though the conceptual frameworks of personalized cognitive models were proposed [99], collecting big amounts of data for quantifying individual models is still a challenge [99]. Researchers suggested combining data-driven and theory-based approaches to reduce the complexity, both in the construction and in the use of the model [99,100].

Social-behavioral models are associated with mental and physical models of humans. Similar to the mental models, though a large amount of qualitative models were available [101], quantifying those models is still in progress. For instance, Tyshchuk and Wallace modeled three human behaviors in social media: (1) obtain and propagate the warning, (2) seek additional information/confirmation, and (3) take the prescribed action [102]. Recently, development in deep learning offers opportunities in modeling groups as well as personal social behaviors, e.g., Phan et al. developed their explainable human behavior prediction models with 33 inputs in the context of health social networks [103]. Meanwhile, social media, e.g., LinkedIn [104], feeds massive amounts of data in building an effective social networking model, e.g., researchers found that people who have weak ties help most in finding a job [105]; Luceri et al. developed a model to predict the influence of social network activities on real-life human behaviors with >80% accuracy [106].

With the underlying that each human model is a partial truth within its defined boundaries, often accompanied by different levels of uncertainties, human beings are 99.9% identical in their genetic makeup [107], which indicates that there is a considerable degree of overlap regarding physiological, physical, mental, and social aspects of humans. In constructing human models, researchers often use population models built on ethnicity groups [108], age, educational background, anthropometric measurements, etc. as an intermediate step between general human models and personalized models [109]. Meanwhile, besides physics principles [4], data-driven approaches, e.g., deep learning, are frequently employed to model various aspects of humans due to their ability to capture unknown principles, complex patterns, and relationships. Consequently, large datasets encompassing different facets of humans, e.g., AffectNet [110], are essential in the future of human modeling.

Besides presenting the status of a specific human in a context regarding specific tasks, a key function of the HDT is to make meaningful predictions about the person regarding a list of requirements, e.g., safety, physical, mental, and/or social status.

As summarized in Sec. 3.4, data-driven methods have attracted much attention in the past decade, and artificial intelligence (AI) models have been built for predicting human physiological [111], physical [64,112], mental [113], and social activities [114]. However, building and updating the model have different requirements. For instance, in the use of a data-driven approach to build an SSM of the human hand [115], a large amount of (3D) data might be needed to construct a model. But in the use of the model, only a few inputs are needed to update the model. Currently, many data-driven models are black-box models and even the creator(s) of the model do not fully understand how different inputs (parameters) are being combined to make reasonable predictions [116].

Explainable AI (XAI) focuses on explaining the model to discover, justify, improve, and control the model [117]. For instance, Lundberg and Lee proposed the SHAP (SHapley Additive exPlanations) method, which utilized a game theoretic approach to explain the output of any machine learning model [118,119]. In detail, model agnostic methods, e.g., permutation importance in Eli5 [120], and model-specific methods, e.g., deep visualization toolbox [121], are often used in XAI. XAI not only helps researchers to gain a deeper understanding of the model and its inputs but also helps them highlight the important factor(s) of the models/inputs [122]. Those highlights might further simplify the (personalized) model by reducing the needed computing power and sensor inputs, which will further minimize ethical concerns in acquiring human data as well.

Visualizing the data of human(s) of an HDT, together with the physical/digital objects around them in a context, can help the user to recognize possible trends/patterns, infer possible causes and effects, recognize possible (cor)relations among different data sets, identify potential outliers/problems, and eventually, trigger possible sensemaking process of the user for different purposes [123,124].

Data and model(s) in an HDT can be visualized in different ways, and researchers developed many data visualization tools to support such activities, e.g., scientific, information, and analytic visualization tools [125]. An example is that Austin et al. [126] used six data visualization tools to identify patterns in whole-person health for adults. A promising development is the use of virtual reality/augmented reality/mixed reality (VR/AR/MR, or XR) tools, which allow the user to not only visualize data in an intuitive manner but also interact with other (H)DTs for different purposes [127], e.g., co-creation [128] and co-simulation [129]. An example is that Aivaliotis et al. proposed an AR suite integrated with the DT of the shop floor, for developing human–robot interaction using Microsoft^® HoloLens^® [130]. Geng et al. also created a modular DT system that integrates VR/AR for remote control and virtual machining [131].

Though researchers and engineers paid considerable effort in developing data visualization tools and methods, there are still some concerns [125], e.g., for the interface of the HDT, currently, data sharing protocols are often ad hoc [129]. Generalized protocols might help the development of HDT in the future. Researchers also acknowledge that presenting a huge amount of multimodality data of the (H)DT in a comfortable and meaningful manner remains a challenge [132]. For instance, VR sickness occurs in many people [133], and the causes might be associated with content, locomotion, and exposure time [134]. All of these pose challenges to the effectiveness and efficiency of presenting the data from (H)DTs [135].

For the development of the HDT, we propose the concept of minimum viable sensing (MVS) based on the concept of minimum viable products [136], however, with different intentions. MVS of an HDT indicates that the sensed and stored data shall be “just enough” for the purpose of the HDT, e.g., predicting the level of comfort of a passenger. MVS asks HDT designers to have a holistic view of the purpose of the HDT, the requirements, the usage scenarios, the available IoT infrastructure, the model, the people, and the ethical concerns for a balance of technology and humanities.

MVS for an HDT can be determined by first exploring the person, the context, and the task of the HDT, and further optimized through the use of XAI and/or other tools. For instance, questions such as “What are the usage scenarios that the HDT be deployed?” “What is the acceptable accuracy?” “What are the available sensing methods?” and “What are the least intrusive ways of deploying wearables on the users?” can help HDT designers to explore multiple possibilities for sensing the required inputs. Meanwhile, designers can start from existing HDT models, or construct a brute-force attack model and then optimize the model with XAI tools. For instance, among 53 2D dimensions of the hand, Yang et al. found that 21 of them have >1% contributions to 3D shapes, and 16 of these 21 have lower measurement variations [122]. Using these 16 dimensions, they can approach 3D shapes with certain accuracy. It is worth mentioning that technically, MVS does not mean a trade-off between accuracy and less amount of data. In fact, in many cases, the accuracy is improved by removing some inputs with large measurement variations [116,122].

The advantages of using MVS are twofold. From a humanities perspective, less data mean fewer ethical concerns in compliance with the GDPR [137], and it is easier to configure wearables (if any) in an ergonomic manner for the user [138]. Technically, less data translates to the use of fewer hardware and software resources on sensors, bandwidth, storage, and computing power, all of which contribute to a more effective and efficient system. However, to achieve MVS, HDT designers need to have a holistic view of the system, pay extra attention to both humanities and technology, and utilize tools such as XAI to generate more insights into the effects of the sensed data.

The content of HDT is not new, and knowledge of different aspects of humans is available. However, the concept of HDT offers an overarching architecture and addresses the importance of time effects, i.e., the information of a human is dynamic, and the model can be continually updated over time. Among different applications of HDT, an extensive literature search did not find a “full” HDT. In most cases, only certain aspects of human attributes are used in a particular context for specific purposes. In the following, we highlight some HDT applications in different fields.

Personalized medicine: In this area, researchers have developed different models on physiological and physical aspects of humans to understand the in situ scenarios of patients for better interventions as summarized in Sec. 3.5. Besides curing, preventive healthcare with wearables also attracted much attention due to many benefits it brings, e.g., longer life span, a better quality of life, and reduced risks in rehabilitation [139]. Meanwhile, HDTs with wearables can be used to support different types of rehabilitation, e.g., sports and cognitive rehabilitation, and provide rehabilitation aids for individuals with disabilities [140]. For instance, Wu and Luo indicated that wearables can monitor physical activities, mental status, etc., and the sensed data can be used to update an HDT for providing suggestions based on in situ data [141].

Smart cities: While researchers and designers focus on building a best-in-class customer experience in smart cities, sensing the dynamic information of city users is the basis for optimally managing the city. For instance, Saeed et al. explored city DT concepts and highlighted the importance of users and their experience in the city of the future [6]. Lee et al. recognized that city user interfaces are crucial enablers for ubiquitous interaction with immersive systems in smart cities [142]. Psyllidis et al. developed a dynamic walk path model for crowd management during the pandemic to ensure social distancing [143]. Villanueva et al. utilized real-time citizen information from social networks and smartphone applications, i.e., HDTs of individuals, to enhance the situation awareness of civil servants regarding crowded events [144].

Manufacturing: The European Commission recently published a white paper on Industry 5.0, emphasizing the importance of a sustainable, human-centric, and resilient industry [145]. While specific details are still being discussed [146,147], it is evident that a human-centric approach [148] and value-driven technology [149] are essential. Recognizing the value of humans in different contexts is crucial for optimizing their performance [150]. For instance, Mourtzis et al. explored the roles, functions, and needs of human operators in future factories utilizing MR tools [151]. Among different activities of humans, human–machine interactions (HMIs) [152,153] and especially human–robot interactions [154–157] attracted much attention. Though different types of models in HDT are needed in modeling the in situ scenarios, cognitive and physical models that accurately and reliably capture the relevant contexts and tasks are of utmost importance.

Sports: HDTs were used in monitoring real-time activities and offering advice based on users’ personal conditions. For instance, Fister et al. applied a LSTM-based cognitive model to leverage in situ sensor information for training cycling athletes [158]. Barricelli et al. created an HDT that mirrored the athletes’ conditions and behaviors, allowing for the prediction of corresponding suggestions [23].

Mobility: Travel is not rational, but it is in our genes [159]. HDT was used in many mobility applications. For instance, in the scenarios of driving, Wang et al. developed a mobility digital twin system consisting of HDTs, DT of vehicles, and a DT of traffic [160] where for the HDTs, all stakeholders involved in the transportation system, e.g., drivers, passengers, including their behaviors, are modeled. A key advantage of using HDTs in mobility is personalization, e.g., Anda et al. proposed the DT travelers, which utilized a two-step framework to synthesize individual travel demands based on data collected from their mobile phones [161].

Metaverse: Human and human-like virtual agents are integral components that enhance the immersive experience and enable rich interactions in the Metaverse. Nguyen emphasized the coexistence of HDTs alongside other DTs in the Metaverse, enabling agent-based modeling and simulation [19]. Moreover, Abraham et al. asserted that DTs and HDTs are the foundation of the Metaverse [162]. Oh et al. highlighted the significant impact of incorporating realistic human shapes from HDTs in generating greater enjoyment and social influence, particularly in neutral and positive contexts [163]. This aspect holds immense potential for various Metaverse applications, including marketing, education, tourism, and healthcare [164].

Design: “Knowing the user” is a fundamental principle in human-centered design [165]. HDTs encompass a comprehensive range of physiological, physical, psychological, and social information about the user, allowing for real-time updates and seamless integration with other digital objects. Personal, real-time, prediction, and automation are the added values of using HDT in designing products, services, and systems [166].

At the product level, HDT offers personalized information to facilitate the design. For example, in the realm of footwear design, Rout et al. [167] utilized 3D scans of a person to create customized shoe lasts. Personalized braces and splints have also emerged in the market, to cater to specific user needs [91]. In service design, personalized marketing strategies have been developed to enhance user experience [168]. However, it is crucial for designers to navigate ethical boundaries and strike a balance between various considerations [169,170]. At the system level, the design of smart cities serves as a notable example, as summarized before. HDTs are also employed in the context of Industry 4.0 for the HMI design [171]. In these scenarios, different users, such as operators and supervisors, and machines, such as robots, must function both independently and synergistically to achieve optimal outcomes.

Moreover, HDT also enables new design tools and triggers new design processes. For instance, remote presence and collaboration have brought forth novel forms of customer experience and value co-creation [42]. This is especially true in Metaverse where HDT(s) can act as virtual agents on behalf of human agents when they are inactive. In other words, HDTs can autonomously simulate human actions, contributing to automated interactions even in the absence of the actual human [172].

Case study—personalized product design: Personalized products refer to products that are specifically designed and manufactured to meet the unique needs and preferences of individual customers. This includes catering to functional requirements as well as esthetic preferences [146]. Among different types of personalization, personalization-in-fit addresses the presence of the personalized forms regarding the interactions between the product and the consumer, the environment and/or other products that are used by the consumer.

In the design of personalization-in-fit products [173], the HDT of the user, which contains information regarding their body shapes, is often used. The creation of the HDT involves building a model of (part of) the user’s body shape. This can be done by directly utilizing body shape data acquired from 3D scans [9] or by adapting a few parameters of a human body SSM in the model library [10]. Figure 4(a) shows a SSM of the human hand that was developed and stored in the model library. In Fig. 4(b), researchers utilized XAI tools to highlight the most important 2D dimensions for minimum viable sensing. In the figure, each 2D dimension has more than 1% dominance value regarding 3D shapes. However, dimension 2, 3, 4, 7, 8, 9, 10, 11, 12, 14, 20, 22, 33, 39, 41 and 43 have low variances in data acquisition and therefore were recommended for data acquisition in building a personalized human hand model. In Fig. 4(c), three personalized hand splints were designed for three different hands using the information of three HDTs, respectively.

While this paper focuses on the technical aspects involved in the development and utilization of HDTs, it is important to acknowledge that many other aspects, e.g., ethics and regulations, are crucial for future research and applications on HDT. For instance, the rapid development of data collection methods and AI [174] often lead to new knowledge and insights into existing data, which can potentially give rise to new ethical issues. Further exploration and discussion of these aspects are strongly recommended. While there is abundant literature on each key enabling technology, the use of these technologies in HDT is highly context and task-dependent. Space limitations prevent in-depth discussions of details, particularly regarding the complexity, accuracy, robustness, and uncertainty involved in human modeling. Further research and analysis are necessary to investigate these aspects.

The number of research works in the field of HDT has significantly increased over the past decade, yet there is still much ground to cover. This paper explores the framework of HDT, its key enabling technology and potential applications. While we propose that the personal data, model, and interface are the three key modules in building an HDT, we also envision IoT infrastructure, data security, wearables, human modeling, XAI, data visualization and XR, and minimum viable sensing as key enabling technologies with the focus on unique attributes of HDT, i.e., personal, private, multidimensional, updateable, context-dependent, descriptive and predictive, mobile, robust, uncertainty, and interactible and integrable.

While this paper provides a glimpse into the field, current applications have indicated that using HDT could offer a multitude of new (design) opportunities in areas such as personalization, real-time interactions, predictive modeling, and automation. Further exploration and research are needed to fully realize the potential of HDT in these domains.

There are no conflicts of interest.

No data, models, or code were generated or used for this paper.