ABSTRACT

Aim

This study aimed to evaluate the readability, quality, and temporal consistency of Turkish responses generated by ChatGPT to frequently asked questions in pediatric dentistry.

Materials and Methods

Twenty open-ended questions were adapted from the frequently asked questions published on the American Academy of Pediatric Dentistry parent information page. Each question was submitted to ChatGPT (GPT-4, OpenAI, USA) in Turkish over seven consecutive days, three times per day, using a new chat session for each interaction, standardized prompts and default system settings. A total of 420 responses were analyzed. Readability was assessed using both the Ateşman and Çetinkaya-Uzun readability formulas, while response quality was evaluated using the Global Quality Score (GQS). Inter-day and intra-day comparisons were performed using repeated-measures ANOVA or Friedman tests according to the data distribution. Intraclass Correlation Coefficient (ICC) analysis was used to evaluate temporal consistency (α=0.05).

Results

A significant intra-day variability in the readability scores was observed during the first four days for the Ateşman and Çetinkaya-Uzun formulas (p<0.05), whereas no significant differences were identified during Days 5-7. Between-day comparisons revealed statistically significant variations in readability scores for both formulations (p<0.001). Ateşman’s scores were found to correspond to “moderate difficulty” and “easy” readability levels, while Çetinkaya-Uzun’s scores indicated an “instructional reading level”. By contrast, no significant intra-day or between-day differences were observed in the GQS scores (p=0.650), and overall quality scores remained consistently high throughout the study period. The ICC analyses demonstrated good-to-excellent consistency for readability and GQS measurements, with overall ICC values ranging between 0.906 and 0.961.

Conclusion

The Turkish responses generated by ChatGPT to frequently asked questions in pediatric dentistry demonstrated moderate readability, high information quality, and good-to-excellent temporal consistency. However, due to the variability in readability scores across repeated assessments and over time, Large Language Models (LLMS) may serve as supportive tools for patient and parent education, however, human oversight remains necessary for the interpretation and clinical use of LLM-generated health information.

Introduction

Large Language Models (LLM) represent a significant innovation in technology, with the objective of emulating cognitive processes which are characteristic of human intelligence through the utilization of computer systems. This development has precipitated substantial transformations across numerous domains, with particular relevance to the field of healthcare in recent years (1). The increasing data capacity, more advanced algorithms, and developments in LLMs have resulted in the wider use of AI applications in the early diagnosis of diseases, clinical decision support systems, treatment planning, and patient information processes (2). These developments provide rapid and straightforward access to information and influence the decision-making processes of patients and their relatives regarding health-related issues (2).

Pediatric dentistry is a specialty in which preventive approaches and parental education play a decisive role in treatment success (3). In this context, parents’ access to accurate, understandable, and accessible information is of great importance for clinical outcomes. Today, internet-based information-seeking behaviors have changed with the widespread use of LLMs, and users have begun to prefer platforms which provide direct answers to their questions rather than relying solely on traditional search engines (4, 5). However, the readability and consistency of the information provided through these platforms have become critical evaluation criteria, particularly in the field of healthcare.

In recent years, studies investigating the capacity of ChatGPT to generate health-related information have increased in the literature (6-10). Nevertheless, when the current literature is examined, it is evident that most studies have focused largely on English-language content, while research addressing Turkish-language health-related content, particularly in the field of pediatric dentistry, remains limited (11, 12). Although recent Turkish studies have evaluated LLMs’ performance for general parental inquiries as well as specific pediatric dental contexts such as fluoride education, sedation-related questions, pulpotomy, and molar incisor hypomineralization, these studies have primarily focused on response accuracy or single-timepoint performance (13-17). Consequently, important aspects of LLM-generated patient education, including temporal consistency across repeated interactions, changes in readability over repeated queries, and the combined evaluation of readability, information quality, and consistency, remain insufficiently investigated in Turkish-language pediatric dentistry. Considering the effect of language structure on readability, the differences in Turkish in terms of word and syllable structure necessitate a separate evaluation of the comprehensibility of texts produced in this language. Therefore, evaluating not only readability but also the informational quality and temporal consistency of LLM-generated Turkish health information has become an important research priority.

A comprehensive evaluation of LLM-generated health information requires the use of validated and language-appropriate assessment tools. For Turkish texts, the Ateşman and Çetinkaya-Uzun readability formulas are among the most widely used readability indices (18, 19). While the Ateşman formula classifies texts according to their overall reading difficulty, the Çetinkaya-Uzun formula estimates the educational level required for comprehension, thereby providing complementary perspectives on textual readability. In addition to readability, the Global Quality Score (GQS) is a validated 5-point Likert-type scale widely used to evaluate the overall quality, educational value, and usefulness of patient-oriented health information, including LLM-generated content (20). Together, these complementary instruments enable a multidimensional assessment of LLM-generated health information by evaluating not only its linguistic accessibility, but also its educational quality and consistency. In order to address the aforementioned knowledge gaps, the present study adopted a repeated-measures design in order to comprehensively evaluate the readability, informational quality, and temporal consistency of ChatGPT-generated responses using validated Turkish readability formulas, the GQS, and Intraclass Correlation Coefficient (ICC) analyses. Accordingly, the present study aimed to evaluate the readability, overall quality, and consistency of ChatGPT’s responses to frequently asked questions by patients and parents in the field of pediatric dentistry. The null hypotheses tested in this study were determined as follows: (i) The Ateşman readability scores of the responses generated by ChatGPT do not show statistically significant differences between different days and repetitions. (ii) The Çetinkaya-Uzun readability scores of the responses generated by ChatGPT do not show statistically significant differences between different days and repetitions. (iii) The GQS scores of the responses generated by ChatGPT do not show statistically significant differences between different days and repetitions. (iv) The responses generated by ChatGPT do not demonstrate a high level of consistency across different time periods.

Materials and Methods

Study Design

The present study was designed as an observational and descriptive investigation with the objective of evaluating the overall quality, consistency, and readability of responses generated by a LLM (ChatGPT Plus, GPT-4o, OpenAI, USA) to frequently asked questions by patients in the field of pediatric dentistry. The LLM was accessed on 10^th of June, 2025 using the default system settings, with no additional prompting or parameter modification.

Question Preparation

In this study, 20 open-ended questions were prepared regarding topics frequently asked by patients and parents in the field of pediatric dentistry. Rather than constituting a novel questionnaire or psychometric instrument, the question set was designed to reflect common parental concerns regarding children’s oral health. The questions employed in this study were adapted to the scope of the research based on the frequently asked questions published on the parent information page of the American Academy of Pediatric Dentistry (21). The final set of 20 questions was subsequently reviewed by two experienced pediatric dentists in order to ensure clinical relevance, clarity, and appropriateness for Turkish-speaking parents before data collection. Any differences in wording or content were resolved through discussion until consensus was reached before the data collection (Supplementary File 1). As no accepted methodology exists for determining the number of questions in LLM evaluation studies, the present sample size of 20 questions was selected in accordance with comparable pediatric dentistry studies using parent-oriented, question-based ChatGPT evaluation designs (13, 14, 17).

Data Collection

The finalized 20 questions were posed to ChatGPT in Turkish on three occasions per day for a duration of seven days. The timing of these data collections was as follows: in the morning between 08:00-10:00, in the afternoon between 12:00-14:00, and in the evening between 18:00-20:00. All questions were entered into the system manually in a standardized format. Responses were generated using the ChatGPT web interface under default system settings, with no custom instructions, system prompts, or parameter modifications. Each interaction was conducted in a new chat session. Before every measurement, a new conversation within the same user account was initiated in order to prevent carry-over effects from previous interactions and to ensure the independence of each generated response. The responses were evaluated by an assistant professor (10 years of clinical experience) and a professor (15 years of clinical experience), both specialized in pediatric dentistry. Any discrepancies in the assessments were resolved through discussion until a consensus was reached.

Ateşman Readability Formula

The Ateşman Readability Formula, developed by Ateşman in 1997 as an adaptation of the Flesch Reading Ease Formula into Turkish, was used to evaluate the readability level of the texts (18). According to the Ateşman readability formula, readability levels were classified as very difficult, difficult, moderately difficult, easy or very easy (18).

Çetinkaya-Uzun Readability Formula

This formula, developed to evaluate the readability of Turkish texts, is based on average word length and sentence length (19). The readability formula was calculated and readability levels were classified as frustration reading level (10^th-12^th grade), instructional reading level (8^th-9^th grade) or independent reading level (5^th-7^th grade) (19).

GQS Scale

The overall quality level of the responses was evaluated using the GQS, a 5-point Likert-type scale (20). The GQS scale levels were classified as excellent quality, low quality, moderate quality or high/excellent quality (20).

Statistical Analysis

The statistical analyses were conducted utilizing IBM SPSS Statistics Version 23.0 (IBM Corp., Armonk, NY, USA). Continuous variables are presented as mean±standard deviation or median (Q1-Q3), while the categorical variables are presented as n (%). In order to assess differences between repeated measurements, Repeated Measures ANOVA was applied when the normal distribution assumptions were met. In instances where the assumption of normality was not met, the Friedman test was employed, and if statistical significance was detected, pairwise comparisons with Bonferroni correction were conducted. For GQS scores, the assumptions for repeated-measures ANOVA were not satisfied; therefore, all within-day comparisons were performed using the Friedman test.

In order to evaluate the consistency of responses over time, the ICC was calculated. The analysis was conducted in accordance with the two-way mixed effects and absolute agreement model [ICC (3,k)] (22). Inter-rater agreement was determined by calculating Cohen’s Kappa coefficient (23). In all analyses, the level of statistical significance was accepted as p<0.05.

Results

The results are presented according to the predefined outcome measures, including readability, overall quality, and temporal consistency of the ChatGPT-generated responses.

The findings obtained using the Ateşman readability formula are presented in Table I. A significant difference was observed among repeated assessments within Days 1-4 (p=0.017, p=0.016, p=0.003, and p=0.018, respectively), whereas no significant intra-day differences were detected for Days 5-7 (p=0.937, p=0.754, and p=0.222, respectively). Furthermore, between-day comparisons revealed statistically significant differences in Ateşman readability scores (p<0.001). The highest mean readability score was observed on Day 2, Repetition 1 (79.2±15.1), whereas the lowest mean score was recorded on Day 7, Repetition 2 (60.3 ±15.6). Despite these statistically significant differences, the readability scores remained within the same readability category throughout the study period, indicating that the observed variations were not associated with a meaningful change in the practical readability of the generated responses.

The results obtained using the Çetinkaya-Uzun readability formula are presented in Table II. A significant intra-day variability was observed among repeated assessments on Days 1-4 (p=0.022, p=0.020, p=0.003, and p=0.009, respectively), while no significant differences were detected on Days 5-7 (p=0.887, p=0.525, and p=0.399, respectively). Furthermore, between-day comparisons revealed statistically significant differences in readability scores (p<0.001). Similarly, although statistically significant differences were observed, these variations did not result in meaningful changes in the corresponding educational reading levels, suggesting that the practical comprehensibility of the responses remained largely stable over time.

The results of the GQS evaluation are presented in Table III. No statistically significant differences were found between repetitions within the same day for any of the days analyzed (respectively: p=0.497; p=0.595; p=0.368; p=0.692; p=0.311; p=0.174; p=0.054). In the analysis in which all repetitions were evaluated together (1^st-21^st repetitions), no significant difference was detected (p=0.650).

The ICC analysis results for the Ateşman readability scores are presented in Table IV. Reliability ranging from moderate to good was observed in the within-day repeated measurements (ICC=0.588-0.883). In the analysis evaluating all repetitions together, the ICC value was 0.958 [95% confidence interval (CI): 0.925-0.980]. All ICC values were found to be statistically significant (p<0.001).

The ICC analysis results for the Çetinkaya-Uzun readability scores are presented in Table V. Moderate, good, and excellent levels of reliability were obtained in the within-day repeated measurements (ICC=0.650-0.919). In the analysis evaluating all measurements together, the ICC value was 0.961 (95% CI: 0.932-0.982). All ICC values obtained in the analyses were statistically significant (p<0.001).

The ICC analysis results for the GQS scores are presented in Table VI. In the analysis evaluating all measurements together, the ICC value was 0.919 (95% CI: 0.919-0.979). All ICC analyses were found to be statistically significant (p<0.001).

Discussion

The findings of the present study demonstrated that ChatGPT-generated responses to frequently asked question in pediatric dentistry generally exhibited a moderate level of readability, high quality scores, and overall good consistency between repetitions. However, variability in readability scores was observed both within the same day and across different days. Accordingly, the first and second null hypotheses, proposing that the Ateşman and Çetinkaya-Uzun readability scores did not differ significantly across different days and repetitions, were rejected. The third null hypothesis, which proposed that GQS scores did not demonstrate significant differences over time, was accepted, whereas the fourth null hypothesis, which proposed that ChatGPT responses did not demonstrate a high level of consistency over time, was rejected. Collectively, these results suggest that although ChatGPT maintains a relatively stable level of informational quality over time, the linguistic complexity of its responses may fluctuate depending on when the same prompt is submitted.

According to the findings obtained in the present study, the readability results demonstrated that the Ateşman readability scores were generally within the range of 60-80, indicating that the responses were between the “moderately difficult” and “easy” reading levels. Similarly, the Çetinkaya-Uzun scores ranging between 35 and 47 suggested that the contents could be classified within the ‘”instructional reading level” category. These results indicate that the responses are understandable for a certain user population. However, the “instructional reading level” identified by the Çetinkaya-Uzun formula approximately corresponds to an eighth- to ninth-grade educational level, suggesting that individuals with limited educational attainment or lower health literacy may still experience difficulties in fully comprehending or applying the information provided. Consequently, readability should be considered as an important determinant of the practical usefulness of LLM-generated patient education materials, in addition to their factual accuracy and informational quality.

Previous studies have similarly reported that the readability levels of health-related content generated by LLMs is generally acceptable, although substantial variation exists depending on the clinical topic, language, and assessment method employed (24-27). In a study conducted by Aypar Akbağ (26), the mean Ateşman readability score of content related to gestational diabetes generated by ChatGPT and Gemini was reported as 77.8. The researchers stated that these contents had a high level of readability and were considered understandable for users (26). Similarly, in a study conducted by Boztaş Demir and Görgülü (27), the Turkish responses generated by ChatGPT and Gemini to patient questions in the field of orthodontics were reported to be generally adequate in terms of accuracy, comprehensiveness, and readability. Erdat et al. (25) also reported that ChatGPT’s responses to questions related to colorectal cancer demonstrated moderate readability and good quality characteristics. Overall, the findings of the present study are consistent with this growing body of evidence indicating that LLM-generated health information is generally understandable and of acceptable educational quality. Nevertheless, direct comparison across studies should be interpreted cautiously as the evaluated clinical topics, response languages, and readability formulas differ considerably.

When the literature specific to pediatric dentistry is considered, an interesting pattern emerges. Although direct numerical comparison is limited by differences in language and readability indices, Kocaoğlu et al. (15) and Aydın Varol et al. (16) similarly reported that ChatGPT-generated responses addressing procedural or guideline-based pediatric dental topics in English generally required university-level reading ability. By contrast, the parental frequently asked questions evaluated in the present study demonstrated moderate-to-easy readability. This discrepancy is likely attributable to the inherently greater linguistic complexity of procedure-oriented subjects, such as pediatric dental sedation and pulpotomy, compared with preventive oral health information intended for parents, rather than reflecting a true language-related difference. These observations further emphasize that the readability of LLM-generated responses depends not only on the characteristics of the language model itself, but also on the complexity and intended audience of the clinical topic being addressed.

One of the notable findings of the present study was that readability scores varied significantly despite the use of identical prompts. Significant differences were observed both between repeated assessments performed on the same day and between different days, indicating that the linguistic characteristics of ChatGPT-generated responses are not entirely stable over time. This situation may be explained by the probabilistic nature of LLMs. LLMs do not always generate the same output for the same input and may demonstrate variability in characteristics such as sentence structure, word choice, and text length. As readability indices are inherently sensitive to these textual characteristics, fluctuations in readability scores are expected even when the underlying informational content remains largely unchanged. These findings suggest that readability assessments performed at a single time point may not adequately represent the linguistic variability of LLM-generated health information. Importantly, the temporal fluctuations observed in the present study provide support for concerns regarding potential model drift and response variability which have previously been discussed in the pediatric dentistry literature. Kocaoğlu et al. (15), for example, identified their cross-sectional design as an important limitation and recommended repeated longitudinal evaluations in order to determine whether chatbot performance changes over time. Similarly, Aydın Varol et al. (16) acknowledged that the temporal consistency of individual chatbots could not be assessed within their study design, while Uçar Gündoğar and Sarıoğlu (17) limited repeated evaluations to a single day, precluding an assessment of between-day variability. By employing a seven-day, three-times-daily repeated-measures protocol, the present study directly addresses this methodological gap and demonstrated that although the overall informational quality of ChatGPT responses remains remarkably stable, their linguistic complexity may fluctuate across repeated interactions. These findings indicate that temporal reproducibility constitutes an independent dimension of LLM performance which cannot be adequately captured by conventional single-timepoint evaluations.

When the quality evaluation was examined, ChatGPT responses demonstrated a consistent performance with high GQS scores across all time periods, and no significant differences were observed between repetitions. Similarly, previous studies have reported that the responses of LLMs in the healthcare field are generally of moderate-to-high quality (28, 29). In a systematic review published in 2024, Li et al. (29) reported that ChatGPT mostly demonstrated “passing” or ‘”moderate” performance across different healthcare applications, although it was not yet considered fully reliable for clinical use. In another systematic review, ChatGPT was reported to have a broad range of applications in healthcare education, research, and clinical practice, offering significant advantages in areas such as improving writing quality, data analysis, and patient education (28). That same study also emphasized that despite high efficiency and explanatory capacity in content generation, risks such as inaccuracies, bias, and “hallucinations” remain important concerns. The phenomenon of “hallucinations”,  defined as the ability of LLMs to generate different or inaccurate responses to the same input, has been highlighted as a potential risk for users, particularly in the healthcare field (28, 29). Therefore, despite the high-quality scores observed, such systems should not be regarded as definitive standalone sources of information. Recent discussions regarding the design philosophy of LLMs further reinforce these concerns. Contemporary LLMs are generally optimized to generate a response to user prompts rather than explicitly indicating uncertainty or declining to answer. Consequently, inaccurate or unsupported information may occasionally be presented in a fluent and confident manner, making it difficult for users to distinguish between reliable and erroneous content. In the context of pediatric dentistry, where parents may rely on online information to make decisions regarding their children’s oral health, this behavior highlights the importance of professional verification and reinforces that LLM-generated information should complement, but not replace, clinical judgment.

The stability of GQS scores observed in the present study parallels findings reported in recent pediatric dentistry investigations. Both Karamüftüoğlu et al. (14) and Aydın Varol et al. (16) similarly observed no significant differences in GQS values, although their comparisons were performed between different chatbot platforms rather than across time. Considered together with the present findings, these studies suggest that GQS may represent a relatively robust measure of the overall educational quality of LLM-generated content, remaining comparatively stable regardless of whether the source of variation is model selection or repeated prompting over time. Furthermore, Kocaoğlu et al. (15) demonstrated no significant association between DISCERN quality scores and FKGL readability scores across multiple chatbots, supporting the concept that readability and informational quality constitute distinct dimensions of LLM-generated health information. The present study further strengthens this interpretation by demonstrating that readability may fluctuate substantially over time while overall quality remains stable, indicating that linguistic accessibility and informational quality should be evaluated independently rather than being considered interchangeable indicators of content performance.

Nevertheless, consistently high GQS scores should not be interpreted as evidence of complete factual reliability. Recent pediatric dentistry studies have demonstrated that even LLM-generated responses rated highly for quality may contain fabricated or inaccurate references. Uçar Gündoğar and Sarıoğlu (17) reported that although DeepSeek R1 demonstrated greater overall consistency and accuracy than ChatGPT-4o when generating information related to molar-incisor hypomineralization, both models exhibited substantial rates of fabricated citations. Similarly, Sağlam et al. (13) found that the accuracy of cited references was consistently lower than the accuracy of the explanatory content itself across all evaluated LLMs. Taken together with the present findings, these observations indicate that fluent language, stable quality scores, and coherent explanations should not be interpreted as guarantees of source reliability. Consequently, LLM-generated health information, particularly when intended for patient education, should continue to undergo professional verification before being incorporated into clinical communication or educational resources.

Upon examination of the consistency analyses evaluated in the present study, it was seen that the responses generated by ChatGPT generally exhibited a satisfactory level of consistency, as evidenced by the high ICC values obtained. In particular, the high ICC values observed in the analyses evaluating all repetitions collectively suggest that the model is generally capable of producing stable and predictable responses to similar prompts. However, the presence of significant differences in readability scores both within the same day and across different days, despite the absence of significant differences in GQS scores over time, suggests that LLMs may exhibit variability in linguistic complexity while maintaining a more stable level of overall information quality. The findings, when considered as a whole, suggest that the evaluation of health-related content generated by LLMs should consider not only a single response, but also temporal variability and response consistency over time.

From a clinical perspective, when the findings obtained from this study are evaluated collectively, LLMs appear to be potential tools which may support rapid access to information for parents in the field of pediatric dentistry. However, the results also demonstrate that these systems should not be considered completely reliable and standardized sources of information. In particular, the fact that readability levels may not be equally suitable for all user groups and that responses may exhibit a certain degree of variability over time suggests that LLMs should be regarded as supportive tools in clinical practice. Although the Cetinkaya-Uzun readability scores corresponded to the instructional reading level (approximately the 8^th-9^th grade), this level may still exceed the reading and health literacy skills of a substantial proportion of caregivers. Consequently, despite the consistently high quality of the information provided, some parents may experience difficulty in fully understanding or applying the generated recommendations. Furthermore, the possibility that semi-technical or clinical expressions may be misinterpreted by individuals with low health literacy should also be taken into consideration. These findings highlight the importance of optimizing the readability of LLM-generated patient education materials in addition to ensuring their informational quality particularly as such tools are intended for use by diverse populations with varying levels of health literacy. Therefore, it is recommended that health-related information provided by LLMs should, when necessary, be supported by expert evaluation and interpreted with caution.

From a methodological perspective, the present study possesses several important strengths. Unlike previous investigations which primarily adopted cross-sectional or single-timepoint designs, the present study incorporated a longitudinal repeated-measures protocol consisting of three independent assessments per day over seven consecutive days. This design enabled simultaneous evaluations of readability, information quality, and temporal reproducibility while minimizing the likelihood that the findings reflected a single stochastic response. Furthermore, the combined use of two validated Turkish readability formulas, the GQS, and intraclass correlation analyses provided a multidimensional assessment framework, allowing for the complementary evaluation of linguistic accessibility, educational quality, and response stability. Such an approach offers a more comprehensive characterization of LLM-generated patient education materials in comparison to studies relying on a single evaluation metric.

Nevertheless, several limitations should be considered when interpreting the findings. First, the present study evaluated only ChatGPT; therefore, the results cannot be generalized to other LLMs, which may demonstrate different linguistic characteristics, quality profiles, or temporal stability. This deliberate methodological choice allowed for a rigorous repeated-measures design focused on the temporal reproducibility of a single model but precluded direct comparisons between competing LLMs. Future studies incorporating multiple LLMs within longitudinal repeated-measures protocols would help determine whether the temporal variability observed in the present study is model-specific or represents a more general characteristic of contemporary LLMs. Second, the evaluation period was limited to seven consecutive days. Given that commercial LLMs undergo continuous algorithmic refinement and periodic updates, longer follow-up periods may reveal additional patterns of temporal variation which could not be captured within the present study. Finally, although readability, information quality, and temporal consistency were comprehensively assessed, this study did not evaluate the factual accuracy, reference validity, or clinical correctness of the generated responses. These dimensions remain essential components of the performance of LLMs and should be incorporated into future investigations in order to provide a more holistic assessment of LLM-assisted patient education.

Conclusion

This study demonstrated that the Turkish responses generated by ChatGPT to frequently asked questions in the field of pediatric dentistry generally exhibited moderate readability, high quality, and good consistency. However, variability in readability scores was observed both within the same day and across different days. The findings obtained indicate that LLMs may be used as supportive tools in patient and parent information processes; however, as the generated content may vary in terms of readability and consistency, these systems should not be regarded as completely reliable sources of information in clinical settings without human supervision and expert evaluation.