Machine translation by ChatGPT: Monitoring of outcome reproducibility

The paper considers the question of monitoring the reproducibility of the results performed by ChatGPT chatbot over a time interval for solving a mathematical task, generating code, and resolving a visual puzzle. A brief review of the experimental data for monitoring the reproducibility of the results for these three applications is given. The presented data show that the outcomes of ChatGPT when solving the same problem may change over time. At the same time, significant changes may occur in a relatively short period of time which emphasizes the need to monitor and evaluate the behavior of the ChatGPT chatbot. The main goal of the paper is to study the reproducibility of the machine translation outcomes performed by ChatGPT over a given time interval. The experimental data obtained during the monitoring of outcome reproducibility demonstrate some changes in the results including the decline in translation quality of the same text fragments over a time interval. To monitor the outcome reproducibility and evaluate the behavior of ChatGPT, a previously developed method for interval evaluation of machine translation is used.