The quality improvement method for detecting attacks on web applications using pre-trained natural language models
O. A. Kovaleva, A. V. Samokhvalov, M. A. Liashkov, S. Yu. Pchelintsev

References
1.	Hacker A. J., “Importance of web application firewall technology for protecting web-based resources”, ICSA Labs an Independent Verizon Business, 2008, 7 https://img2.helpnetsecurity.com/dl/articles/ICSA_Whitepaper.pdf (accessed December 28, 2022)
2.	Sureda Riera T., Bermejo Higuera J. R., Bermejo Higuera J., Martinez Herraiz J. J., Sicilia Montalvo J. A., “Prevention and fighting against web attacks through anomaly detection technology. A systematic review”, Sustainability, 12:12 (2020), 4945
3.	Betarte G., Martinez R., Pardo A., “Web application attacks detection using machine learning techniques”, 17th IEEE International Conference on Machine Learning and Applications (ICMLA) (Orlando, 2018), 1065–1072
4.	Betarte G., Gimenez E., Martinez R., Pardo A., “Improving web application rewalls through anomaly detection”, 17th IEEE International Conference on Machine Learning and Applications (ICMLA) (Orlando, 2018), 2018
5.	Martinez R., Enhancing web application attack detection using machine learning, UdelaR – Area Informatica del Pedeciba, Montevideo, 2019, 82 pp.
6.	Liu Y., Ott M., Goyal N., Du J., Joshi M., Chen D., Levy O., Lewis M., Zettlemoyer L., Stoyanov V., “RoBERTa: A robustly optimized BERT pretraining approach”, ICLR 2020 Conference Blind Submission (Addis Ababa, 2020) https://openreview.net/forum?id=SyxS0T4tvS (accessed January 15, 2023)
7.	Devlin J., Chang M. W., Lee K., Toutanova K., “BERT: Pre-training of deep bidirectional transformers for language understanding”, NAACL-HLT 2019 (Minneapolis, 2019), 4171–4186
8.	Radford A., Wu J., Child R., Luan D., Amodei D., Sutskever I., “Language models are unsupervised multitask learners”, OpenAI Blog, 1:8 (2019), 9
9.	Peters M. E., Neumann M., Iyyer M., Gardner M., Clark C., Lee K., Zettlemoyer L., “Deep contextualized word representations”, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (New Orleans, 2018), 2227–2237
10.	Mikolov T., Chen K., Corrado G., Dean J., Efficient estimation of word representations in vector space, 2013, arXiv: 1301.3781v3 [cs.CL]
11.	Bengio Y., Ducharme R., Vincent P., Janvin C., “A neural probabilistic language model”, Journal of Machine Learning Research, 3 (2003), 1137–1155
12.	Olah C., Deep learning, NLP, and representations, GitHub blog, posted on 2014 July, 7, https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/ (accessed January 15, 2023)
13.	Luong M. T., Socher R., Manning C. D., “Better word representations with recursive neural networks for morphology”, Proceedings of the Seventeenth Conference on Computational Natural Language Learning, 2013, 104–113
14.	Zou W. Y., Socher R., Cer D., Manning C. D., “Bilingual word embeddings for phrase-based machine translation”, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013, 1393–1398
15.	Ethayarajh K., “How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings”, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP (Hong Kong, 2019), 55–65
16.	Kruegel C., Vigna G., “Anomaly detection of web-based attacks”, Proceedings of CCS, 2003, 251–261
17.	Corona I., Ariu D., Giacinto G., “HMM-Web: A framework for the detection of attacks against web applications”, Proceedings of ICC, 2009, 1–6
18.	Torrano-Giménez C., Pérez-Villegas A., Marañón G. Á., “An anomaly-based approach for intrusion detection in web traffic”, Journal of Information Assurance and Security, 5 (2010), 446–454
19.	Yuan G., Li B., Yao Y., Zhang S., “Deep learning enabled subspace spectral ensemble clustering approach for web anomaly detection”, 2017 International Joint Conference on Neural Networks (IJCNN) (Anchorage, AK, USA, 2017), 3896–3903
20.	Yu Y., Yan H., Guan H., Zhou H., “DeepHTTP: Anomalous HTTP Traffic Detection and Malicious Pattern Mining Based on Deep Learning”, IET Information Security, Communications in Computer and Information Science, 1299, Springer, Singapore, 2020
21.	Qin Z. Q., Ma X. K., Wang Y. J., “Attentional payload anomaly detector for web applications”, International Conference on Neural Information Processing, Springer, 2018, 588–599
22.	Vartouni A. M., Teshnehlab M., Kashi S. S., “Leveraging deep neural networks for anomaly-based web application firewall”, IET Information Security, 2019, no. 13, 352–361
23.	Sennrich R., Haddow B., Birch A., “Neural machine translation of rare words with subword units”, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Berlin), 2015, 1715–1725
24.	Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A. N., Kaiser L., Polosukhin I., “Attention is all you need”, Advances in Neural Information Processing Systems, 30, 2017
25.	Scholkopf B., Platt J. C., Shawe-Taylor J., Smola A. J., Williamson R. C., “Estimating the support of a high-dimensional distribution”, Neural Computation, 2001, no. 13, 1443–1471