Adaptive crawler for external hyperlinks search and acquisition

We consider the web-crawler designed to find information about outgoing hyperlinks from a set of monothematic web-sites. Adaptive behavior of the crawler is formulated in terms of the multi-armed bandit problem. The experiments show that the choice of the adaptive algorithm of rational behavior depends on the subject of the considered set of web-sites.