Nonlinear Stability Analysis of a Regular Vortex Pentagon Outside a Circle

A nonlinear stability analysis of the stationary rotation of a system of five identical point vortices lying uniformly on a circle of radius $R_0$ outside a circular domain of radius $R$ is performed. The problem is reduced to the problem of stability of an equilibrium position of a Hamiltonian system with a cyclic variable. The stability of stationary motion is interpreted as Routh stability. Conditions for stability, formal stability and instability are obtained depending on the values of the parameter $q=R^2/R^2_0$.