The deformation of two-dimensional vortex patches in the vicinity of fluid boundaries is investigated. The presence of a boundary causes an initially circular patch of uniform vorticity to deform. Sufficiently far away from the boundary, the deformed shape is well approximated by an ellipse. This leading order elliptical deformation is investigated via the elliptic moment model of Melander, Zabusky, and Styczek [J. Fluid Mech. 167, 95 (1986)10.1017/S0022112086002744]. When the boundary is straight, the centre of the elliptic patch remains at a constant distance from the boundary, and the motion is integrable. Furthermore, since the straining flow acting on the patch is constant in time, the problem is that of an elliptic vortex patch in constant strain, which was analysed by Kida [J. Phys. Soc. Jpn. 50, 3517 (1981)10.1143/JPSJ.50.3517]. For more complicated boundary shapes, such as a square corner, the motion is no longer integrable. Instead, there is an adiabatic invariant for the motion. This adiabatic invariant arises due to the separation in times scales between the relatively rapid time scale associated with the rotation of the patch and the slower time scale associated with the self-advection of the patch along the boundary. The interaction of a vortex patch with a circular island is also considered. Without a background flow, the conservation of angular impulse implies that the motion is again integrable. The addition of an irrotational flow past the island can drive the patch towards the boundary, leading to the possibility of large deformations and breakup.