The 25-year-old, who has also appeared against the Pilgrims in Oxford City colours, has put pen-to-paper on a one-year contract to become manager Adam Murray's seventh signing of the summer.
Clifton won the Southern League championship with Brackley in 2012 — and helped the Saints reach the National League North promotion final a year later.
Reflecting on Clifton's arrival, Murray said: "He's a player that brings experience of what it takes to be successful in this league.
"He's someone that is multi-functional and can play comfortably anywhere down the right-hand side. He has a hunger to achieve and really wants to be part of what we are trying to build.
"James will add quality and knowledge to a growing and youthful squad".