The striker, on loan at Eastbourne Borough, put three goals past Hemel Hempstead at the weekend.
With the Hawks struggling for form and for goals, their manager Lee Bradbury decided to cut short his spell along the south coast.
Rutherford now wants to play a part in turning Havant’s season around.
He tweeted: “Loved my short time. All the best to the club and all the players and everyone involved.”
An Eastbourne statement confirmed the news.
“We thank the Hawks for allowing Alfie to represent The Sports over the past weeks and wish Alfie all the best on his return to Hampshire,” it read.
Havant have a crucial home game with Eastbourne Borough on Tuesday night.