Jay Saunders' side won 4-2 at Whaddon Road in front of over 500 travelling fans.
Two goals in two minutes from Delano Sam-Yorke and Joe Pigott gave Maidstone the lead before Zavon Hines made it three before half-time.
Kevin Dawson pulled one back but Sam-Yorke restored the visitors' lead and was then sent off.
Alex Finney inadvertently turned into his own-net before Cheltenham's goal scorer Dawson also saw red.
Manager Saunders said: "I thought all over the park we were excellent today and even when we went down to 10 men we dug deep.
"That's what you've got to do and to come away from home anywhere and score four goals is great.
"To do it at a League Two club speaks volumes about the performance the boys put in."
Beaten boss Gary Johnson said: "I know the lads don't go out there purposely not to try.
"It was just very, very poor defending, and we couldn't score on our chances.
"Three-nil down at half-time is a mountain to come back, but funnily enough we had enough chances to get it back.
"We didn't have the quality to stick the ball in the net when we needed to."