After 46 minutes and 2-0 up, the Heed might have thought they had the three points in the bag but instead their winning goal came in the 94th minute - watch match highlights here.
An own-goal from Chester's Andrew Hall's gave Gateshead the lead before half-time and then Richard Peniket doubled their advantage just after the break.
Ross Hannah got one back for the visitors and then a stunner from Matthew Waters in the 90th minute looked to have gained the Blues a point.
There was to be another dramatic twist, though, as another own-goal, this time from Conor Mitchell, gave Heed the win.
"I suppose that's why people come to watch football because you never know what's going to happen and certainly that game had a lot of twists and turns," said Aspin.
"I thought we played some decent football in terms of we created more chances, had more impetus going forward but obviously after going 2-0 in front and getting into a really good position, once Chester scored, it pushed us back and when you concede at the death you think that's it.
"We got a fortunate winning goal but obviously I'm delighted to take the three points."
Gateshead have gained an extraordinary five own-goals so far this season and Aspin added: "Own-Goals, he's running away at the top of the scorers chart. If he keeps scoring we should be able to sell him for a lot of money!"