Goals from Inih Effiong and Joe Ward gave new Cards boss Anthony Limbrick the perfect start but Aspin was left unhappy.
Danny Johnson did equalise for Gateshead in the first-half although Ward's strike before half-time proved match winning.
It's early days at the moment, but Aspin knows where his side need to improve, he said: "I thought that the two goals we conceded were very preventable, the first goal in particular was very bad defending on our behalf.
"We got back into the game and I felt we could have gone in level at the break ut we conceded, which should have been prevented.
"Second half, they had one chance which hit the post but we looked the better side and got forward more so I think we should have got something from the game.
"Football usually comes down to that you do in both penalty areas and unfortunately we gave away two bad goals at one end and weren't clinical enough at the other. That's what the game has hinged on.
"We know where we need to improve and the games come thick and fast so we'll be looking back at the game and start preparing for Tuesday night against Guiseley."