Moors made the long trip up to Gateshead and played against ten men for most of the game following Rees Greenwood’s early dismissal.
Yates’ side took the lead twice - the second time in the 87th minute - but Wes York’s deflected goal at the death denied Solihull the three points.
The late setback means the Moors remain fourth bottom of the table with just a point now separating themselves and Barrow.
“I’m fuming, absolutely fuming,” Yates told the club’s audioboom. “We could have won that game 15-2 and we should never have conceded two goals.
“We’ve had chance after chance and I’m not having it. I’m gutted because so much of our play was so good. To concede in the manner we did, especially the last one, that’s why we are where we are.
“We were playing ten men. Don’t get me wrong, they’re a good team but they’ve not hurt us all game. It should have been double figures.
“We’ve not won the game when it would have taken us out of the bottom four. The players need to take a long, hard look at themselves for the number of chances they missed and the two goals they conceded.”
For more news, reaction and action from across the National League, like us on Facebook and follow us on Twitter and Instagram.