Joe Widdowson, Jordan Maguire-Drew and Marvin Ekpiteta all hit the upright for the O's who sprung into life after the break.
They were looking to complete an incredible double after winning the National League title last month.
Danny Rowe proved to be the difference in the end but Edinburgh won't let it get to his players.
“When you arrive here you are focused on solely one thing, to make sure you leave here winners," he said.
“It was a slow start but we certainly deserved a little bit more than we got for our second-half performance, I thought we were very good.
“But I’m proud of the football club, I’m proud of my players, they’ve been incredible all season and the fans again today - the numbers that we’ve been backed by is quite incredible.
“We accomplished the goal to get back into the Football League, obviously this was the next to try and complete the double and it hasn’t worked that way.
"The players are hurting but they have to take a lot of credit for the season they’ve had.”