The Spireites, relegation from League Two last season, lost their sixth National League match on the spin on Tuesday night.
Going down at Boreham Wood, their fantastic start has totally evaporated and their fourth 1-0 defeat in a row as paints a picture of where things are going wrong for a club who won their first three fixtures.
Allen was never fooled by their explosive start but even he did see a return of zero points from a possible 18 coming.
“It’s a tough one to take,” a very tough one to take,” said the manager.
“The Chesterfield supporters are very loyal and I think they know there’s a lot to be done having been relegated with eight points less than anyone else.
“But I know now for sure that there’s a lot more to be done than perhaps what I first expected.”
Although his side didn’t play badly in Hertfordshire, it’s now four games without a goal.
“It was a really poor first 20 minutes. We didn’t go to our game plan that we put together but after that we played very well,” he told us.
“But the goals haven’t gone in and they need to. There are signs of encouragement but so there should be. We brought some very good players, but there’s a lot of work ahead of us.
“This is a very competitive, athletic and mobile league. You are coming to these places with fit, young and strong players on their way up. You have to match teams before you can get it down and play.”