Seaforth Pavilion, used as a clubhouse by both the Annan Youth Cycling Club and Seaforth Tennis Club, was hit by the blaze on Tuesday night.
The incident is being treated as suspicious by police.
And representatives from both clubs say they are “saddened” and “shocked” by the dangerous act.
Scott Irving from Annan Youth Cycling Club, who is a fireman in Dumfries, said: “I’m saddened to see it. I know this pavilion has been the subject of fireraising in the past and that’s why the council were going to shut it down, but that’s why we got involved and got access to the facility.
“We’ve been using it as our clubhouse for the past three years and prior to that we coached out the back of our cars in a playing field, so this clubhouse has been a real asset to us.”
He added: “The clubs are both in the middle of raising funds for a refurbishment and we’ve been raising this money for a couple of years now.
“Money will now need to be taken from those funds for these repairs which is a shame.”
But, remaining upbeat about the situation, he added: “Even in its current condition it’s an asset though — both the tennis and the cycling clubs use it and the kids involved see it as their own.”
A passer-by discovered the fire and alerted the emergency services and Scott immediately.
Scott said: “A member of the public was walking by and heard crackling and hissing, and then realised that the pavilion was on fire.”
He added: “I made my way down straight away and I arrived here just after the fire engines and I could see clearly then that it was well burnt through.
“But the fire services quick actions stopped the fire from growing and obviously if that man wasn’t walking by then it could’ve been burning for a longer period of time.”
Elaine Lamont from Seaforth Tennis Club cannot believe that people in Annan would want to ruin something so important to the community.
She said: “I guess it’s just a feeling of disappointment to be honest. Both clubs have been working together to build a facility where young people can really benefit through tennis, cycling and volunteering and it’s so disappointing that people would do such a thing.”
She added: “Both clubs have developed hugely over the last few years and for something like this to happen is almost hard to believe — it’s shocking.”
Constable Emma Morgan at Annan said: “We are currently carrying out enquiries into this fireraising and want to hear from anyone who may have been in the vicinity of the tennis pavilion last night.
“In particular, we want to trace a group of youths, not further described. who were seen running from the area at the rear of nearby Annan Academy around this time.
“This was a deliberate act which could have had very serious consequences.”