The attack on the 64-year-old happened at the Whitesands, near the bus stances, at 1.20 pm on Monday.
He was knocked to the ground but was not injured.
Witnesses are being sought and Constable Gary Bil said: “It appears that the victim was attacked by a man who was in a group of four people, three males and a female. Whilst he was not injured in this attack, he was extremely shaken and we would like to hear from anyone who saw this incident.
“There are no descriptions of those in the group or of the attacker at ths stage.”
Anyone with information is urged to call police on 101.