Sounds like it could be a dry solder joint somewhere (either in the amp or speakers)…
Tracing it’s not easy, but if you’re patient and careful you can do it (look for a solder joint that has a dull matt appearance – a good solder joint should be shiny). Be very careful with the inside of the amp – make sure you turn it off and let it sit for a bit before opening it, and be careful what you touch… If you’re in doubt, get a professional to do it for you (because amps can carry very dangerous charge levels especially in their power supplies).
Oh, and swap the speaker cabs round before doing this to check if it moves with the speaker (in which case look for a dry joint in the speaker box somewhere)…