it is a bad thing and it probably is an indication that one of the objects you are using is doing something "prohibited" in the audio thread, for instance reading or writing to disk or doing networking.
[soundfiler] reads a file in the audio thread, but once the file content is loaded in a table, the table can be safely accessed without mode switches.
[readsf~] instead is safe in that it reads from disk in a separate thread.
Can you think of any of the objects you are using which may be reading/writing from disk or network? It can help to delete objects or groups of objects one at a time until you find the culprit.