Usability and accessibility aren’t binary. When you discover a handful of usability issues, some are more important than others. It’s the same with accessibility. Some usability issues are a brick wall for users. They can’t get past. Others simply slow down, confuse or annoy. It’s exactly the same with accessibility issues.
Therefore some usability issues will be more pressing than some accessibility issues and vice versa.