If someone has your bank account number, but not your routing number (BSB number Australia, Sort Code UK), the financial harm they can inflict is limited. For instance, it would be difficult to withdraw money from your account or create new debits without being able to identify which bank your account is at. But if someone has both your bank account and routing number, they can make fraudulent purchases.
Most banks do not supply your bank account and routing number in the file of your transactions, instead they mask it.
While your bank account number isn’t going to give scammers instant access to your finances, they can use it as leverage to obtain further information that they can use to access to your finances (online banking login details/debit card pin numbers/etc).
For example, they may send you a phishing email or text, impersonating your financial institution, asking you to “confirm” your login details. They include your real account number to trick you into trusting the legitimacy of the message. They will try to create a sense of urgency to encourage you to provide the information there and then.