I Have a Question

If someone has all your bank details and setted it up to do bank transfers and can put money into your bank account could they also if you had money in transfer your money to there account without your permission just a question.