IMO the firm has been in conflict the whole time. Yes I think you should be pissed off. Yes I think they were unethical before and they're unethical now. My guess is regardless this is a well know law firm and the local bar would be uninterested in any complaint even though IMO one would be well founded.
However logically there is no reason they cannot represent you if you join another agency. Assuming they think it was ok to represent both agency and client before, the only real conflict from their 'POV' is in the decision to choose another agency. If you do that on your own or consult another lawyer solely on that decision once you have made the switch so long as there are no allegations remaining against the old agency that they would have to deal with then I don't see why they can't ethically represent you vs the SD. Unless of course they are the ones interposing business over their ethical duty to their client by trying to stop you from switching.
However any time an attorney has to make any decision or consultation affecting another client he has to step aside for that decision. I don't understand how he has been consulting you from the beginning without crossing the line whenever he had to advise you about the work the agency was doing. They must be doing this with a lot of their clients. Pisses me off really. In that respect the firm would actually remove the conflict if you signed with another agency, they would then be acting ethically, but they don't want that apparently.