I'm new to Rosetta and I'm not sure if it is feasible for solving my problem. I have an octameric complex consisting of eight short (20 amino acid) helices packed together. The peptide exists in two conformations in the complex - say A and B. This means that the asymmetric unit is made up of an 'AB' dimer. Could anyone tell me if having more than one chain as the asymmetric unit will be a problem with defining the symmetry or docking? When generating the symmetry definition file with make_symmdef_file_denovo.py would I specify the number of subunits as 4 (as I have four asymmetric units) or 8 (since there are 8 subunits)?
As I believe I have two possible symmetry types (c4 and d2) is it reasonable to assume I should just try both and then evaluate which is correct based on the lowest energy structures?
Finally, as I know each subunit is an alpha helix I'm not sure if the docking protocol starting with a canonical alpha helix monomer structure, or fold-and-dock is more suitable. Can I restrict only the backbone residues in order to maintain the helical structure but allow flexibility in the side chains.
The Rosetta symmetry machinery should be able to handle a heterodimer as the asymmetric unit -- although the automated scripts to set up the symmetry might not be put together with that use case in mind. At the worst, you can always approximate the AB dimer as a single chain (e.g. rename/renumber chain B to also be part of chain A). Rosetta shouldn't care about the gap between the two helicies, although it might not optimize the rigid body positions quite as well as if they were treated as separate chains.
If you have 4 asymmetric subunits, you're not going to be C2 symmetry, as that has 2 subunits with a 180 degree symmetry. With 4 subunits, arranged evenly at 90 degrees around a central axis, you'll probably be looking at either C4 or D2 symmetry. The difference is in the orientation of the subunits. A D2 symmetry has an up-down-up-down orientation pattern as you go around the ring, whereas a C4 symmetry has an up-up-up-up orientation. If the same side of the asymmetric unit is facing the same direction, it's C4. If it's alternating sides, it's D2.