Handling peptide with noncanonical aminoacids

Rosetta can indeed handle non-canonical amino acids. The trick is to create a "params" file which describes the residue in a format that Rosetta can use.

There's several different approaches you can use, depending on how extensively you want to model the non-canonical amino acid. I'd recommend looking at the demos in rosetta_demos/protocol_capture/2010/using_ncaas_protein_peptide_interface_design and rosetta_demos/public/design_with_ncaa, and reading some of the papers which have come out of Richard Bonneau's lab at NYU. The trickiest bit is setting up the rotamer library for use. You may be able to skip this by adding the line "ROTAMER_AA PRO" to the params file, instead of the NCAA_ROTLIB lines, to reuse the proline rotamer set (if that is something that would work for you). Be cautioned that this setting hasn't been heavily tested.