I noticed that occasionally I get a negative fa_rep values when using ddGFilter. Under which circumstances it can occur? is it a bug/outlier?
As you might be aware, the fa_rep of a single scored structure is always going to be positive. The only way you'll get a negative fa_rep value would be in situations where you're subtracting two scores from each other ... like in the ddGFilter case, where you're subtracting the score of the apo/separated structure from that of the holo/bound structure.
So a negative fa_rep in the ddG value indicates that the fa_rep of the separated structure is *greater* than that of the bound structure. This is a little counter intuitive, as the separated strucutres should have fewer atoms in contact, and less chance of overlap. ... and that would be true if the ddG process was strictly separating the two halves without any other change. However, the ddGFilter has an optional apo repack step, which might allow some sidechains to pack into a conformation which has a higher fa_rep score.
But you'd only get a higher fa_rep score if *other* score terms are getting better. You can compare the score breakdown between the with-apo-repack and without-apo-repack structures to see which score terms might be at issue, but off the cuff I'm guessing it might be something like the fa_dun term. In the bound state you may have a rotamer with a very poor fa_dun score, but which may not be able to repack due to major clashes with the binding partner. But in the apo state those clashes are releaved, and there's an alternate state which is higher in fa_rep, but more than makes up for it by lowering the fa_dun score.
This situation might be especially prevalent in situations where you're taking the bound state from a crystal structure without relaxing it into the Rosetta energy function first. There the bound state may be in a particularly bad internal conformation (but coincidentally decent fa_rep), and the apo repacking can greatly improve the energy, but may have to increase the fa_rep to do so.
As a final note, keep in mind that packing is stochastic. Another possibility is that your apo repacking simply ended in a coincidentally bad conformation with a higher fa_rep score. You might want to repeat the ddG calculations a few times to see if the fa_rep score is consistently negative.