I used to have the same problem before , And this is my solution
df1=df[['X','Y']].apply(sorted,1)
df.groupby([df1.X,df1.Y])['count'].sum().reset_index(name="count")
Out[400]:
X Y count
0 A B 3
1 C D 8
solved Aggregate symmetric pairs pandas