You can use pandas groupby
method with list comprehension which will do the JOb like Below:
>>> df
X Y
0 Yes 1
1 No 2
2 Yes 3
3 Yes 4
4 No 2
5 No 1
6 Yes 0
7 No 4
8 No 4
9 No 5
>>> {k: v["Y"].tolist() for k,v in df.groupby("X")}
{'No': [2, 2, 1, 4, 4, 5], 'Yes': [1, 3, 4, 0]}
solved Pandas: filter data frame by category