Using the following answer from n1k31t4 in: https://datascience.stackexchange.com/questions/37717/imputation-missing-values-other-than-using-mean-median-in-python
I was able to solve my problem.
df[col]=df.groupby(['X', 'Y'])[col].transform(lambda x: x.median() if (np.abs(x)>3).any() else x)
solved How can I impute values to outlier cells based on groups? [closed]