You are building the point and the polygon – instead of having proper geometry column – which is inefficient.
You haven’t talked about spatial index, so the intersect operation will be slow.
You haven’t specified the complexity of the polygons: if they have many many vertices, an intersect operation will be slow. This can be improved by subdividing them first.
So technically speaking, using indexed and eventually subdivided polygons make such operation blazing fast.
BUT, addresses are typically not perfect. Some may be incomplete, other may be just wrong (ex: someone entered the neighboring town name). Selecting by geometry will give different results than selecting by attribute, so you may want to analyze your data quality before deciding, and to carefully compare the result discrepancies between both methods.
1
solved How do I search for restaurants in a region in a country?