Clean up filtering logic (#1183)

* Clean up filtering logic

Should also make compilation faster as we no longer reparse for strict deps

Fixes #1081

* Review feedback
4 files changed