Clean up filtering logic (#1183) * Clean up filtering logic Should also make compilation faster as we no longer reparse for strict deps Fixes #1081 * Review feedback