Fix diff_optimizer with no bridge

[blegat]

Weirdly enough, it works after the first commit but fails once I remove the outer cache.
I think this is because POI first adds a quadratic constraint and then modifies it. Apparently it's not complaining if you have two layers of caches ?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.01%. Comparing base (7953a55) to head (35f399e).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #309   +/-   ##
=======================================
  Coverage   89.01%   89.01%           
=======================================
  Files          16       16           
  Lines        2057     2057           
=======================================
  Hits         1831     1831           
  Misses        226      226           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.