POI assumes that inner VariableIndex match outer VariableIndex

[joaquimg]

I still don't know where the error is.

Here is what I know:

1. The seemingly inoffensive(we know they are never inoffensive) update in POI: Update POI DiffOpt.jl#316
   Led to a fail in MOI tests in DiffOpt:

test_conic_linear_VectorOfVariables_2: Error During Test at /home/runner/.julia/packages/MathOptInterface/zq9bo/src/Test/Test.jl:264
  Got exception outside of a @test
  KeyError: key MOI.VariableIndex(4) not found
  Stacktrace:
    [1] getindex(c::MathOptInterface.Utilities.CleverDicts.CleverDict{MathOptInterface.VariableIndex, MathOptInterface.VariableIndex, typeof(MathOptInterface.Utilities.CleverDicts.key_to_index), typeof(MathOptInterface.Utilities.CleverDicts.index_to_key)}, key::MathOptInterface.VariableIndex)
      @ MathOptInterface.Utilities.CleverDicts ~/.julia/packages/MathOptInterface/zq9bo/src/Utilities/CleverDicts.jl:164
    [2] getindex
      @ ~/.julia/packages/MathOptInterface/zq9bo/src/Utilities/copy/index_map.jl:64 [inlined]
    [3] _extract_row_data(dest::HiGHS.Optimizer, src::MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}, mapping::MathOptInterface.Utilities.IndexMap, rowlower::Vector{Float64}, rowupper::Vector{Float64}, I::Vector{Int32}, J::Vector{Int32}, V::Vector{Float64}, ::Type{MathOptInterface.EqualTo{Float64}})
      @ HiGHS ~/.julia/packages/HiGHS/umjNf/src/MOI_wrapper.jl:2997
    [4] copy_to(dest::HiGHS.Optimizer, src::MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}})
      @ HiGHS ~/.julia/packages/HiGHS/umjNf/src/MOI_wrapper.jl:3065

I reduced the test to:

using Test
import MathOptInterface as MOI
import DiffOpt
import HiGHS

model = DiffOpt.diff_optimizer(HiGHS.Optimizer)

MOI.set(model, MOI.Silent(), true)

T = Float64

x = MOI.add_variable(model)
@test MOI.get(model, MOI.NumberOfVariables()) == 1

ys, vc = MOI.add_constrained_variables(model, MOI.Nonpositives(1))
y = ys[1]

@test MOI.get(model, MOI.NumberOfVariables()) == 2
z, s = MOI.add_variables(model, 2)
@test MOI.get(model, MOI.NumberOfVariables()) == 4
MOI.set(
    model,
    MOI.ObjectiveFunction{MOI.ScalarAffineFunction{T}}(),
    MOI.ScalarAffineFunction(
        MOI.ScalarAffineTerm.(T[3, 2, -4], [x, y, z]),
        T(0),
    ),
)
MOI.set(model, MOI.ObjectiveSense(), MOI.MIN_SENSE)
c = MOI.add_constraint(
    model,
    MOI.VectorAffineFunction(
        MOI.VectorAffineTerm.(
            [1, 1, 2, 3, 3],
            MOI.ScalarAffineTerm.(T[1, -1, 1, 1, 1], [x, s, y, x, z]),
        ),
        T[4, 3, -12],
    ),
    MOI.Zeros(3),
)

vz = MOI.add_constraint(model, [z], MOI.Nonnegatives(1))

vov = MOI.VectorOfVariables([s])

vs = MOI.add_constraint(model, vov, MOI.Zeros(1))

@test MOI.get(model, MOI.TerminationStatus()) == MOI.OPTIMIZE_NOT_CALLED

MOI.optimize!(model)

And the same error is trigged.

Inpective the innermost CachingOptimizer we get:

julia> model.optimizer.optimizer.optimizer.model
MOIU.CachingOptimizer
├ state: EMPTY_OPTIMIZER
├ mode: AUTOMATIC
├ model_cache: MOIU.UniversalFallback{MOIU.Model{Float64}}
│ ├ ObjectiveSense: MIN_SENSE
│ ├ ObjectiveFunctionType: MOI.ScalarAffineFunction{Float64}
│ ├ NumberOfVariables: 3
│ └ NumberOfConstraints: 5
│   ├ MOI.ScalarAffineFunction{Float64} in MOI.EqualTo{Float64}: 3
│   ├ MOI.VariableIndex in MOI.GreaterThan{Float64}: 1
│   └ MOI.VariableIndex in MOI.LessThan{Float64}: 1
└ optimizer: A HiGHS model with 0 columns and 0 rows.

Where there are only 3 variables:

julia> model.optimizer.optimizer.optimizer.model.model_cache.model.variables
MathOptInterface.Utilities.VariablesContainer{Float64}(UInt16[0x0000, 0x0004, 0x0002], [-Inf, -Inf, 0.0], [Inf, 0.0, Inf])

but the first of the affine constraints contains a 4th variable!

julia> model.optimizer.optimizer.optimizer.model.model_cache.model.constraints.moi_scalaraffinefunction.moi_equalto.constraints
MathOptInterface.Utilities.CleverDicts.CleverDict{MathOptInterface.ConstraintIndex{MathOptInterface.ScalarAffineFunction{Float64}, MathOptInterface.EqualTo{Float64}}, Tuple{MathOptInterface.ScalarAffineFunction{Float64}, MathOptInterface.EqualTo{Float64}}, typeof(MathOptInterface.Utilities.CleverDicts.key_to_index), typeof(MathOptInterface.Utilities.CleverDicts.index_to_key)} with 3 entries:
  ConstraintIndex{ScalarAffineFunction{Float64}, EqualTo{Float64}}(1) => (0.0 + 1.0 MOI.VariableIndex(1) - 1.0 MOI.VariableIndex(4), EqualTo{Float64}(-4.0))
  ConstraintIndex{ScalarAffineFunction{Float64}, EqualTo{Float64}}(2) => (0.0 + 1.0 MOI.VariableIndex(2), EqualTo{Float64}(-3.0))
  ConstraintIndex{ScalarAffineFunction{Float64}, EqualTo{Float64}}(3) => (0.0 + 1.0 MOI.VariableIndex(1) + 1.0 MOI.VariableIndex(3), EqualTo{Float64}(12.0))

[joaquimg]

simplified to:

import MathOptInterface as MOI
import DiffOpt
import HiGHS

model = DiffOpt.diff_optimizer(HiGHS.Optimizer)

z, s = MOI.add_variables(model, 2)

c = MOI.add_constraint(
    model,
    MOI.VectorAffineFunction(
        MOI.VectorAffineTerm.(
            [1, 2],
            MOI.ScalarAffineTerm.([-1., 1.], [ s, z]),
        ),
        [4.,-12.],
    ),
    MOI.Zeros(2),
)

vov = MOI.VectorOfVariables([s])

vs = MOI.add_constraint(model, vov, MOI.Zeros(1))

MOI.optimize!(model)

which throws:

julia> MOI.optimize!(model)
ERROR: KeyError: key MOI.VariableIndex(2) not found
Stacktrace:
  [1] getindex(c::MathOptInterface.Utilities.CleverDicts.CleverDict{…}, key::MathOptInterface.VariableIndex)
    @ MathOptInterface.Utilities.CleverDicts C:\JG\Julia\dev\MathOptInterface\src\Utilities\CleverDicts.jl:164
  [2] getindex
    @ C:\JG\Julia\dev\MathOptInterface\src\Utilities\copy\index_map.jl:64 [inlined]
  [3] _extract_row_data(dest::HiGHS.Optimizer, src::MathOptInterface.Utilities.UniversalFallback{…}, mapping::MathOptInterface.Utilities.IndexMap, rowlower::Vector{…}, rowupper::Vector{…}, I::Vector{…}, J::Vector{…}, V::Vector{…}, ::Type{…})
    @ HiGHS C:\JG\Julia\packages\HiGHS\umjNf\src\MOI_wrapper.jl:2997
  [4] copy_to(dest::HiGHS.Optimizer, src::MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}})
    @ HiGHS C:\JG\Julia\packages\HiGHS\umjNf\src\MOI_wrapper.jl:3065
  [5] optimize!
    @ C:\JG\Julia\dev\MathOptInterface\src\MathOptInterface.jl:121 [inlined]
  [6] optimize!(m::MathOptInterface.Utilities.CachingOptimizer{HiGHS.Optimizer, MathOptInterface.Utilities.UniversalFallback{MathOptInterface.Utilities.Model{Float64}}})
    @ MathOptInterface.Utilities C:\JG\Julia\dev\MathOptInterface\src\Utilities\cachingoptimizer.jl:370
  [7] optimize!
    @ C:\JG\Julia\dev\MathOptInterface\src\Bridges\bridge_optimizer.jl:367 [inlined]
  [8] optimize!(model::ParametricOptInterface.Optimizer{Float64, MathOptInterface.Bridges.LazyBridgeOptimizer{MathOptInterface.Utilities.CachingOptimizer{…}}})
    @ ParametricOptInterface C:\JG\Julia\packages\ParametricOptInterface\JYJKg\src\MOI_wrapper.jl:1967
  [9] optimize!
    @ C:\JG\Julia\dev\MathOptInterface\src\MathOptInterface.jl:122 [inlined]
 [10] optimize!(m::MathOptInterface.Utilities.CachingOptimizer{ParametricOptInterface.Optimizer{…}, MathOptInterface.Utilities.UniversalFallback{…}})
    @ MathOptInterface.Utilities C:\JG\Julia\dev\MathOptInterface\src\Utilities\cachingoptimizer.jl:370
 [11] optimize!(model::DiffOpt.Optimizer{MathOptInterface.Utilities.CachingOptimizer{ParametricOptInterface.Optimizer{…}, MathOptInterface.Utilities.UniversalFallback{…}}})
    @ DiffOpt C:\JG\Julia\dev\DiffOpt\src\moi_wrapper.jl:518

[joaquimg]

It can be made even simpler (but maybe too much?)

import MathOptInterface as MOI
import DiffOpt
import HiGHS

model = DiffOpt.diff_optimizer(HiGHS.Optimizer)

s = MOI.add_variable(model)

c = MOI.add_constraint(
    model,
    MOI.VectorAffineFunction(
        MOI.VectorAffineTerm.(
            [1],
            MOI.ScalarAffineTerm.([-1.], [ s]),
        ),
        [4.],
    ),
    MOI.Zeros(1),
)

vov = MOI.VectorOfVariables([s])

vs = MOI.add_constraint(model, vov, MOI.Zeros(1))

MOI.optimize!(model)

It errors the same way:

ERROR: KeyError: key MOI.VariableIndex(1) not found

[odow]

I'll take a look, but my starting assumption is that this is a bug in POI.

[joaquimg]

It is probably in POI, I just bissected to:
#167
before this commit (https://github.com/jump-dev/ParametricOptInterface.jl/tree/af1b2379af14ae9d88e25993e8f54c970eecb1bb) it works

I still find it weird how its gets to CachingOptimizer in a inconsistent state

[odow]

julia> import MathOptInterface as MOI

julia> import ParametricOptInterface as POI

julia> import HiGHS

julia> inner = MOI.instantiate(HiGHS.Optimizer; with_bridge_type = Float64)
MOIB.LazyBridgeOptimizer{HiGHS.Optimizer}
├ Variable bridges: none
├ Constraint bridges: none
├ Objective bridges: none
└ model: A HiGHS model with 0 columns and 0 rows.

julia> model = POI.Optimizer(inner)
ParametricOptInterface.Optimizer{Float64, MOIB.LazyBridgeOptimizer{HiGHS.Optimizer}}
├ ObjectiveSense: FEASIBILITY_SENSE
├ ObjectiveFunctionType: MOI.ScalarAffineFunction{Float64}
├ NumberOfVariables: 0
└ NumberOfConstraints: 0

julia> x, _ = MOI.add_constrained_variables(model, MOI.Zeros(1))
(MathOptInterface.VariableIndex[MOI.VariableIndex(1)], MathOptInterface.ConstraintIndex{MathOptInterface.VectorOfVariables, MathOptInterface.Zeros}(-1))

julia> f = MOI.Utilities.vectorize([-1.0 * x[1] + 4.0])
┌                              ┐
│4.0 - 1.0 MOI.VariableIndex(1)│
└                              ┘

julia> MOI.add_constraint(model, f, MOI.Zeros(1))
ERROR: The index MOI.VariableIndex(1) is invalid. Note that an index becomes invalid after it has been deleted.
Stacktrace:
  [1] _info
    @ ~/git/jump-dev/HiGHS/src/MOI_wrapper.jl:795 [inlined]
  [2] column
    @ ~/git/jump-dev/HiGHS/src/MOI_wrapper.jl:805 [inlined]
  [3] _indices_and_coefficients(indices::Vector{…}, coefficients::Vector{…}, model::HiGHS.Optimizer, f::MathOptInterface.ScalarAffineFunction{…})
    @ HiGHS ~/git/jump-dev/HiGHS/src/MOI_wrapper.jl:1743
  [4] _indices_and_coefficients(model::HiGHS.Optimizer, f::MathOptInterface.ScalarAffineFunction{Float64})
    @ HiGHS ~/git/jump-dev/HiGHS/src/MOI_wrapper.jl:1757
  [5] add_constraint(model::HiGHS.Optimizer, f::MathOptInterface.ScalarAffineFunction{…}, s::MathOptInterface.EqualTo{…})
    @ HiGHS ~/git/jump-dev/HiGHS/src/MOI_wrapper.jl:1776
  [6] add_constraint
    @ ~/git/jump-dev/MathOptInterface/src/Bridges/bridge_optimizer.jl:2082 [inlined]
  [7] #normalize_and_add_constraint#51
    @ ~/git/jump-dev/MathOptInterface/src/Utilities/constraints.jl:25 [inlined]
  [8] normalize_and_add_constraint
    @ ~/git/jump-dev/MathOptInterface/src/Utilities/constraints.jl:19 [inlined]
  [9] bridge_constraint(::Type{…}, model::MathOptInterface.Bridges.LazyBridgeOptimizer{…}, f::MathOptInterface.VectorAffineFunction{…}, set::MathOptInterface.Zeros)
    @ MathOptInterface.Bridges.Constraint ~/git/jump-dev/MathOptInterface/src/Bridges/Constraint/bridges/ScalarizeBridge.jl:46
 [10] add_bridged_constraint(b::MathOptInterface.Bridges.LazyBridgeOptimizer{…}, BridgeType::Type, f::MathOptInterface.VectorAffineFunction{…}, s::MathOptInterface.Zeros)
    @ MathOptInterface.Bridges ~/git/jump-dev/MathOptInterface/src/Bridges/bridge_optimizer.jl:1950
 [11] add_constraint(b::MathOptInterface.Bridges.LazyBridgeOptimizer{…}, f::MathOptInterface.VectorAffineFunction{…}, s::MathOptInterface.Zeros)
    @ MathOptInterface.Bridges ~/git/jump-dev/MathOptInterface/src/Bridges/bridge_optimizer.jl:2080
 [12] _add_constraint_direct_and_cache_map!
    @ ~/git/jump-dev/ParametricOptInterface/src/MOI_wrapper.jl:796 [inlined]
 [13] add_constraint(model::ParametricOptInterface.Optimizer{…}, f::MathOptInterface.VectorAffineFunction{…}, set::MathOptInterface.Zeros)
    @ ParametricOptInterface ~/git/jump-dev/ParametricOptInterface/src/MOI_wrapper.jl:915
 [14] top-level scope
    @ REPL[48]:1
Some type information was truncated. Use `show(err)` to see complete types.

[odow]

Hmm. I guess this is because of the ZerosBridge. Could be an issue in MOI or POI.

[odow]

All of POI is pretty severely broken. In this case it returned an error because the inner variable did not exist, but there is potential for other errors to have been introduced where we silently returned the wrong answer.

If you do

import MathOptInterface as MOI
import ParametricOptInterface as POI
import HiGHS
inner = MOI.instantiate(HiGHS.Optimizer; with_bridge_type = Float64)
model = POI.Optimizer(inner)
x, _ = MOI.add_constrained_variables(model, MOI.Zeros(1))

It assumes that the x returned to the user is the same x as was added to the optimiser. See this function:

ParametricOptInterface.jl/src/MOI_wrapper.jl

Lines 795 to 798 in 84af4e9

	function _add_constraint_direct_and_cache_map!(model::Optimizer, f, set)
	ci = MOI.add_constraint(model.optimizer, f, set)
	_add_to_constraint_map!(model, ci)
	return ci

this isn't safe to do if there are any x => y in model.variables with x != y.

[joaquimg]

I am a bit confused because you are saying the variable indexes are bad, but you posted a reference to constraint indexes

[odow]

The f that gets passed in is in user-space. But model.optimizer requires inner-space.

It needs to be something like:

function _add_constraint_direct_and_cache_map!(model::Optimizer, f, set) 
     g = MOI.Utilities.map_indices(model.variables, f)
     ci = MOI.add_constraint(model.optimizer, g, set) 

But not in a few other places as well.

[joaquimg]

Ok, now I got what you meant

[joaquimg]

Got it.
Yes that is a possible fix.
But we can do that without all the mapping.
I will take a look.

[joaquimg]

Ok, the assumption was:
Whatever was below POI would return variable indices in order.

It seems that is mostly the case. But not always the case, for instance, with LazyBridge (negative indexing?)