|
| 1 | +--- |
| 2 | +tags: |
| 3 | + - OMSCS |
| 4 | + - Algorithms |
| 5 | +--- |
| 6 | +# 07.2 - LP - Linear Programming |
| 7 | +> Linear program is a general technique for solving optimization problems. |
| 8 | +
|
| 9 | +## Introduction |
| 10 | +- Examples |
| 11 | + - Max-Flow |
| 12 | + - Simple Production |
| 13 | +- General formulation / Standard Form |
| 14 | +- Simplex |
| 15 | +- LP Duality |
| 16 | +- Max-SAT approximation |
| 17 | + |
| 18 | +## Max Flow via LP |
| 19 | +- **Input:** directed $G=(V,E)$ with capacities $c_e \gt 0$ for $e \in E$ |
| 20 | +- LP: $m$ variables: $f_e$ for every $e \in E$ |
| 21 | +- Objective Function: $\max \sum_{sv \in E} f_{sv}$ |
| 22 | +- Subject To |
| 23 | + - for every $e \in E:0 \le f_e \le c_e$ |
| 24 | + - for every $v \in (V-\{s,t\})$ |
| 25 | + - $\sum_{wv \in E} f_{wv}=\sum_{vz \in E}f_{vz}$ |
| 26 | + |
| 27 | +## Simple 2D Example |
| 28 | +- Basic Production Problem |
| 29 | +- Company makes A and B |
| 30 | +- How many of each to maximize profit? |
| 31 | + - Each unit of A makes profit $1 |
| 32 | + - Each unit of B makes profit $6 |
| 33 | +- Demand |
| 34 | + - $\le 300$ units of A |
| 35 | + - $\le 200$ units of B |
| 36 | +- Supply |
| 37 | + - $\le 700$ hours |
| 38 | + - A takes 1 hour |
| 39 | + - B takes 3 hours |
| 40 | + |
| 41 | +- Variables |
| 42 | + - $x_1=$ number of units of A to produce per day |
| 43 | + - $x_2=$ number of units of B to produce per day |
| 44 | +- Objective |
| 45 | + - $\max (x_1 + 6x_2)$ |
| 46 | +- Constraints |
| 47 | + - $0 \le x_1 \le 300$ |
| 48 | + - $0 \le x_2 \le 200$ |
| 49 | + - $x_1 + 3x_2 \le 700$ |
| 50 | + |
| 51 | +### Geometric View |
| 52 | + |
| 53 | +Let's ignore the objective function and look at the constraints. We can plot the constraints in 2D space, resulting in a "Geometric View" of the problem. In the graph below, the red region is the "feasible region." |
| 54 | + |
| 55 | +![[Pasted image 20260330205615.png]] |
| 56 | + |
| 57 | +How do we find the optimum? The goal is to maximize $c=x_1+6x_2$, where $c$ a profit projection. We can plot this line for various outcomes to see where it intersects this region. $c=1300$ is the last point at which $x_1+6x_2$ intersects the region of feasibility. It intersects this region at $x_1=100$ and $x_2=200$, resulting in a profit of $1300$. |
| 58 | + |
| 59 | +![[Pasted image 20260330215220.png]] |
| 60 | + |
| 61 | +## Key Issues |
| 62 | +- optimum may be non-integer |
| 63 | +- If we end up with a fractional point, we could try to round it. |
| 64 | +- Linear Programming (LP) $\in P$ |
| 65 | +- Integer Linear Programming (ILP) is NP-Complete |
| 66 | +- Vertex = corner |
| 67 | + - The optimum must lie at a vertex on the region of feasibility. |
| 68 | + - There may exist a continuum of solutions, but the vertices are "at least as good" as any other point in the region of feasibility. |
| 69 | + - If the objective function aligns with one of the edges of the feasibility region, then all of the points on that edge are equally good. |
| 70 | +- The feasible region is convex. It's defined by the intersection of half-spaces. This leads to the optimal solutions lying at vertices of the region. |
| 71 | +- This leads us to an example algorithm. Starting from any vertex on the polygon, we can check its neighbors to see if they result in a better solution. The example above has only 5 vertices, so that's just 5 values for $(x_1,x_2)$ which need to be checked. |
| 72 | + |
| 73 | +## Simple 3D Example |
| 74 | +- Products A, B, and C |
| 75 | +- Profit |
| 76 | + - $1 for A |
| 77 | + - $6 for B |
| 78 | + - $10 for C |
| 79 | +- Demand |
| 80 | + - $\le 300$ for A |
| 81 | + - $\le 200$ for B |
| 82 | + - No limit for C |
| 83 | +- Supply |
| 84 | + - $\le 1000$ hours total |
| 85 | + - A takes 1 hour |
| 86 | + - B takes 3 hours |
| 87 | + - C takes 2 hours |
| 88 | +- Packaging |
| 89 | + - $\le 500$ units per day |
| 90 | + - B takes 1 unit of packaging |
| 91 | + - C takes 3 units of packaging |
| 92 | + |
| 93 | +### LP Formulation |
| 94 | +- $x_1,x_2,x_3=A,B,C$ |
| 95 | +- objective: $\max x_1 + 6x_2 + 10x_3$ |
| 96 | +- Constraints |
| 97 | + - $x_1 \le 300$ |
| 98 | + - $x_2 \le 200$ |
| 99 | + - $x_1+3x_2+2x_3 \le 1000$ |
| 100 | + - $x_2+3x_3 \le 500$ |
| 101 | + - $x_1,x_2,x_3 \ge 0$ |
| 102 | +### Geometric View |
| 103 | +![[Pasted image 20260401135010.png]] |
| 104 | + |
| 105 | +## Standard Form |
| 106 | +- $n$ variables: $X=x_1,x_2,...,x_n$ |
| 107 | +- Objective function: $\max\space c_1x_1+c_2x_2+...+c_nx_n$ |
| 108 | +- s.t. |
| 109 | + |
| 110 | +$$ |
| 111 | +a_{11}x_1+a_{12}x_2+...+a_{1n}x_n \le b_1 |
| 112 | +$$ |
| 113 | +$$\vdots$$ |
| 114 | +$$ |
| 115 | +a_{m1}x_1+a_{m2}x_2+...+a_{mn}x_n \le b_m |
| 116 | +$$ |
| 117 | +- $x_1,...,x_n \ge 0$ |
| 118 | + |
| 119 | +### Linear Algebra View |
| 120 | +- Maximize $c^Tx$ |
| 121 | + - Variables $x=[x_1,...,x_n]$ |
| 122 | + - Objective Function: $c=[c_1,...,c_n]$ |
| 123 | +- Such that: $Ax \le b$ |
| 124 | + - Constraint Matrix: $m \times n$ matrix $A$ |
| 125 | + - Constraints: $b=[b_1, ..., b_m]$ |
| 126 | +- Nonnegativity constraint: $x \ge 0$ |
| 127 | +- $x,c,b$ are columnar. Transpose them in your mind s.t. the equations work. |
| 128 | +- $x=[0,...,0]$ is a known feasible point. |
| 129 | + |
| 130 | +### Convert |
| 131 | +Suppose we want to minimize an objective function, rather than maximize it. |
| 132 | + |
| 133 | +- $\min \space c^Tx \iff \max \space -c^Tx$ |
| 134 | +- What do we do with $\ge$ constraints? |
| 135 | + - Example: $a_{1}x_1+a_{2}x_2+...+a_{n}x_n \ge b$ |
| 136 | + - This becomes: $-a_{1}x_1-a_{2}x_2-...-a_{n}x_n \le -b$ |
| 137 | +- What do we do with equality constraints? |
| 138 | + - Example: $a_{1}x_1+a_{2}x_2+...+a_{n}x_n = b$ |
| 139 | + - This becomes |
| 140 | + - $a_{1}x_1+a_{2}x_2+...+a_{n}x_n \le b$ |
| 141 | + - and $a_{1}x_1+a_{2}x_2+...+a_{n}x_n \ge b$ |
| 142 | + - Which becomes: |
| 143 | + - $-a_{1}x_1-a_{2}x_2-...-a_{n}x_n \le b$ |
| 144 | +- What about strict inequalities? |
| 145 | + - Example: $x < 100$ |
| 146 | + - This is an ill-defined constraint. |
| 147 | + - Strict inequalities are not allowed in linear programming. |
| 148 | +- Unconstrained variable $x$? |
| 149 | + - We can add $x^+$ and $x^-$ |
| 150 | + - $x^+ \ge 0$ |
| 151 | + - $x^- \ge 0$ |
| 152 | + - Replace $x$: $x=x^+-x^-$ |
| 153 | + |
| 154 | +### Geometric View |
| 155 | +- $n$ variables $\rightarrow$ $n$ dimensions |
| 156 | +- $n+m$ constraints |
| 157 | +- feasible region is the intersection of $n+m$ halfspaces. This is a convex polyhedron. |
| 158 | +- How to get vertices of feasible region? |
| 159 | +- The number of vertices equates to the number of points satisfying... |
| 160 | + - ... $n$ constraints with $=$ |
| 161 | + - ... $m$ constraints with $\le$ |
| 162 | +- There are $\le {{n+m} \choose {n}}$ vertices. This is exponential in $n$ |
| 163 | +- How many neighbors does each vertex have? $\le nm$ |
| 164 | + |
| 165 | +## LP Algorithms |
| 166 | +- Polynomial-time algorithms |
| 167 | + - Ellipsoid algorithms |
| 168 | + - Interior point methods |
| 169 | +- Simplex Algorithm |
| 170 | + - Worst-case: exponential time |
| 171 | + - widely used |
| 172 | + - Output is guaranteed to be optimum. |
| 173 | + - Fast on huge LP's |
| 174 | + |
| 175 | +## Simplex Algorithm |
| 176 | +- Start at $x=0$ |
| 177 | + - If this point does not satisfy all constraints, then we know that the feasible region is empty, and therefore the LP is "infeasible" |
| 178 | +- Look at neighboring vertices, find one with strictly higher objective value, then move there. |
| 179 | +- Repeat. |
| 180 | +- If there are multiple neighbors with higher objective value |
| 181 | + - Choose a random one? |
| 182 | + - Choose the highest? |
| 183 | + - Choose the lowest? |
| 184 | + - This heuristic is important for optimizing the Simplex for the current problem. |
| 185 | +- If there are no better neighbors, then we've found the global maxima. This is dependent on the feasible region being convex. |
0 commit comments