apply GPU transformation to avoid KV cache broadcast#36775
apply GPU transformation to avoid KV cache broadcast#36775nazanin-beheshti wants to merge 14 commits into
Conversation
|
@Lyamin-Roman Would you please review one more time? |
| auto sdpa = std::make_shared<ov::intel_gpu::op::SDPA>(inputs, is_causal, in0_order, in1_order, in2_order, out_order); | ||
|
|
||
| model_ref = std::make_shared<ov::Model>(ov::OutputVector{ sdpa }, ov::ParameterVector{ input_q, input_k, input_v }); | ||
| comparator.enable(FunctionsComparator::ATTRIBUTES); |
There was a problem hiding this comment.
Add accuracy check too
comparator.enable(FunctionsComparator::CmpValues::ACCURACY);
There was a problem hiding this comment.
I got an exception Exception from src\inference\src\cpp\infer_request.cpp:224:
Check 'it != map.end()' failed at src\plugins\template\backend\int_executable.cpp:225:
Interpreter backend doesn't implement evaluate method for OP SDPA
There was a problem hiding this comment.
@Lyamin-Roman the current transformation is similar with https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/intel_gpu/tests/unit/transformations/unsqueeze_broadcast_reshape_sdpa_fusion_test.cpp which doesn't include comparator.enable(FunctionsComparator::CmpValues::ACCURACY); in the unit test as well.
There was a problem hiding this comment.
@Kotomi-Du, Yes, ok, it looks like there is a reference only for PA, but accuracy test is still needed, and I suggest expanding the src/plugins/intel_gpu/tests/functional/subgraph_tests/sdpa.cpp test, which also tests the aforementioned UnsqueezeBroadcastReshapeSDPAFusion under is_complex_gqa flag
| auto reshape1_pattern_key_m = any_input(); | ||
| auto reshape_4d_to_5d_key_m = wrap_type<ov::op::v1::Reshape>({ in_key, reshape1_pattern_key_m }, reshape_4d_to_5d); | ||
|
|
||
| auto concat_key_m = wrap_type<ov::op::v0::Concat>({ reshape_4d_to_5d_key_m, reshape_4d_to_5d_key_m, reshape_4d_to_5d_key_m, reshape_4d_to_5d_key_m }); |
There was a problem hiding this comment.
it's a hard code for GQA specifically q_head/k_head=4. Please refine the match pattern
3 participants
Details:
The transformation is added in GPU plugin to identify the pattern added in here https://github.com/Kotomi-Du/openvino/blob/741382eafa64a55d8dd2f1dc43a45726fe4306c6/src/common/transformations/src/transformations/op_conversions/group_query_attention_decomposition.cpp#L208 and remove the extra nodes for better efficiency and performance.
Tickets:
AI Assistance: