[feature][CGS][application-manager] add secondary yarn queue selection with permission check

[v-kkhuang]

Search before asking

• I had searched in the issues and found no similar issues.

Linkis Component

• linkis-commons
• linkis-computation-governance
• linkis-dist
• linkis-engineconn-plugin
• linkis-extensions
• linkis-orchestrator
• linkis-public-enhancements
• linkis-spring-cloud-services
• linkis-web

Description

In production environments, YARN queues may become unavailable due to resource constraints, maintenance, or other issues. Currently, Linkis does not have a mechanism to automatically switch to a backup queue when the primary queue is unavailable, leading to task failures.

Additionally, not all users or engine types should be allowed to use secondary queues due to resource management and cost control requirements.

Steps to reproduce

N/A - This is a feature request

Expected behavior

When enabled, the system should:

1. Check if the task creator is in the allowed list for secondary queue usage
2. Check if the engine type is supported
3. Query LinkisManager for the optimal queue (primary or secondary)
4. Automatically use the secondary queue when the primary queue is unavailable
5. Include permission checks to prevent unauthorized access to secondary queues

Your environment

• Linkis version used: 2.0.0
• Environment name and version:
  • hadoop 3.3.4
  • jdk 1.8.0_xxx

Anything else

This change adds:

1. QueueSelectionInterceptor at Entrance layer for intelligent queue selection
2. SecondaryYarnConf protocol for RPC communication
3. Configuration options to enable/disable the feature and control access
4. Permission checks based on creator and engine type
5. Integration with LinkisManager for queue availability detection
6. Comprehensive documentation, requirements, and test cases

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[github-actions]

😊 Welcome to the Apache Linkis community!!

We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context.
We will be here shortly.

If you are interested in contributing to our website project, please let us know!
You can check out our contributing guide on
👉 How to Participate in Project Contribution.

Community

WeChat Assistant	WeChat Public Account

Mailing Lists

Name	Description	Subscribe	Unsubscribe	Archive
dev@linkis.apache.org	community activity information	subscribe	unsubscribe	archive