Console Proxy - No authorization to connect

[fermosan]

ISSUE TYPE

• Bug Report

COMPONENT NAME

ConsoleProxy

CLOUDSTACK VERSION

main (7936b0)

CONFIGURATION

Advanced Networking
No SSL

OS / ENVIRONMENT

UBUNTU 22.04 / KVM

SUMMARY

running a clean install on current 4.18 tree looks like something is not working in consoleproxy.
(noVNC and classic as well)

An identical 4.17 environment works fine.

STEPS TO REPRODUCE

Deploydb for cloudstack current
Create Zone / Storage 
Join KVM Host
Enable Zone
Try to open console for system vms

EXPECTED RESULTS

Console Window opens for SystemVMS

ACTUAL RESULTS

Authorization Fails. 
**Logs Consoleproxy System VM:**
INFO  [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp1810267116-38:null) Get websocket connection request from remote IP : 10.8.80.3
WARN  [cloud.consoleproxy.ConsoleProxy] (qtp1810267116-38:null) External authenticator failed authentication request for vm d59c31a5-e4c8-454f-a9ff-432bc8910fcb with sid 7lXWphOeekTsH0G6x3xWVQ
[cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp1810267116-38:null) Failed to create viewer due to External authenticator failed request for vm d59c31a5-e4c8-454f-a9ff-432bc8910fcb with sid 7lXWphOeekTsH0G6x3xWVQ
com.cloud.consoleproxy.AuthenticationException: External authenticator failed request for vm d59c31a5-e4c8-454f-a9ff-432bc8910fcb with sid 7lXWphOeekTsH0G6x3xWVQ

**Logs Management vmops**
WARN  [c.c.a.d.ParamGenericValidationWorker] (qtp1637207591-343:ctx-79672d38 ctx-0d2376ce) (logid:b4beffac) Received unknown parameters for command createConsoleEndpoint. Unknown parameters : client-inet-address
INFO  [o.a.c.c.ConsoleAccessManagerImpl] (qtp1637207591-343:ctx-79672d38 ctx-0d2376ce) (logid:b4beffac) Parse host info returned from executing GetVNCPortCommand. host info: 172.31.255.211
WARN  [c.c.a.m.AgentManagerImpl] (qtp1637207591-343:ctx-79672d38 ctx-0d2376ce) (logid:b4beffac) Unsupported Command: Unsupported command issued: com.cloud.agent.api.BadCommand.  Are you sure you got the right type of server?
ERROR [c.c.c.AgentHookBase] (AgentManager-Handler-3:null) (logid:) Invalid session, only one session allowed per token
<!-- Paste verbatim command output between quotes below -->

[fermosan]

@nvazquez
As @weizhouapache mentioned in Dev Mailing list maybe related to: @6577

[nvazquez]

Hi @fermosan can you share the /var/log/cloud.log file from your CPVM?

[fermosan]

Hi @fermosan can you share the /var/log/cloud.log file from your CPVM?

cloud.log

@nvazquez There you are

[nvazquez]

Thanks @fermosan, quick question in case you have not regenerated your system VMs can you please do (at least for CPVM in this case)? I see from the stack trace that it may not be using the latest code

[fermosan]

The system vms have been regenerated a few times. I think we are using template KVM 17.0.
Is there any newer ? i think not.

[nvazquez]

@fermosan I think the code running on your CPVM does not contain the latest changes. I have checked the following in your logs:

2022-10-14 05:59:39,720 WARN  [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp1810267116-38:null) Failed to create viewer due to External authenticator failed request for vm d59c31a5-e4c8-454f-a9ff-432bc8910fcb with sid 7lXWphOeekTsH0G6x3xWVQ
com.cloud.consoleproxy.AuthenticationException: External authenticator failed request for vm d59c31a5-e4c8-454f-a9ff-432bc8910fcb with sid 7lXWphOeekTsH0G6x3xWVQ
	at com.cloud.consoleproxy.ConsoleProxy.authenticationExternally(ConsoleProxy.java:528)
	at com.cloud.consoleproxy.ConsoleProxy.getNoVncViewer(ConsoleProxy.java:562)
	at com.cloud.consoleproxy.ConsoleProxyNoVNCHandler.onConnect(ConsoleProxyNoVNCHandler.java:136

Both lines 528 and 562 point to closing brackets on the latest main branch code:
https://github.com/apache/cloudstack/blob/main/services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxy.java#L528

https://github.com/apache/cloudstack/blob/main/services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxy.java#L562

Can you share the commit hash of the main branch you are using? I see the latest changes of that file for the console enhancements were merged on September 14, is there any chance your main branch is older than that date? If that's the case can you please rebase to the latest main branch?

[fermosan]

@nvazquez Hello. Branch is rebased with main (as you can see in #6550
Existing VMs deleted.
Commit hash for main i am using: [289a43f]
Still consoles do not fire up with the same issue.

[nvazquez]

Thanks @fermosan I think the issue is the agent.zip and cloud-scripts.tgz files are not up to date, these files replace the previous systemvm.iso. Those are located on the KVM hosts at /usr/share/cloudstack-common/vms/. Can you regenerate them by building cloudstack with the -P systemvm option? The new files will be located on <CS_DIR>/systemvm/dist and would need to be copied to the KVM hosts /usr/share/cloudstack-common/vms/ dir. After the copy the CPVM needs to be destroyed/regenerated

[fermosan]

@nvazquez You nailed it. Consoles are working again. Thanks so much. I am closing this.

[nvazquez]

Great news, thanks @fermosan