node 3 reporting too many open files

[ashraffouda]

node id : 3
net: development
monitoring url: https://mon.grid.tf/explore?orgId=1&left=%7B%22datasource%22:%22Loki-ZOS%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnetwork%3D%5C%22development%5C%22,%20node%3D%5C%22HSCP7NggzvSUDynLhvWzddHyZv9nEK9fnNfmyYJwwyjm%5C%22%7D%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D

this node keeps reporting too many open files on the logs

[Eslam-Nawara]

Work Completed

after investigation found out that environment.GetConfig doesn't close response body properly and after adding a new feature to use configurable RelaysURLs from zos-config repo the node kept opening files without closing them causing the node to no longer be able to register of update itself

now fixed that and will be monitoring devnet nodes to check if this solves the problem

[Eslam-Nawara]

WIP
I was investigating this issue and found out the node opens too much connections with devnet stacks, now investigating why it's opening all this connections

the problem is, the api-gateway module keeps trying to create new rmb peer and it keeps failing over and over again which cause a huge number of open connections to the chain that never get to be closed, so now I stopped trying to open that peer, rmb is not working for zos4 either ways, so until we start using new open rpc this part will be commented

[Eslam-Nawara]

Work Completed

merged and updated devnet and will be updating the rest of the networks once we verify that the issue is fixed in devnet

[Eslam-Nawara]

Verified

last update was almost 2 weeks ago and after monitoring the node they seem to be working fine with no nodes reporting too many open files anymore