Show why healthcheck_cmd is failing

[letmerecall]

Description

Is there any way to show the output of the healthcheck_cmd ? Like why it's failing?
If not, it'd be great if we can have something like that.

Why it's important?

I want to perform an integration test on a container which take around 2 minutes to warm up. In case I have some issue with the healthcheck (hence the container) all I am left with is this cryptic message with not much information to actually debug the issue.

(monty) ➜  tox
no-env-8080 installed: attrs==21.4.0,certifi==2021.10.8,charset-normalizer==2.0.12,idna==3.3,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pyparsing==3.0.7,pytest==7.0.1,pytest-dotenv==0.5.2,python-dotenv==0.19.2,requests==2.27.1,tomli==2.0.1,urllib3==1.26.8
no-env-8080 docker: run 'holy-grail:pytest' (from 'holy-grail')
no-env-8080 docker: health check 'holy-grail:pytest' (from 'holy-grail')
no-env-8080 docker: remove '26bb537d52' (from 'holy-grail)
______________________________________________________________________________ summary ______________________________________________________________________________
ERROR:   no-env-8080: 'holy-grail:pytest' (from 'holy-grail') failed health check
Traceback (most recent call last):
  File "/Users/girish/miniconda3/envs/monty/bin/tox", line 8, in <module>
    sys.exit(cmdline())
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/tox/session/__init__.py", line 44, in cmdline
    main(args)
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/tox/session/__init__.py", line 69, in main
    exit_code = session.runcommand()
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/tox/session/__init__.py", line 197, in runcommand
    return self.subcommand_test()
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/tox/session/__init__.py", line 225, in subcommand_test
    run_sequential(self.config, self.venv_dict)
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/tox/session/commands/run/sequential.py", line 22, in run_sequential
    runtestenv(venv, config)
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/tox/session/commands/run/sequential.py", line 73, in runtestenv
    config.pluginmanager.hook.tox_runtest_pre(venv=venv)
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/Users/girish/miniconda3/envs/monty/lib/python3.8/site-packages/tox_docker/__init__.py", line 338, in tox_runtest_pre
    raise HealthCheckFailed(msg)
tox_docker.HealthCheckFailed: 'holy-grail:pytest' (from 'holy-grail') failed health check

[terrynguyen255]

Please make this happen. I'm so frustrated to blindly fixing my tox don't know why a container fails its healthcheck

[notzippy]

Im pretty sure the healthcheck results are stored in the docker logs, so when the healthcheck fails ask the container for the logs and output that as an error log.

[terrynguyen255]

Hi @notzippy ,

Thank you for the response.
Unfortunately, I think your answer does not work. Because after a docker container failed, all the containers will be removed. Therefore, you cannot access there log

[notzippy]

@Nogias9x At the point in code that I referenced, the containers are still running and have not been removed. If I have time Ill submit a PR to demonstrate

[tkdchen]

As of 5.0.0, once the status is unhealthy, tox-docker removes the containers by default.

The output messages .[].State.Health.Log[].Output should be gathered and reported.

[thenewguy]

Is there any workaround to display the output using tox-docker-4.1.0? I am working with a project that cannot upgrade at the moment due to a feature removed from tox after version tox-3.28.0

[dcrosta]

This is an interesting idea, and probably doable. I'd accept a PR which adds this behavior.

In the meantime as a workaround, you can use tox --docker-dont-stop=CONTAINER ... so that you can docker inspect the container after the healthcheck fails.

[thenewguy]

This is an interesting idea, and probably doable. I'd accept a PR which adds this behavior.

In the meantime as a workaround, you can use tox --docker-dont-stop=CONTAINER ... so that you can docker inspect the container after the healthcheck fails.

Should the dont-stop option work here? I tried that before asking about a workaround because it didn't work for me. I couldn't see the container when listing all for an id to inspect

[dcrosta]

I think it should. I’m guessing from yo ur reply that it doesn’t — we should probably consider that a bug.

[thenewguy]

I think it should. I’m guessing from yo ur reply that it doesn’t — we should probably consider that a bug.

I will confirm and let you know