Add error handling for IPython display errors in notebook cell execution

[cderv]

Note

This is only a proposal and this needs to be discussed.

Closes #12228

With this change, quarto render would fail for any error not caught by nbclient as cell execution error.

nbclient does not error out when a cell has an error due to a display error. This PR adds error handling for display errors in notebook cells so that quarto render will error out. error: true controls the behavior to allow or not allow the error.

Tests are added to shown the new behavior.

We may not want that solution but this is probably coherent with error config as true or false expectation.

If we do not do this, I think it is still not good to output the display error as a cell error without a way to remove it. We could also tweak our processing so that we remove this output in some situations and show the display error as a warning.

---

This PR comes with a few tweaks to smoke-all to be able to do more from in-document testing specification and also have a better configuration with a name (so YAML validation for tests will become a reality at some point. I am making the changes slowly across our tests)

[cderv]

@cscheid thanks again for the live review. I was indeed missing a test and not handling cell level precedence. This is now fixed, and test added.

[cderv]

I tried to re-run great_tables example using v 0.16.0 and I noticed truncated output, just above the quarto error and stacktrace. So I am trying to understand this.

❯ quarto render index.qmd --execute-debug
- Connecting to kernel
- Kernel connection successful
- Sending execute command to kernel
- Execute command sent, reading response

Executing 'index.quarto_ipynb'
  Cell 1/1: ''...- Error response received
ERROR:

An error occurred while executing the following cell:
------------------
from great_tables import GT, nanoplot_options
import polars as pl
weather_2 = pl.DataFrame(
    {
        "station": ["Station " + str(x) for x in range(1, 4)],
        "temperatures": [
            {
                "x": [6.1, 8.0, 10.1, 10.5, 11.2, 12.4, 13.1, 15.3],
                "y": [24.2, 28.2, 30.2, 30.5, 30.5, 33.1, 33.5, 32.7],
            },
            {
                "x": [7.1, 8.2, 10.3, 10.75, 11.25, 12.5, 13.5, 14.2],
                "y": [18.2, 18.1, 20.3, 20.5, 21.4, 21.9, 23.1, 23.3],
            },
            {
                "x": [6.3, 7.1, 10.3, 11.0, 12.07, 13.1, 15.12, 16.42],
                "y": [15.2, 17.77, 21.42, 21.63, 25.23, 26.84, 27.2, 27.44],
            },
        ]
    }
)

(
    GT(weather_2)
    .fmt_nanoplot(
        columns="temperatures",
        plot_type="line",
        expand_x=[5, 16],
        expand_y=[10, 40],
        options=nanoplot_options(
            show_data_area=False,
            show_data_line=False
        )
    )
)
------------------


---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\IPython\core\formatters.py:406, in BaseFormatter.__call__(self, obj)
    404     method = get_real_method(obj, self.print_method)
    405     if method is not None:
--> 406         return method()
    407     return None
    408 else:

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\gt.py:293, in GT._repr_html_(self)
    290 make_page = defaults["make_page"]
    291 all_important = defaults["all_important"]
--> 293 rendered = self.as_raw_html(
    294     make_page=make_page,
    295     all_important=all_important,
    296 )
    298 return rendered

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_export.py:220, in as_raw_html(self, inline_css, make_page, all_important)
    130 def as_raw_html(
    131     self: GT,
    132     inline_css: bool = False,
    133     make_page: bool = False,
    134     all_important: bool = False,
    135 ) -> str:
    136     """
    137     Get the HTML content of a GT object.
    138
   (...)    218     ```
    219     """
--> 220     built_table = self._build_data(context="html")
    222     if not inline_css:
    223         html_table = built_table._render_as_html(
    224             make_page=make_page,
    225             all_important=all_important,
    226         )

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tabERROR: Error
    at renderFiles (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/render-files.ts:350:23)
    at eventLoopTick (ext:core/01_core.js:175:7)
    at async renderProject (file:///C:/Users/chris/DOCUME~1/DEV_R/QUARTO~1/src/command/render/project.ts:464:23)
    at async Command.actionHandler (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/cmd.ts:251:26)
    at async Command.execute (https://deno.land/x/cliffy@v1.0.0-rc.3/command/command.ts:1948:7)
    at async Command.parseCommand (https://deno.land/x/cliffy@v1.0.0-rc.3/command/command.ts:1780:14)
    at async quarto (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/quarto.ts:190:5)
    at async file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/quarto.ts:219:5
    at async file:///C:/Users/chris/DOCUME~1/DEV_R/QUARTO~1/src/core/main.ts:41:14
    at async mainRunner (file:///C:/Users/chris/DOCUME~1/DEV_R/QUARTO~1/src/core/main.ts:43:5)

[cderv]

An another mystery as this is not happening in VSCODE debug mode

Full Log

Debugger listening on ws://127.0.0.1:9229/ws/016514b4-e981-430c-a912-e4ccdf8c6227
Visit chrome://inspect to connect to the debugger.
Deno is waiting for debugger to connect.
Debugger session started.

Starting python3 kernel...Done

Executing 'index.quarto_ipynb'
  Cell 1/1: ''...

An error occurred while executing the following cell:
------------------
from great_tables import GT, nanoplot_options
import polars as pl
weather_2 = pl.DataFrame(
    {
        "station": ["Station " + str(x) for x in range(1, 4)],
        "temperatures": [
            {
                "x": [6.1, 8.0, 10.1, 10.5, 11.2, 12.4, 13.1, 15.3],
                "y": [24.2, 28.2, 30.2, 30.5, 30.5, 33.1, 33.5, 32.7],
            },
            {
                "x": [7.1, 8.2, 10.3, 10.75, 11.25, 12.5, 13.5, 14.2],
                "y": [18.2, 18.1, 20.3, 20.5, 21.4, 21.9, 23.1, 23.3],
            },
            {
                "x": [6.3, 7.1, 10.3, 11.0, 12.07, 13.1, 15.12, 16.42],
                "y": [15.2, 17.77, 21.42, 21.63, 25.23, 26.84, 27.2, 27.44],
            },
        ]
    }
)

(
    GT(weather_2)
    .fmt_nanoplot(
        columns="temperatures",
        plot_type="line",
        expand_x=[5, 16],
        expand_y=[10, 40],
        options=nanoplot_options(
            show_data_area=False,
            show_data_line=False
        )
    )
)
------------------


---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\IPython\core\formatters.py:406, in BaseFormatter.__call__(self, obj)
    404     method = get_real_method(obj, self.print_method)
    405     if method is not None:
--> 406         return method()
    407     return None
    408 else:

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\gt.py:293, in GT._repr_html_(self)
    290 make_page = defaults["make_page"]
    291 all_important = defaults["all_important"]
--> 293 rendered = self.as_raw_html(
    294     make_page=make_page,
    295     all_important=all_important,
    296 )
    298 return rendered

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_export.py:220, in as_raw_html(self, inline_css, make_page, all_important)
    130 def as_raw_html(
    131     self: GT,
    132     inline_css: bool = False,
    133     make_page: bool = False,
    134     all_important: bool = False,
    135 ) -> str:
    136     """
    137     Get the HTML content of a GT object.
    138 
   (...)    218     ```
    219     """
--> 220     built_table = self._build_data(context="html")
    222     if not inline_css:
    223         html_table = built_table._render_as_html(
    224             make_page=make_page,
    225             all_important=all_important,
    226         )

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\gt.py:311, in GT._build_data(self, context)
    308 def _build_data(self, context: str) -> Self:
    309     # Build the body of the table by generating a dictionary
    310     # of lists with cells initially set to nan values
--> 311     built = self._render_formats(context)
    313     if context == "latex":
    314         built = _migrate_unformatted_to_output(
    315             data=built, data_tbl=self._tbl_data, formats=self._formats, context=context
    316         )

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\gt.py:304, in GT._render_formats(self, context)
    301 new_body = self._body.copy()
    303 # TODO: this body method performs a mutation. Should we make a copy of body?
--> 304 new_body.render_formats(self._tbl_data, self._formats, context)
    305 new_body.render_formats(self._tbl_data, self._substitutions, context)
    306 return self._replace(_body=new_body)

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_gt_data.py:172, in Body.render_formats(self, data_tbl, formats, context)
    170     raise Exception("Internal Error")
    171 for col, row in fmt.cells.resolve():
--> 172     result = eval_func(_get_cell(data_tbl, row, col))
    173     if isinstance(result, FormatterSkipElement):
    174         continue

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_formats.py:4716, in fmt_nanoplot.<locals>.fmt_nanoplot_fn(x, context, plot_type, plot_height, missing_vals, reference_line, reference_area, all_single_y_vals, options_plots)
   4713     y_vals = x
   4714     x_vals = None
-> 4716 nanoplot = _generate_nanoplot(
   4717     y_vals=y_vals,
   4718     y_ref_line=reference_line,
   4719     y_ref_area=reference_area,
   4720     x_vals=x_vals,
   4721     expand_x=expand_x,
   4722     expand_y=expand_y,
   4723     missing_vals=missing_vals,
   4724     all_single_y_vals=all_single_y_vals,
   4725     plot_type=plot_type,
   4726     svg_height=plot_height,
   4727     **options_plots,
   4728 )
   4730 return nanoplot

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_utils_nanoplots.py:915, in _generate_nanoplot(y_vals, y_ref_line, y_ref_area, x_vals, expand_x, expand_y, missing_vals, all_y_vals, all_single_y_vals, plot_type, data_line_type, currency, y_val_fmt_fn, y_axis_fmt_fn, y_ref_line_fmt_fn, data_point_radius, data_point_stroke_color, data_point_stroke_width, data_point_fill_color, data_line_stroke_color, data_line_stroke_width, data_area_fill_color, data_bar_stroke_color, data_bar_stroke_width, data_bar_fill_color, data_bar_negative_stroke_color, data_bar_negative_stroke_width, data_bar_negative_fill_color, reference_line_color, reference_area_fill_color, vertical_guide_stroke_color, vertical_guide_stroke_width, show_data_points, show_data_line, show_data_area, show_reference_line, show_reference_area, show_vertical_guides, show_y_axis_guide, interactive_data_values, svg_height)
    911 # If x values are present then normalize them between [0, 1]; if
    912 # there are no x values, generate equally-spaced x values according
    913 # to the number of y values
    914 if plot_type == "line" and x_vals is not None:
--> 915     if expand_x is not None and _val_is_str(expand_x):
    916         # TODO: the line below lacked tests, and called non-existent methods.
    917         # replace with something that doesn't use pandas and returns the correct thing.
    918 
    919         # Assume that string values are dates and convert them to timestamps
    920         # expand_x = pd.to_datetime(expand_x, utc=True).timestamp()
    921         raise NotImplementedError("Currently, passing expand_x as a string is unsupported.")
    923     # Scale to proportional values

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_utils_nanoplots.py:41, in _val_is_str(x)
     39 # If a list then signal a failure
     40 if isinstance(x, list):
---> 41     raise ValueError("The input cannot be a list. It must be a single value.")
     43 return isinstance(x, (str))

ValueError: The input cannot be a list. It must be a single value.

ERROR: Error
    at renderFiles (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/render-files.ts:350:23)
    at eventLoopTick (ext:core/01_core.js:175:7)
    at async renderProject (file:///C:/Users/chris/DOCUME~1/DEV_R/QUARTO~1/src/command/render/project.ts:464:23)
    at async Command.actionHandler (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/cmd.ts:251:26)
    at async Command.execute (https://deno.land/x/cliffy@v1.0.0-rc.3/command/command.ts:1948:7)
    at async Command.parseCommand (https://deno.land/x/cliffy@v1.0.0-rc.3/command/command.ts:1780:14)
    at async quarto (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/quarto.ts:190:5)
    at async file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/quarto.ts:219:5
    at async file:///C:/Users/chris/DOCUME~1/DEV_R/QUARTO~1/src/core/main.ts:41:14
    at async mainRunner (file:///C:/Users/chris/DOCUME~1/DEV_R/QUARTO~1/src/core/main.ts:43:5)

[cderv]

So I don't manage to solve the truncated console output for long error message from python. This does not happen in the log file when setting --log, so this is a way to see the full error message.

I believe what happens is something about writing to console and buffer size. It seems there is a limit where msg needs to be written by chunk.

So if we manage to write by chunk, we may solve this.

Also I noticed a difference:

• keep-alive kernel will output to error()
• oneshot kernel will output to info()

We also may want to fix this.

I am thinking we should merge this, and then handle in another issue the long output error truncation.

[cderv]

I am now thinking this is about Deno.stderr

quarto-cli/src/core/log.ts

Lines 158 to 162 in e7809ce

	log(msg: string): void {
	Deno.stderr.writeSync(
	new TextEncoder().encode(msg),
	);
	}

which is not writing the full buffer in a single call as the doc says 🤔

[cderv]

Yes this is it !! It will be hard to test though 🤔

But here is the output now

> quarto render index.qmd --execute-daemon-restart --log test.log
Check file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/quarto.ts

Starting python3 kernel...Done

Executing 'index.quarto_ipynb'
  Cell 1/1: ''...ERROR: 

An error occurred while executing the following cell:
------------------
from great_tables import GT, nanoplot_options
import polars as pl
weather_2 = pl.DataFrame(
    {
        "station": ["Station " + str(x) for x in range(1, 4)],
        "temperatures": [
            {
                "x": [6.1, 8.0, 10.1, 10.5, 11.2, 12.4, 13.1, 15.3],
                "y": [24.2, 28.2, 30.2, 30.5, 30.5, 33.1, 33.5, 32.7],
            },
            {
                "x": [7.1, 8.2, 10.3, 10.75, 11.25, 12.5, 13.5, 14.2],
                "y": [18.2, 18.1, 20.3, 20.5, 21.4, 21.9, 23.1, 23.3],
            },
            {
                "x": [6.3, 7.1, 10.3, 11.0, 12.07, 13.1, 15.12, 16.42],
                "y": [15.2, 17.77, 21.42, 21.63, 25.23, 26.84, 27.2, 27.44],
            },
        ]
    }
)

(
    GT(weather_2)
    .fmt_nanoplot(
        columns="temperatures",
        plot_type="line",
        expand_x=[5, 16],
        expand_y=[10, 40],
        options=nanoplot_options(
            show_data_area=False,
            show_data_line=False
        )
    )
)
------------------


---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\IPython\core\formatters.py:406, in BaseFormatter.__call__(self, obj)
    404     method = get_real_method(obj, self.print_method)
    405     if method is not None:
--> 406         return method()
    407     return None
    408 else:

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\gt.py:293, in GT._repr_html_(self)
    290 make_page = defaults["make_page"]
    291 all_important = defaults["all_important"]
--> 293 rendered = self.as_raw_html(
    294     make_page=make_page,
    295     all_important=all_important,
    296 )
    298 return rendered

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_export.py:220, in as_raw_html(self, inline_css, make_page, all_important)
    130 def as_raw_html(
    131     self: GT,
    132     inline_css: bool = False,
    133     make_page: bool = False,
    134     all_important: bool = False,
    135 ) -> str:
    136     """
    137     Get the HTML content of a GT object.
    138 
   (...)    218     ```
    219     """
--> 220     built_table = self._build_data(context="html")
    222     if not inline_css:
    223         html_table = built_table._render_as_html(
    224             make_page=make_page,
    225             all_important=all_important,
    226         )

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\gt.py:311, in GT._build_data(self, context)
    308 def _build_data(self, context: str) -> Self:
    309     # Build the body of the table by generating a dictionary
    310     # of lists with cells initially set to nan values
--> 311     built = self._render_formats(context)
    313     if context == "latex":
    314         built = _migrate_unformatted_to_output(
    315             data=built, data_tbl=self._tbl_data, formats=self._formats, context=context
    316         )

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\gt.py:304, in GT._render_formats(self, context)
    301 new_body = self._body.copy()
    303 # TODO: this body method performs a mutation. Should we make a copy of body?
--> 304 new_body.render_formats(self._tbl_data, self._formats, context)
    305 new_body.render_formats(self._tbl_data, self._substitutions, context)
    306 return self._replace(_body=new_body)

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_gt_data.py:172, in Body.render_formats(self, data_tbl, formats, context)
    170     raise Exception("Internal Error")
    171 for col, row in fmt.cells.resolve():
--> 172     result = eval_func(_get_cell(data_tbl, row, col))
    173     if isinstance(result, FormatterSkipElement):
    174         continue

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_formats.py:4716, in fmt_nanoplot.<locals>.fmt_nanoplot_fn(x, context, plot_type, plot_height, missing_vals, reference_line, reference_area, all_single_y_vals, options_plots)
   4713     y_vals = x
   4714     x_vals = None
-> 4716 nanoplot = _generate_nanoplot(
   4717     y_vals=y_vals,
   4718     y_ref_line=reference_line,
   4719     y_ref_area=reference_area,
   4720     x_vals=x_vals,
   4721     expand_x=expand_x,
   4722     expand_y=expand_y,
   4723     missing_vals=missing_vals,
   4724     all_single_y_vals=all_single_y_vals,
   4725     plot_type=plot_type,
   4726     svg_height=plot_height,
   4727     **options_plots,
   4728 )
   4730 return nanoplot

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_utils_nanoplots.py:915, in _generate_nanoplot(y_vals, y_ref_line, y_ref_area, x_vals, expand_x, expand_y, missing_vals, all_y_vals, all_single_y_vals, plot_type, data_line_type, currency, y_val_fmt_fn, y_axis_fmt_fn, y_ref_line_fmt_fn, data_point_radius, data_point_stroke_color, data_point_stroke_width, data_point_fill_color, data_line_stroke_color, data_line_stroke_width, data_area_fill_color, data_bar_stroke_color, data_bar_stroke_width, data_bar_fill_color, data_bar_negative_stroke_color, data_bar_negative_stroke_width, data_bar_negative_fill_color, reference_line_color, reference_area_fill_color, vertical_guide_stroke_color, vertical_guide_stroke_width, show_data_points, show_data_line, show_data_area, show_reference_line, show_reference_area, show_vertical_guides, show_y_axis_guide, interactive_data_values, svg_height)
    911 # If x values are present then normalize them between [0, 1]; if
    912 # there are no x values, generate equally-spaced x values according
    913 # to the number of y values
    914 if plot_type == "line" and x_vals is not None:
--> 915     if expand_x is not None and _val_is_str(expand_x):
    916         # TODO: the line below lacked tests, and called non-existent methods.
    917         # replace with something that doesn't use pandas and returns the correct thing.
    918 
    919         # Assume that string values are dates and convert them to timestamps
    920         # expand_x = pd.to_datetime(expand_x, utc=True).timestamp()
    921         raise NotImplementedError("Currently, passing expand_x as a string is unsupported.")
    923     # Scale to proportional values

File ~\Documents\DEV_OTHER\00-TESTS\test-quarto\.venv\Lib\site-packages\great_tables\_utils_nanoplots.py:41, in _val_is_str(x)
     39 # If a list then signal a failure
     40 if isinstance(x, list):
---> 41     raise ValueError("The input cannot be a list. It must be a single value.")
     43 return isinstance(x, (str))

ValueError: The input cannot be a list. It must be a single value.

ERROR: Error
    at renderFiles (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/render-files.ts:351:23)
    at eventLoopTick (ext:core/01_core.js:175:7)
    at async renderProject (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/project.ts:463:23)
    at async Command.actionHandler (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/cmd.ts:251:26)
    at async Command.execute (https://deno.land/x/cliffy@v1.0.0-rc.3/command/command.ts:1948:7)
    at async Command.parseCommand (https://deno.land/x/cliffy@v1.0.0-rc.3/command/command.ts:1780:14)
    at async quarto (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/quarto.ts:191:5)
    at async file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/quarto.ts:220:5
    at async file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/core/main.ts:41:14
    at async mainRunner (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/core/main.ts:43:5)

[cderv]

This is better overall to handle error in Jupyter as other error, but we did use trace() in Jupyter to log as info... 🤔

However, trace() should have been trace level only

quarto-cli/src/resources/jupyter/log.py

Lines 43 to 45 in ec6cfd0

	def trace(msg):
	prev_frame = inspect.stack()[1]
	log(TRACE, "%s:%s - %s" % (prev_frame.filename, prev_frame.lineno, msg))

Though I see now that this output we had for each quarto render

Executing 'index.quarto_ipynb'
  Cell 1/1: ''...Done

is coming from

quarto-cli/src/resources/jupyter/notebook.py

Lines 326 to 328 in ec6cfd0

	# we are done w/ setup (with no restarts) so it's safe to print 'Executing...'
	if not quiet:
	status("\nExecuting '{0}'\n".format(input))

quarto-cli/src/resources/jupyter/notebook.py

Lines 277 to 284 in ec6cfd0

	progress = (not quiet) and cell.cell_type == 'code' and index > 0
	if progress:
	status(" Cell {0}/{1}: '{2}'{3}...".format(
	current_code_cell - 1,
	total_code_cells - 1,
	cell_label,
	padding
	))

So there is also I broke with Status 🤔 🤦‍♂️

I should have split this PR in several I think

[cderv]

And this is because we stream status to stderr 😭

quarto-cli/src/resources/jupyter/jupyter.py

Lines 251 to 254 in ec6cfd0

	# stream status to stderr
	def status(msg):
	sys.stderr.write(msg)
	sys.stderr.flush()

And we do the same for errors

quarto-cli/src/resources/jupyter/jupyter.py

Lines 226 to 246 in ec6cfd0

	# run a notebook directly (not a server)
	def run_notebook(options, status):
	# run notebook w/ some special exception handling. note that we don't
	# log exceptions here b/c they are considered normal course of execution
	# for errors that occur in notebook cells
	try:
	trace('Running notebook_execute')
	notebook_execute(options, status)
	except Exception as e:
	trace(f'run_notebook caught exception: {type(e).__name__}')
	# CellExecutionError for execution at the terminal includes a bunch
	# of extra stack frames internal to this script. remove them
	msg = str(e)
	kCellExecutionError = "nbclient.exceptions.CellExecutionError: "
	loc = msg.find(kCellExecutionError)
	if loc != -1:
	msg = msg[loc + len(kCellExecutionError):]
	sys.stderr.write("\n\n" + msg + "\n")
	sys.stderr.flush()
	sys.exit(1)

We do it for some warnings too:

quarto-cli/src/resources/jupyter/notebook.py

Lines 687 to 694 in ec6cfd0

	try:
	params_cell_yaml = parse_string("\n".join(params_cell_yaml))
	del params_cell_yaml['label']
	params_cell_yaml = safe_dump(params_cell_yaml).strip().splitlines()
	except Exception as e:
	sys.stderr.write("\nWARNING: Invalid YAML option format in cell:\n" + "\n".join(params_cell_yaml) + "\n")
	sys.stderr.flush()
	params_cell_yaml = []

quarto-cli/src/resources/jupyter/notebook.py

Lines 751 to 760 in ec6cfd0

	# if we have yaml then parse it
	if len(yaml_lines) > 0:
	yaml_code = "\n".join(yaml_lines)
	yaml_options = parse_string(yaml_code)
	if (type(yaml_options) is dict):
	return yaml_options
	else:
	sys.stderr.write("\nWARNING: Invalid YAML option format in cell:\n" + yaml_code + "\n")
	sys.stderr.flush()
	return dict()

So I think all this digging to understand is not vain: My initial fix is probably not completely correct.

Correct solution: Use logError() ?

quarto-cli/src/resources/jupyter/log.py

Lines 40 to 41 in ec6cfd0

	def log_error(msg, exc_info = False):
	logging.getLogger().log(logging.ERROR, msg, exc_info = exc_info, stack_info = not exc_info)

Like we do in two places

quarto-cli/src/resources/jupyter/jupyter.py

Lines 189 to 195 in ec6cfd0

	def run_server(options):
	try:
	with execute_server(options) as server:
	while True:
	server.handle_request()
	except Exception as e:
	log_error("Unable to run server", exc_info = e)

quarto-cli/src/resources/jupyter/jupyter.py

Lines 288 to 290 in ec6cfd0

	except Exception as e:
	log_error("Unable to run notebook", exc_info = e)
	sys.exit(1)

More work for tomorrow...

[cderv]

So I don't manage to get to the bottom of how the stderr and other stream are redirected in Jupyter context.

What I see is that, this is different between two mode

• Using a keep alive kernel, the error is written to console, and also to the log file. But possibly because we receive json message in typescript and then correctly error
  

  quarto-cli/src/execute/jupyter/jupyter-kernel.ts

  Lines 129 to 131 in 58200c4

  	if (msg.type === "error") {
  	trace(options, "Error response received");
  	error(msg.data, { colorize: false });

• Using a oneshot kernel, we don't get those messages, and stderr from execJupyter() is in fact expected to be written in info() and not as an error. Because we rely on it for status() in python.
  Somehow some of the messages are shown in console, but not in log file. It is something I don't understand why some of the info shows, and others not.

So this PR is good now, but the error message from the new exception raised won't be shown in log file.