Conflict with Pandas

[jkuruzovich]

I have a strange error that I've been able to reproduce on 2 machines but it doesn't occur on others. For example, I have the issue running on my local laptop but then not with other environments.

The behavior is such that Gofer Grader runs fine until pandas is imported. Then the issue below occurs every time you run the grading.

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-7-bdbe578848a5> in <module>()
----> 1 _ = ok.grade('q21')

~/anaconda3/envs/auto/lib/python3.6/site-packages/client/api/notebook.py in grade(self, question, global_env)
     56             # inspect trick to pass in its parents' global env.
     57             global_env = inspect.currentframe().f_back.f_globals
---> 58         result = check(path, global_env)
     59         # We display the output if we're in IPython.
     60         # This keeps backwards compatibility with okpy's grade method

~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/ok.py in check(test_file_path, global_env)
    294         # inspect trick to pass in its parents' global env.
    295         global_env = inspect.currentframe().f_back.f_globals
--> 296     return tests.run(global_env, include_grade=False)

~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/ok.py in run(self, global_environment, include_grade)
    143         failed_tests = []
    144         for t in self.tests:
--> 145             passed, hint = t.run(global_environment)
    146             if passed:
    147                 passed_tests.append(t)

~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/ok.py in run(self, global_environment)
     85     def run(self, global_environment):
     86         for i, t in enumerate(self.tests):
---> 87             passed, result = run_doctest(self.name + ' ' + str(i), t, global_environment)
     88             if not passed:
     89                 return False, OKTest.result_fail_template.render(

~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/ok.py in run_doctest(name, doctest_string, global_environment)
     43     runresults = io.StringIO()
     44     with redirect_stdout(runresults), redirect_stderr(runresults), hide_outputs():
---> 45         doctestrunner.run(test, clear_globs=False)
     46     with open('/dev/null', 'w') as f, redirect_stderr(f), redirect_stdout(f):
     47         result = doctestrunner.summarize(verbose=True)

~/anaconda3/envs/auto/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback)
     86         if type is None:
     87             try:
---> 88                 next(self.gen)
     89             except StopIteration:
     90                 return False

~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/utils.py in hide_outputs()
     46         yield
     47     finally:
---> 48         flush_inline_matplotlib_plots()
     49         ipy.display_formatter.formatters = old_formatters

~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/utils.py in flush_inline_matplotlib_plots()
     21     try:
     22         import matplotlib as mpl
---> 23         from ipykernel.pylab.backend_inline import flush_figures
     24     except ImportError:
     25         return

~/anaconda3/envs/auto/lib/python3.6/site-packages/ipykernel/pylab/backend_inline.py in <module>()
    167             ip.events.register('post_run_cell', configure_once)
    168 
--> 169 _enable_matplotlib_integration()
    170 
    171 def _fetch_figure_metadata(fig):

~/anaconda3/envs/auto/lib/python3.6/site-packages/ipykernel/pylab/backend_inline.py in _enable_matplotlib_integration()
    158         try:
    159             activate_matplotlib(backend)
--> 160             configure_inline_support(ip, backend)
    161         except (ImportError, AttributeError):
    162             # bugs may cause a circular import on Python 2

~/anaconda3/envs/auto/lib/python3.6/site-packages/IPython/core/pylabtools.py in configure_inline_support(shell, backend)
    409     if new_backend_name != cur_backend:
    410         # Setup the default figure format
--> 411         select_figure_formats(shell, cfg.figure_formats, **cfg.print_figure_kwargs)
    412         configure_inline_support.current_backend = new_backend_name

~/anaconda3/envs/auto/lib/python3.6/site-packages/IPython/core/pylabtools.py in select_figure_formats(shell, formats, **kwargs)
    215     from matplotlib.figure import Figure
    216 
--> 217     svg_formatter = shell.display_formatter.formatters['image/svg+xml']
    218     png_formatter = shell.display_formatter.formatters['image/png']
    219     jpg_formatter = shell.display_formatter.formatters['image/jpeg']

KeyError: 'image/svg+xml'

[vipasu]

Thanks for sharing. We have seen a similar issue before. Not sure exactly what is causing the problem, but can you try putting "%matplotlib inline" before importing pandas? This is assuming you're in a jupyter notebook. Best, Vincent

…

On Mon, Sep 23rd, 2019 at 6:29 AM, Jason Kuruzovich ***@***.***> wrote: I have a strange error that I've been able to reproduce on 2 machines but it doesn't occur on others. For example, I have the issue running on my local laptop but then not with other environments. The behavior is such that Gofer Grader runs fine until pandas is imported. Then the issue below occurs every time you run the grading. --------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-7-bdbe578848a5> in <module>() ----> 1 _ = ok.grade('q21') ~/anaconda3/envs/auto/lib/python3.6/site-packages/client/api/notebook.py in grade(self, question, global_env) 56 # inspect trick to pass in its parents' global env. 57 global_env = inspect.currentframe().f_back.f_globals ---> 58 result = check(path, global_env) 59 # We display the output if we're in IPython. 60 # This keeps backwards compatibility with okpy's grade method ~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/ok.py in check(test_file_path, global_env) 294 # inspect trick to pass in its parents' global env. 295 global_env = inspect.currentframe().f_back.f_globals --> 296 return tests.run(global_env, include_grade=False) ~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/ok.py in run(self, global_environment, include_grade) 143 failed_tests = [] 144 for t in self.tests: --> 145 passed, hint = t.run(global_environment) 146 if passed: 147 passed_tests.append(t) ~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/ok.py in run(self, global_environment) 85 def run(self, global_environment): 86 for i, t in enumerate(self.tests): ---> 87 passed, result = run_doctest(self.name + ' ' + str(i), t, global_environment) 88 if not passed: 89 return False, OKTest.result_fail_template.render( ~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/ok.py in run_doctest(name, doctest_string, global_environment) 43 runresults = io.StringIO() 44 with redirect_stdout(runresults), redirect_stderr(runresults), hide_outputs(): ---> 45 doctestrunner.run(test, clear_globs=False) 46 with open('/dev/null', 'w') as f, redirect_stderr(f), redirect_stdout(f): 47 result = doctestrunner.summarize(verbose=True) ~/anaconda3/envs/auto/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback) 86 if type is None: 87 try: ---> 88 next(self.gen) 89 except StopIteration: 90 return False ~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/utils.py in hide_outputs() 46 yield 47 finally: ---> 48 flush_inline_matplotlib_plots() 49 ipy.display_formatter.formatters = old_formatters ~/anaconda3/envs/auto/lib/python3.6/site-packages/gofer/utils.py in flush_inline_matplotlib_plots() 21 try: 22 import matplotlib as mpl ---> 23 from ipykernel.pylab.backend_inline import flush_figures 24 except ImportError: 25 return ~/anaconda3/envs/auto/lib/python3.6/site-packages/ipykernel/pylab/backend_inline.py in <module>() 167 ip.events.register('post_run_cell', configure_once) 168 --> 169 _enable_matplotlib_integration() 170 171 def _fetch_figure_metadata(fig): ~/anaconda3/envs/auto/lib/python3.6/site-packages/ipykernel/pylab/backend_inline.py in _enable_matplotlib_integration() 158 try: 159 activate_matplotlib(backend) --> 160 configure_inline_support(ip, backend) 161 except (ImportError, AttributeError): 162 # bugs may cause a circular import on Python 2 ~/anaconda3/envs/auto/lib/python3.6/site-packages/IPython/core/pylabtools.py in configure_inline_support(shell, backend) 409 if new_backend_name != cur_backend: 410 # Setup the default figure format --> 411 select_figure_formats(shell, cfg.figure_formats, **cfg.print_figure_kwargs) 412 configure_inline_support.current_backend = new_backend_name ~/anaconda3/envs/auto/lib/python3.6/site-packages/IPython/core/pylabtools.py in select_figure_formats(shell, formats, **kwargs) 215 from matplotlib.figure import Figure 216 --> 217 svg_formatter = shell.display_formatter.formatters['image/svg+xml'] 218 png_formatter = shell.display_formatter.formatters['image/png'] 219 jpg_formatter = shell.display_formatter.formatters['image/jpeg'] KeyError: 'image/svg+xml' — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub ( #23?email_source=notifications&email_token=AAONNLSXUWIWFWMCD5A2ZPLQLDAFPA5CNFSM4IZK3NE2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HNBEGUQ ) , or mute the thread ( https://github.com/notifications/unsubscribe-auth/AAONNLRBIAYTCQFDR2SZRB3QLDAFPANCNFSM4IZK3NEQ ).

[jkuruzovich]

Vincent,
Vincent, Thanks for the reply. I didn't see it come through. Yes, using Jupyter.
Not sure but it is this section of the code that is causing problems. Commenting it out didn't seem to hurt any functionality but fixed the problem.
rpi-techfundamentals@f8566ce

I'm using gopher grader with autograding this semester. Thanks for your work on it.
Best,
Jason

[matthew-brett]

I have the same problem. I'm actually running a version of a data-8 lab notebook - here:

https://github.com/matthew-brett/cfd2019/blob/master/_ok_exercises/arrays/arrays_solution.ipynb

If I'm running from within the notebook:

• Adding %matplotlib inline before or after the Pandas import fixes it (but I would prefer to avoid that if possible, it's confusing for the students at this stage);
• Essentially disabling gofer.utils.flush_inline_matplotlib_plots as in the patch above, also fixes it.

However, if I'm running outside the notebook, with something like

[nav] In [1]: from glob import glob 
         ...: from gofer.ok import grade_notebook 
         ...: tests = glob('arrays/tests/q*.py') 
         ...: grade_notebook('arrays/arrays_solution.ipynb', tests)

then the cell following the Pandas import fails, as do several others, regardless of the presence of %matplotlib inline or the patch disabling gofer.utils.flush_inline_matplotlib_plots.

[jkuruzovich]

I also wasn't able to just do the matplotlib solution. We have a forked version were I've essentially commented out the code that is causing a problem. I've been using this forked version without any issues for a few assignments, and I'm not exactly sure what the offending code does.
Forked repo.

Specific code I commented out.

I'm glad to do a pull request but I imagine the code I commented out might be useful in some type of graphing context and would break someone else's work.

Anyway, you can always install from the forked repo as an alternative.
pip install git+https://github.com/rpi-techfundamentals/Gofer-Grader

[matthew-brett]

@jkuruzovich - thanks for the feedback. Sure enough, your patch does work, if I'm running inside the notebook (open notebook, kernel restart and run all). But - it doesn't seem to work from outside the notebook, as a Python script.

[matthew-brett]

And now I also find that I do not get the same error on another machine, running outside the notebook, via a script:

https://github.com/matthew-brett/gofer_pandas/blob/master/grade_nb.py

I think that I got the error, and then it resolved, on a single machine, but, because I had not recorded my exact notebook state in git, I cannot be absolutely sure. That made me wonder whether it was to do with some cache for ok grader, or the notebook.

[matthew-brett]

Sorry - I was wrong. The failures from the command line were only coincidentally related to Pandas. They were caused by the fact that the notebook code was not running in the notebook directory, but in the working directory of the grading script, so pd.read_csv was not finding the data files. I've now fixed this in #26.