Commit reports for 0.1.1

[vmarkovtsev]

We don't have them for the recently released 0.1.1

[zurk]

I add for quality-report but I miss logs for noisy.
Well, I have it but there is only

INFO:quality_report_noisy:Repositories: https://github.com/warenlg/axios
https://github.com/warenlg/jquery
INFO:quality_report_noisy:Fetching https://github.com/warenlg/axios
DEBUG:quality_report_noisy:Running: git clone --single-branch --branch master https://github.com/warenlg/axios /tmp/tmpus1xiwzs/axios
Cloning into '/tmp/tmpus1xiwzs/axios'...
DEBUG:quality_report_noisy:Running: git clone --single-branch --branch style-noise-1-per-file https://github.com/warenlg/axios /tmp/tmpus1xiwzs/axios_noisy
Cloning into '/tmp/tmpus1xiwzs/axios_noisy'...

inside.

I think logging was broken. If we really need it, we can fix and restore it for v010.
@vmarkovtsev should I do it?

[vmarkovtsev]

Yeah, but this is the issue about 0.1.1 reports...

Regarding logs in 0.1.0, ok.

[zurk]

Ok, it was rush release and there were some problems.
To generate reports we should fix #663 first.
As I checked several days ago I have the same issue for some repos for a quality report. So we fail to generate reports now.

[m09]

Should be unblocked after the merge of #740.

[m09]

Reports are running for 0.1.1 in the container report_0.1.1. Below I give the three scripts I used to install the old version and cherry-pick some commits.

Outer script

docker network create --driver bridge report_0.1.1
docker run --network report_0.1.1 -d --rm --name report_0.1.1_bblfshd --privileged bblfsh/bblfshd\:v2.11.0 --log-level DEBUG
docker exec report_0.1.1_bblfshd bblfshctl driver install javascript docker://bblfsh/javascript-driver:v1.2.0
docker run --network report_0.1.1 -e LOOKOUT_BBLFSHD=ipv4://report_0.1.1_bblfshd:9432 --name report_0.1.1 --entrypoint /bin/bash -it --rm -v /storage/hugo:/storage -v /home/hugo:/home ubuntu:18.04

Inner script

export BROWSER=/browser
export LC_ALL=en_US.UTF-8
apt update
apt-get install -y --no-install-suggests --no-install-recommends \
        libgomp1 libsnappy1v5 libsnappy-dev gcc g++ make git \
        python3-dev libxml2 libxml2-dev zlib1g-dev ca-certificates \
        locales libxml2 libxml2-dev gcc g++ wget python3 python3-dev \
        python3-distutils emacs
echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && locale-gen
wget -O - https://bootstrap.pypa.io/get-pip.py | python3
git clone https://github.com/src-d/style-analyzer.git
cd style-analyzer/
git checkout 45d23c8d8a9369b5445a8586e8a4f16a292c9270
git config --global user.email "142691+m09@users.noreply.github.com"
git config --global user.name m09
git cherry-pick 6a46f26d5fe32040f505df0f1b7f16e8d6a6a26d
git cherry-pick 4b10fb4def7602ca50d4fc19a634756b36820930
patch lookout/style/format/uast_stability_checker.py /home/cherry-pick-0.1.1.patch
git add lookout
git commit --no-edit
pip install -e '.[all_cpu]'

Content of cherry-pick-0.1.1.patch

73,79d72
< <<<<<<< HEAD
<             uast, errors = parse_uast(stub, content[start:end], filename="",
<                                       language=self._feature_extractor.language)
<             if not errors:
<                 result = uast, start, end
<                 break
< =======
81c74
<                 uast, errors = parse_uast(stub, content[start:end], filename="", unicode=True,
---
>                 uast, errors = parse_uast(stub, content[start:end], filename="",
86d78
< >>>>>>> 4b10fb4... Fix UASTFixer reliance on non-existing invariant of parent covers children

[zurk]

And I assume to run report generation you run something like

make report-release

?

[m09]

with a few env variables set, yes

[m09]

To document, last version ran for 0.1.1 report before I got my access to science-3 nuked was:

Outer script

docker network create --driver bridge report_0.1.1
docker run --network report_0.1.1 -d --rm --name report_0.1.1_bblfshd --privileged bblfsh/bblfshd\:v2.11.0 --log-level DEBUG
docker exec report_0.1.1_bblfshd bblfshctl driver install javascript docker://bblfsh/javascript-driver:v1.2.0
docker run --network report_0.1.1 -e LOOKOUT_BBLFSHD=ipv4://report_0.1.1_bblfshd:9432 --name report_0.1.1 --entrypoint /bin/bash -it --rm -v /storage/hugo:/storage -v /home/hugo:/home ubuntu:18.04

Inner script

export BROWSER=/browser
export LC_ALL=en_US.UTF-8
apt update
apt-get install -y --no-install-suggests --no-install-recommends \
        libgomp1 libsnappy1v5 libsnappy-dev gcc g++ make git \
        python3-dev libxml2 libxml2-dev zlib1g-dev ca-certificates \
        locales libxml2 libxml2-dev gcc g++ wget python3 python3-dev \
        python3-distutils emacs
echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && locale-gen
wget -O - https://bootstrap.pypa.io/get-pip.py | python3
git clone https://github.com/src-d/style-analyzer.git
cd style-analyzer/
git checkout 45d23c8d8a9369b5445a8586e8a4f16a292c9270
git config --global user.email "142691+m09@users.noreply.github.com"
git config --global user.name m09
git cherry-pick 6a46f26d5fe32040f505df0f1b7f16e8d6a6a26d
git cherry-pick 4b10fb4def7602ca50d4fc19a634756b36820930
patch lookout/style/format/uast_stability_checker.py /home/cherry-pick-0.1.1.patch
patch lookout/style/format/benchmarks/quality_report.py /home/quality-report-0.1.1.patch
git add lookout
git commit --no-edit
pip install -e '.[all_cpu]'

Content of cherry-pick-0.1.1.patch

73,79d72
< <<<<<<< HEAD
<             uast, errors = parse_uast(stub, content[start:end], filename="",
<                                       language=self._feature_extractor.language)
<             if not errors:
<                 result = uast, start, end
<                 break
< =======
81c74
<                 uast, errors = parse_uast(stub, content[start:end], filename="", unicode=True,
---
>                 uast, errors = parse_uast(stub, content[start:end], filename="",
86d78
< >>>>>>> 4b10fb4... Fix UASTFixer reliance on non-existing invariant of parent covers children

Content of quality-report-0.1.1.patch

433c433
<             with io.StringIO() as output:
---
>             with io.StringIO() as output_io:
469,470c469,470
<                       file=output)
<                 summary = output.getvalue()
---
>                       file=output_io)
>                 summary = output_io.getvalue()

[vmarkovtsev]

@zurk @m09 Any updates?

[zurk]

@m09 can I ask you to check report at our new cluster? A Directory with all old from science-3 should be there.

If there are no reports for some reason or you are not sure they are correct, tell me, I will repeat experiment one more time.