Merge pull request #131 from ilyak93/running-on-docker

Instructions for running it on Docker

Author: deqings

Docker/Dockerfile

@@ -0,0 +1,37 @@
+FROM nvidia/cuda:8.0-cudnn7-devel-ubuntu16.04
+
+RUN apt-get update && apt-get install -y \
+    rsync \
+    htop \
+    git \
+    openssh-server \
+    nano \
+    cmake \
+    python-opencv \
+    python2.7 \
+    unzip \
+    time
+
+
+RUN apt-get -qq -y install curl bzip2 && \
+    curl -sSL https://repo.continuum.io/miniconda/Miniconda2-4.6.14-Linux-x86_64.sh -o /tmp/miniconda.sh && \
+    bash /tmp/miniconda.sh -bfp /usr/local && \
+    rm -rf /tmp/miniconda.sh && \
+    conda install -y python=2 && \
+    conda update conda && \
+    apt-get -qq -y autoremove && \
+    apt-get autoclean && \
+    rm -rf /var/lib/apt/lists/* /var/log/dpkg.log && \
+    conda clean --all --yes
+    
+
+# install external packages 
+ADD torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl
+RUN pip install torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl
+RUN pip install torchvision==0.2.1 visdom==0.1.8.8 dominate==2.3.5 opencv-python==4.1.0.25 cffi==1.12.2
+
+ADD external_packages external_packages
+
+ENV NVIDIA_VISIBLE_DEVICES all
+ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
+

Docker/README.md

@@ -0,0 +1,105 @@
+# Running PWC-Net on Docker and working with a remote interpreter in PyCharm on Linux:
+
+### Worked for me on the following system:
+
+Ubuntu 20.04.4 LTS
+
+docker 20.10.17, build 100c701
+
+nvidia-docker 2.11.0-1
+
+GPU: GeForce RTX 2080 Ti
+
+ 
+### Full installation and and settings configuration instructions:
+
+Step 1: Look at [NVIDIA official guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) for installing `nvdia-docker2` (which include instructions for installing docker)
+
+make sure [step 3](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#step-3-testing-the-installation) of it runs as expected. 
+
+Step 2: [Download](https://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl) the appropriate torch version for this project.
+
+Step 3: Put the `torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl`, the Dockerfile and the external packages folder in the same folder anywhere you like.
+
+Step 4: got to the folder where your Dockerfile located through the terminal and run `docker build -t choose_any_tag_name .`
+
+Now you should be able to see your docker img tag in Docker in your PyCharm services:
+
+![image](https://user-images.githubusercontent.com/50303550/185483233-b77c45b1-75a9-404d-8fdb-c4a7aad2f72f.png)
+
+Step 5: As another sainity check, `Create Container` (right mouse click on the img tag) with the following run options arguments `--gpus=all --runtime=nvidia` (click on `Modify Options`, then on `Run Options`):
+
+![image](https://user-images.githubusercontent.com/50303550/185483940-f272742a-b15d-44bc-aa1d-a222b9d1e433.png)
+
+You may set the container name and the Docker name as you desire.
+
+
+Step 6: Create a project in PyCharm and put all the files from the repository in there except the `external packages`, which you should take from this folder.
+
+Step 7: in file `models/PWC-Net.py` at the head of it, add:
+
+`import os ;
+os.system("bash /external_packages/correlation-pytorch-master/make_cuda.sh")
+`
+
+Step 7: Go to Python Interpreter window in PyCharm and click on `Add Interpreter` and choose `On Docker`:
+
+![image](https://user-images.githubusercontent.com/50303550/185485149-87092a37-5d2c-48c1-a4bc-57c63412666f.png)
+
+
+ choose option `pull` and write your image tag and then add `:latest` (it should pop up from the list as you can see below) and click `next`:
+
+![image](https://user-images.githubusercontent.com/50303550/185485744-6e3b9d5b-ff78-4c78-9c4a-c35b90e4d3af.png)
+
+it should end with `Introspection completed ; Removing introspection container`. click again `next`.
+
+
+choose `System Enterpreter`, then click on the `...` and enter `/usr/local/bin/python` (this should be the path on your docker to the python interpreter).
+Notice that you didn't choose it yet, just click on the toggle arrow and click on the path you've just added.
+
+
+### You are done and you should see how the interpreter now updates in libraries and in a couple of minutes it is ready for running and debugging. 
+
+If everything worked, the first thing you would see each time your run the console is how the external packages are compiled with nvcc and the correlation layer is installed. 
+Notice this must happen each run because docker running a clean version of the container with everything installed from your build command in step 4.
+
+For compiling the external_packages on the docker in file `make_cuda.sh` I changed the setup command to `pip install .` and also set the absolute path from `~` directory on the docker.
+also in `external_packages/correlation-pytorch-master/correlation-pytorch/correlation_package/_ext/corr/__init__.py` 
+
+I modificated the import to be:
+
+`from torch.utils.ffi import _wrap_function ;
+import imp ;
+_corr = imp.load_dynamic('_corr','/usr/local/lib/python2.7/site-packages/correlation_package/_ext/corr/_corr.so') ;
+from _corr import lib as _lib, ffi as _ffi` ;
+
+eventually that how it worked for me. 
+
+
+# Super important note:
+
+notice the authors used a GPU of Pascal series for which CUDNN of particular versions only suitable and there for CUDA and torch.
+My GPU wasn't compatible, because for torch 0.2.0, CUDA8.0 and compatibele CUDNN a GPU with compute capability of 6.x is needed.
+But some how, except very slow few first GPU commands such as `.cuda()` it worked. 
+
+If you want to be able to run it without such side-effects, you will probably have to run on it on some older GPU with compute capability of 6.x, 
+and the sure solution is to run it with a similiar GPU version to which the authors mention in the paper. 
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

Docker/external_packages/correlation-pytorch-master/.make_cuda.sh.swp

@@ -0,0 +1,6 @@
+This directory contains eggs that were downloaded by setuptools to build, test, and run plug-ins.
+
+This directory caches those eggs to prevent repeated downloads.
+
+However, it is safe to delete this directory.
+

Docker/external_packages/correlation-pytorch-master/correlation-pytorch/.eggs/README.txt

@@ -0,0 +1,26 @@
+
+Except when otherwise stated (look for LICENSE files in directories or
+information at the beginning of each file) all software and
+documentation is licensed as follows: 
+
+    The MIT License
+
+    Permission is hereby granted, free of charge, to any person 
+    obtaining a copy of this software and associated documentation 
+    files (the "Software"), to deal in the Software without 
+    restriction, including without limitation the rights to use, 
+    copy, modify, merge, publish, distribute, sublicense, and/or 
+    sell copies of the Software, and to permit persons to whom the 
+    Software is furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included 
+    in all copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS 
+    OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 
+    THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 
+    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER 
+    DEALINGS IN THE SOFTWARE.
+

Docker/external_packages/correlation-pytorch-master/correlation-pytorch/.eggs/cffi-1.15.1-py2.7-linux-x86_64.egg/EGG-INFO/LICENSE

@@ -0,0 +1,35 @@
+Metadata-Version: 2.1
+Name: cffi
+Version: 1.15.1
+Summary: Foreign Function Interface for Python calling C code.
+Home-page: http://cffi.readthedocs.org
+Author: Armin Rigo, Maciej Fijalkowski
+Author-email: python-cffi@googlegroups.com
+License: MIT
+Platform: UNKNOWN
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 2
+Classifier: Programming Language :: Python :: 2.7
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Classifier: License :: OSI Approved :: MIT License
+Requires-Dist: pycparser
+
+CFFI
+====
+
+Foreign Function Interface for Python calling C code.
+Please see the `Documentation <http://cffi.readthedocs.org/>`_.
+
+Contact
+-------
+
+`Mailing list <https://groups.google.com/forum/#!forum/python-cffi>`_
+
+

Docker/external_packages/correlation-pytorch-master/correlation-pytorch/.eggs/cffi-1.15.1-py2.7-linux-x86_64.egg/EGG-INFO/PKG-INFO

@@ -0,0 +1,27 @@
+_cffi_backend.so,sha256=8YrQLM1nwNmAVj9VdUqADK9LBgSuXOnh-oL3B8b4nMg,847804
+cffi-1.15.1.dist-info/top_level.txt,sha256=rE7WR3rZfNKxWI9-jn6hsHCAl7MDkB-FmuQbxWjFehQ,19
+cffi-1.15.1.dist-info/entry_points.txt,sha256=Q9f5C9IpjYxo0d2PK9eUcnkgxHc9pHWwjEMaANPKNCI,76
+cffi-1.15.1.dist-info/WHEEL,sha256=xIMo9FffJ9e90LlGcE7k4UedKvWdvgFuFEg1EAZP6Xc,110
+cffi-1.15.1.dist-info/LICENSE,sha256=BLgPWwd7vtaICM_rreteNSPyqMmpZJXFh72W3x6sKjM,1294
+cffi-1.15.1.dist-info/METADATA,sha256=q6tBTBdFHdedHoe-W_-D9ioSiZWZaEQ4MJGUm-HxNOM,1141
+cffi-1.15.1.dist-info/RECORD,,
+cffi/vengine_gen.py,sha256=5dX7s1DU6pTBOMI6oTVn_8Bnmru_lj932B6b4v29Hlg,26684
+cffi/backend_ctypes.py,sha256=h5ZIzLc6BFVXnGyc9xPqZWUS7qGy7yFSDqXe68Sa8z4,42454
+cffi/cparser.py,sha256=rO_1pELRw1gI1DE1m4gi2ik5JMfpxouAACLXpRPlVEA,44231
+cffi/_cffi_errors.h,sha256=zQXt7uR_m8gUW-fI2hJg0KoSkJFwXv8RGUkEDZ177dQ,3908
+cffi/api.py,sha256=yxJalIePbr1mz_WxAHokSwyP5CVYde44m-nolHnbJNo,42064
+cffi/_embedding.h,sha256=9tnjF44QRobR8z0FGqAmAZY-wMSBOae1SUPqHccowqc,17680
+cffi/pkgconfig.py,sha256=LP1w7vmWvmKwyqLaU1Z243FOWGNQMrgMUZrvgFuOlco,4374
+cffi/commontypes.py,sha256=QS4uxCDI7JhtTyjh1hlnCA-gynmaszWxJaRRLGkJa1A,2689
+cffi/model.py,sha256=_GH_UF1Rn9vC4AvmgJm6qj7RUXXG3eqKPc8bPxxyBKE,21768
+cffi/_cffi_include.h,sha256=tKnA1rdSoPHp23FnDL1mDGwFo-Uj6fXfA6vA6kcoEUc,14800
+cffi/parse_c_type.h,sha256=OdwQfwM9ktq6vlCB43exFQmxDBtj2MBNdK8LYl15tjw,5976
+cffi/lock.py,sha256=l9TTdwMIMpi6jDkJGnQgE9cvTIR7CAntIJr8EGHt3pY,747
+cffi/recompiler.py,sha256=YgVYTh2CrXIobo-vMk7_K9mwAXdd_LqB4-IbYABQ488,64598
+cffi/vengine_cpy.py,sha256=YglN8YS-UaHEv2k2cxgotNWE87dHX20-68EyKoiKUYA,43320
+cffi/ffiplatform.py,sha256=HMXqR8ks2wtdsNxGaWpQ_PyqIvtiuos_vf1qKCy-cwg,4046
+cffi/setuptools_ext.py,sha256=RUR17N5f8gpiQBBlXL34P9FtOu1mhHIaAf3WJlg5S4I,8931
+cffi/cffi_opcode.py,sha256=v9RdD_ovA8rCtqsC95Ivki5V667rAOhGgs3fb2q9xpM,5724
+cffi/error.py,sha256=v6xTiS4U0kvDcy4h_BDRo5v39ZQuj-IMRYLv5ETddZs,877
+cffi/__init__.py,sha256=6xB_tafGvhhM5Xvj0Ova3oPC2SEhVlLTEObVLnazeiM,513
+cffi/verifier.py,sha256=ESwuXWXtXrKEagCKveLRDjFzLNCyaKdqAgAlKREcyhY,11253

Docker/external_packages/correlation-pytorch-master/correlation-pytorch/.eggs/cffi-1.15.1-py2.7-linux-x86_64.egg/EGG-INFO/RECORD

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: bdist_wheel (0.36.2)
+Root-Is-Purelib: false
+Tag: cp27-cp27mu-manylinux1_x86_64
+

Docker/external_packages/correlation-pytorch-master/correlation-pytorch/.eggs/cffi-1.15.1-py2.7-linux-x86_64.egg/EGG-INFO/WHEEL

@@ -0,0 +1,3 @@
+[distutils.setup_keywords]
+cffi_modules = cffi.setuptools_ext:cffi_modules
+

Docker/external_packages/correlation-pytorch-master/correlation-pytorch/.eggs/cffi-1.15.1-py2.7-linux-x86_64.egg/EGG-INFO/entry_points.txt

@@ -0,0 +1 @@
+pycparser