Merge branch 'master' into autopulate-2.0

Author: ttngu207

.codespellrc

@@ -0,0 +1,5 @@
+[codespell]
+skip = .git,*.pdf,*.svg,*.csv,*.ipynb,*.drawio
+# Rever -- nobody knows
+# numer -- numerator variable
+ignore-words-list = rever,numer

.devcontainer/Dockerfile

@@ -4,7 +4,7 @@ FROM mcr.microsoft.com/devcontainers/python:3.7-bullseye
 RUN \
     apt update && \
     apt-get install bash-completion graphviz default-mysql-client -y && \
-    pip install flake8 black faker ipykernel nose nose-cov datajoint && \
+    pip install flake8 black faker ipykernel pytest pytest-cov nose nose-cov datajoint && \
     pip uninstall datajoint -y
 
 ENV DJ_HOST fakeservices.datajoint.io

.devcontainer/devcontainer.json

@@ -16,7 +16,7 @@
 		"ghcr.io/guiyomh/features/vim:0": {}
 	},
 	"onCreateCommand": "pip install -e .",
-	"postStartCommand": "MYSQL_VER=8.0 MINIO_VER=RELEASE.2022-08-11T04-37-28Z docker compose -f local-docker-compose.yml down && MYSQL_VER=8.0 MINIO_VER=RELEASE.2022-08-11T04-37-28Z docker compose -f local-docker-compose.yml up --build --wait",
+	"postStartCommand": "MYSQL_VER=8.0 MINIO_VER=RELEASE.2022-08-11T04-37-28Z docker compose -f local-docker-compose.yml down && docker volume prune -f && MYSQL_VER=8.0 MINIO_VER=RELEASE.2022-08-11T04-37-28Z docker compose -f local-docker-compose.yml up --build --wait",
 	"forwardPorts": [
 		80,
 		443,

.github/workflows/development.yaml

@@ -92,6 +92,17 @@ jobs:
                  --count --max-complexity=62 --max-line-length=127 --statistics
           black datajoint --check -v
           black tests --check -v
+          black tests_old --check -v
+  codespell:
+    name: Check for spelling errors
+    permissions:
+      contents: read
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: Codespell
+        uses: codespell-project/actions-codespell@v2
   publish-docs:
     if: |
       github.event_name == 'push' &&

.github/workflows/docs.yaml

@@ -0,0 +1,19 @@
+name: Manual docs release
+on:
+  workflow_dispatch:
+jobs:
+  publish-docs:
+    runs-on: ubuntu-latest
+    env:
+      DOCKER_CLIENT_TIMEOUT: "120"
+      COMPOSE_HTTP_TIMEOUT: "120"
+    steps:
+      - uses: actions/checkout@v3
+      - name: Deploy docs
+        run: |
+          export MODE=BUILD
+          export PACKAGE=datajoint
+          export UPSTREAM_REPO=https://github.com/${GITHUB_REPOSITORY}.git
+          export HOST_UID=$(id -u)
+          docker compose -f docs/docker-compose.yaml up --exit-code-from docs --build
+          git push origin gh-pages

.vscode/settings.json

@@ -11,5 +11,11 @@
     "[markdown]": {
         "editor.defaultFormatter": "disable"
     },
+    "[yaml]": {
+        "editor.defaultFormatter": "disable"
+    },
+    "[dockercompose]": {
+        "editor.defaultFormatter": "disable"
+    },
     "files.autoSave": "off"
 }

CHANGELOG.md

@@ -1,5 +1,22 @@
 ## Release notes
 
+### Upcoming
+- Added - Codespell GitHub Actions workflow
+- Added - GitHub Actions workflow to manually release docs
+- Changed - Update `datajoint/nginx` to `v0.2.6`
+- Changed - Migrate docs from `https://docs.datajoint.org/python` to `https://datajoint.com/docs/core/datajoint-python`
+
+### 0.14.1 -- Jun 02, 2023
+- Fixed - Fix altering a part table that uses the "master" keyword - PR [#991](https://github.com/datajoint/datajoint-python/pull/991)
+- Fixed - `.ipynb` output in tutorials is not visible in dark mode ([#1078](https://github.com/datajoint/datajoint-python/issues/1078)) PR [#1080](https://github.com/datajoint/datajoint-python/pull/1080)
+- Fixed - preview table font for darkmode PR [#1089](https://github.com/datajoint/datajoint-python/pull/1089)
+- Changed - Readme to update links and include example pipeline image
+- Changed - Docs to add landing page and update navigation
+- Changed - `.data` method to `.stream` in the `get()` method for S3 (external) objects PR [#1085](https://github.com/datajoint/datajoint-python/pull/1085)
+- Fixed - Docs to rename `create_virtual_module` to `VirtualModule`
+- Added - Skeleton from `datajoint-company/datajoint-docs` repository for docs migration
+- Added - Initial `pytest` for `test_connection`
+
 ### 0.14.0 -- Feb 13, 2023
 - Added - `json` data type ([#245](https://github.com/datajoint/datajoint-python/issues/245)) PR [#1051](https://github.com/datajoint/datajoint-python/pull/1051)
 - Fixed - Activating a schema requires all tables to exist even if `create_tables=False` PR [#1058](https://github.com/datajoint/datajoint-python/pull/1058)
@@ -13,14 +30,14 @@
 - Deprecated - `table._update()` PR [#1073](https://github.com/datajoint/datajoint-python/pull/1073)
 - Deprecated - old-style foreign key syntax PR [#1073](https://github.com/datajoint/datajoint-python/pull/1073)
 - Deprecated - `dj.migrate_dj011_external_blob_storage_to_dj012()` PR [#1073](https://github.com/datajoint/datajoint-python/pull/1073)
-* Added - Method to set job keys to "ignore" status - PR [#1068](https://github.com/datajoint/datajoint-python/pull/1068)
+- Added - Method to set job keys to "ignore" status - PR [#1068](https://github.com/datajoint/datajoint-python/pull/1068)
 
 ### 0.13.8 -- Sep 21, 2022
 - Added - New documentation structure based on markdown PR [#1052](https://github.com/datajoint/datajoint-python/pull/1052)
 - Fixed - Fix queries with backslashes ([#999](https://github.com/datajoint/datajoint-python/issues/999)) PR [#1052](https://github.com/datajoint/datajoint-python/pull/1052)
 
 ### 0.13.7 -- Jul 13, 2022
-- Fixed - Fix networkx incompatable change by version pinning to 2.6.3 (#1035) PR #1036
+- Fixed - Fix networkx incompatible change by version pinning to 2.6.3 (#1035) PR #1036
 - Added - Support for serializing numpy datetime64 types (#1022) PR #1036
 - Changed - Add traceback to default logging PR #1036
 
@@ -72,7 +89,7 @@
 - Fixed - `schema.list_tables()` is not topologically sorted (#838) PR #893
 - Fixed - Diagram part tables do not show proper class name (#882) PR #893
 - Fixed - Error in complex restrictions (#892) PR #893
-- Fixed - WHERE and GROUP BY clases are dropped on joins with aggregation (#898, #899) PR #893
+- Fixed - WHERE and GROUP BY classes are dropped on joins with aggregation (#898, #899) PR #893
 
 ### 0.13.0 -- Mar 24, 2021
 - Re-implement query transpilation into SQL, fixing issues (#386, #449, #450, #484, #558). PR #754

LNX-docker-compose.yml

@@ -44,7 +44,7 @@ services:
       interval: 15s
   fakeservices.datajoint.io:
     <<: *net
-    image: datajoint/nginx:v0.2.4
+    image: datajoint/nginx:v0.2.6
     environment:
       - ADD_db_TYPE=DATABASE
       - ADD_db_ENDPOINT=db:3306
@@ -89,7 +89,8 @@ services:
         pip install --user nose nose-cov
         pip install -e .
         pip list --format=freeze | grep datajoint
-        nosetests -vsw tests --with-coverage --cover-package=datajoint
+        pytest -sv --cov-report term-missing --cov=datajoint tests
+        nosetests -vsw tests_old --with-coverage --cover-package=datajoint
     # ports:
     #   - "8888:8888"
     user: ${HOST_UID}:anaconda

README.md

@@ -1,33 +1,50 @@
 [![DOI](https://zenodo.org/badge/16774/datajoint/datajoint-python.svg)](https://zenodo.org/badge/latestdoi/16774/datajoint/datajoint-python)
-[![Build Status](https://travis-ci.org/datajoint/datajoint-python.svg?branch=master)](https://travis-ci.org/datajoint/datajoint-python)
 [![Coverage Status](https://coveralls.io/repos/datajoint/datajoint-python/badge.svg?branch=master&service=github)](https://coveralls.io/github/datajoint/datajoint-python?branch=master)
 [![PyPI version](https://badge.fury.io/py/datajoint.svg)](http://badge.fury.io/py/datajoint)
-[![Requirements Status](https://requires.io/github/datajoint/datajoint-python/requirements.svg?branch=master)](https://requires.io/github/datajoint/datajoint-python/requirements/?branch=master)
 [![Slack](https://img.shields.io/badge/slack-chat-green.svg)](https://datajoint.slack.com/)
 
 # Welcome to DataJoint for Python!
 
-DataJoint for Python is a framework for scientific workflow management based on relational principles. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, computing, and querying data.
+DataJoint for Python is a framework for scientific workflow management based on 
+relational principles. DataJoint is built on the foundation of the relational data 
+model and prescribes a consistent method for organizing, populating, computing, and 
+querying data.
 
-DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at Baylor College of Medicine for the distributed processing and management of large volumes of data streaming from regular experiments. Starting in 2011, DataJoint has been available as an open-source project adopted by other labs and improved through contributions from several developers.
-Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com).
+DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at 
+Baylor College of Medicine for the distributed processing and management of large 
+volumes of data streaming from regular experiments. Starting in 2011, DataJoint has 
+been available as an open-source project adopted by other labs and improved through 
+contributions from several developers.
+Presently, the primary developer of DataJoint open-source software is the company 
+DataJoint (https://datajoint.com).
 
-- [Getting Started](https://datajoint.com/docs/core/datajoint-python/latest/getting-started/)
-- [DataJoint Elements](https://datajoint.com/docs/elements/) - Catalog of example pipelines
-- [DataJoint CodeBook](https://codebook.datajoint.io) - Interactive online tutorials
-- Contribute
+## Data Pipeline Example
 
-  - [Development Environment](https://datajoint.com/docs/core/datajoint-python/latest/develop/)
-  - [Guidelines](https://datajoint.com/docs/community/contribute/)
+![pipeline](https://raw.githubusercontent.com/datajoint/datajoint-python/master/images/pipeline.png)
+
+[Yatsenko et al., bioRxiv 2021](https://doi.org/10.1101/2021.03.30.437358)
+
+## Getting Started
+
+- Install with Conda
 
-- Legacy Resources (To be replaced by above)
-  - [Documentation](https://docs.datajoint.org)
-  - [Tutorials](https://tutorials.datajoint.org)
+     ```bash
+     conda install -c conda-forge datajoint
+     ```
 
-## Citation
+- Install with pip
 
-- If your work uses DataJoint for Python, please cite the following Research Resource Identifier (RRID) and manuscript.
+     ```bash
+     pip install datajoint
+     ```
 
-- DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for Python (version `<Enter version number>`)
+- [Documentation & Tutorials](https://datajoint.com/docs/core/datajoint-python/)
+
+- [Interactive Tutorials](https://github.com/datajoint/datajoint-tutorials) on GitHub Codespaces
+
+- [DataJoint Elements](https://datajoint.com/docs/elements/) - Catalog of example pipelines for neuroscience experiments
+
+- Contribute
+  - [Development Environment](https://datajoint.com/docs/core/datajoint-python/latest/develop/)
 
-- Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
+  - [Guidelines](https://datajoint.com/docs/about/contribute/)

datajoint/__init__.py

@@ -1,5 +1,5 @@
 """
-DataJoint for Python is a framework for building data piplines using MySQL databases
+DataJoint for Python is a framework for building data pipelines using MySQL databases
 to represent pipeline structure and bulk storage systems for large objects.
 DataJoint is built on the foundation of the relational data model and prescribes a
 consistent method for organizing, populating, and querying data.