diff --git a/docs/setup-guide/multi-node-dev-cluster.md b/docs/setup-guide/multi-node-dev-cluster.md
deleted file mode 100644
index c3d0922c48d..00000000000
--- a/docs/setup-guide/multi-node-dev-cluster.md
+++ /dev/null
@@ -1,100 +0,0 @@
-# How to Setup a Dev Cluster of multi-nodes
-
-It may be desired to setup a dev cluster having one or more worker nodes in order to play with comprehensive features of Arktos. The simple and easy-to-use arktos-up.sh feels short in this case; this doc describes the minimum effort to run a cluster having 1 master node and extra worker nodes joining after the master is up.
-
-This doc, as at the moment written, does not mandate which cni plugin to be used; it is up to the user. We have verified that Flannel works well with multi-tenancy Arktos cluster; the instructions laid out here are based on our experience with Flannel at GCP env.
-
-Assuming you have got the Arktos repo downloaded to your local disk and the current folder is at the root of the repo,
-
-0. Make sure the following directories are empty. If not, clean them up.
-```
-/opt/cni/bin/
-/etc/cni/net.d/
-```
-
-1. bootstrap the cluster by starting the master node (no CNI plgin at first)
-```bash
-export ARKTOS_NO_CNI_PREINSTALLED=y
-./hack/arktos-up.sh
-```
-
-Note: arktos-up.sh should be stuck in "Waiting for node ready at api server" messages. Don't worry, the apiserver is already up at this point, just the master node status is not "Ready" as we have not installed the network plugin yet. 
-
-2. Open another terminal to the master node to install CNI plugin
-```bash
-./cluster/kubectl.sh apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
-```
-
-After that, "arktos-up.sh" should get rid of "Waiting for node ready at api server" messages and be successfully started.
-
-3. In the lab machine to be added as a worker node, ensure following worker secret files copied from the master node:
-If the worker node is a GCP instance  
-```bash
-mkdir -p /tmp/arktos
-gcloud compute scp <master-node-instance>:/var/run/kubernetes/kubelet.kubeconfig /tmp/arktos/
-gcloud compute scp <master-node-instance>:/var/run/kubernetes/client-ca.crt /tmp/arktos/
-```
-
-If the worker node is an AWS EC2, you can download files /var/run/kubernetes/kubelet.kubeconfig and /var/run/kubernetes/client-ca.crt from the master node, and then upload to /tmp/arktos/ in the worker node.
-
-At the moment this doc is written, kube-proxy does not support multi-tenancy yet, and it won't be deployed in the cluster. After kube-proxy issue has been fixed (we will allocate resource to cope with it soon), probably we will also need to copy over kube-proxy related secret and artifacts.
-
-As a temporary measure to accommodate Flannel pods to access api server through the kubernetes service IP (the notorious 10.0.0.1) at the absense of kube-proxy, we need to place a bit trick by running below command __at the worker node__, assuming the kupe-apiserver is listening at https://10.138.0.19:6443 (please substitute with the proper value of your own cluster):
-```bash
-sudo iptables -t nat -A OUTPUT -p tcp -d 10.0.0.1 --dport 443 -j DNAT --to-destination 10.138.0.19:6443
-```
-
-NOTE: you need to re-run the above command once you restart the machine.
-
-Please be advised that this is a temporary quirk only; after we have the full service support by proper kube-proxy or other means, we don't need it any more.
-
-4. start the worker node and register into cluster
-
-First Make sure the following directories in the worker node are empty. If not, clean them up.
-```
-/opt/cni/bin/
-/etc/cni/net.d/
-```
-
-Then at worker node, run following commands:
-```bash
-export ARKTOS_NO_CNI_PREINSTALLED=y
-export KUBELET_IP=<worker-ip>
-./hack/arktos-worker-up.sh
-```
-
-After the script returns, go to master node terminal and run command "[arktos_repo]/cluster/kubectl.sh get nodes", you should see the work node is displayed and its status should be "Ready".
-
-5. label worker node as vm runtime capable (optional)
-
-If you would like to allow this work node to run VM-based pods, please run below command at the master console:
-```bash
-./cluster/kubectl.sh label node <worker-node-name> extraRuntime=virtlet
-```
-The work-node-name is the new worker just added; its name can be found by ```./cluster/kubectl.sh get node```.
-
-You should be able to notice that node in READY state after a while; you can run container pods now:
-```bash
-./cluster/kubectl.sh run nginx --image=nginx --replicas=2
-./cluster/kubectl.sh get pod -o wide
-```
-
-To run a small cirros VM pod, you can use below yaml content:
-```yaml
-apiVersion: v1
-kind: Pod
-metadata:
-  name: cirros-vm
-spec:
-  virtualMachine:
-    name: vm
-    keyPairName: "foo"
-    image: download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
-    imagePullPolicy: IfNotPresent
-```
-
-# How to set up multiple partitioned apiservers
-
-It may be desired to setup a cluster having 2 or 3 apiservers in order to play with the scalability features of Arktos. It is simple and easy to implement it using arktos-up.sh, arktos-apiserver-partition.sh, and install-etcd.sh. The  doc describes the minimum effort to run a cluster having 2 apiservers (1 apiserver first, plus a apiserver who joins later).
-
-Instructions are in [API Server Partition](arktos-apiserver-partition.md)
diff --git a/docs/setup-guide/multi-node-dev-scale-up-cluster-with-Mizar.md b/docs/setup-guide/multi-node-dev-scale-up-cluster-with-Mizar.md
new file mode 100644
index 00000000000..1e6dced0e44
--- /dev/null
+++ b/docs/setup-guide/multi-node-dev-scale-up-cluster-with-Mizar.md
@@ -0,0 +1,381 @@
+# How to Setup a Dev Cluster of multi-node on AWS EC2 instance running Ubuntu 18.04, 20.04 x86 
+
+0. Pre-requisite - setup local development environment with Mizar
+
+0.1) Create AWS EC2 instance 
+```
+     - OS: Ubuntu 18.04 and Ubuntu 20.04
+     - Instance Type: t2.2xlarge
+     - Storage Size: 128GB or more
+```
+
+0.2) Follow up the step 1 in following procedure to upgrade kernel to 5.6.0-rc2
+
+```
+   https://github.com/Click2Cloud-Centaurus/arktos/blob/default-cni-mizar/docs/setup-guide/arktos-with-mizar-cni.md
+```
+
+```bash
+   uname -a
+   wget https://raw.githubusercontent.com/CentaurusInfra/mizar/dev-next/kernelupdate.sh
+   sudo bash kernelupdate.sh
+   uname -a
+```  
+
+0.3) Follow up the step 2 in following procedure to clone the Arktos repository and install the required dependencies
+```
+   https://github.com/Click2Cloud-Centaurus/arktos/blob/default-cni-mizar/docs/setup-guide/arktos-with-mizar-cni.md
+```
+
+```bash
+   git clone https://github.com/CentaurusInfra/arktos.git ~/go/src/k8s.io/arktos 
+   sudo bash $HOME/go/src/k8s.io/arktos/hack/setup-dev-node.sh
+   echo export PATH=$PATH:/usr/local/go/bin\ >> ~/.profile
+   echo cd \$HOME/go/src/k8s.io/arktos >> ~/.profile
+   source ~/.profile
+```
+
+0.4) Start Arktos cluster in default mode (without mizar)
+```bash
+   sudo chmod o+rw /var/run/docker.sock; ls -al /var/run/docker.sock
+   make clean
+   ./hack/arktos-up.sh
+```
+
+OR
+
+```bash
+   sudo groupadd docker
+   sudo usermode -aG docker $USER
+   make clean
+   ./hack/arktos-up.sh
+```
+
+   In another window:
+```bash
+   ./hack/arktos-up.sh get all --all-namespaces
+   ./hack/arktos-up.sh get all --all-namespaces -AT
+   ./cluster/kubectl.sh run nginx --image=nginx --replicas=2
+   ./cluster/kubectl.sh get pods -o wide
+   ./cluster/kubectl.sh exec -ti nginx-5d79788459-gz74w -n default -- /bin/bash
+   # curl 10.88.0.5  (You will see "Welcome to nginx!" title)
+   # curl 10.88.0.4  (You will see "Welcome to nginx!" title) 
+   # curl 10.88.0.3  (You will see error "# curl 10.88.0.4  (You will see "Welcome to nginx!" title)")
+```
+
+
+
+0.5) Apply PR1114 to give "Support for Mizar CNI in arktos-up"
+     PR1114: https://github.com/CentaurusInfra/arktos/pull/1114
+
+```bash
+   cd ~/go/src/k8s.io/arktos
+   git checkout master
+   git fetch origin pull/1114/head:pr1114
+   git checkout -b CarlXie_20211101-Mizar
+   git rebase pr1114
+   git log
+   git show c4a0ff73ced143fc954c6e34670dd10780f1eb5a
+```
+
+0.6) Start Arktos cluster with Mizar
+     Note: You will see the warning "Waiting for node ready at api server" for 5 minutes more because it takes time for Mizar to compile codes and Arktos cluster will be up
+
+```bash
+   sudo chmod o+rw /var/run/docker.sock; ls -al /var/run/docker.sock  (if reboot machine)
+   sudo rm -rf /opt/cni/bin/*; sudo ls -alg /opt/cni/bin/  (keep clean)
+   sudo rm -rf /etc/cni/net.d/*; sudo ls -alg /etc/cni/net.d/  (keep clean)
+   make clean
+   CNIPLUGIN=mizar ./hack/arktos-up.sh
+```
+
+0.7) Follow up the following procedure to verify whether Mizar CRDs (vpcs,subnets,droplets,bouncers,dividers and endpoints) are all in "Provisioned" states. If not, reboot machine and go back to above step 0.6) until all Mizar CRDs are in "Provisioned" states
+
+```
+     https://github.com/CentaurusInfra/mizar/wiki/Mizar-Cluster-Health-Criteria
+```
+
+```bash
+   ./cluster/kubectl.sh get crds
+```
+```
+    NAME                            AGE
+    bouncers.mizar.com              10m
+    dividers.mizar.com              10m
+    droplets.mizar.com              10m
+    endpoints.mizar.com             10m
+    networks.arktos.futurewei.com   10m
+    subnets.mizar.com               10m
+    vpcs.mizar.com                  10m
+```
+```bash
+   ./cluster/kubectl.sh get crds
+   ./cluster/kubectl.sh get vpcs
+   ./cluster/kubectl.sh get subnets
+   ./cluster/kubectl.sh get droplets
+   ./cluster/kubectl.sh get bouncers
+   ./cluster/kubectl.sh get dividers
+```
+```bash
+   ./cluster/kubectl.sh get endpoints.mizar.com
+   ./cluster/kubectl.sh get networks
+```
+
+0.8) Verify whether the pods of Mizar and Mizar Operator are all in "Running" states
+```bash
+   ./cluster/kubectl.sh get pods -o wide
+   ./cluster/kubectl.sh get pods -o wide -AT
+```
+
+1. Create two network pods and test each other using ping command to test - 1. Basic pod connectivity: pods in “system” tenant should be able to communicate with each other 
+```
+     https://github.com/CentaurusInfra/mizar/wiki/Mizar-Cluster-Health-Criteria
+```
+
+netpod-single-node.yaml
+```
+apiVersion: v1
+kind: Pod
+metadata:
+  name: netpod1
+  labels:
+    app: netpod
+spec:
+  restartPolicy: OnFailure
+  terminationGracePeriodSeconds: 10
+  containers:
+  - name: netctr
+    image: mizarnet/testpod
+    ports:
+    - containerPort: 9001
+      protocol: TCP
+    - containerPort: 5001
+      protocol: UDP
+    - containerPort: 7000
+      protocol: TCP
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: netpod2
+  labels:
+    app: netpod
+spec:
+  restartPolicy: OnFailure
+  terminationGracePeriodSeconds: 10
+  containers:
+  - name: netctr
+    image: mizarnet/testpod
+    ports:
+    - containerPort: 9001
+      protocol: TCP
+    - containerPort: 5001
+      protocol: UDP
+    - containerPort: 7000
+      protocol: TCP
+
+```
+```bash
+   vi ~/TMP/netpod-single-node.yaml (copy from netpod-single-node.yaml)
+   cat ~/TMP/netpod-single-node.yaml
+   ./cluster/kubectl.sh apply -f ~/TMP/netpod-single-node.yaml
+   ./cluster/kubectl.sh get pods -o wide
+   ./cluster/kubectl.sh exec -ti netpod1 -- ping -c2 20.0.0.18 (IP of pod netpod2)
+   ./cluster/kubectl.sh exec -ti netpod2 -- ping -c2 20.0.0.26 (IP of pod netpod1)
+   ./cluster/kubectl.sh exec -ti netpod1 -n default -- /bin/bash
+   # curl 20.0.0.18:7000
+   netpod2
+   # curl 20.0.0.26:7000
+   netpod1
+   # exit
+```
+
+
+2. Follow up the procedure of issue 1142 at https://github.com/CentaurusInfra/arktos/issues/1142 to test - 2.	General pod connectivity: pods in same non-system tenant should be able to communicate;
+
+2.1) Create new tenant 'mytenant'
+
+```bash
+   ./cluster/kubectl.sh create tenant mytenant
+```
+```
+   ./cluster/kubectl.sh get network -T
+   ./cluster/kubectl.sh get service --all-namespaces --tenant mytenant -o wide
+   ./cluster/kubectl.sh get pod --all-namespaces --tenant mytenant -o wide
+   ./cluster/kubectl.sh get deployment --all-namespaces  --tenant mytenant -o wide
+   ./cluster/kubectl.sh get endpoints --all-namespaces --tenant mytenant -o wide
+
+```
+
+2.2) Create the Pod yaml file by adding the tenant 'mytenant' and apply this Pod yaml file
+```bash
+   cat ~/TMP/netpod-arktos-team-single-node.yaml
+```
+```
+apiVersion: v1
+kind: Pod
+metadata:
+  name: my-netpod1
+  labels:
+    app: my-netpod
+  tenant: mytenant
+spec:
+  restartPolicy: OnFailure
+  terminationGracePeriodSeconds: 10
+  containers:
+  - name: netctr
+    image: mizarnet/testpod
+    ports:
+    - containerPort: 9001
+      protocol: TCP
+    - containerPort: 5001
+      protocol: UDP
+    - containerPort: 7000
+      protocol: TCP
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: my-netpod2
+  labels:
+    app: my-netpod
+  tenant: mytenant
+spec:
+  restartPolicy: OnFailure
+  terminationGracePeriodSeconds: 10
+  containers:
+  - name: netctr
+    image: mizarnet/testpod
+    ports:
+    - containerPort: 9001
+      protocol: TCP
+    - containerPort: 5001
+      protocol: UDP
+    - containerPort: 7000
+      protocol: TCP
+```
+```bash
+   ./cluster/kubectl.sh apply -f ~/TMP/netpod-arktos-team-single-node.yaml
+```
+
+2.3) Check whether new pods are in Running state
+
+```bash
+   ./cluster/kubectl.sh get pods -AT
+```
+```
+TENANT     NAMESPACE     NAME                               HASHKEY               READY   STATUS              RESTARTS   AGE
+mytenant   default       my-netpod1                         5743084656396193665   0/1     ContainerCreating   0          48s
+mytenant   default       my-netpod2                         1296240596527502600   0/1     ContainerCreating   0          48s
+mytenant   kube-system   coredns-default-798fbcc5f4-qsrdd   7084949829346585805   0/1     ContainerCreating   0          10m
+system     default       mizar-daemon-wfsdx                 3626414391297043142   1/1     Running             0          126m
+system     default       mizar-operator-6b78d7ffc4-kv6n9    8147745129439049838   1/1     Running             0          126m
+system     default       netpod1                            6285438668331818604   1/1     Running             0          17m
+system     default       netpod2                            996610216049115966    1/1     Running             0          17m
+system     kube-system   coredns-default-798fbcc5f4-gs92h   4660664115769723653   1/1     Running             0          126m
+system     kube-system   kube-dns-554c5866fc-4hx2m          6383051253301223595   3/3     Running             0          126m
+system     kube-system   virtlet-6bc92                      1075534439738712156   3/3     Running             0          121m
+```
+
+```bash
+   ./cluster/kubectl.sh get pods --tenant mytenant
+```
+```
+my-netpod1   5743084656396193665   0/1     ContainerCreating   0          76s
+my-netpod2   1296240596527502600   0/1     ContainerCreating   0          76s
+```
+
+```bash
+   cat /tmp/kubelet.log | grep my-netpod1 |tail -9
+```
+```
+I1101 20:48:06.397537   11072 volume_manager.go:358] Waiting for volumes to attach and mount for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)"
+I1101 20:48:06.397690   11072 volume_manager.go:391] All volumes are attached and mounted for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)"
+I1101 20:48:06.397708   11072 kuberuntime_manager.go:529] No sandbox for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)" can be found. Need to start a new one
+I1101 20:48:06.397723   11072 kuberuntime_manager.go:948] computePodActions got {KillPod:true CreateSandbox:true SandboxID: Attempt:0 NextInitContainerToStart:nil ContainersToStart:[0] ContainersToKill:map[] ContainersToUpdate:map[] ContainersToRestart:[] Hotplugs:{NICsToAttach:[] NICsToDetach:[]}} for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)"
+I1101 20:48:06.398626   11072 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-netpod1", UID:"82f46ddd-2b12-48bd-b3c3-5aaf12265fdb", APIVersion:"v1", ResourceVersion:"2948", FieldPath:"", Tenant:"mytenant"}): type: 'Warning' reason: 'GettingClusterDNS' pod: "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)". For verification - ClusterDNS IP : "10.0.0.94"
+E1101 20:48:09.424185   11072 kuberuntime_sandbox.go:86] CreatePodSandbox for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "7292679bedba779c3e93d22be35ed24bb71d6fdc9decd35810a23424b6548423": rpc error: code = DeadlineExceeded desc = Deadline Exceeded
+E1101 20:48:09.424241   11072 kuberuntime_manager.go:1024] createPodSandbox for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "7292679bedba779c3e93d22be35ed24bb71d6fdc9decd35810a23424b6548423": rpc error: code = DeadlineExceeded desc = Deadline Exceeded
+E1101 20:48:09.424310   11072 pod_workers.go:196] Error syncing pod 82f46ddd-2b12-48bd-b3c3-5aaf12265fdb ("my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)"), skipping: failed to "CreatePodSandbox" for "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)" with CreatePodSandboxError: "CreatePodSandbox for pod \"my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)\" failed: rpc error: code = Unknown desc = failed to setup network for sandbox \"7292679bedba779c3e93d22be35ed24bb71d6fdc9decd35810a23424b6548423\": rpc error: code = DeadlineExceeded desc = Deadline Exceeded"
+I1101 20:48:09.424478   11072 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-netpod1", UID:"82f46ddd-2b12-48bd-b3c3-5aaf12265fdb", APIVersion:"v1", ResourceVersion:"2948", FieldPath:"", Tenant:"mytenant"}): type: 'Warning' reason: 'FailedCreatePodSandBox' Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7292679bedba779c3e93d22be35ed24bb71d6fdc9decd35810a23424b6548423": rpc error: code = DeadlineExceeded desc = Deadline Exceeded
+```
+
+
+3. Follow up the procedure of issue 1142 at https://github.com/CentaurusInfra/arktos/issues/1142 to test - 3.	General pod isolation: a pod in one tenant may not communicate with pods in other tenants;
+
+```bash
+   Stop test when step 2 failed to test
+```
+
+4. Build two node scale-up cluster to test - 4.	Worker node joining: new worker node should be able to join cluster, and basic pod connectivity should be provided.
+
+4.1) Suppose you have a simple node cluster with Mizar and verify successfully using the proecedure at https://github.com/CentaurusInfra/mizar/wiki/Mizar-Cluster-Health-Criteria
+
+4.2) Create a new worker node by following up the step 1, 2 at the procedure at https://github.com/Click2Cloud-Centaurus/arktos/blob/default-cni-mizar/docs/setup-guide/arktos-with-mizar-cni.md
+
+4.3) Add security group rules on both nodes to ensure all traffic can go through each other
+
+4.4) Follow up the step 3, 4 at the procedure at https://github.com/q131172019/arktos/blob/CarlXie_singleNodeArktosCluster/docs/setup-guide/multi-node-dev-scale-up-cluster.md to let work node join cluster 
+
+
+4.5) Check the status of two nodes
+```bash
+./cluster/kubectl.sh get nodes
+```
+```
+NAME               STATUS   ROLES    AGE     VERSION
+ip-172-31-28-132   Ready    <none>   59m     v0.9.0
+ip-172-31-28-9     Ready    <none>   5m37s   v0.9.0
+```
+
+4.6)  Deploy Nginx application
+```bash
+./cluster/kubectl.sh run nginx --image=nginx --replicas=10
+```
+4.7) Check the status of Nginx pods
+```bash
+./cluster/kubectl.sh get pods -o wide
+```
+```
+NAME                              HASHKEY               READY   STATUS              RESTARTS   AGE     IP              NODE               NOMINATED NODE   READINESS GATES
+mizar-daemon-jm68c                8489468700754586427   1/1     Running             1          6m22s   172.31.28.9     ip-172-31-28-9     <none>           <none>
+mizar-daemon-qth7w                9137939745832437098   1/1     Running             0          59m     172.31.28.132   ip-172-31-28-132   <none>           <none>
+mizar-operator-6b78d7ffc4-fm4pp   3559156546840379852   1/1     Running             0          59m     172.31.28.132   ip-172-31-28-132   <none>           <none>
+netpod1                           6169807998527740827   1/1     Running             0          56m     20.0.0.41       ip-172-31-28-132   <none>           <none>
+netpod2                           8273861328765425857   1/1     Running             0          56m     20.0.0.18       ip-172-31-28-132   <none>           <none>
+nginx-5d79788459-48l6q            1654048144042850706   0/1     ContainerCreating   0          10s     <none>          ip-172-31-28-9     <none>           <none>
+nginx-5d79788459-5g7w5            7238259642648495854   1/1     Running             0          10s     20.0.0.37       ip-172-31-28-132   <none>           <none>
+nginx-5d79788459-5lrp9            2544086530344874677   0/1     ContainerCreating   0          10s     <none>          ip-172-31-28-9     <none>           <none>
+nginx-5d79788459-bvqmt            8343757557393359410   1/1     Running             0          10s     20.0.0.45       ip-172-31-28-132   <none>           <none>
+nginx-5d79788459-fqhhx            3882393408941073858   0/1     ContainerCreating   0          10s     <none>          ip-172-31-28-9     <none>           <none>
+nginx-5d79788459-l46z5            5947515209238197563   0/1     ContainerCreating   0          10s     <none>          ip-172-31-28-9     <none>           <none>
+nginx-5d79788459-l9m77            4466493257680056631   0/1     ContainerCreating   0          10s     <none>          ip-172-31-28-9     <none>           <none>
+nginx-5d79788459-n6v7x            6802894149274701769   0/1     ContainerCreating   0          10s     <none>          ip-172-31-28-9     <none>           <none>
+nginx-5d79788459-spvfs            1493173479578877424   0/1     ContainerCreating   0          10s     <none>          ip-172-31-28-9     <none>           <none>
+nginx-5d79788459-zf92h            7575485586534491466   0/1     ContainerCreating   0          10s     <none>          ip-172-31-28-9     <none>           <none>
+ubuntu@ip-172-31-28-132:~/go/src/k8s.io/arktos$
+```
+4.8) Check the error of containerd
+```bash
+journalctl -u containerd
+```
+```
+Nov 01 22:31:19 ip-172-31-28-9 containerd[2324]: time="2021-11-01T22:31:19.318504821Z" level=info msg="No cni config template is specified, wait for other system components to drop the config."
+Nov 01 22:31:19 ip-172-31-28-9 containerd[2324]: time="2021-11-01T22:31:19.419328912Z" level=info msg="No cni config template is specified, wait for other system components to drop the config."
+Nov 01 22:31:19 ip-172-31-28-9 containerd[2324]: time="2021-11-01T22:31:19.520129201Z" level=info msg="No cni config template is specified, wait for other system components to drop the config."
+Nov 01 22:31:19 ip-172-31-28-9 containerd[2324]: time="2021-11-01T22:31:19.721247645Z" level=info msg="No cni config template is specified, wait for other system components to drop the config."
+Nov 01 22:31:19 ip-172-31-28-9 containerd[2324]: time="2021-11-01T22:31:19.821904690Z" level=info msg="No cni config template is specified, wait for other system components to drop the config."
+Nov 01 22:31:19 ip-172-31-28-9 containerd[2324]: time="2021-11-01T22:31:19.922523863Z" level=info msg="No cni config template is specified, wait for other system components to drop the config."
+Nov 01 22:31:20 ip-172-31-28-9 containerd[2324]: time="2021-11-01T22:31:20.023186444Z" level=info msg="No cni config template is specified, wait for other system components to drop the config."
+Nov 01 22:31:20 ip-172-31-28-9 containerd[2324]: time="2021-11-01T22:31:20.123857308Z" level=info msg="No cni config template is specified, wait for other system components to drop the config."
+```
+
+4.9) Check the error in kubelet log at /tmp/kubelet.log
+```bash
+grep nginx-5d79788459-48l6q /tmp/kubelet.worker.log |tail -3
+```
+```
+E1103 23:01:50.925691    3384 kuberuntime_manager.go:1024] createPodSandbox for pod "nginx-5d79788459-48l6q_default_system(9fef41f2-b9a0-4203-a26c-a3f28a5df19f)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "b81277168b31e4d5bae52855bff3a1e7e97fa35130db727dfbf8b3c8c833bd1b": rpc error: code = DeadlineExceeded desc = Deadline Exceeded
+E1103 23:01:50.925771    3384 pod_workers.go:196] Error syncing pod 9fef41f2-b9a0-4203-a26c-a3f28a5df19f ("nginx-5d79788459-48l6q_default_system(9fef41f2-b9a0-4203-a26c-a3f28a5df19f)"), skipping: failed to "CreatePodSandbox" for "nginx-5d79788459-48l6q_default_system(9fef41f2-b9a0-4203-a26c-a3f28a5df19f)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-5d79788459-48l6q_default_system(9fef41f2-b9a0-4203-a26c-a3f28a5df19f)\" failed: rpc error: code = Unknown desc = failed to setup network for sandbox \"b81277168b31e4d5bae52855bff3a1e7e97fa35130db727dfbf8b3c8c833bd1b\": rpc error: code = DeadlineExceeded desc = Deadline Exceeded"
+I1103 23:01:50.925803    3384 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"nginx-5d79788459-48l6q", UID:"9fef41f2-b9a0-4203-a26c-a3f28a5df19f", APIVersion:"v1", ResourceVersion:"1927", FieldPath:"", Tenant:"system"}): type: 'Warning' reason: 'FailedCreatePodSandBox' Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b81277168b31e4d5bae52855bff3a1e7e97fa35130db727dfbf8b3c8c833bd1b": rpc error: code = DeadlineExceeded desc = Deadline Exceeded
+```
diff --git a/docs/setup-guide/multi-node-dev-scale-up-cluster.md b/docs/setup-guide/multi-node-dev-scale-up-cluster.md
new file mode 100644
index 00000000000..f7de55bb363
--- /dev/null
+++ b/docs/setup-guide/multi-node-dev-scale-up-cluster.md
@@ -0,0 +1,629 @@
+# How to Setup a Scale-UP Dev Cluster of multi-nodes with flannel in two modes
+
+It may be desired to setup a dev cluster having one or more worker nodes in order to play with comprehensive features of Arktos. The simple and easy-to-use arktos-up.sh feels short in this case; this doc describes the minimum effort to run a cluster having 1 master node and extra worker nodes joining after the master is up.
+
+This doc, as at the moment written, does not mandate which cni plugin to be used; it is up to the user. We have verified that Flannel works well with multi-tenancy Arktos cluster; the instructions laid out here are based on our experience with Flannel at GCP env or AWS env with Ubuntu 18.04 x86 image.
+
+Assuming you have got the Arktos repo downloaded to your local disk and the current folder (i.e ~/go/src/arktos/) is at the root of the repo, and 
+
+- Setup master node and worker nodes based on [set up developer environment](setup-dev-env.md) to install needed packages (docker, make, gcc, jq and golang).
+- And master node has been tested successfully based on [set up single node cluster](single-node-dev-scale-up-cluster.md)
+- SSH/SCP should work from worker nodes to master node in order to copy the worker secret files from the master node (in AWS, use private key file of keypair to access to master/worker nodes).
+- Needed ports are opened between master node and worker nodes
+  * Allow access to kube-api port 6443 on master node from work nodes (in AWS, add this rule into inbound rules of security group for master node)
+  * Allow access to kubelet port 10251 and kube-proxy port 10255 on worker nodes from master node(in AWS, add this rule into inbound rules of security group for worker nodes)
+- On all nodes using AWS EC2 instances, Stop Source / destination check
+  * Instance -> Networking -> Change source / destination check -> click check box at Stop to stop Source / destination check
+- On all nodes, after rebooting the permisson of others for file /var/run/docker.sock should be readable and writable.
+
+```bash
+   sudo chmod o+rw /var/run/docker.sock; sudo ls -alg /var/run/docker.sock
+```
+
+OR
+
+```bash
+   sudo groupadd docker
+   sudo usermode -aG docker $USER
+```
+
+
+Method #1 - DaemonSet Mode
+
+On master node:
+
+1.0) Make sure the following directories are empty. If not, clean them up using sudo permisson
+```bash
+   sudo ls -alg /opt/cni/bin/; sudo rm -r /opt/cni/bin/*; sudo ls -alg /opt/cni/bin/
+   sudo ls -alg /etc/cni/net.d/; sudo rm -r /etc/cni/net.d/bridge.conf;sudo rm -r /etc/cni/net.d/10-flannel.conflist sudo ls -alg /etc/cni/net.d/
+```
+
+1.1) bootstrap the cluster by starting the master node (no CNI plugin at first)
+   Run the following commands when you are first time
+```bash
+   export ARKTOS_NO_CNI_PREINSTALLED=y
+   make clean
+   ./hack/arktos-up.sh
+```
+
+   After you first run './hack/arktos-up.sh' successfully, you can use './hack/arktos-up.sh -O' intead
+```bash
+   export ARKTOS_NO_CNI_PREINSTALLED=y
+   ./hack/arktos-up.sh -O
+```
+
+   Note: arktos-up.sh should be stuck in "Waiting for node ready at api server" messages. Don't worry, the apiserver is already up at this point, just the master node status is not "Ready"  and pod/kube-dns in name space 'kube-system' is not in state of 'Running' as we have not installed the network plugin yet. 
+
+1.2) Open another terminal on the master node to install CNI plugin of flannel and check the status of node
+```bash
+   ./cluster/kubectl.sh apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
+```
+
+   After that, "arktos-up.sh" should get rid of "Waiting for node ready at api server" messages and be successfully started. You can check the status of all resources including pod/kube-dns and flannel network
+
+```bash
+   ./cluster/kubectl.sh get all --all-namespaces
+   ./cluster/kubectl.sh get pods --all-namespaces
+   ps -ef |grep flannel |grep -v grep
+   sudo cat /etc/cni/net.d/10-flannel.conflist
+   cat /tmp/flanneld.log
+   cat /run/flannel/subnet.env
+
+   ifconfig -a
+   ip route
+```
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+
+   Note: Please reference the blogs [Kubernetes: Flannel networking](https://blog.laputa.io/kubernetes-flannel-networking-6a1cb1f8ec7c) and [Flannel Networking Demystify: Support three backend mode - VXLAN, UDP and Host-GW](https://msazure.club/flannel-networking-demystify/) if you want to know more how flannel network works in kubernetes with VXLAN backend mode.
+
+   Note: CNI plugin of Calico is not supported because the resource 'EndpointSlices' is not supported by Arktos so far.
+
+
+1.3) On worker node: ensure following worker secret files copied from the master node
+
+   If the worker node is a GCP instance  
+```bash
+   gcloud beta compute ssh --zone "us-west2-a" "<worker node machine name>"  --project "<gce project name>"
+```
+
+   If the worker node is an AWS EC2, you can download files /var/run/kubernetes/kubelet.kubeconfig and /var/run/kubernetes/client-ca.crt from the master node, and then upload to /tmp/arktos/ in the worker node.
+
+```bash
+   mkdir -p /tmp/arktos
+   scp -i "<private key of keypair of master node>" ubuntu@<master-node-instance>:/var/run/kubernetes/kubelet.kubeconfig /tmp/arktos/kubelet.kubeconfig
+   scp -i "<private key of keypair of master node>" ubuntu@<master-node-instance>:/var/run/kubernetes/client-ca.crt /tmp/arktos/client-ca.crt
+```
+
+
+On worker node: start the worker node to register into cluster
+
+1.4) First make sure the following directories in the worker node are empty. If not, clean them up using sudopermission.
+```bash
+   sudo ls -alg /opt/cni/bin/; sudo rm -r /opt/cni/bin/*; sudo ls -alg /opt/cni/bin/
+   sudo ls -alg /etc/cni/net.d/; sudo rm -r /etc/cni/net.d/bridge.conf;sudo rm -r /etc/cni/net.d/10-flannel.conflist sudo ls -alg /etc/cni/net.d/
+```
+
+1.5) Start the worker node and register into cluster
+   Run the following commands when you are first time
+```bash
+   export ARKTOS_NO_CNI_PREINSTALLED=y
+   export KUBELET_IP=`hostname -i`;echo $KUBELET_IP
+   make clean
+   ./hack/arktos-worker-up.sh
+```
+OR
+   After you first run './hack/arktos-up.sh' successfully, you can use './hack/arktos-up.sh -O' intead
+```bash
+   export ARKTOS_NO_CNI_PREINSTALLED=y
+   export KUBELET_IP=`hostname -i`;echo $KUBELET_IP
+   ./hack/arktos-worker-up.sh -O
+```
+
+   Check the status of flannel installed on worker node:
+```bash
+   ps -ef |grep flannel |grep -v grep
+   sudo cat /etc/cni/net.d/10-flannel.conflist
+   cat /tmp/flanneld.log
+   cat /run/flannel/subnet.env
+
+   ifconfig -a
+   ip route
+```
+
+On the master node:
+
+1.6) Check status of nodes
+
+    You should see the work node is displayed and its status should be "Ready". But probably you will see the worker node is in "NotReady" state.
+
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+```bash
+   NAME               STATUS     ROLES    AGE     VERSION
+   ip-172-31-22-85    Ready      <none>   13m     v0.9.0
+   ip-172-31-29-128   NotReady   <none>   3m39s   v0.9.0
+```
+
+   Check the kubelet log on worker node:
+```bash
+   grep ip-172-31-29-128 /tmp/kubelet.worker.log |tail -2
+```
+```bash
+   E1116 05:30:12.750708   17517 controller.go:129] failed to ensure node lease exists, will retry in 7s, error: leases.coordination.k8s.io "ip-172-31-29-128" is forbidden: User "system:node:ip-172-31-22-85" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
+   E1116 05:30:19.752760   17517 controller.go:129] failed to ensure node lease exists, will retry in 7s, error: leases.coordination.k8s.io "ip-172-31-29-128" is forbidden: User "system:node:ip-172-31-22-85" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
+```
+
+1.7) Run the following command to create clusterrolebinding 'system-node-role-bound' to bind the group 'system:nodes' to clusterrole 'system:node' so that the master node has corresponding permission to get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease"
+
+```bash
+   ./cluster/kubectl.sh create clusterrolebinding system-node-role-bound --clusterrole=system:node --group=system:nodes
+   ./cluster/kubectl.sh get clusterrolebinding/system-node-role-bound -o yaml
+```
+
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+
+```bash
+   NAME               STATUS   ROLES    AGE    VERSION
+   ip-172-31-22-85    Ready    <none>   18m    v0.9.0
+   ip-172-31-29-128   Ready    <none>   8m1s   v0.9.0
+```
+
+1.8) Check the status of flannel pod running on worker node and fix the issue
+
+```bash
+   ./cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+```bash
+   NAMESPACE     NAME                               HASHKEY               READY   STATUS    RESTARTS   AGE     IP              NODE               NOMINATED NODE   READINESS GATES
+   kube-system   coredns-default-6c8b75ff85-ffxk8   2098388192664718022   1/1     Running   0          19m     10.244.0.65     ip-172-31-22-85    <none>           <none>
+   kube-system   kube-dns-554c5866fc-m9jgx          2830239144273075720   3/3     Running   0          19m     10.244.0.66     ip-172-31-22-85    <none>           <none>
+   kube-system   kube-flannel-ds-fwxln              5019221519058267550   0/1     Error     6          9m24s   172.31.29.128   ip-172-31-29-128   <none>           <none>
+   kube-system   kube-flannel-ds-nj9lr              5649932688255010940   1/1     Running   0          14m     172.31.22.85    ip-172-31-22-85    <none>           <none>
+   kube-system   virtlet-drnzl                      7477111360539274047   3/3     Running   0          13m     172.31.22.85    ip-172-31-22-85    <none>           <none>
+   kube-system   virtlet-g5x6k                      6083367271442737085   3/3     Running   0          9m13s   172.31.29.128   ip-172-31-29-128   <none>           <none>
+```
+
+   Check the log of pod 'kube-flannel-ds-fwxln' on worker node:
+```bash
+   sudo cat /var/log/pods/system_kube-system_kube-flannel-ds-fwxln_697ad8fe-9f60-4641-bd07-8b65397ba7f8/kube-flannel/5.log
+```
+
+```bash
+   2021-11-16T05:31:53.503829615Z stderr F I1116 05:31:53.503552       1 main.go:217] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: help:false version:false autoDetectIPv4:false autoDetectIPv6:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env subnetDir: publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 charonExecutablePath: charonViciUri: iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
+   2021-11-16T05:31:53.50389087Z stderr F W1116 05:31:53.503668       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
+   2021-11-16T05:32:23.506033195Z stderr F E1116 05:32:23.505730       1 main.go:234] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-fwxln': Get "https://10.0.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-fwxln": dial tcp 10.0.0.1:443: i/o timeout
+```
+
+1.9) On worker node, add one more iptable rule
+As a temporary measure to accommodate Flannel pods to access api server through the kubernetes service IP (the notorious 10.0.0.1) at the absense of kube-proxy, we need to place a bit trick by running below command at the worker node, assuming the kupe-apiserver is listening at https://172.31.22.85:6443 (please substitute with the proper value of your own cluster)
+
+```bash
+   sudo iptables -t nat -A OUTPUT -p tcp -d 10.0.0.1 --dport 443 -j DNAT --to-destination 172.31.22.85:6443
+```
+
+1.10) On master node, delete old pod of flannel running on worker node
+```bash
+   cluster/kubectl.sh delete pod/kube-flannel-ds-fwxln -n kube-system
+```
+```bash
+   cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+
+```bash
+   NAMESPACE     NAME                               HASHKEY               READY   STATUS    RESTARTS   AGE   IP              NODE               NOMINATED NODE   READINESS GATES
+   kube-system   coredns-default-6c8b75ff85-ffxk8   2098388192664718022   1/1     Running   0          22m   10.244.0.65     ip-172-31-22-85    <none>           <none>
+   kube-system   kube-dns-554c5866fc-m9jgx          2830239144273075720   3/3     Running   0          22m   10.244.0.66     ip-172-31-22-85    <none>           <none>
+   kube-system   kube-flannel-ds-fg2fl              7051182752843277513   1/1     Running   0          3s    172.31.29.128   ip-172-31-29-128   <none>           <none>
+   kube-system   kube-flannel-ds-nj9lr              5649932688255010940   1/1     Running   0          17m   172.31.22.85    ip-172-31-22-85    <none>           <none>
+   kube-system   virtlet-drnzl                      7477111360539274047   3/3     Running   0          17m   172.31.22.85    ip-172-31-22-85    <none>           <none>
+   kube-system   virtlet-g5x6k                      6083367271442737085   3/3     Running   0          12m   172.31.29.128   ip-172-31-29-128   <none>           <none>
+```
+
+   Check the log of pod - kube-flannel-ds-fg2fl running on worker node
+```bash
+   cluster/kubectl.sh logs pod/kube-flannel-ds-fg2fl -n kube-system
+```
+```bash
+I1116 05:39:20.311817       1 main.go:217] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: help:false version:false autoDetectIPv4:false autoDetectIPv6:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env subnetDir: publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 charonExecutablePath: charonViciUri: iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
+W1116 05:39:20.311943       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
+I1116 05:39:20.603991       1 kube.go:120] Waiting 10m0s for node controller to sync
+I1116 05:39:20.604071       1 kube.go:378] Starting kube subnet manager
+I1116 05:39:21.604353       1 kube.go:127] Node controller sync successful
+I1116 05:39:21.604401       1 main.go:237] Created subnet manager: Kubernetes Subnet Manager - ip-172-31-29-128
+I1116 05:39:21.604408       1 main.go:240] Installing signal handlers
+I1116 05:39:21.605025       1 main.go:459] Found network config - Backend type: vxlan
+I1116 05:39:21.605652       1 main.go:651] Determining IP address of default interface
+I1116 05:39:21.606171       1 main.go:698] Using interface with name eth0 and address 172.31.29.128
+I1116 05:39:21.606201       1 main.go:720] Defaulting external address to interface address (172.31.29.128)
+I1116 05:39:21.606208       1 main.go:733] Defaulting external v6 address to interface address (<nil>)
+I1116 05:39:21.606309       1 vxlan.go:137] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
+I1116 05:39:21.703010       1 kube.go:339] Setting NodeNetworkUnavailable
+I1116 05:39:21.707423       1 main.go:408] Current network or subnet (10.244.0.0/16, 10.244.1.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
+I1116 05:39:21.919287       1 iptables.go:240] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
+I1116 05:39:22.001812       1 iptables.go:240] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
+I1116 05:39:22.003069       1 iptables.go:240] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
+I1116 05:39:22.004036       1 iptables.go:240] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j MASQUERADE --random-fully
+I1116 05:39:22.005067       1 main.go:340] Setting up masking rules
+I1116 05:39:22.005853       1 main.go:361] Changing default FORWARD chain policy to ACCEPT
+I1116 05:39:22.005971       1 main.go:374] Wrote subnet file to /run/flannel/subnet.env
+I1116 05:39:22.005992       1 main.go:378] Running backend.
+I1116 05:39:22.006016       1 main.go:396] Waiting for all goroutines to exit
+I1116 05:39:22.006045       1 vxlan_network.go:60] watching for new subnet leases
+I1116 05:39:22.101236       1 iptables.go:216] Some iptables rules are missing; deleting and recreating rules
+I1116 05:39:22.101270       1 iptables.go:240] Deleting iptables rule: -s 10.244.0.0/16 -j ACCEPT
+I1116 05:39:22.101576       1 iptables.go:216] Some iptables rules are missing; deleting and recreating rules
+I1116 05:39:22.101597       1 iptables.go:240] Deleting iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
+I1116 05:39:22.102424       1 iptables.go:240] Deleting iptables rule: -d 10.244.0.0/16 -j ACCEPT
+I1116 05:39:22.102637       1 iptables.go:240] Deleting iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
+I1116 05:39:22.103307       1 iptables.go:228] Adding iptables rule: -s 10.244.0.0/16 -j ACCEPT
+I1116 05:39:22.103616       1 iptables.go:240] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.1.0/24 -j RETURN
+I1116 05:39:22.201201       1 iptables.go:228] Adding iptables rule: -d 10.244.0.0/16 -j ACCEPT
+I1116 05:39:22.201243       1 iptables.go:240] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
+I1116 05:39:22.202553       1 iptables.go:228] Adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
+I1116 05:39:22.204499       1 iptables.go:228] Adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
+I1116 05:39:22.301498       1 iptables.go:228] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.1.0/24 -j RETURN
+I1116 05:39:22.303743       1 iptables.go:228] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
+```
+
+   Check the status of nodes:
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+
+```bash
+   NAME               STATUS   ROLES    AGE   VERSION
+   ip-172-31-22-85    Ready    <none>   26m   v0.9.0
+   ip-172-31-29-128   Ready    <none>   16m   v0.9.0
+```
+
+1.11) Start 2nd worker node to join cluster by following up the above steps 1.3) through step 1.10).
+
+   on master node: Check the status of nodes and pods
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+
+```bash
+   NAME               STATUS   ROLES    AGE   VERSION
+   ip-172-31-22-85    Ready    <none>   32m   v0.9.0
+   ip-172-31-24-185   Ready    <none>   79s   v0.9.0
+   ip-172-31-29-128   Ready    <none>   22m   v0.9.0
+```
+
+```bash
+   ./cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+
+```bash
+   NAMESPACE     NAME                               HASHKEY               READY   STATUS    RESTARTS   AGE     IP              NODE               NOMINATED NODE   READINESS GATES
+   kube-system   coredns-default-6c8b75ff85-ffxk8   2098388192664718022   1/1     Running   0          32m     10.244.0.65     ip-172-31-22-85    <none>           <none>
+   kube-system   kube-dns-554c5866fc-m9jgx          2830239144273075720   3/3     Running   0          32m     10.244.0.66     ip-172-31-22-85    <none>           <none>
+   kube-system   kube-flannel-ds-7h6nv              8115799907707745269   1/1     Running   0          100s    172.31.24.185   ip-172-31-24-185   <none>           <none>
+   kube-system   kube-flannel-ds-fg2fl              7051182752843277513   1/1     Running   0          9m34s   172.31.29.128   ip-172-31-29-128   <none>           <none>
+   kube-system   kube-flannel-ds-nj9lr              5649932688255010940   1/1     Running   0          27m     172.31.22.85    ip-172-31-22-85    <none>           <none>
+   kube-system   virtlet-6drfl                      8480505743384093590   3/3     Running   0          89s     172.31.24.185   ip-172-31-24-185   <none>           <none>
+   kube-system   virtlet-drnzl                      7477111360539274047   3/3     Running   0          26m     172.31.22.85    ip-172-31-22-85    <none>           <none>
+   kube-system   virtlet-g5x6k                      6083367271442737085   3/3     Running   0          22m     172.31.29.128   ip-172-31-29-128   <none>           <none>
+```
+
+1.12) Test whether the ngnix application can be deployed successfully
+```bash
+   ./cluster/kubectl.sh run nginx --image=nginx --replicas=10
+   ./cluster/kubectl.sh get pods -n default -o wide
+```
+
+```bash
+   NAME                     HASHKEY               READY   STATUS    RESTARTS   AGE   IP            NODE               NOMINATED NODE   READINESS GATES
+   nginx-5d79788459-2ssb9   4714815331945249933   1/1     Running   0          9s    10.244.1.24   ip-172-31-29-128   <none>           <none>
+   nginx-5d79788459-5d9p7   3245382200846785472   1/1     Running   0          9s    10.244.0.67   ip-172-31-22-85    <none>           <none>
+   nginx-5d79788459-8gfrf   4231171286710049709   1/1     Running   0          9s    10.244.1.23   ip-172-31-29-128   <none>           <none>
+   nginx-5d79788459-bh2dl   2227188740354401744   1/1     Running   0          9s    10.244.1.25   ip-172-31-29-128   <none>           <none>
+   nginx-5d79788459-g4v56   4357703955321943380   1/1     Running   0          9s    10.244.2.3    ip-172-31-24-185   <none>           <none>
+   nginx-5d79788459-k74rq   4931046737888471496   1/1     Running   0          9s    10.244.2.5    ip-172-31-24-185   <none>           <none>
+   nginx-5d79788459-n7vhx   7564874686242188556   1/1     Running   0          9s    10.244.1.27   ip-172-31-29-128   <none>           <none>
+   nginx-5d79788459-r2wlr   3098101036923465124   1/1     Running   0          9s    10.244.2.2    ip-172-31-24-185   <none>           <none>
+   nginx-5d79788459-svzgd   2241339277610995766   1/1     Running   0          9s    10.244.2.4    ip-172-31-24-185   <none>           <none>
+   nginx-5d79788459-trlw9   1683625407392884361   1/1     Running   0          9s    10.244.1.26   ip-172-31-29-128   <none>           <none>
+```
+
+1.13) Test whether nginx pods on three nodes can talk each other using 'curl'
+   If you like, you need test connectivities among 6 subnets on 3 nods
+```bash
+   10.244.1.24 ---> 10.244.0.x(67)
+   10.244.1.24 ---> 10.244.2.x (2,3,4,5)
+   10.244.0.67 ---> 10.244.1.x (23,24,25,26,27)
+   10.244.0.67 --> 10.244.2.x (2,3,4,5)
+   10.244.2.3 --> 10.244.0.x(67)
+   10.244.2.3 --> 10.244.1.x (23,24,25,26,27)
+```
+
+For example:
+```bash
+   ./cluster/kubectl.sh exec -ti nginx-5d79788459-2ssb9 -- curl 10.244.0.67
+```
+```bash
+<!DOCTYPE html>
+<html>
+<head>
+<title>Welcome to nginx!</title>
+<style>
+html { color-scheme: light dark; }
+body { width: 35em; margin: 0 auto;
+font-family: Tahoma, Verdana, Arial, sans-serif; }
+</style>
+</head>
+<body>
+<h1>Welcome to nginx!</h1>
+<p>If you see this page, the nginx web server is successfully installed and
+working. Further configuration is required.</p>
+
+<p>For online documentation and support please refer to
+<a href="http://nginx.org/">nginx.org</a>.<br/>
+Commercial support is available at
+<a href="http://nginx.com/">nginx.com</a>.</p>
+
+<p><em>Thank you for using nginx.</em></p>
+</body>
+</html>
+```
+
+1.14) Clean up nginx deployment
+```bash
+   cluster/kubectl.sh delete deployment/nginx
+```
+
+```bash
+  cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+
+```bash
+```
+
+1.15) Please follow up the steps to do [end-to-end verification of service in scale-up cluster](https://github.com/CentaurusInfra/arktos/issues/1142)
+
+
+Method #2 - Process Mode (to support M TPs X N RPs)
+
+On master node:
+
+2.0) Make sure the following directories are empty. If not, clean them up using sudo permisson
+```bash
+   sudo ls -alg /opt/cni/bin/; sudo rm -r /opt/cni/bin/*; sudo ls -alg /opt/cni/bin/
+   sudo ls -alg /etc/cni/net.d/; sudo rm -r /etc/cni/net.d/bridge.conf;sudo rm -r /etc/cni/net.d/10-flannel.conflist sudo ls -alg /etc/cni/net.d/
+```
+
+2.1) bootstrap the cluster by starting the master node and CNI plugin - flannel is automatically installed in process mode
+   Run the following commands when you are first time
+```bash
+   make clean
+   CNIPLUGIN=flannel ./hack/arktos-up.sh
+```
+
+   After you first run './hack/arktos-up.sh' successfully, you can use './hack/arktos-up.sh -O' intead
+```bash
+   CNIPLUGIN=flannel ./hack/arktos-up.sh -O
+```
+
+2.2) Open another terminal on the master node to check the status of node
+```bash
+   ./cluster/kubectl.sh get all --all-namespaces
+   ./cluster/kubectl.sh get pods --all-namespaces
+   ps -ef |grep flannel |grep -v grep
+   sudo cat /etc/cni/net.d/10-flannel.conflist
+   cat /tmp/flanneld.log
+   cat /run/flannel/subnet.env
+
+   ifconfig -a
+   ip route
+```
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+```bash
+   NAME              STATUS   ROLES    AGE    VERSION
+   ip-172-31-22-85   Ready    <none>   9m7s   v0.9.0
+```
+
+   Note: Please reference the blogs [Kubernetes: Flannel networking](https://blog.laputa.io/kubernetes-flannel-networking-6a1cb1f8ec7c) and [Flannel Networking Demystify: Support three backend mode - VXLAN, UDP and Host-GW](https://msazure.club/flannel-networking-demystify/) if you want to know more how flannel network works in kubernetes with Host-GW backend mode.
+
+   Note: CNI plugin of Calico is not supported because the resource 'EndpointSlices' is not supported by Arktos so far.
+
+2.3) On worker node: ensure following worker secret files copied from the master node
+
+   If the worker node is a GCP instance
+```bash
+   gcloud beta compute ssh --zone "us-west2-a" "<worker node machine name>"  --project "<gce project name>"
+```
+
+   If the worker node is an AWS EC2, you can download files /var/run/kubernetes/kubelet.kubeconfig and /var/run/kubernetes/client-ca.crt from the master node, and then upload to /tmp/arktos/ in the worker node.
+
+```bash
+   mkdir -p /tmp/arktos
+   scp -i "<private key of keypair of master node>" ubuntu@<master-node-instance>:/var/run/kubernetes/kubelet.kubeconfig /tmp/arktos/kubelet.kubeconfig
+   scp -i "<private key of keypair of master node>" ubuntu@<master-node-instance>:/var/run/kubernetes/client-ca.crt /tmp/arktos/client-ca.crt
+```
+
+
+On worker node: start the worker node to register into cluster
+
+2.4) First make sure the following directories in the worker node are empty. If not, clean them up using sudopermission.
+```bash
+   sudo ls -alg /opt/cni/bin/; sudo rm -r /opt/cni/bin/*; sudo ls -alg /opt/cni/bin/
+   sudo ls -alg /etc/cni/net.d/; sudo rm -r /etc/cni/net.d/bridge.conf;sudo rm -r /etc/cni/net.d/10-flannel.conflist sudo ls -alg /etc/cni/net.d/
+```
+
+2.5) Start the worker node and register into cluster
+   Run the following commands when you are first time
+```bash
+   export ARKTOS_NO_CNI_PREINSTALLED=y
+   export KUBELET_IP=`hostname -i`;echo $KUBELET_IP
+   make clean
+   API_HOST_IP_EXTERNAL=<IP of master API host> CNIPLUGIN=flannel ./hack/arktos-worker-up.sh
+```
+   Note: for example, in step 2.2) 172.31.22.85 is IP of master API host - ip-172-31-22-85
+OR
+   After you first run './hack/arktos-up.sh' successfully, you can use './hack/arktos-up.sh -O' intead
+```bash
+   export ARKTOS_NO_CNI_PREINSTALLED=y
+   export KUBELET_IP=`hostname -i`;echo $KUBELET_IP
+   API_HOST_IP_EXTERNAL=<IP of master API host> CNIPLUGIN=flannel ./hack/arktos-worker-up.sh -O
+```
+  Note: for example, in step 2.2) 172.31.22.85 is IP of master API host - ip-172-31-22-85
+
+   Check the status of flannel installed on worker node:
+```bash
+   ps -ef |grep flannel |grep -v grep
+   sudo cat /etc/cni/net.d/10-flannel.conflist
+   cat /tmp/flanneld.log
+   cat /run/flannel/subnet.env
+
+   ifconfig -a
+   ip route
+```
+
+On the master node:
+
+2.6) Check status of nodes
+
+    You should see the work node is displayed and its status should be "Ready". But probably you will see the worker node is in "NotReady" state.
+
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+```bash
+   NAME               STATUS     ROLES    AGE   VERSION
+   ip-172-31-22-85    Ready      <none>   71m   v0.9.0
+   ip-172-31-29-128   NotReady   <none>   59m   v0.9.0
+```
+
+   Check the kubelet log on worker node:
+```bash
+   grep ip-172-31-29-128 /tmp/kubelet.worker.log |tail -2
+```
+```bash
+   E1116 05:30:12.750708   17517 controller.go:129] failed to ensure node lease exists, will retry in 7s, error: leases.coordination.k8s.io "ip-172-31-29-128" is forbidden: User "system:node:ip-172-31-22-85" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
+   E1116 05:30:19.752760   17517 controller.go:129] failed to ensure node lease exists, will retry in 7s, error: leases.coordination.k8s.io "ip-172-31-29-128" is forbidden: User "system:node:ip-172-31-22-85" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease": can only access node lease with the same name as the requesting node
+```
+
+2.7) Run the following command to create clusterrolebinding 'system-node-role-bound' to bind the group 'system:nodes' to clusterrole 'system:node' so that the master node has corresponding permission to get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease"
+
+```bash
+   ./cluster/kubectl.sh create clusterrolebinding system-node-role-bound --clusterrole=system:node --group=system:nodes
+   ./cluster/kubectl.sh get clusterrolebinding/system-node-role-bound -o yaml
+```
+
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+
+```bash
+   NAME               STATUS   ROLES    AGE    VERSION
+   ip-172-31-22-85    Ready    <none>   87m    v0.9.0
+   ip-172-31-29-128   Ready    <none>   75m    v0.9.0
+```
+
+2.8) Start 2nd worker node to join cluster by following up the above steps 2.3) through step 2.7).
+
+   on master node: Check the status of nodes and pods
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+
+```bash
+   NAME               STATUS   ROLES    AGE     VERSION
+   ip-172-31-22-85    Ready    <none>   129m    v0.9.0
+   ip-172-31-24-185   Ready    <none>   5m15s   v0.9.0
+   ip-172-31-29-128   Ready    <none>   126m    v0.9.0
+```
+
+```bash
+   ./cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+
+2.9) Test whether the ngnix application can be deployed successfully
+```bash
+   ./cluster/kubectl.sh run nginx --image=nginx --replicas=10
+   ./cluster/kubectl.sh get pods -n default -o wide
+```
+
+```bash
+   NAME                     HASHKEY               READY   STATUS    RESTARTS   AGE   IP            NODE               NOMINATED NODE   READINESS GATES
+   nginx-5d79788459-2z9xm   7135149431889485631   1/1     Running   0          34s   10.244.5.4    ip-172-31-24-185   <none>           <none>
+   nginx-5d79788459-6l8g2   3843739298418711470   1/1     Running   0          34s   10.244.1.19   ip-172-31-29-128   <none>           <none>
+   nginx-5d79788459-7klr7   2834848570761789551   1/1     Running   0          34s   10.244.0.58   ip-172-31-22-85    <none>           <none>
+   nginx-5d79788459-d48dk   9074133381883477042   1/1     Running   0          34s   10.244.5.5    ip-172-31-24-185   <none>           <none>
+   nginx-5d79788459-jknqx   7965692118631704777   1/1     Running   0          34s   10.244.5.3    ip-172-31-24-185   <none>           <none>
+   nginx-5d79788459-jtm8t   3306282595644386936   1/1     Running   0          34s   10.244.1.21   ip-172-31-29-128   <none>           <none>
+   nginx-5d79788459-lg9sn   8426441781750165096   1/1     Running   0          34s   10.244.1.18   ip-172-31-29-128   <none>           <none>
+   nginx-5d79788459-lhtdh   1475896372455155057   1/1     Running   0          34s   10.244.1.20   ip-172-31-29-128   <none>           <none>
+   nginx-5d79788459-lnlv7   5397625964048573894   1/1     Running   0          34s   10.244.5.2    ip-172-31-24-185   <none>           <none>
+   nginx-5d79788459-mgx9l   8679199556520419985   1/1     Running   0          34s   10.244.1.22   ip-172-31-29-128   <none>           <none>
+```
+
+2.10) Test whether nginx pods on three nodes can talk each other using 'curl'
+   If you like, you need test connectivities among 6 subnets on 3 nods
+```bash
+   10.244.5.4  --> 10.244.0.x(58)
+   10.244.5.4  --> 10.244.1.x (18,19,20,21,22)
+   10.244.0.58 --> 10.244.5.x (2,3,4,5)
+   10.244.0.58 --> 10.244.1.x (18,19,20,21,22)
+   10.244.1.19 --> 10.244.0.x(58)
+   10.244.1.19 --> 10.244.5.x (2,3,4,5)
+```
+
+For example:
+```bash
+   ./cluster/kubectl.sh exec -ti nginx-5d79788459-2z9xm -- curl 10.244.0.58
+```
+```bash
+<!DOCTYPE html>
+<html>
+<head>
+<title>Welcome to nginx!</title>
+<style>
+html { color-scheme: light dark; }
+body { width: 35em; margin: 0 auto;
+font-family: Tahoma, Verdana, Arial, sans-serif; }
+</style>
+</head>
+<body>
+<h1>Welcome to nginx!</h1>
+<p>If you see this page, the nginx web server is successfully installed and
+working. Further configuration is required.</p>
+
+<p>For online documentation and support please refer to
+<a href="http://nginx.org/">nginx.org</a>.<br/>
+Commercial support is available at
+<a href="http://nginx.com/">nginx.com</a>.</p>
+
+<p><em>Thank you for using nginx.</em></p>
+</body>
+</html>
+```
+
+2.11) Clean up nginx deployment
+```bash
+   cluster/kubectl.sh delete deployment/nginx
+```
+```bash
+  cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+
+2.12) Please follow up the steps to do [end-to-end verification of service in scale-up cluster](https://github.com/CentaurusInfra/arktos/issues/1142)
+
+      Note: Currently the tests on master node in end-to-end verification of service in scale-up cluster - steps 2.4.6) and 2.4.7) do pass - login nginx pods and can ping service names in same tenant
+            Currently the tests on two worker nodes in end-to-end verification of service in scale-up cluster - steps 2.4.6) and 2.4.7) do not pass - login nginx pods and can not ping service names in same tenant
+            -- YunWen recommends starting kubelete on worker nodes
+               Yes, after the starting kube-proxy is added into script hack/arktos-worker-up.sh, above issue is really fixed after complete tests/verificatins
+
+            -- After starting kube-proxy on worker node, the worker node joining to cluster still needs step 2.7).
+
+                  ./cluster/kubectl.sh create clusterrolebinding system-node-role-bound --clusterrole=system:node --group=system:nodes
+                  ./cluster/kubectl.sh get clusterrolebinding/system-node-role-bound -o yaml
+
+            Currently other tests do pass
diff --git a/docs/setup-guide/scale-out-local-dev-setup.md b/docs/setup-guide/scale-out-local-dev-setup.md
index 38f6eb5ded3..4cba7669459 100644
--- a/docs/setup-guide/scale-out-local-dev-setup.md
+++ b/docs/setup-guide/scale-out-local-dev-setup.md
@@ -1,4 +1,4 @@
-# Setting up local dev environment for scale out
+# Setting up local dev environment for scale out with flannel in two modes - process mode and daemonset mode
 
 ## Scenarios
 
@@ -21,28 +21,54 @@
 
 1. Set up environment variables (no changes have been made for RP2 nor tested)
 
-```
-export TENANT_PARTITION_IP=[TP1_IP],[TP2_IP]
-export RESOURCE_PARTITION_IP=[RP1_IP]
+```bash
+   export TENANT_PARTITION_IP=[TP1_IP],[TP2_IP]
+   export RESOURCE_PARTITION_IP=[RP1_IP]
 ```
 
 1. Run ./hack/scalability/setup_haproxy.sh (depends on your HA proxy version and environment setup, you might need to comment out some code in the script)
 
+
+### Method #1 - Install Flannel in process mode
+Issues:
+1. Two worker nodes successfully join RP1 cluster in "Ready" state when flannel is installed successfully in process mode, 
+   BUT nginx pods bounding to two worker nodes are in Pending state
+2. Scheduler on TP1 is working because nginx pods are bounding to two worker nodes, BUT the directory /var/log/pods on two nodes are empty.
+
+### Patching Network Routing Across RPs
+Depending on your situation, you may need to change instruction properly - the bottom line is pods from one RP should be able to access pods of other RP.
+
+Below is what we did in our test lab using AWS EC2 resources, where RP1/RP2 nodes are in same subnet.
+On both RP nodes,
+1. stop the source/dest check (on AWS console, using EC2 instance Action menu, choosing Networking | Change source-destination check);
+2. manually add relevant routing entries of each node, so that each routing table is complete for all nodes of whole cluster across RPs, e.g. (assuming pod cidr of rp0 is 10.244.0.0/16, rp1 10.245.0.0/16)
+
+on RP1,
+```
+   sudo ip r add 10.245.0.0/24 via [RP2-IP]
+```
+
+on RP2
+```
+   sudo ip r add 10.244.0.0/24 via [RP1-IP]
+```
+
 ### Setting up TPs
 1. Make sure hack/arktos-up.sh can be run at the box
 
-1. Set up environment variables
+2. Set up environment variables
+```bash
+   # optional, used for cloud KCM only but not tested
+   export SCALE_OUT_PROXY_IP=[PROXY_IP]
+   export SCALE_OUT_PROXY_PORT=8888
+   export TENANT_SERVER_NAME=tp-name (e.g. tp1)
 
-```
-# optional, used for cloud KCM only but not tested
-export SCALE_OUT_PROXY_IP=[PROXY_IP]
-export SCALE_OUT_PROXY_PORT=8888
-export TENANT_SERVER_NAME=tp-name (e.g. tp1)
+   # required
+   export SCALE_OUT_TP_ENABLE_DAEMONSET=false
 
-# required
-export IS_RESOURCE_PARTITION=false
-export RESOURCE_SERVER=[RP1_IP]<,[RP2_IP]>
-export TENANT_PARTITION_SERVICE_SUBNET=[service-ip-cidr]
+   export IS_RESOURCE_PARTITION=false
+   export RESOURCE_SERVER=[RP1_IP]<,[RP2_IP]>
+   export TENANT_PARTITION_SERVICE_SUBNET=[service-ip-cidr]
 ```
 
 an examplative allocation for 2 TPs could be
@@ -51,23 +77,24 @@ an examplative allocation for 2 TPs could be
 | --- | --- |
 | 10.0.0.0/16 | 10.1.0.0/16 |
 
-1. Run ./hack/arktos-up-scale-out-poc.sh
-
-1. Expected last line of output: "Tenant Partition Cluster is Running ..."
+3. Run ./hack/arktos-up-scale-out-poc.sh
+```bash
+   Expected last line of output: "Tenant Partition Cluster is Running ..."
+```
 
 Note:
-
-1. As certificates generating and sharing is confusing and time consuming in local test environment. We will use insecure mode for local test for now. Secured mode can be added back later when main goal is acchieved.
+As certificates generating and sharing is confusing and time consuming in local test environment. We will use insecure mode for local test for now. Secured mode can be added back later when main goal is acchieved.
 
 ### Setting up RPs
 1. Make sure hack/arktos-up.sh can be run at the box
 
-1. Set up environment variables
+2. Set up environment variables
+```bash
+   export SCALE_OUT_TP_ENABLE_DAEMONSET=false
 
-```
-export IS_RESOURCE_PARTITION=true
-export TENANT_SERVER=[TP1_IP]<,[TP2_IP]>
-export RESOURCE_PARTITION_POD_CIDR=[pod-cidr]
+   export IS_RESOURCE_PARTITION=true
+   export TENANT_SERVER=[TP1_IP]<,[TP2_IP]>
+   export RESOURCE_PARTITION_POD_CIDR=[pod-cidr]
 ```
 
 an examplative allocation of pod cidr for 2 RPs could be
@@ -76,9 +103,255 @@ an examplative allocation of pod cidr for 2 RPs could be
 | --- | --- |
 | 10.244.0.0/16 | 10.245.0.0/16 |
 
-1. Run ./hack/arktos-up-scale-out-poc.sh
+3. Run ./hack/arktos-up-scale-out-poc.sh
+```bash
+   Expected last line of output: "Resource Partition Cluster is Running ..."
+```
+
+4. On RP1 node, check node status
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+```bash
+   NAME               STATUS   ROLES    AGE     VERSION
+   ip-172-31-13-237   Ready    <none>   67s     v0.9.0
+```
+
+5. On TP1 node, check pod status
+```bash
+   ./cluster/kubectl.sh get pod --all-namespaces
+```
+```bash
+   NAMESPACE     NAME                                              HASHKEY               READY   STATUS    RESTARTS   AGE
+   kube-system   coredns-default-ip-172-31-5-56-67d8d65fb8-kjdrn   236997205987588351    1/1     Running   0          5m18s
+   kube-system   kube-dns-554c5866fc-rmqvg                         140848597806714248    3/3     Running   0          5m18s
+   kube-system   virtlet-vk2sz                                     5275988391050895787   3/3     Running   0          82s   
+```
+
+6. Test whether the ngnix application can be deployed successfully
+```bash
+   ./cluster/kubectl.sh run nginx --image=nginx --replicas=2
+   ./cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+```bash
+   NAMESPACE     NAME                                              HASHKEY               READY   STATUS    RESTARTS   AGE     IP              NODE               NOMINATED NODE   READINESS GATES
+   default       nginx-5d79788459-hdhbz                            5700817263648618771   1/1     Running   0          2m16s   10.244.0.191    ip-172-31-13-237   <none>           <none>
+   default       nginx-5d79788459-qdqpb                            1425984288120945573   1/1     Running   0          2m16s   10.244.0.190    ip-172-31-13-237   <none>           <none>
+   kube-system   coredns-default-ip-172-31-5-56-67d8d65fb8-kjdrn   236997205987588351    1/1     Running   0          9m2s    10.244.0.188    ip-172-31-13-237   <none>           <none>
+   kube-system   kube-dns-554c5866fc-rmqvg                         140848597806714248    3/3     Running   0          9m2s    10.244.0.189    ip-172-31-13-237   <none>           <none>
+   kube-system   virtlet-vk2sz                                     5275988391050895787   3/3     Running   0          5m6s    172.31.13.237   ip-172-31-13-237   <none>           <none>
+```
+   Delete nginx application:
+```bash
+   ./cluster/kubectl.sh delete deployment/nginx
+   ./cluster/kubectl.sh get pods -n default -o wide
+```
+ 
+7. On TP1 node, check the status of flannel pods running on worker nodes
+```bash
+   ./cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+```bash
+   NAMESPACE     NAME                                              HASHKEY               READY   STATUS    RESTARTS   AGE   IP              NODE               NOMINATED NODE   READINESS GATES
+   kube-system   coredns-default-ip-172-31-5-56-67d8d65fb8-kjdrn   236997205987588351    1/1     Running   0          25m     10.244.0.188    ip-172-31-13-237   <none>           <none>
+   kube-system   kube-dns-554c5866fc-rmqvg                         140848597806714248    3/3     Running   0          25m     10.244.0.189    ip-172-31-13-237   <none>           <none>
+   kube-system   virtlet-hm984                                     2793640128098565838   0/3     Pending   0          3m33s   <none>          ip-172-31-26-244   <none>           <none>
+   kube-system   virtlet-vk2sz                                     5275988391050895787   3/3     Running   0          21m     172.31.13.237   ip-172-31-13-237   <none>           <none>
+   kube-system   virtlet-xdlr5                                     8190198442816427150   0/3     Pending   0          4m23s   <none>          ip-172-31-29-26    <none>           <none>
+```
+
+### Join worker nodes into RP1 cluster 
+1. Make sure hack/arktos-up.sh can be run at the box
+
+2. Ensure following worker secret files copied from the master node
+```bash
+   mkdir -p /tmp/arktos
+   scp -i "/home/ubuntu/AWS/keypair/CarlXieKeyPairAccessFromWin.pem" ubuntu@ip-172-31-13-237:/var/run/kubernetes/kubelet.kubeconfig /tmp/arktos/kubelet.kubeconfig
+   scp -i "/home/ubuntu/AWS/keypair/CarlXieKeyPairAccessFromWin.pem" ubuntu@ip-172-31-13-237:/var/run/kubernetes/client-ca.crt /tmp/arktos/client-ca.crt
+   ls -alg /tmp/arktos
+```
+```bash
+   -rw-r--r-- 1 ubuntu 1310 Nov 19 00:11 client-ca.crt
+   -rw-r--r-- 1 ubuntu  312 Nov 19 00:11 kubelet.kubeconfig
+```
+
+3. Clean up the directories - /etc/cni/net.d/ and /opt/cni/bin/ as well as the processes - kubelet and flannel
+```bash
+   sudo ls -alg /opt/cni/bin/; sudo rm -r /opt/cni/bin/*; sudo ls -alg /opt/cni/bin/
+
+   sudo ls -alg /etc/cni/net.d/
+   sudo rm -r /etc/cni/net.d/bridge.conf
+   sudo rm -r /etc/cni/net.d/10-flannel.conflist
+   sudo ls -alg /etc/cni/net.d/
+```
+```bash
+   sudo kill -9 `ps -ef |grep kubelet |grep -v grep |awk '{print $2}'`
+   sudo kill -9 `ps -ef |grep flannel |grep -v grep |awk '{print $2}'`
+```
+
+4. Set up environment variables
+```bash
+   export ARKTOS_NO_CNI_PREINSTALLED=y
+   export SCALE_OUT_TP_ENABLE_DAEMONSET=false
+   export KUBELET_IP=`hostname -i`; echo $KUBELET_IP
+```
+
+5. Start worker node to join RP1 cluster including start of kubelet and kube-proxy
+```bash
+   ./hack/arktos-worker-up.sh
+```
+Note: here we try to join two worker nodes into RP1 clister
+
+6. On RP1 node, check node status
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+```bash
+   NAME               STATUS     ROLES    AGE   VERSION
+   ip-172-31-13-237   Ready    <none>   19m     v0.9.0
+   ip-172-31-26-244   Ready    <none>   89s     v0.9.0
+   ip-172-31-29-26    Ready    <none>   2m19s   v0.9.0
+```
+
+7. On TP1 node, check the status of flannel pods running on worker nodes
+```bash
+   ./cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+```bash
+   NAMESPACE     NAME                                              HASHKEY               READY   STATUS    RESTARTS   AGE   IP              NODE               NOMINATED NODE   READINESS GATES
+   kube-system   coredns-default-ip-172-31-5-56-67d8d65fb8-kjdrn   236997205987588351    1/1     Running   0          25m     10.244.0.188    ip-172-31-13-237   <none>           <none>
+   kube-system   kube-dns-554c5866fc-rmqvg                         140848597806714248    3/3     Running   0          25m     10.244.0.189    ip-172-31-13-237   <none>           <none>
+   kube-system   virtlet-hm984                                     2793640128098565838   0/3     Pending   0          3m33s   <none>          ip-172-31-26-244   <none>           <none>
+   kube-system   virtlet-vk2sz                                     5275988391050895787   3/3     Running   0          21m     172.31.13.237   ip-172-31-13-237   <none>           <none>
+   kube-system   virtlet-xdlr5                                     8190198442816427150   0/3     Pending   0          4m23s   <none>          ip-172-31-29-26    <none>           <none>
+```
+
+8. Test whether the ngnix application can be deployed on RP1 cluster successfully
+```bash
+   ./cluster/kubectl.sh run nginx --image=nginx --replicas=10
+   ./cluster/kubectl.sh get pods -n default --tenant system -o wide
+```
+```bash
+   NAME                     HASHKEY               READY   STATUS    RESTARTS   AGE    IP             NODE               NOMINATED NODE   READINESS GATES
+   nginx-5d79788459-6rtnl   3730908148668899933   1/1     Running   0          3d3h   10.244.1.4     ip-172-31-29-26    <none>           <none>
+   nginx-5d79788459-8cx96   7055915722265553081   1/1     Running   0          3d3h   10.244.2.3     ip-172-31-26-244   <none>           <none>
+   nginx-5d79788459-994nl   5869074441326266477   1/1     Running   0          3d3h   10.244.2.2     ip-172-31-26-244   <none>           <none>
+   nginx-5d79788459-f9zqc   8691715263191300667   1/1     Running   0          3d3h   10.244.1.5     ip-172-31-29-26    <none>           <none>
+   nginx-5d79788459-hbzbj   5139024971118873047   1/1     Running   0          3d3h   10.244.2.5     ip-172-31-26-244   <none>           <none>
+   nginx-5d79788459-kjnjl   3031080773184543669   1/1     Running   0          3d3h   10.244.2.4     ip-172-31-26-244   <none>           <none>
+   nginx-5d79788459-vc7g5   1034646888068516150   1/1     Running   0          3d3h   10.244.1.2     ip-172-31-29-26    <none>           <none>
+   nginx-5d79788459-wj5fr   2675269125612355016   1/1     Running   0          3d3h   10.244.0.192   ip-172-31-13-237   <none>           <none>
+   nginx-5d79788459-x5bjs   8380430494939361657   1/1     Running   0          3d3h   10.244.1.3     ip-172-31-29-26    <none>           <none>
+   nginx-5d79788459-x8lhc   1892452665592184108   1/1     Running   0          3d3h   10.244.2.6     ip-172-31-26-244   <none>           <none>
+```
+
+9. Check the log files in /var/log/pods on three RP1 nodes (ip-172-31-13-237, ip-172-31-29-26, ip-172-31-26-244)
+```bash
+   sudo cat /var/log/pods/system_default_nginx-5d79788459-wj5fr_3dfd5ed6-6b1b-4682-8eaf-94bb57524a63/nginx/0.log
+```
+
+```bash
+2021-11-19T03:09:39.500406298Z stdout F /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
+2021-11-19T03:09:39.500437097Z stdout F /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
+2021-11-19T03:09:39.502061568Z stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
+2021-11-19T03:09:39.506749842Z stdout F 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
+2021-11-19T03:09:39.511831273Z stdout F 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
+2021-11-19T03:09:39.511995564Z stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
+2021-11-19T03:09:39.514361468Z stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
+2021-11-19T03:09:39.515548993Z stdout F /docker-entrypoint.sh: Configuration complete; ready for start up
+2021-11-19T03:09:39.520020136Z stderr F 2021/11/19 03:09:39 [notice] 1#1: using the "epoll" event method
+2021-11-19T03:09:39.520031396Z stderr F 2021/11/19 03:09:39 [notice] 1#1: nginx/1.21.4
+2021-11-19T03:09:39.520035949Z stderr F 2021/11/19 03:09:39 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
+2021-11-19T03:09:39.520040123Z stderr F 2021/11/19 03:09:39 [notice] 1#1: OS: Linux 5.4.0-1059-aws
+2021-11-19T03:09:39.520044855Z stderr F 2021/11/19 03:09:39 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
+2021-11-19T03:09:39.520053491Z stderr F 2021/11/19 03:09:39 [notice] 1#1: start worker processes
+2021-11-19T03:09:39.52019447Z stderr F 2021/11/19 03:09:39 [notice] 1#1: start worker process 31
+2021-11-19T03:09:39.520301688Z stderr F 2021/11/19 03:09:39 [notice] 1#1: start worker process 32
+2021-11-19T03:09:39.520435814Z stderr F 2021/11/19 03:09:39 [notice] 1#1: start worker process 33
+2021-11-19T03:09:39.520544384Z stderr F 2021/11/19 03:09:39 [notice] 1#1: start worker process 34
+2021-11-19T03:09:39.520668522Z stderr F 2021/11/19 03:09:39 [notice] 1#1: start worker process 35
+2021-11-19T03:09:39.520836267Z stderr F 2021/11/19 03:09:39 [notice] 1#1: start worker process 36
+2021-11-19T03:09:39.520920495Z stderr F 2021/11/19 03:09:39 [notice] 1#1: start worker process 37
+2021-11-19T03:09:39.521111475Z stderr F 2021/11/19 03:09:39 [notice] 1#1: start worker process 38
+```
+
+```bash
+   sudo cat /var/log/pods/system_default_nginx-5d79788459-6rtnl_bb4775a4-a9c7-4487-868e-5ca2f9389fb5/nginx/0.log
+```
+```bash
+2021-11-22T06:13:58.808003171Z stdout F /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
+2021-11-22T06:13:58.808035483Z stdout F /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
+2021-11-22T06:13:58.80952329Z stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
+2021-11-22T06:13:58.816425099Z stdout F 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
+2021-11-22T06:13:58.823330754Z stdout F 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
+2021-11-22T06:13:58.823545216Z stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
+2021-11-22T06:13:58.826990761Z stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
+2021-11-22T06:13:58.828414357Z stdout F /docker-entrypoint.sh: Configuration complete; ready for start up
+2021-11-22T06:13:58.834993952Z stderr F 2021/11/22 06:13:58 [notice] 1#1: using the "epoll" event method
+2021-11-22T06:13:58.836495476Z stderr F 2021/11/22 06:13:58 [notice] 1#1: nginx/1.21.4
+2021-11-22T06:13:58.836505289Z stderr F 2021/11/22 06:13:58 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
+2021-11-22T06:13:58.836510621Z stderr F 2021/11/22 06:13:58 [notice] 1#1: OS: Linux 5.4.0-1059-aws
+2021-11-22T06:13:58.836564739Z stderr F 2021/11/22 06:13:58 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
+2021-11-22T06:13:58.836572291Z stderr F 2021/11/22 06:13:58 [notice] 1#1: start worker processes
+2021-11-22T06:13:58.836576914Z stderr F 2021/11/22 06:13:58 [notice] 1#1: start worker process 32
+2021-11-22T06:13:58.83658122Z stderr F 2021/11/22 06:13:58 [notice] 1#1: start worker process 33
+2021-11-22T06:13:58.836585738Z stderr F 2021/11/22 06:13:58 [notice] 1#1: start worker process 34
+2021-11-22T06:13:58.836590167Z stderr F 2021/11/22 06:13:58 [notice] 1#1: start worker process 35
+2021-11-22T06:13:58.83659473Z stderr F 2021/11/22 06:13:58 [notice] 1#1: start worker process 36
+2021-11-22T06:13:58.836599047Z stderr F 2021/11/22 06:13:58 [notice] 1#1: start worker process 37
+2021-11-22T06:13:58.836604594Z stderr F 2021/11/22 06:13:58 [notice] 1#1: start worker process 38
+2021-11-22T06:13:58.836621963Z stderr F 2021/11/22 06:13:58 [notice] 1#1: start worker process 39
+```
+
+```
+   sudo cat /var/log/pods/system_default_nginx-5d79788459-8cx96_7a6c7d61-2572-48ea-8d4f-06e750d18b73/nginx/0.log
+
+```
+```
+2021-11-22T05:02:45.622666173Z stdout F /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
+2021-11-22T05:02:45.622699113Z stdout F /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
+2021-11-22T05:02:45.623558087Z stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
+2021-11-22T05:02:45.631778942Z stdout F 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
+2021-11-22T05:02:45.63827023Z stdout F 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
+2021-11-22T05:02:45.639181646Z stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
+2021-11-22T05:02:45.643319505Z stdout F /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
+2021-11-22T05:02:45.64568454Z stdout F /docker-entrypoint.sh: Configuration complete; ready for start up
+2021-11-22T05:02:45.651482771Z stderr F 2021/11/22 05:02:45 [notice] 1#1: using the "epoll" event method
+2021-11-22T05:02:45.651505399Z stderr F 2021/11/22 05:02:45 [notice] 1#1: nginx/1.21.4
+2021-11-22T05:02:45.651511264Z stderr F 2021/11/22 05:02:45 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
+2021-11-22T05:02:45.651516704Z stderr F 2021/11/22 05:02:45 [notice] 1#1: OS: Linux 5.4.0-1059-aws
+2021-11-22T05:02:45.65152124Z stderr F 2021/11/22 05:02:45 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
+2021-11-22T05:02:45.651540832Z stderr F 2021/11/22 05:02:45 [notice] 1#1: start worker processes
+2021-11-22T05:02:45.651751533Z stderr F 2021/11/22 05:02:45 [notice] 1#1: start worker process 33
+2021-11-22T05:02:45.651921707Z stderr F 2021/11/22 05:02:45 [notice] 1#1: start worker process 34
+2021-11-22T05:02:45.654343921Z stderr F 2021/11/22 05:02:45 [notice] 1#1: start worker process 35
+2021-11-22T05:02:45.65435891Z stderr F 2021/11/22 05:02:45 [notice] 1#1: start worker process 36
+2021-11-22T05:02:45.654364973Z stderr F 2021/11/22 05:02:45 [notice] 1#1: start worker process 37
+2021-11-22T05:02:45.654370066Z stderr F 2021/11/22 05:02:45 [notice] 1#1: start worker process 38
+2021-11-22T05:02:45.654375341Z stderr F 2021/11/22 05:02:45 [notice] 1#1: start worker process 39
+2021-11-22T05:02:45.654380663Z stderr F 2021/11/22 05:02:45 [notice] 1#1: start worker process 40
+```
+
+
+10. On TP1 node, delete nginx application
+```bash
+   ./cluster/kubectl.sh delete deployment/nginx
+   ./cluster/kubectl.sh get pods -n default -o wide
+```
+
+11. Please follow up the steps to do [end-to-end verification of service in scale-out cluster](https://github.com/CentaurusInfra/arktos/issues/1143)
+    Note: To be done
+
+
+### Method #2 - Install Flannel in daemonset mode (For 1 TP X 1+N RPs is under working)
 
-1. Expected last line of output: "Resource Partition Cluster is Running ..."
+Issues:
+1. ./hack/arktos-up-scale-out-poc.sh should not install flannel in process mode when SCALE_OUT_TP_ENABLE_DAEMONSET=true
+   Need bash code change
+2. arktos-flannel.deamonset.yaml does not work for two worker nodes because they join RP1 cluster in "NotReady" state because flannel pods are Pending
+
+Progress:
+3. arktos-flannel.deamonset.yaml works for RP1 cluser master node after manual changes and nginx pods can be deployed on RP1 cluster master node
 
 ### Patching Network Routing Across RPs
 Depending on your situation, you may need to change instruction properly - the bottom line is pods from one RP should be able to access pods of other RP.
@@ -89,17 +362,234 @@ On both RP nodes,
 2. manually add relevant routing entries of each node, so that each routing table is complete for all nodes of whole cluster across RPs, e.g. (assuming pod cidr of rp0 is 10.244.0.0/16, rp1 10.245.0.0/16)
 
 on RP1,
-```
-sudo ip r add 10.245.0.0/24 via [RP2-IP]
+```bash
+   sudo ip r add 10.245.0.0/24 via [RP2-IP]
 ```
 
 on RP2
+```bash
+   sudo ip r add 10.244.0.0/24 via [RP1-IP]
 ```
-sudo ip r add 10.244.0.0/24 via [RP1-IP]
+
+### Setting up TPs
+1. Make sure hack/arktos-up.sh can be run at the box
+
+2. Set up environment variables
+
+```bash
+   # optional, used for cloud KCM only but not tested
+   export SCALE_OUT_PROXY_IP=[PROXY_IP]
+   export SCALE_OUT_PROXY_PORT=8888
+   export TENANT_SERVER_NAME=tp-name (e.g. tp1)
+
+   # required
+   export ARKTOS_NO_CNI_PREINSTALLED=y
+   export SCALE_OUT_TP_ENABLE_DAEMONSET=true
+
+   export IS_RESOURCE_PARTITION=false
+   export RESOURCE_SERVER=[RP1_IP]<,[RP2_IP]>
+   export TENANT_PARTITION_SERVICE_SUBNET=[service-ip-cidr]
 ```
 
-### Test Cluster
+an examplative allocation for 2 TPs could be
+
+| tp1 | tp2 |
+| --- | --- |
+| 10.0.0.0/16 | 10.1.0.0/16 |
+
+3. Run ./hack/arktos-up-scale-out-poc.sh 
+```bash
+   Expected last line of output: "Tenant Partition Cluster is Running ..."
+```
+
+Note:
+As certificates generating and sharing is confusing and time consuming in local test environment. We will use insecure mode for local test for now. Secured mode can be added back later when main goal is acchieved.
+
+### Setting up RPs
+1. Make sure hack/arktos-up.sh can be run at the box
+
+2. Set up environment variables
+
+```bash
+export ARKTOS_NO_CNI_PREINSTALLED=y
+export SCALE_OUT_TP_ENABLE_DAEMONSET=true
+
+export IS_RESOURCE_PARTITION=true
+export TENANT_SERVER=[TP1_IP]<,[TP2_IP]>
+export RESOURCE_PARTITION_POD_CIDR=[pod-cidr]
+```
+
+an examplative allocation of pod cidr for 2 RPs could be
+
+| rp1 | rp2 |
+| --- | --- |
+| 10.244.0.0/16 | 10.245.0.0/16 |
+
+3. Run ./hack/arktos-up-scale-out-poc.sh (changes: flannel is not installed in process mode if SCALE_OUT_TP_ENABLE_DAEMONSET=true)
+```bash
+   Expected last line of output: "Waiting for node ready" because flannel network plugin is not installed yet
+```
+
+4. On TP1 node, run ./cluster/kubectl.sh apply -f arktos-flannel.deamonset.yaml
+```bash
+   ./cluster/kubectl.sh apply -f arktos-flannel.deamonset.yaml
+   ./cluster/kubectl.sh get ds
+   ./cluster/kubectl.sh get pods --all-namespaces
+```
+5. On RP1 node, you will see expected last line of output: "Resource Partition Cluster is Running ..."
+   and check node status
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+```bash
+   NAME               STATUS   ROLES    AGE     VERSION
+   ip-172-31-13-237   Ready    <none>   3m53s   v0.9.0
+```
+6. Test whether the ngnix application can be deployed successfully
+```bash
+   ./cluster/kubectl.sh run nginx --image=nginx --replicas=2
+   ./cluster/kubectl.sh get pods -n --all-namespaces -o wide
+```
+```bash
+   NAMESPACE     NAME                                              HASHKEY               READY   STATUS    RESTARTS   AGE     IP              NODE               NOMINATED NODE   READINESS GATES
+   default       nginx-5d79788459-dwcmx                            8933199855509516061   1/1     Running   0          13s     10.244.0.187    ip-172-31-13-237   <none>           <none>
+   default       nginx-5d79788459-h8j5h                            5289368905140319516   1/1     Running   0          13s     10.244.0.186    ip-172-31-13-237   <none>           <none>
+   kube-system   coredns-default-ip-172-31-5-56-67d8d65fb8-ntxmz   2333267612604322023   1/1     Running   0          11m     10.244.0.182    ip-172-31-13-237   <none>           <none>
+   kube-system   kube-dns-554c5866fc-425fz                         1986526451047783165   3/3     Running   0          11m     10.244.0.181    ip-172-31-13-237   <none>           <none>
+   kube-system   kube-flannel-ds-kq7bf                             2755923839726458021   1/1     Running   0          4m48s   172.31.13.237   ip-172-31-13-237   <none>           <none>
+   kube-system   virtlet-xwt84                                     7194442764852737610   3/3     Running   0          4m43s   172.31.13.237   ip-172-31-13-237   <none>           <none>
+```
+   Delete nginx application:
+```bash
+   ./cluster/kubectl.sh delete deployment/nginx
+   ./cluster/kubectl.sh get pods -n default -o wide
+```
+
+### Join worker node into RP1 cluster
+1. Make sure hack/arktos-up.sh can be run at the box
 
+2. Ensure following worker secret files copied from the master node
+```bash
+   mkdir -p /tmp/arktos
+   scp -i "/home/ubuntu/AWS/keypair/CarlXieKeyPairAccessFromWin.pem" ubuntu@ip-172-31-13-237:/var/run/kubernetes/kubelet.kubeconfig /tmp/arktos/kubelet.kubeconfig
+   scp -i "/home/ubuntu/AWS/keypair/CarlXieKeyPairAccessFromWin.pem" ubuntu@ip-172-31-13-237:/var/run/kubernetes/client-ca.crt /tmp/arktos/client-ca.crt
+   ls -alg /tmp/arktos
+```
+```bash
+   -rw-r--r-- 1 ubuntu 1310 Nov 19 00:11 client-ca.crt
+   -rw-r--r-- 1 ubuntu  312 Nov 19 00:11 kubelet.kubeconfig
+```
+
+3. Clean up the directories - /etc/cni/net.d/ and /opt/cni/bin/ as well as the processes - kubelet and flannel
+```bash
+   sudo ls -alg /opt/cni/bin/; sudo rm -r /opt/cni/bin/*; sudo ls -alg /opt/cni/bin/
+
+   sudo ls -alg /etc/cni/net.d/
+   sudo rm -r /etc/cni/net.d/bridge.conf
+   sudo rm -r /etc/cni/net.d/10-flannel.conflist
+   sudo ls -alg /etc/cni/net.d/
+```
+```bash
+   sudo kill -9 `ps -ef |grep kubelet |grep -v grep |awk '{print $2}'`
+   sudo kill -9 `ps -ef |grep flannel |grep -v grep |awk '{print $2}'`
+```
+
+4. Set up environment variables
+```bash
+   export ARKTOS_NO_CNI_PREINSTALLED=y
+   export SCALE_OUT_TP_ENABLE_DAEMONSET=true
+   export KUBELET_IP=`hostname -i`; echo $KUBELET_IP
+```
+
+5. Start worker node to join RP1 cluster
+```bash
+   ./hack/arktos-worker-up.sh
+```
+Note: here we try to join two worker nodes into RP1 clister
+
+6. On RP1 node, check node status
+```bash
+   ./cluster/kubectl.sh get nodes
+```
+```bash
+   NAME               STATUS     ROLES    AGE   VERSION
+   ip-172-31-13-237   Ready      <none>   23m     v0.9.0
+   ip-172-31-26-244   NotReady   <none>   4m10s   v0.9.0
+   ip-172-31-29-26    NotReady   <none>   12m     v0.9.0
+```
+
+7. On TP1 node, check the status of flannel pods running on worker nodes
+```bash
+   ./cluster/kubectl.sh get pods --all-namespaces -o wide
+```
+```bash
+   NAMESPACE     NAME                                              HASHKEY               READY   STATUS    RESTARTS   AGE   IP              NODE               NOMINATED NODE   READINESS GATES
+   kube-system   coredns-default-ip-172-31-5-56-67d8d65fb8-ntxmz   2333267612604322023   1/1     Running   0          97m   10.244.0.182    ip-172-31-13-237   <none>           <none>
+   kube-system   kube-dns-554c5866fc-425fz                         1986526451047783165   3/3     Running   0          97m   10.244.0.181    ip-172-31-13-237   <none>           <none>
+   kube-system   kube-flannel-ds-4bm9z                             8382328744370754654   0/1     Pending   0          73m   <none>          ip-172-31-26-244   <none>           <none>
+   kube-system   kube-flannel-ds-kq7bf                             2755923839726458021   1/1     Running   0          90m   172.31.13.237   ip-172-31-13-237   <none>           <none>
+   kube-system   kube-flannel-ds-mp2vr                             389084117387529988    0/1     Pending   0          81m   <none>          ip-172-31-29-26    <none>           <none>
+   kube-system   virtlet-xwt84                                     7194442764852737610   3/3     Running   0          90m   172.31.13.237   ip-172-31-13-237   <none>           <none>
+```
+   Check the flannel pod bound to worker node#1 - ip-172-31-29-26 and this means scheduler works
+```bash
+   grep kube-flannel-ds-mp2vr /tmp/*.log
+```
+```bash
+/tmp/etcd.log:2021-11-19 00:14:13.769884 D | etcdserver/api/v3rpc: start time = 2021-11-19 00:14:13.769439287 +0000 UTC m=+965.241791675, time spent = 410.69µs, remote = 172.31.5.56:44484, response type = /etcdserverpb.KV/Txn, request count = 1, request size = 4532, response count = 0, response size = 40, request content = compare:<target:MOD key:"/registry/pods/system/kube-system/kube-flannel-ds-mp2vr" mod_revision:0 > success:<request_put:<key:"/registry/pods/system/kube-system/kube-flannel-ds-mp2vr" value_size:4469 >> failure:<>
+/tmp/etcd.log:2021-11-19 00:14:13.774461 D | etcdserver/api/v3rpc: start time = 2021-11-19 00:14:13.773581695 +0000 UTC m=+965.245934082, time spent = 838.191µs, remote = 172.31.5.56:44484, response type = /etcdserverpb.KV/Txn, request count = 1, request size = 4699, response count = 0, response size = 40, request content = compare:<target:MOD key:"/registry/pods/system/kube-system/kube-flannel-ds-mp2vr" mod_revision:581 > success:<request_put:<key:"/registry/pods/system/kube-system/kube-flannel-ds-mp2vr" value_size:4636 >> failure:<request_range:<key:"/registry/pods/system/kube-system/kube-flannel-ds-mp2vr" > >
+/tmp/etcd.log:2021-11-19 00:14:13.778167 D | etcdserver/api/v3rpc: start time = 2021-11-19 00:14:13.777433639 +0000 UTC m=+965.249785996, time spent = 703.326µs, remote = 172.31.5.56:44600, response type = /etcdserverpb.KV/Txn, request count = 1, request size = 1098, response count = 0, response size = 40, request content = compare:<target:MOD key:"/registry/events/system/kube-system/kube-flannel-ds-mp2vr.16b8ca4306decbd0" mod_revision:0 > success:<request_put:<key:"/registry/events/system/kube-system/kube-flannel-ds-mp2vr.16b8ca4306decbd0" value_size:1006 lease:4007215437127664378 >> failure:<>
+/tmp/kube-apiserver0.log:I1119 00:14:13.774921   21594 wrap.go:47] POST /api/v1/tenants/system/namespaces/kube-system/pods/kube-flannel-ds-mp2vr/binding: (2.17936ms) 201 [hyperkube/v0.9.0 (linux/amd64) kubernetes/$Format/scheduler 172.31.5.56:35922]
+/tmp/kube-controller-manager.log:I1119 00:14:13.772080   22176 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-flannel-ds", UID:"949ae64b-5a5e-4bbd-90e6-4afac60a9664", APIVersion:"apps/v1", ResourceVersion:"422", FieldPath:"", Tenant:"system"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-flannel-ds-mp2vr
+/tmp/kube-controller-manager.log:I1119 00:14:13.775441   22176 vm_controller.go:62] in vm controller, pod kube-flannel-ds-mp2vr is updated
+/tmp/kube-scheduler.log:I1119 00:14:13.770969   22179 eventhandlers.go:183] add event for unscheduled pod system/kube-system/kube-flannel-ds-mp2vr
+/tmp/kube-scheduler.log:I1119 00:14:13.771835   22179 scheduler.go:576] Attempting to schedule pod: system/kube-system/kube-flannel-ds-mp2vr
+/tmp/kube-scheduler.log:I1119 00:14:13.772241   22179 default_binder.go:53] Attempting to bind system/kube-system/kube-flannel-ds-mp2vr to ip-172-31-29-26
+/tmp/kube-scheduler.log:I1119 00:14:13.775202   22179 eventhandlers.go:215] delete event for unscheduled pod system/kube-system/kube-flannel-ds-mp2vr
+/tmp/kube-scheduler.log:I1119 00:14:13.775217   22179 eventhandlers.go:239] add event for scheduled pod system/kube-system/kube-flannel-ds-mp2vr
+/tmp/kube-scheduler.log:I1119 00:14:13.776080   22179 scheduler.go:741] pod system/kube-system/kube-flannel-ds-mp2vr is bound successfully on node "ip-172-31-29-26", 2 nodes evaluated, 1 nodes were found feasible.
+
+```
+   Check the flannel pod bound to worker node#1 - ip-172-31-26-244 and this means scheduler works
+```bash
+   grep kube-flannel-ds-4bm9z /tmp/*.log  
+```
+```bash
+/tmp/etcd.log:2021-11-19 00:22:07.565517 D | etcdserver/api/v3rpc: start time = 2021-11-19 00:22:07.565207744 +0000 UTC m=+1439.037560188, time spent = 270.78µs, remote = 172.31.5.56:44484, response type = /etcdserverpb.KV/Txn, request count = 1, request size = 4534, response count = 0, response size = 40, request content = compare:<target:MOD key:"/registry/pods/system/kube-system/kube-flannel-ds-4bm9z" mod_revision:0 > success:<request_put:<key:"/registry/pods/system/kube-system/kube-flannel-ds-4bm9z" value_size:4471 >> failure:<>
+/tmp/etcd.log:2021-11-19 00:22:07.569952 D | etcdserver/api/v3rpc: start time = 2021-11-19 00:22:07.569546404 +0000 UTC m=+1439.041898773, time spent = 364.907µs, remote = 172.31.5.56:44484, response type = /etcdserverpb.KV/Txn, request count = 1, request size = 4702, response count = 0, response size = 40, request content = compare:<target:MOD key:"/registry/pods/system/kube-system/kube-flannel-ds-4bm9z" mod_revision:637 > success:<request_put:<key:"/registry/pods/system/kube-system/kube-flannel-ds-4bm9z" value_size:4639 >> failure:<request_range:<key:"/registry/pods/system/kube-system/kube-flannel-ds-4bm9z" > >
+/tmp/etcd.log:2021-11-19 00:22:07.575337 D | etcdserver/api/v3rpc: start time = 2021-11-19 00:22:07.573315982 +0000 UTC m=+1439.045668436, time spent = 1.978649ms, remote = 172.31.5.56:44600, response type = /etcdserverpb.KV/Txn, request count = 1, request size = 1099, response count = 0, response size = 40, request content = compare:<target:MOD key:"/registry/events/system/kube-system/kube-flannel-ds-4bm9z.16b8cab1573a4438" mod_revision:0 > success:<request_put:<key:"/registry/events/system/kube-system/kube-flannel-ds-4bm9z.16b8cab1573a4438" value_size:1007 lease:4007215437127665368 >> failure:<>
+/tmp/kube-apiserver0.log:I1119 00:22:07.570424   21594 wrap.go:47] POST /api/v1/tenants/system/namespaces/kube-system/pods/kube-flannel-ds-4bm9z/binding: (1.625024ms) 201 [hyperkube/v0.9.0 (linux/amd64) kubernetes/$Format/scheduler 172.31.5.56:37682]
+/tmp/kube-controller-manager.log:I1119 00:22:07.568051   22176 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-flannel-ds", UID:"949ae64b-5a5e-4bbd-90e6-4afac60a9664", APIVersion:"apps/v1", ResourceVersion:"584", FieldPath:"", Tenant:"system"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-flannel-ds-4bm9z
+/tmp/kube-controller-manager.log:I1119 00:22:07.571175   22176 vm_controller.go:62] in vm controller, pod kube-flannel-ds-4bm9z is updated
+/tmp/kube-scheduler.log:I1119 00:22:07.566906   22179 eventhandlers.go:183] add event for unscheduled pod system/kube-system/kube-flannel-ds-4bm9z
+/tmp/kube-scheduler.log:I1119 00:22:07.567769   22179 scheduler.go:576] Attempting to schedule pod: system/kube-system/kube-flannel-ds-4bm9z
+/tmp/kube-scheduler.log:I1119 00:22:07.568218   22179 default_binder.go:53] Attempting to bind system/kube-system/kube-flannel-ds-4bm9z to ip-172-31-26-244
+/tmp/kube-scheduler.log:I1119 00:22:07.570654   22179 scheduler.go:741] pod system/kube-system/kube-flannel-ds-4bm9z is bound successfully on node "ip-172-31-26-244", 3 nodes evaluated, 1 nodes were found feasible.
+/tmp/kube-scheduler.log:I1119 00:22:07.570993   22179 eventhandlers.go:215] delete event for unscheduled pod system/kube-system/kube-flannel-ds-4bm9z
+/tmp/kube-scheduler.log:I1119 00:22:07.571023   22179 eventhandlers.go:239] add event for scheduled pod system/kube-system/kube-flannel-ds-4bm9z
+```
+
+8. On worker node #1 - ip-172-31-29-26, check the log of kube-flannel-ds-mp2vr and directory /var/log/pods is blank 
+```bash
+   ls -al /var/log/pods
+```
+```bash
+total 8
+drwxr-xr-x  2 root root   4096 Nov 17 23:48 .
+drwxrwxr-x 13 root syslog 4096 Nov 18 06:25 ..
+```
+
+9.  On worker node #1 - ip-172-31-26-244, check the log of kube-flannel-ds-4bm9z` and directory /var/log/pods is blank
+```bash
+   ls -al /var/log/pods
+```
+```bash
+total 8
+drwxr-xr-x  2 root root   4096 Nov 18 00:09 .
+drwxrwxr-x 13 root syslog 4096 Nov 18 06:25 ..
+```
+
+### Test Cluster
 1. Use kubectl with kubeconfig. For example:
 
 ```
diff --git a/docs/setup-guide/setup-dev-env.md b/docs/setup-guide/setup-dev-env.md
index 7615a735d07..b6049c3dbac 100644
--- a/docs/setup-guide/setup-dev-env.md
+++ b/docs/setup-guide/setup-dev-env.md
@@ -1,28 +1,22 @@
-## Set up developer environment
+## Set up developer environment (updated on 2021-11-16)
 
-Note: tested on AWS EC2 Ubuntu 16.04 x86 image.
- 
+Note: Tested on AWS EC2 Ubuntu 16.04/18.04/20.04 x86 image.
 
 ### Clone repo
 ```
 $ mkdir -p go/src/
 $ cd go/src/
-$ git clone https://github.com/futurewei-cloud/arktos
+$ git clone https://github.com/centaurusinfra/arktos
 ```
 
-Note: the following steps can be simplified by running hack/setup-dev-node.sh
-
-### Install Golang
+### Install needed packages (docker, make, gcc, jq and golang)
 ```
-$ sudo apt-get update
-# $ sudo apt-get -y upgrade // optional
-$ cd /tmp
-$ wget https://dl.google.com/go/go1.12.9.linux-amd64.tar.gz
-$ tar -xvf go1.12.9.linux-amd64.tar.gz
-$ sudo mv go /usr/local
-$ rm go1.12.9.linux-amd64.tar.gz
+$ cd arktos
+$ ./hack/setup-dev-node.sh
 ```
-Add the following lines to ~/.profile
+
+### Update your account's profile
+Add the following lines into the profile ~/.profile
 ```
 GOROOT=/usr/local/go
 GOPATH=$HOME/go
@@ -31,54 +25,12 @@ PATH=$GOPATH/bin:$GOROOT/bin:$PATH
 Update the current shell session
 ```
 $ source ~/.profile
+$ echo $PATH
 ```
 
-### Install gcc and make. There might be an issue to build images. It can be fixed by running "git tag -a v2.7.4"
-```
-$ sudo apt install build-essential
-```
-
-### Install Docker
-```
-# sudo apt-get update -y -q
-
-# sudo apt-get install \
-    apt-transport-https \
-    ca-certificates \
-    curl \
-    gnupg-agent \
-    software-properties-common -y -q
-
-$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
-# sudo apt-key fingerprint 0EBFCD88
-
-# sudo add-apt-repository \
-   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
-   $(lsb_release -cs) \
-   stable"
-
-# sudo apt-get update -y -q
-# sudo apt-get install docker-ce docker-ce-cli containerd.io -y -q
-# sudo gpasswd -a $USER docker
-```
-
-### Install crictl
-```
-$ cd /tmp
-$ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.17.0/crictl-v1.17.0-linux-amd64.tar.gz
-# sudo tar zxvf crictl-v1.17.0-linux-amd64.tar.gz -C /usr/local/bin
-$ rm -f crictl-v1.17.0-linux-amd64.tar.gz
-
-$ touch /tmp/crictl.yaml
-$ echo runtime-endpoint: unix:///run/containerd/containerd.sock >> /tmp/crictl.yaml
-$ echo image-endpoint: unix:///run/containerd/containerd.sock >> /tmp/crictl.yaml
-$ echo timeout: 10 >> /tmp/crictl.yaml
-$ echo debug: true >> /tmp/crictl.yaml
-# sudo mv /tmp/crictl.yaml /etc/crictl.yaml
-
-# mkdir -p /etc/containerd
-# sudo rm -rf /etc/containerd/config.toml
-# sudo containerd config default > /tmp/config.toml
-# sudo mv /tmp/config.toml /etc/containerd/config.toml
-# sudo systemctl restart containerd
+### Ensure the permisson of others for file /var/run/docker.sock should be readable and writable
+Normally if the machine is rebooted, the permission of this file is changed to default permission.
+Please run the command to add the permission for 'others' using sudo
+```bash
+sudo chmod o+rw /var/run/docker.sock; sudo ls -al /var/run/docker.sock
 ```
diff --git a/docs/setup-guide/single-node-dev-scale-up-cluster-with-Mizar.md b/docs/setup-guide/single-node-dev-scale-up-cluster-with-Mizar.md
new file mode 100644
index 00000000000..813b118c6b8
--- /dev/null
+++ b/docs/setup-guide/single-node-dev-scale-up-cluster-with-Mizar.md
@@ -0,0 +1,546 @@
+# How to Setup a Dev Cluster of single node on AWS EC2 instance running Ubuntu 18.04, 20.04 x86 
+
+0. Pre-requisite - setup local development environment with Mizar
+
+0.1) Create AWS EC2 instance 
+```
+     - OS: Ubuntu 18.04 and Ubuntu 20.04
+     - Instance Type: t2.2xlarge
+     - Storage Size: 128GB or more
+
+```
+
+0.2) Follow up the step 1 in following procedure to upgrade kernel to 5.6.0-rc2
+
+```
+   https://github.com/Click2Cloud-Centaurus/arktos/blob/default-cni-mizar/docs/setup-guide/arktos-with-mizar-cni.md
+```
+
+```bash
+   uname -a
+   wget https://raw.githubusercontent.com/CentaurusInfra/mizar/dev-next/kernelupdate.sh
+   sudo bash kernelupdate.sh
+   uname -a
+```  
+
+0.3) Follow up the step 2 in following procedure to clone the Arktos repository and install the required dependencies
+```
+   https://github.com/Click2Cloud-Centaurus/arktos/blob/default-cni-mizar/docs/setup-guide/arktos-with-mizar-cni.md
+```
+
+```bash
+   git clone https://github.com/CentaurusInfra/arktos.git ~/go/src/k8s.io/arktos 
+   sudo bash $HOME/go/src/k8s.io/arktos/hack/setup-dev-node.sh
+   echo export PATH=$PATH:/usr/local/go/bin\ >> ~/.profile
+   echo cd \$HOME/go/src/k8s.io/arktos >> ~/.profile
+   source ~/.profile
+```
+
+0.4) Start Arktos cluster in default mode (without mizar)
+```bash
+   sudo groupadd docker
+   sudo usermode -aG docker $USER
+   make clean
+   ./hack/arktos-up.sh
+```
+
+OR
+
+```bash
+   sudo chmod o+rw /var/run/docker.sock; ls -al /var/run/docker.sock
+   make clean
+   ./hack/arktos-up.sh
+```
+   In another window:
+```bash
+   ./hack/arktos-up.sh get all --all-namespaces
+   ./hack/arktos-up.sh get all --all-namespaces -AT
+   ./cluster/kubectl.sh run nginx --image=nginx --replicas=2
+   ./cluster/kubectl.sh get pods -o wide
+   ./cluster/kubectl.sh exec -ti nginx-5d79788459-gz74w -n default -- /bin/bash
+   # curl 10.88.0.5  (You will see "Welcome to nginx!" title)
+   # curl 10.88.0.4  (You will see "Welcome to nginx!" title) 
+   # curl 10.88.0.3  (You will see error "# curl 10.88.0.4  (You will see "Welcome to nginx!" title)")
+```
+
+
+
+0.5) Apply PR1114 to give "Support for Mizar CNI in arktos-up"
+     PR1114: https://github.com/CentaurusInfra/arktos/pull/1114
+
+```bash
+   cd ~/go/src/k8s.io/arktos
+   git checkout master
+   git fetch origin pull/1114/head:pr1114
+   git checkout -b CarlXie_20211101-Mizar
+   git rebase pr1114
+   git log
+   git show c4a0ff73ced143fc954c6e34670dd10780f1eb5a
+```
+
+0.6) Start Arktos cluster with Mizar
+     Note: You will see the warning "Waiting for node ready at api server" for 5 minutes more because it takes time for Mizar to compile codes and Arktos cluster will be up
+
+```bash
+   sudo chmod o+rw /var/run/docker.sock; ls -al /var/run/docker.sock  (if reboot machine)
+   sudo rm -rf /opt/cni/bin/*; sudo ls -alg /opt/cni/bin/  (keep clean)
+   sudo rm -rf /etc/cni/net.d/*; sudo ls -alg /etc/cni/net.d/  (keep clean)
+   make clean
+   CNIPLUGIN=mizar ./hack/arktos-up.sh
+```
+
+0.7) Follow up the following procedure to verify whether Mizar CRDs (vpcs,subnets,droplets,bouncers,dividers and endpoints) are all in "Provisioned" states. If not, reboot machine and go back to above step 0.6) until all Mizar CRDs are in "Provisioned" states
+
+```
+     https://github.com/CentaurusInfra/mizar/wiki/Mizar-Cluster-Health-Criteria
+```
+
+```bash
+   ./cluster/kubectl.sh get crds
+```
+```
+    NAME                            AGE
+    bouncers.mizar.com              10m
+    dividers.mizar.com              10m
+    droplets.mizar.com              10m
+    endpoints.mizar.com             10m
+    networks.arktos.futurewei.com   10m
+    subnets.mizar.com               10m
+    vpcs.mizar.com                  10m
+```
+```bash
+   ./cluster/kubectl.sh get crds
+   ./cluster/kubectl.sh get vpcs
+   ./cluster/kubectl.sh get subnets
+   ./cluster/kubectl.sh get droplets
+   ./cluster/kubectl.sh get bouncers
+   ./cluster/kubectl.sh get dividers
+```
+```bash
+   ./cluster/kubectl.sh get endpoints.mizar.com
+   ./cluster/kubectl.sh get networks
+```
+
+0.8) Verify whether the pods of Mizar and Mizar Operator are all in "Running" states
+```bash
+   ./cluster/kubectl.sh get pods -o wide
+   ./cluster/kubectl.sh get pods -o wide -AT
+```
+
+1. Create two network pods and test each other using ping command to test - 1. Basic pod connectivity: pods in “system” tenant should be able to communicate with each other 
+```
+     https://github.com/CentaurusInfra/mizar/wiki/Mizar-Cluster-Health-Criteria
+```
+
+netpod-single-node.yaml
+```
+apiVersion: v1
+kind: Pod
+metadata:
+  name: netpod1
+  labels:
+    app: netpod
+spec:
+  restartPolicy: OnFailure
+  terminationGracePeriodSeconds: 10
+  containers:
+  - name: netctr
+    image: mizarnet/testpod
+    ports:
+    - containerPort: 9001
+      protocol: TCP
+    - containerPort: 5001
+      protocol: UDP
+    - containerPort: 7000
+      protocol: TCP
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: netpod2
+  labels:
+    app: netpod
+spec:
+  restartPolicy: OnFailure
+  terminationGracePeriodSeconds: 10
+  containers:
+  - name: netctr
+    image: mizarnet/testpod
+    ports:
+    - containerPort: 9001
+      protocol: TCP
+    - containerPort: 5001
+      protocol: UDP
+    - containerPort: 7000
+      protocol: TCP
+
+```
+```bash
+   vi ~/TMP/netpod-single-node.yaml (copy from netpod-single-node.yaml)
+   cat ~/TMP/netpod-single-node.yaml
+   ./cluster/kubectl.sh apply -f ~/TMP/netpod-single-node.yaml
+   ./cluster/kubectl.sh get pods -o wide
+   ./cluster/kubectl.sh exec -ti netpod1 -- ping -c2 20.0.0.18 (IP of pod netpod2)
+   ./cluster/kubectl.sh exec -ti netpod2 -- ping -c2 20.0.0.26 (IP of pod netpod1)
+   ./cluster/kubectl.sh exec -ti netpod1 -n default -- /bin/bash
+   # curl 20.0.0.18:7000
+   netpod2
+   # curl 20.0.0.26:7000
+   netpod1
+   # exit
+```
+
+
+2. Follow up the procedure of issue 1142 at https://github.com/CentaurusInfra/arktos/issues/1142 to test - 2.	General pod connectivity: pods in same non-system tenant should be able to communicate;
+
+2.1) Create new tenant 'mytenant'
+
+```bash
+   ./cluster/kubectl.sh create tenant mytenant
+```
+```
+   ./cluster/kubectl.sh get network -T
+   ./cluster/kubectl.sh get service --all-namespaces --tenant mytenant -o wide
+   ./cluster/kubectl.sh get pod --all-namespaces --tenant mytenant -o wide
+   ./cluster/kubectl.sh get deployment --all-namespaces  --tenant mytenant -o wide
+   ./cluster/kubectl.sh get endpoints --all-namespaces --tenant mytenant -o wide
+
+```
+
+2.2) Create the Pod yaml file by adding the tenant 'mytenant' and apply this Pod yaml file
+```bash
+   cat ~/TMP/netpod-arktos-team-single-node.yaml
+```
+```
+apiVersion: v1
+kind: Pod
+metadata:
+  name: my-netpod1
+  labels:
+    app: my-netpod
+  tenant: mytenant
+spec:
+  restartPolicy: OnFailure
+  terminationGracePeriodSeconds: 10
+  containers:
+  - name: netctr
+    image: mizarnet/testpod
+    ports:
+    - containerPort: 9001
+      protocol: TCP
+    - containerPort: 5001
+      protocol: UDP
+    - containerPort: 7000
+      protocol: TCP
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: my-netpod2
+  labels:
+    app: my-netpod
+  tenant: mytenant
+spec:
+  restartPolicy: OnFailure
+  terminationGracePeriodSeconds: 10
+  containers:
+  - name: netctr
+    image: mizarnet/testpod
+    ports:
+    - containerPort: 9001
+      protocol: TCP
+    - containerPort: 5001
+      protocol: UDP
+    - containerPort: 7000
+      protocol: TCP
+```
+```bash
+   ./cluster/kubectl.sh apply -f ~/TMP/netpod-arktos-team-single-node.yaml
+```
+
+2.3) Check whether new pods are in Running state
+
+```bash
+   ./cluster/kubectl.sh get pods -AT
+```
+```
+TENANT     NAMESPACE     NAME                               HASHKEY               READY   STATUS              RESTARTS   AGE
+mytenant   default       my-netpod1                         5743084656396193665   0/1     ContainerCreating   0          48s
+mytenant   default       my-netpod2                         1296240596527502600   0/1     ContainerCreating   0          48s
+mytenant   kube-system   coredns-default-798fbcc5f4-qsrdd   7084949829346585805   0/1     ContainerCreating   0          10m
+system     default       mizar-daemon-wfsdx                 3626414391297043142   1/1     Running             0          126m
+system     default       mizar-operator-6b78d7ffc4-kv6n9    8147745129439049838   1/1     Running             0          126m
+system     default       netpod1                            6285438668331818604   1/1     Running             0          17m
+system     default       netpod2                            996610216049115966    1/1     Running             0          17m
+system     kube-system   coredns-default-798fbcc5f4-gs92h   4660664115769723653   1/1     Running             0          126m
+system     kube-system   kube-dns-554c5866fc-4hx2m          6383051253301223595   3/3     Running             0          126m
+system     kube-system   virtlet-6bc92                      1075534439738712156   3/3     Running             0          121m
+```
+
+```bash
+   ./cluster/kubectl.sh get pods --tenant mytenant
+```
+```
+my-netpod1   5743084656396193665   0/1     ContainerCreating   0          76s
+my-netpod2   1296240596527502600   0/1     ContainerCreating   0          76s
+```
+
+```bash
+   cat /tmp/kubelet.log | grep my-netpod1 |tail -9
+```
+```
+I1101 20:48:06.397537   11072 volume_manager.go:358] Waiting for volumes to attach and mount for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)"
+I1101 20:48:06.397690   11072 volume_manager.go:391] All volumes are attached and mounted for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)"
+I1101 20:48:06.397708   11072 kuberuntime_manager.go:529] No sandbox for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)" can be found. Need to start a new one
+I1101 20:48:06.397723   11072 kuberuntime_manager.go:948] computePodActions got {KillPod:true CreateSandbox:true SandboxID: Attempt:0 NextInitContainerToStart:nil ContainersToStart:[0] ContainersToKill:map[] ContainersToUpdate:map[] ContainersToRestart:[] Hotplugs:{NICsToAttach:[] NICsToDetach:[]}} for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)"
+I1101 20:48:06.398626   11072 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-netpod1", UID:"82f46ddd-2b12-48bd-b3c3-5aaf12265fdb", APIVersion:"v1", ResourceVersion:"2948", FieldPath:"", Tenant:"mytenant"}): type: 'Warning' reason: 'GettingClusterDNS' pod: "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)". For verification - ClusterDNS IP : "10.0.0.94"
+E1101 20:48:09.424185   11072 kuberuntime_sandbox.go:86] CreatePodSandbox for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "7292679bedba779c3e93d22be35ed24bb71d6fdc9decd35810a23424b6548423": rpc error: code = DeadlineExceeded desc = Deadline Exceeded
+E1101 20:48:09.424241   11072 kuberuntime_manager.go:1024] createPodSandbox for pod "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)" failed: rpc error: code = Unknown desc = failed to setup network for sandbox "7292679bedba779c3e93d22be35ed24bb71d6fdc9decd35810a23424b6548423": rpc error: code = DeadlineExceeded desc = Deadline Exceeded
+E1101 20:48:09.424310   11072 pod_workers.go:196] Error syncing pod 82f46ddd-2b12-48bd-b3c3-5aaf12265fdb ("my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)"), skipping: failed to "CreatePodSandbox" for "my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)" with CreatePodSandboxError: "CreatePodSandbox for pod \"my-netpod1_default_mytenant(82f46ddd-2b12-48bd-b3c3-5aaf12265fdb)\" failed: rpc error: code = Unknown desc = failed to setup network for sandbox \"7292679bedba779c3e93d22be35ed24bb71d6fdc9decd35810a23424b6548423\": rpc error: code = DeadlineExceeded desc = Deadline Exceeded"
+I1101 20:48:09.424478   11072 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-netpod1", UID:"82f46ddd-2b12-48bd-b3c3-5aaf12265fdb", APIVersion:"v1", ResourceVersion:"2948", FieldPath:"", Tenant:"mytenant"}): type: 'Warning' reason: 'FailedCreatePodSandBox' Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "7292679bedba779c3e93d22be35ed24bb71d6fdc9decd35810a23424b6548423": rpc error: code = DeadlineExceeded desc = Deadline Exceeded
+
+```
+
+2' Follow the following two documents to test with Mizar team Phu - 2.   General pod connectivity: pods in same non-system tenant should be able to communicate;
+
+```bash
+   https://mizar.readthedocs.io/en/latest/user/getting_started/
+```
+
+```bash
+   https://github.com/CentaurusInfra/mizar/blob/dev-next/docs/design/mp_arktos.md
+```
+
+2'.1) Creating a new VPC using yaml file with procedure at https://mizar.readthedocs.io/en/latest/user/getting_started/
+
+``` bash
+    cat ~/TMP/file_name_here.vpc-tenant-phu.yaml
+```
+
+```
+apiVersion: mizar.com/v1
+kind: Vpc
+metadata:
+  name: vpc-tenant-phu
+spec:
+  ip: "24.0.0.0"
+  prefix: "16"
+  dividers: 1
+  status: "Init"
+```
+
+2'.2) Creating a new Subnet using yaml file with procedure at https://mizar.readthedocs.io/en/latest/user/getting_started/
+
+```bash
+    cat ~/TMP/file_name_here.subnets-phu.yaml
+
+```
+
+Note: The 9019248 below is CNI number of above VPC 'vpc-tenant-phu'.
+```
+apiVersion: mizar.com/v1
+kind: Subnet
+metadata:
+  name: net-tenant-phu
+spec:
+  vni: "9019248"
+  ip: "24.0.24.0"
+  prefix: "24"
+  bouncers: 1
+  vpc: "vpc-tenant-phu"
+  status: "Init"
+```
+
+2'.3) Creating a new tenant 'tenant-phu'
+
+```bash
+   ./cluster/kubectl.sh create tenant tenant-phu
+   ./cluster/kubectl.sh get tenant
+   ./cluster/kubectl.sh get network --tenant tenant-phu
+```
+
+2'.4) Creating a new Network using yaml file with procedure at https://github.com/CentaurusInfra/mizar/blob/dev-next/docs/design/mp_arktos.md
+
+```bash
+   cat ~/TMP/file_name_here.network-tenant-phu.yaml
+```
+
+```
+apiVersion: arktos.futurewei.com/v1
+kind: Network
+metadata:
+  name: network-tenant-phu
+  tenant: tenant-phu
+spec:
+  type: mizar
+  vpcID: vpc-tenant-phu
+```
+
+2'.5) Creating a new Network using yaml file with procedure at https://github.com/CentaurusInfra/mizar/blob/dev-next/docs/design/mp_arktos.md
+
+```bash
+   cat ~/TMP/file_name_here.pod-tenant-phu.yaml
+```
+
+```
+   apiVersion: v1
+kind: Pod
+metadata:
+  name: nginx-tenant-phu
+  namespace: default
+  tenant: tenant-phu
+  labels:
+    arktos.futurewei.com/network: network-tenant-phu
+spec:
+  containers:
+  - name: nginx
+    image: nginx
+    ports:
+      - containerPort: 443
+```
+
+2'.6) Check the pod of 'nginx-tenant-phu' and see it is in ContainerCreating
+
+```bash
+   cluster/kubectl.sh get pod --tenant system |grep nginx-tenant-phu
+   cluster/kubectl.sh get pod -AT |grep nginx-tenant-phu
+```
+
+2'.7) Check the log of pod mizar-operator
+
+```bash
+   ./cluster/kubectl.sh get pods -AT |grep mizar-operator
+   ./cluster/kubectl.sh log mizar-operator-6b78d7ffc4-9fdh9
+```
+
+```
+<snip>
+Scheduled 1 tasks of which:
+* 1 ran successfully:
+    - 1 BouncerProvisioned(param=<mizar.common.wf_param.HandlerParam object at 0x7f20c8235c10>)
+
+This progress looks :) because there were no failed tasks or missing dependencies
+
+===== Luigi Execution Summary =====
+
+[2021-11-04 17:57:01,937] kopf.objects         [INFO    ] [default/net-tenant-phu-b-9fcc6f7c-f5a9-414d-8efc-7b73576e5e9d] Handler 'bouncer_opr_on_bouncer_provisioned' succeeded.
+[2021-11-04 17:57:01,938] kopf.objects         [INFO    ] [default/net-tenant-phu-b-9fcc6f7c-f5a9-414d-8efc-7b73576e5e9d] Updating is processed: 1 succeeded; 0 failed.
+```
+
+2'.8) Check the log of pod mizar-daemon
+
+```bash
+   ./cluster/kubectl.sh get pods -AT |grep mizar-daemon
+   ./cluster/kubectl.sh log mizar-daemon-5tnlb
+```
+
+```
+<snip>
+INFO:root:Consuming interfaces for pod: nginx-mytenant-22-default-mytenant Current Queue: []
+INFO:root:Deleting interfaces for pod coredns-default-798fbcc5f4-qdkpl-kube-system-mytenant-1 with interfaces []
+ERROR:root:Timeout, no new interface to consume! coredns-default-798fbcc5f4-qdkpl-kube-system-mytenant-1 []
+INFO:root:Consumed
+INFO:root:Deleting interfaces for pod coredns-network-tenant-phu-6776c9c4cb-c7l7c-kube-system-tenant-phu with interfaces []
+```
+
+
+3. Follow up the procedure of issue 1142 at https://github.com/CentaurusInfra/arktos/issues/1142 to test - 3.	General pod isolation: a pod in one tenant may not communicate with pods in other tenants;
+```bash
+   Stop test when step 2 failed to test
+```
+
+4. Build two node scale-up cluster to test - 4.	Worker node joining: new worker node should be able to join cluster, and basic pod connectivity should be provided.
+```
+   Please see the procedure at https://github.com/q131172019/arktos/blob/CarlXie_singleNodeArktosCluster/docs/setup-guide/multi-node-dev-scale-up-cluster-with-Mizar.md to create new worker node to join cluster
+```
+
+5. General pod connectivity: pods in system tenant should be able to communicate in another single-node cluster with Mizar
+
+Note: if you do this test in same cluster as step 2, when you create new VPC for system tenant, ensure the ip in spec part below is not same as 20.0.0.0.
+
+5.1) Creating a new VPC using yaml file with procedure at https://mizar.readthedocs.io/en/latest/user/getting_started/
+
+``` bash
+    cat ~/TMP/mizar-system-tenant/vpc.yaml
+```
+
+```
+apiVersion: mizar.com/v1
+kind: Vpc
+metadata:
+  name: vpc-ying
+spec:
+  ip: "21.0.0.0"
+  prefix: "16"
+  dividers: 1
+  status: "Init"
+```
+
+5.2) Creating a new Subnet using yaml file with procedure at https://mizar.readthedocs.io/en/latest/user/getting_started/
+
+```bash
+    cat ~/TMP/mizar-system-tenant/subnet.yaml
+
+```
+
+Note: The 9131671 below is CNI number of above VPC 'vpc-ying'.
+```
+apiVersion: mizar.com/v1
+kind: Subnet
+metadata:
+  name: net-ying
+spec:
+  vni: "9131671"
+  ip: "21.0.0.0"
+  prefix: "24"
+  bouncers: 1
+  vpc: "vpc-ying"
+  status: "Init"
+```
+
+5.3) Creating a new Network using yaml file with procedure at https://github.com/CentaurusInfra/mizar/blob/dev-next/docs/design/mp_arktos.md
+
+```bash
+   cat ~/TMP/mizar-system-tenant/network.yaml
+```
+
+```
+apiVersion: arktos.futurewei.com/v1
+kind: Network
+metadata:
+  name: network-ying
+spec:
+  type: mizar
+  vpcID: vpc-ying
+```
+
+5.4) Creating a new Netwrk using yaml file with procedure at https://github.com/CentaurusInfra/mizar/blob/dev-next/docs/design/mp_arktos.md
+
+```bash
+   cat ~/TMP/mizar-system-tenant/pod.yaml
+```
+
+```
+cat ~/TMP/mizar-system-tenant/pod.yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  name: ying-nginx
+  labels:
+    arktos.futurewei.com/network: network-ying
+spec:
+  containers:
+  - name: nginx
+    image: nginx
+    ports:
+      - containerPort: 443
+```
+
+5.5) Check the pod of 'ying-nginx' and see it is in ContainerCreating
+
+```bash
+   cluster/kubectl.sh get pod --tenant system |grep ying-nginx
+   cluster/kubectl.sh get pod -AT |grep ying-nginx
+
+```
+
diff --git a/docs/setup-guide/single-node-dev-scale-up-cluster.md b/docs/setup-guide/single-node-dev-scale-up-cluster.md
new file mode 100644
index 00000000000..ca626d527cc
--- /dev/null
+++ b/docs/setup-guide/single-node-dev-scale-up-cluster.md
@@ -0,0 +1,59 @@
+# How to Setup a Dev Cluster of single node on AWS EC2 instance running Ubuntu 20.04, 18.04, 16.04 x86 
+
+0. Pre-requisite - setup local development environment based on [set up developer environment](setup-dev-env.md)
+
+1. Run script to create a single arktos cluster
+
+```bash
+   ./hack/arktos-up.sh
+```
+
+   Note: If your Dev Cluster of single node is on AWS EC2 instance running Ubuntu 18.04, after you run the command './hack/arktos-up.sh', you may experience the following error. Just simply remove the symbolic link and re-run the command './hack/arktos-up.sh'.
+
+         ln: failed to create symbolic link '/home/ubuntu/go/src/arktos/_output/bin': File exists 
+
+```bash
+   make clean
+   ./hack/arktos-up.sh
+```
+
+2. Open another terminal to use arktos cluster
+```bash
+  ./cluster/kubectl.sh
+  ./cluster/kubectl.sh get nodes
+  ./cluster/kubectl.sh get all --all-namespaces
+```
+```bash
+Alternatively, you can write to the default kubeconfig:
+
+  export KUBERNETES_PROVIDER=local
+
+  ./cluster/kubectl.sh config set-cluster local --server=https://<hostname>:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt
+  ./cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt
+  ./cluster/kubectl.sh config set-context local --cluster=local --user=myself
+  ./cluster/kubectl.sh config use-context local
+  ./cluster/kubectl.sh config get-contexts
+  ./cluster/kubectl.sh
+  ./cluster/kubectl.sh get nodes
+  ./cluster/kubectl.sh get all --all-namespaces
+```
+
+3. Test whether the ngnix application can be deployed successfully
+```bash
+   ./cluster/kubectl.sh run nginx --image=nginx --replicas=2
+```
+```bash
+   ./cluster/kubectl.sh get pod -n default -o wide
+```
+```bash
+   ./cluster/kubectl.sh exec -ti <1st pod> -- curl <IP of 2nd nginx pod>
+```
+```bash
+   ./cluster/kubectl.sh exec -ti <2nd pod> -- curl <IP of 1st nginx pod>
+```
+
+4. Clean up the ngnix deployment
+```bash
+   ./cluster/kubectl.sh delete deployment/nginx
+   ./cluster/kubectl.sh get pod -n default -o wide
+```
diff --git a/hack/scalability/setup_haproxy.sh b/hack/scalability/setup_haproxy.sh
index 7a1f22a1db6..75f30089b06 100755
--- a/hack/scalability/setup_haproxy.sh
+++ b/hack/scalability/setup_haproxy.sh
@@ -79,8 +79,8 @@ function direct-haproxy-logging {
   fi
 
   sudo /bin/su -c "echo '
-$ModLoad imudp
-$UDPServerRun 514
+\$ModLoad imudp
+\$UDPServerRun 514
 local0.* -/var/log/haproxy.log
 ' >> $haproxy_conf"
 
@@ -135,4 +135,4 @@ config_haproxy
 
 run_command_exit_if_failed sudo systemctl restart haproxy
 
-printf "Haproxy is in $(systemctl is-active haproxy) status. Log file: /var/log/haproxy.log.\n"
\ No newline at end of file
+printf "Haproxy is in $(systemctl is-active haproxy) status. Log file: /var/log/haproxy.log.\n"
diff --git a/hack/setup-dev-node.sh b/hack/setup-dev-node.sh
index 9587ba77004..6bfb588dc81 100755
--- a/hack/setup-dev-node.sh
+++ b/hack/setup-dev-node.sh
@@ -22,7 +22,7 @@ set -o pipefail
 
 echo "The script is to help install prerequisites of Arktos development environment"
 echo "on a fresh Linux installation."
-echo "It's been tested on Ubuntu 16.04 LTS and 18.04 LTS."
+echo "It's been tested on Ubuntu 16.04 LTS, 18.04 and 20.04 LTS."
 
 GOLANG_VERSION=${GOLANG_VERSION:-"1.13.9"}
 
@@ -42,5 +42,14 @@ wget https://dl.google.com/go/go${GOLANG_VERSION}.linux-amd64.tar.gz -P /tmp
 sudo tar -C /usr/local -xzf /tmp/go${GOLANG_VERSION}.linux-amd64.tar.gz
 
 echo "Done."
-echo "Please run and add 'export PATH=\$PATH:/usr/local/go/bin' into your shell profile."
+echo ""
+echo "Please add the following lines into your shell profile ~/.profile."
+echo "    GOROOT=/usr/local/go"
+echo "    GOPATH=\$HOME/go"
+echo "    export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin"
+echo ""
+echo "Update the current shell session."
+echo "    $ source ~/.profile"
+echo "    $ echo \$PATH"
+echo ""
 echo "You can proceed to run arktos-up.sh if you want to launch a single-node cluster."