Skip to content

Commit 6f15a24

Browse files
bitsondatadevbitsondatadev
committed
Init commit
0 parents  commit 6f15a24

5 files changed

Lines changed: 210 additions & 0 deletions

File tree

‎.github/workflows/ci.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: ci
2+
on:
3+
push:
4+
branches:
5+
- main
6+
permissions:
7+
contents: write
8+
jobs:
9+
deploy:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v4
13+
- name: Configure Git Credentials
14+
run: |
15+
git config user.name github-actions[bot]
16+
git config user.email 41898282+github-actions[bot]@users.noreply.github.com
17+
- uses: actions/setup-python@v5
18+
with:
19+
python-version: 3.x
20+
- run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
21+
- uses: actions/cache@v4
22+
with:
23+
key: mkdocs-material-${{ env.cache_id }}
24+
path: .cache
25+
restore-keys: |
26+
mkdocs-material-
27+
- run: pip install mkdocs-material
28+
- run: mkdocs gh-deploy --force

‎docs/design.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Design
2+
3+
In this draft design, I'll lay out the rack design and hardware necessary to power dozens of servers.
4+
5+
## Bill of Materials
6+
- [Mobile 1/2 cabinet](https://www.amazon.com/NavePoint-Server-Cabinet-Casters-Shelves/dp/B01I48EJOW/) (or 1/4 cabinet if you want to do a smaller version)
7+
- 2x [6U DIN Mount with cable management](https://www.amazon.com/Rackmount-Din-Rail-Panel-6U/dp/B00SG3PGWK/)
8+
- [Raspberry Pi 4 CM4](https://www.adafruit.com/product/4564)
9+
- [Raspberry Pi 4 Carrier Board](https://www.waveshare.com/cm4-io-base-b.htm)
10+
- POE Switch (Ex. [NETGEAR 24 Port POE Switch](https://www.amazon.com/NETGEAR-24-Port-Gigabit-Ethernet-Unmanaged/dp/B07Z8P4ZPW/))
11+
- [Raspberry Pi DIN Rail Mount](https://www.amazon.com/Winford-Engineering-Raspberry-L-Bracket-Compliant/dp/B083YSWYW1/)
12+
- [NVMe 2242](https://www.amazon.com/Sabrent-DRAM-Less-Internal-Performance-SB-1342-512/dp/B07XVR1KKR/)
13+
- [PoE USB C Splitter](https://www.amazon.com/dp/B0CHW5K5F4) Be careful with PoE USB Splitters. Not all are created equal. The one I linked is 5v at 4 Amp. The [Utronics](https://www.amazon.com/UCTRONICS-PoE-Splitter-USB-C-Compliant/dp/B087F4QCTR/) is only 2.4 Amp. I've also had some of the Utronics fail on me, so it's worth it to get a beefier splitter, especially if you have more to power.
14+
15+
16+
## The Rack
17+
18+
Do you need to do 1/2 a cabinet?
19+
20+
No. You could do this smaller.
21+
22+
But for this example, I'll do 1/2 a cabinet with room for *things* like battery backup / UPS...
23+
24+
25+
- 1/2 Cabinet: $374.00
26+
- 6U DIN Rack/Cable Mgmt: $199.00
27+
28+
Total: $574.00
29+
30+
## The basic host
31+
The basic host configuration is as follows
32+
- PoE Powered Raspberry Pi
33+
- NVMe Attached Disk
34+
- Vertically mounted on DIN rails
35+
- No case required - better airflow and cooling
36+
37+
This ultra low profile design dispenses with the bulky and failure prone PoE HATs. The PoE hats often have fan failures and take up valuable HAT space for other things. The carrier board in this BOM has the NVMe on the bottom of the board. With this design, the disk and PoE do not take up any HAT space and also don't glog up the rack with USB cables. The NVMe is also attached with PCIe speed rather than USB.
38+
39+
Cost
40+
- NVMe (512G): 69$
41+
- Raspberry Pi 4 (8G): 75$
42+
- Waveshare Carrier Board: 30$
43+
- DIN Mount: 15$
44+
- Network Port (Switch / Num Ports): $ 11.45 per port (1)
45+
- POE Splitter: 20$
46+
47+
Total: $221.00 / node
48+
49+
Could you get used gear for this cheap? Yes. Would it be more powerful? Yes. But...
50+
- It's bulky as hell (rackmount or old desktop)
51+
- Power hungry
52+
- Is not expandible
53+
- Doesn't grow past the built in number of SATA ports
54+
55+
56+
## Density
57+
With the above BOM (Bill of Materials) we'll be able to get roughly 18 servers in to 6U. This assumes 2" width per server. If you choose to go with SSD instead of NVMe, it can be cheaper, but you'll need additional DIN mounts and it takes up more rack space. Also add on the cost of the USB cables.
58+
59+
Basically you'd need a 24port switch for every 6U.
60+
61+
So in 7U, you'd have the switch, and 18 servers, with cable management.
62+
63+
In that 22U cabinet, you could fit *54 servers*.... now we're talking.
64+
65+
## UPS
66+
Rack Mounted DIY Battery Packs
67+
68+
Yes, we're going _that_ deep. If you want to of course.
69+
70+
Building batteries from old laptop and car cells is a way to learn how do design power capacity for your datacenter. This is important information if you ever need to design or scope out a cage at a datacenter. Building batteries is also just fun. You can get really creative here with batteries and use all kinds of cells. My personal interest is in batteries that won't catch fire or explode. I'd also prefer batteries that don't off-gas in my office/microdatacenter space.
71+
72+
If you're not familiar with battery types, it's a great time to learn the difference. My personal favorite is [Lithium-titanate batteries](https://en.wikipedia.org/wiki/Lithium-titanate_battery). These are used in things like weather stations where they are not accessible and have very high recharge cycles. They're basically 20+ year batteries. Their power density isn't as high, but they're safe and won't explode and have a much larger temperature range than typical lithium batteries. LiFePO4 are also safe batteries for this kind of thing.

‎docs/index.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Micro Data Center Community
2+
3+
## What is a micro datacenter?
4+
5+
This is largely [Rob Gil's](https://github.com/robgil/microdatacenter) opinion if you're looking for someone to blame, but for the purposes of this project, we'll define it as the following.
6+
7+
> A micro datacenter is a fully functioning, ultra low power, and distributed datacenter. It contains all the functionality of a real datacenter, but small enough that you can run it off a 20A 110V outlet.
8+
9+
Also
10+
11+
> it's bare metal...
12+
13+
Why do I need a datacenter?
14+
15+
As more and more services increase their costs, build walled gardens of content, and exploit our information for profit, many are looking to move to an alternative they can trust, themselves. It isn't necessarily easy to know how to do this in a time and cost efficient manner, especially in an eco-friendly matter. But it does gain you a lot of freedoms. For many of us, it is even fun!
16+
17+
- Privacy and the ability to migrate off of large centralized services to minimize your footprint in surveillance capitalism.
18+
- Lower chatty traffic between the devices on your network and them phoning home.
19+
- Lower your monthly subscription spend.
20+
- Own your digital content.
21+
- Run services to support your own business.
22+
- Small business networking.
23+
- Run AI on your data without training someone else's.
24+
25+
26+
Sure, but what is the maintenance cost? What if I have a power outage or a hardware mailfunction?
27+
28+
This is where having a community is valuable that goes beyond just sharing some quick tips on installation, but also shares design and strategy to mitigate issues like data loss and down time without dedicating yourself to a life of on call duty. When you inevitibly do face an outage, you have a lot of folks hanging around who have experienced much of the same, so they can help dig you out while you're in panic mode, and then cover how to avoid that same issue moving forward.
29+
30+
Many of us starting this community have experience with how large corporations ensure that consumers rarely experience downtime and fortunately for you, they layed a bunch of us off, so we are ready to share this wisdom with many of you! Who knows, as we grow and develop intertwining systems as a community, we may be able to build networks akin to the [ipfs](https://en.wikipedia.org/wiki/InterPlanetary_File_System) that use trusted freinds to hold copies of your data if something goes terribly wrong.
31+
32+
Why a microdatacenter?
33+
34+
Most folks, myself included, have always run some form of lab at home. This comes with a host of challenges and annoyances.
35+
- Enormous power requirements for anything more than 1/2 a rack of servers
36+
- Expensive, power hungry servers
37+
- Old servers that are out of support, clunky, expensive to fix and upgrade, etc
38+
- Overloading those servers with VMs
39+
- Many folks run VMWare ESX, but that can be pricey
40+
- Upgrades? What's that?
41+
- Who actually has the funds to regularly ugprade their lab?
42+
43+
## Upcoming developements
44+
45+
This site is under heavy development, but first, why yet another homelab/selfhosted community?
46+
47+
Our goal is to do a better job at consolidating the tutorials, mental models, and language in the amazing and existing communities ([r/homelab](https://www.reddit.com/r/homelab/), [r/minilab](https://www.reddit.com/r/minilab/), [r/selfhosted](https://www.reddit.com/r/selfhosted/), [r/HomeServer](https://www.reddit.com/r/HomeServer/), [openminilabs](https://github.com/Wakoma/openminilab), [mini-rack](https://github.com/geerlingguy/mini-rack), [r/HomeNetworking](https://www.reddit.com/r/HomeNetworking/)) index those communities, and serve as the most ideal way to get started in this potentially large and complicated space of hosting your own services.
48+
49+
Some nearterm goals:
50+
51+
1. Simplify the existing "single" design into a getting started design that showcases a relatively cheap and affordable build, but starts to modularize the focus of various guides that build on the simple getting started guide.
52+
1. Break out guides into different taxonomies (Hardware:Motherboard/Memory/Disk/CPU/GPUs/TPUs/Thermal/Network/Power/Disks/Rack/etc.., Ops:VMs/Containers/Managers/Hypervisor/Configuration/KVM/Netboot/etc.., Software:Storage/Git/Forejo/Certbot/ Personal Software: Nextcloud/Immich/Monero/)
53+
1. Create a Recommendations index for technologies, people, and other stuff to watch and over time provide the ability for community driven benchmarks or showcases.
54+
1. Build out a Discord Forum to discuss topics in a more managed way as oppossed to the Reddit design.
55+
1. Recycle guides - initially, these guides focus on small, quiet, energy efficient builds using smaller micro servers like Raspberry Pi. This is not only pragmatic from a cost and energy savings, but supports smaller chip and dev board manufactures (especially built on open instruction sets like RISC-V). However, there is another consideration as we transition to a smaller interconnected network of nodes, and that is what do we do with servers that will inevitibly start spinning down and heading towards the landfill for children in Ghana to dissassemble? Well, we will also talk about standard homelab transformation of the most common larger systems. Although these won't be energy efficient, we as a community can also look into ways of opitmizing the most power hungry boards to something a bit closer to enable their reuse. When boards are at their end of life, we can also start to look at distributed recycling guides.
56+
57+
Some longterm goals:
58+
59+
1. Compile and provide guides on basic and intermediate Linux commands so that anyone, not just software engineers, can self-educate on the necessary understanding of systems administration.
60+
1. Compile and provide guides on computer, network, nad security basics so that anyone hosting their services can avoid the most common pitfalls in their security model.

‎docs/software-installation.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Software Installation and Configuration
2+
3+
## Server Management and Imaging
4+
[Tinkerbell](https://tinkerbell.org/) is an automated DHCP, NETBOOT, and PXE physical host provisioning service. To keep costs down, you can run this on your desktop in VMs initially. You can also run it on some of your nodes if you have a multi purpose node (something like a shared DHCP, DNS, Tinkerbell, image store, etc)
5+
6+
## Routing
7+
[FRRouting](https://frrouting.org/) is an Open Source routing stack. This is the stack I would recommend for any home lab. This can be run on any linux host and is substantially cheaper than buying or re-using some purpose build switch or router. The goal of this project is to learn the internals, so running FRR is a good way to familiarize yourself with networking protocols.
8+
9+
As we delve in to more detail, FRR becoming increasingly important to learn cloud networking concepts. I'm not talking about simple concepts like IGWs or VPC Route Tables. I mean the underpinning of VPCs in general. In a later article, I'll describe how you can create your own VPC networking with eVPN and VXLAN using FRRouting. This is the basis for undestanding multi tenant network architecture (hint, it's all encapsulated in the datacenter).
10+
11+
### Public IP Space
12+
So you want to learn IPv6? Well, you could get your own ASN and IP space for about $1000.
13+
- ASN: 500$
14+
- /36 IPv6: 500$
15+
16+
_/36 is the smallest ARIN will allocate at the time of writing_
17+
18+
Seems a little steep for most home SREs. But imagine all the money you'll save in not buying rackmount servers or powering them.
19+
20+
With the routing setup and OpenSwitch, you can advertise your IP space through a provider like Equinix Metal (ask me how I know). You'll be able to have your own publicly routable IPv6 space. You can also use AWS BYOIP and [TransitGateway Connect](https://aws.amazon.com/about-aws/whats-new/2020/12/introducing-aws-transit-gateway-connect-to-simplify-sd-wan-branch-connectivity/)
21+
22+
If you're more adventurous, have a bottomless wallet, or already have IPv4 space, you can do that too.
23+
24+
### Accessing Your Microdatacenter (no static addressing from your home ISP)
25+
You can use either wireguard or IPSec up to an Equinix Metal host or to AWS (or any other provider that can handle BGP)
26+
27+
## Elastic IPs
28+
Elastic IPs, which you may be familar with from AWS. What this really is is just a public IP that is NAT'd to an ENI/Private IP. How do you achieve this in your microdatacenter? We don't have fancy network cards that offload this, so in order to do this, it's as simple as running BGP on every node.
29+
30+
For kubernetes workloads, we'll use Cilium to advertise the public IP (Elastic IP) via BGP to the core network and out to the internet (if you're doing public peering)
31+
32+
33+
## Firewall
34+
Firewalls we'll keep simple to understand them. No, we're not going to use PFSense for these exercises. We'll do it the hard way for the purposes of learning.
35+
36+
nftables, bpfilter, iptables, etc
37+
38+
## k8s
39+
Kubernetes has always been a bit of a struggle in home labs. It's annoying running every service and having enough infrastructure to do so. The balance I've found is utilizing [k3s](https://k3s.io/)
40+
41+
This has support for things like Cilium, which ties in nicely to the routing stack.
42+
43+
## Distributed Storage
44+
How do you do persistent storage? What can you use for shared storage and object stores?
45+
46+
[minio](https://min.io/) is the simplest to manage and utilize. It has an S3 like API and is a lot less work than other tools like GlusterFS and Ceph.

‎mkdocs.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
site_name: Micro Data Center Guides
2+
site_url: http://microdatacenter.build
3+
theme:
4+
name: material

0 commit comments

Comments
 (0)