Homelab Documentation

Appearance

Network Setup ​

IMPORTANT

What this document is and is not for. ​

This document is meant to provide an accurate in-depth read of my network architecture and configuration state.

Reasoning behind decisions made in reaching this state is kept in this document.

Physical topology of my full home is kept in this document.

An overview of traffic flow through my network as a whole is kept in this document.

OPNSense Configuration ​

VLANs & Subnets
VLANs and Subnet

WARNING

Some of the service network listed here are planned but not currently active. See the below list for planned to go up very soon but are not yet active services.

  • expenseowl
  • NAS
  • passwordpusher
  • neko
  • homepage
  • commafeed
  • rssbridge
  • tagspaces
  • synthing
  • jellyfin
  • pinchflat
  • kasm workspaces

They are planned to be fully working one - two weeks.

VLAN Subnets & Assignments ​

TIP

Network Naming Schema Explained

networks backends: 10.20.<macvlan-number>.<address-range>

  • day: 1
    • server: 0-6
    • network center: 8-14
    • undecided: 16-22
    • edgeshark: 24-30
    • monitor outpost: 32-38
  • dusk: 2
    • server: 0-6
    • network center: 8-14
    • user gateway: 16-22
    • edgeshark: 24-30
    • monitor center: 32-38
  • night: 3
    • server: 0-6
    • network center: 8-14
    • undecided: 16-22
    • edgeshark: 24-30
    • monitor outpost: 32-38
  • web_dev: 4
    • server: 0-6

Subnet Details ​

Those in use below.

NameTagSubnet Address (CIDR)Default GatewayAvailable Range
Day10110.20.1.0/2910.20.1.12 - 6
Day_Network_Center10210.20.1.8/2910.20.1.910 - 14
Day_Edgeshark10410.20.1.24/2910.20.1.2526 - 30
Day_Monitor_Outpost10510.20.1.32/2910.20.1.3334 - 38
Dusk20110.20.2.0/2910.20.2.12 - 6
Dusk_Network_Center20210.20.2.8/2910.20.2.910 - 14
Dusk_User_Gateway20310.20.2.16/2910.20.2.1718 - 22
Dusk_Edgeshark20410.20.2.24/2910.20.2.2526 - 30
Dusk_Monitor_Center20510.20.2.32/2910.20.2.3334 - 38
Night30110.20.3.0/2910.20.3.12 - 6
Night_Network_Center30210.20.3.8/2910.20.3.910 - 14
Night_Edgeshark30410.20.3.24/2910.20.3.2526 - 30
Night_Monitor_Outpost30510.20.3.32/2910.20.3.3334 - 38
Web_Dev40110.20.4.0/2910.20.4.12 - 6
Assignments Details ​

NOTE

Static DHCP Mappings for Host VMs

Used by main.tf files for Terraform to generate interfaces.

VLAN NameMAC AddressIP Address
Day7a:7b:e2:51:43:9010.20.1.6
Day_Network_Centerbe:bb:37:47:6a:8410.20.1.10
Day_Edgesharkee:76:24:18:a8:0510.20.1.26
Day_Monitor_Outpostea:35:e6:41:05:2110.20.1.34
Duskunassignedunassigned
Dusk_Network_Center0a:1a:19:4f:3d:a010.20.2.10
Dusk_User_Gatewayb6:36:f2:e6:16:6510.20.2.18
Dusk_Edgesharkc6:3c💿de:22:0a10.20.2.26
Dusk_Monitor_Centeree:05:14:9c:4f:0810.20.2.34
Nighte6:20:5f:93:ec:ec10.20.3.6
Night_Network_Centerca:42:3c:61:6d:8f10.20.3.10
Night_Edgeshark3e:1c:43:2e:50:5a10.20.3.26
Night_Monitor_Outpost6a:c9:bd:ad:39:1610.20.3.34
Web_Deva2:30:57:48:b8:9210.20.4.6

NOTE

Static DHCP Mappings for Containers

Used in host_vars Ansible files to generate Docker MACVLANs.

VLAN NameHost NameSidecars ContainersMac AddressIP Address
Day_Network_Centerts-network-centernonec6:19:a8:7f:0e:3910.20.1.11
Day_Edgesharkts-sharkpacketflixa6:d2:73:22:72:d510.20.1.27
Day_Monitor_Outpostts-monitor-outposttraefikda:91:79:c6:65:b610.20.1.35
Day_Monitor_Outpostts-monitor-alloyalloyd6💿54:bc:a3:8510.20.1.36
Dusk_Network_Centerts-network-centernoneda:f3:16:64:f3:3010.20.2.11
Dusk_User_Gatewayts-gatenginx
dnsmasq		56:9d:02:b2:98:a410.20.2.19
Dusk_Edgesharkts-sharkpacketflixee:fc:b4:df:9e:9c10.20.2.27
Dusk_Monitor_Centerts-monitor-centeralloy26:b9:e5:80:b3:c310.20.2.35
Night_Network_Centerts-network-centernonefe:56:b6:04:f0:db10.20.3.11
Night_Edgesharkts-sharkpacketflix2a:ef:f6:79:56:4810.20.3.27
Night_Monitor_Outpostts-monitor-outposttraefik1a:7b:14:b8:0f:5d10.20.3.35
Night_Monitor_Outpostts-monitor-alloyalloy2a:d4:1a:0f:09:4010.20.3.36

Docker Configuration ​

NOTE

Network Naming Schema Explained

networks backends: 10.<vlan-number>.<backend-number>.address-range duplicates are intentional where a service requires multiple networks macvlans obey naming scheme above for vlans

  • vlan & special group numbers
    • web-dev: 100
    • day: 110
    • dusk: 120
    • night: 130
  • backends - docker network subnets
    • network services: 0-19
      • traefik_tailscale: 0
      • dns: 1 (fix later: no longer in use)
      • edgeshark: 2
      • edgeshark: 3 (fix later: should be removed in future)
      • gate: 4 (fix later: hard coded into compose)
    • other operational services: 20-29
      • monitor center/outpost: 20
      • nginx_file_server: 21
    • end services: 30-49
      • commafeed: 30
      • searxng: 31
      • expenseowl: 32
      • homepage: 33
      • vikunja: 34
      • jellyfin: 35
      • passwordpusher: 36
      • neko: 37
      • homepage: 38

Table of all docker networks currently in use. ​

TIP

  • using # instead of {} because this way table formats correctly
  • # is filled based on Network Naming Schema
Host(s)Network NameTypeSubnet Address (CIDR)Default GatewayAvailable Range
Alltraefik_tailscaleBridge10.#.0.0/2910.#.0.12 - 6
Allmacvlan_network_centerMacvlan10.20.#.8/2910.20.#.910 - 14
day
night		monitor_outpostBridge10.#.20.0/2910.#.20.12 - 6
day
night		macvlan_outpostMacvlan10.20.#.32/2910.20.#.3334 - 38
dayt3_searxngBridge10.#.31.0/2910.#.31.12 - 6
dayt3_vikunjaBridge10.#.34.0/2910.#.34.12 - 6
duskgate-tailscale-networkBridge10.110.4.0/2910.110.4.12 - 6
duskmacvlan_user_gateMacvlan10.20.2.16/2910.20.2.1718 - 22
duskmonitor_centerBridge10.120.20.0/2810.120.20.12 - 6
duskmacvlan_monitor_centerMacvlan10.20.2.32/2910.20.2.3334 - 38
Host addresses in use within the above Docker Networks. ​

TIP

  • host addresses refers to the last number in the address here
  • the order of Containers and IP Host Addresses matches to pair each entry together
  • leaving Macvlan networks out of this since those assignments are documented in the above VLAN Assignments section

Keep in mind the traefik container in many of these represents the traefik_tailscale network traefik container that has an additional interface in the given network with the given address.

  • Containers that have multiple network interfaces will be highlighted.
Host(s)Network NameContainersIP Host Addresses
Alltraefik_tailscaletraefik
tailscale
dnsmasq		2
4
5
day
night		monitor_outposttraefik
tailscale_traefik
tailscale_alloy
cadvisor
dnsmasq		2
3
4
5
6
dayt3_searxngtraefik
searxng		3
5
dayt3_vikunjatraefik
vikunja		3
5
duskgate-tailscale-networktailscale_nginx_dnsmasq2
duskmonitor_centertailscale
traefik
dns
grafana
loki
mimir
alloy
cadvisor		2
3
4
5
6
7
8
9

Tailscale: Groups, Tags, Hosts ​

Groups ​

Group NameUsers
personalmy trusted devices
familymy immediate family
outsideanyone other than those above

NOTE

groups family and outside are not currently in use because

  • the homelab is not setup to support more people than me securely yet
  • because of the above I am currently the only user of my Homelab

Tags & Hosts ​

Tags ​

TIP

  • IP Addresses are set by the nodeAttrs block in the Tailscale Access Controls config.
  • IP Pool is used for assigned addresses to a given target, the target being a tag in my case
Tag NameHosts Applied ToDevice Network(s)IP Pool
ssh-serverEach Homelab Proxmox VMVLANs: Day,Dusk,Night"100.80.0.0/29"
day-shark-serverdoes not exist currently

tailscale container +
packetflix sidecar		does not exist currently

edgeshark		"100.80.31.1/32"
day-server-network-centertailscale containertraefik_tailscaledoes not exist currently
dusk-server-network-centertailscale containertraefik_tailscaledoes not exist currently
night-server-network-centertailscale containertraefik_tailscaledoes not exist currently
dusk-gate-servertailscale container +
nginx sidecar		gate-tailscale-network"100.80.1.1/32"
dusk-server-monitor-alloytailscale container +
alloy sidecar		monitor_center"100.80.20.3/32"
day-server-monitor-traefiktailscale container +
traefik sidecar		monitor_outpost"100.80.20.1/32"
day-server-monitor-alloytailscale container +
alloy sidecar		monitor_outpost"100.80.20.2/32"
night-server-monitor-traefiktailscale container +
traefik sidecar		monitor_outpost"100.80.20.4/32"
night-server-monitor-alloytailscale container +
alloy sidecar		monitor_outpost"100.80.20.5/32"
me-clientn/an/an/a
webvm-serverWeb Dev Proxmox VMVLAN: Web_Dev"100.90.0.0/32"
ci-devGithub Actions Runnersn/an/a

WARNING

  • day-shark-server needs to be matched by tags for dusk and night, all with up to date nodeAttr IP Pool assignments
  • make IP Pool assignments for the network-center tags

Host Addresses ​

Host NameIP AddressContainer_NetworkContainer_Name
day-traefik10.110.0.2traefik_tailscaletraefik
day-dns10.110.0.5traefik_tailscalednsmasq
day-monitor-outpost-traefik10.110.20.2monitor_outposttraefik
dusk-traefik10.120.0.2traefik_tailscaletraefik
dusk-dns10.120.0.5traefik_tailscalednsmasq
dusk-monitor-center-traefik10.120.20.3monitor_centertraefik
night-traefik10.130.0.2traefik_tailscaletraefik
night-dns10.130.0.5traefik_tailscalednsmasq
night-monitor-outpost-traefik10.130.20.2monitor_outposttraefik

Host Networks ​

Note: these are not used in grants currently but handy to have access to

Host NameSubnetHost
Matching Docker_Network		Autoapproved True/FasleTag
Associated
day-traefik-net10.110.0.0/29day
traefik_tailscale		trueday-server-network-center
day-monitor-outpost-net10.110.20.0/29day
monitor_outpost		trueday-server-network-center
dusk-traefik-net10.120.0.0/29dusk
traefik_tailscale		truedusk-server-network-center
dusk-monitor-center-net10.120.20.0/28dusk
monitor_center		truedusk-server-network-center
night-traefik-net10.130.0.0/29night
traefik_tailscale		truenight-server-network-center
night-monitor-outpost-net10.130.20.0/29night
monitor_outpost		truenight-server-network-center

TIP

The {host}-network-center tag is used for all of these routes. This is because the compose that includes the Tailscale instance with this tag has jinja templating for populating all the routes automatically from the host variable I provide.

SSH ​

SourceDestinationUsers
group:personaltag:ssh-server"autogroup:nonroot", "root"
group:personaltag:webvm-server"autogroup:nonroot", "root"

Tailscale: Grants & Traffic Flow ​

Tailscale Mesh
Tailscale Mesh
SourceDestinationHost
Destination Network		Protocols:Ports Allowed
group:personaltag:ssh-serverVLANs: Day,Dusk,Night*:22
group:personaltag:day-server-network-centerday
traefik_tailscale		*😗
group:personaltag:day-shark-serverdoes not exist currently

edgeshark		*😗
group:personaltag:dusk-gate-serverdusk
gate-tailscale-network		udp:53
group:personaltag:night-server-monitor-traefiknight
monitor_outpost		*:9002
group:personaltag:night-server-network-centernight
traefik_tailscale		*:9002
group:personaltag:dusk-gate-serverdusk
gate-tailscale-network		tcp:80, tcp:443
tag:me-clienttag:day-server-network-centerday
traefik_tailscale		*😗
tag:me-clienttag:dusk-gate-serverdusk
gate-tailscale-network		udp:53
tag:me-clienttag:dusk-gate-serverdusk
gate-tailscale-network		tcp:80, tcp:443
tag: day-shark-servergroup:personaln/a*😗
group:personaltag:webvm-serverVLAN: Web_Dev*:22, tcp:80, tcp:443

WARNING

Need to join grants for destination:tag:dusk-gate-server.

Extra-VLAN Container to Container ​

SourceDestinationHost
Source Network		Host
Destination Network		Protocols:Ports Allowed
tag:day-server-monitor-traefikhost address: night-dnsday
monitor_outpost		night
traefik_tailscale		udp:53
tag:day-server-monitor-traefikhost address:dusk-traefikday
monitor_outpost		dusk
traefik_tailscale		tcp:80, tcp:443
tag:day-server-monitor-alloyhost address: night-dns

host address: dusk-dns		day
monitor_outpost		night
traefik_tailscale

dusk
traefik_tailscale		udp:53
tag:dusk-server-monitor-alloyhost address: day-dns

host address: night-dns		dusk
monitor_center		day
traefik_tailscale

night
traefik_tailscale		udp:53
tag:night-server-monitor-traefikhost address: day-dns

host address: dusk-dns		night
monitor_outpost		day
traefik_tailscale

dusk
traefik_tailscale		udp:53
tag:night-server-monitor-traefikhost address:dusk-traefiknight
monitor_outpost		dusk
traefik_tailscale		tcp:80, tcp:443
tag:night-server-monitor-alloyhost address:day-dnsnight
monitor_outpost		day
traefik_tailscale		udp:53
tag:dusk-gate-serverhost address: day-traefik
dusk
monitor_outpost		day
traefik_tailscale		tcp:80, tcp:443
tag:dusk-gate-serverhost address: night-traefik
dusknighttcp:80, tcp:443

WARNING

  • Investigate the udp:53 grants.

TIP

Q: Why does tag:night-server-monitor-traefik reach out to host address:dusk-traefik instead of a new host address:dusk-monitor-traefik with a separate Traefik instance there.

A: Because this would require an additional Traefik which feels like it provides little benefit for me in terms of security. I still will probably set this additional instance up when my threat model goes up as I start locking down the Homelab.

Intra-VLAN Container to Container ​

SourceDestinationSource NetworkDestination NetworkProtocols:Ports Allowed
tag:day-server-monitor-traefikhost address:day-dnsmonitor_outposttraefik_tailscaleudp:53
tag:day-server-monitor-alloyhost address:day-dnsmonitor_outposttraefik_tailscaleudp:53
tag:dusk-server-monitor-alloyhost address: dusk-traefikmonitor_centertraefik_tailscaletcp:80, tcp:443, *:9100
tag:dusk-server-monitor-alloytag:dusk-gate-servermonitor_centergate-tailscale-network*:9002
tag:dusk-server-monitor-alloyhost address: dusk-dnsmonitor_centertraefik_tailscaleudp:53
tag:dusk-server-monitor-alloytag:dusk-server-network-centermonitor_centertraefik_tailscale*:9002
tag:night-server-monitor-traefikhost address: night-dnsmonitor_outposttraefik_tailscaleudp:53
tag:night-server-monitor-alloyhost address: night-dnsmonitor_outposttraefik_tailscaleudp:53
tag:night-server-monitor-alloyhost address: night-traefikmonitor_outposttraefik_tailscaletcp:80, tcp:443, *:9100
tag:night-server-monitor-alloytag:night-server-monitor-traefikmonitor_outpostmonitor_outpost*:9002
tag:night-server-monitor-alloytag:night-server-network-centermonitor_outposttraefik_tailscale*:9002
tag:dusk-gate-serverhost address: dusk-traefikgate-tailscale-networktraefik_tailscaletcp:80, tcp:443

WARNING

  • Investigate why no grants point to dusk-dns either here in Intra-VLAN grants and Extra-VLAN.
  • Investigate why the monitor-alloy tag can go to the traefik_tailscale network with more than *:9100.
  • tag:day-server-monitor-alloy needs grants allowing Intra-VLAN scraping.
  • Investigate why tag:night-server-monitor-alloy is using Tailscale to go to another container in its own Docker Network.

Keys ​

Total keys used: 10

Note: A role can have multiple keys and in turn multiple tags (tag per key) only when there are multiple tailscale containers in that role.

Containers with ts_auth_key. ​
Ansible Role NameKey DetailsTailscale Sidecar Details
network_center1 key (accounting for all hosts 3 keys)no sidecars - single key "host-server-network-center"
nginx_gate1 key (accounting for all hosts 1 key)nginx sidecar - single key "dusk-server-gate"
dnsmasq sidecar - single key "dusk-server-gate"
monitor_center1 key (accounting for all hosts 1 key)alloy sidecar - single key "dusk-monitor-alloy"
monitor_outpost2 keys (accounting for all hosts 4 keys)traefik sidecar - first key "host-monitor-traefik"
alloy sidecar - second key "host-monitor-alloy"

To find them use (ctrl+f "TS_AUTHKEY: " case sensitive whole word) in this repo. It will show all docker composes where a Tailscale key is used.

Outbound-Capable Tailscale Containers ​

Container NameCapability Purpose
monitor_centeralloy container (sidecar): to scrape traefik and tailscale container not in its local docker networks
monitor_outposttraefik container (sidecar): to send data to monitor_center
edgesharkpacketflix (sidecar): requires outbound capability (back and forth of inbound outbound traffic) to function
nginx_gatenginx (sidecar): to forward traffic to the appropiate traefik container

WARNING

Investigate the capability purpose of dnsmasq container (sidecar). Test later with existing sidecar setup and advertised subnet setup.

Direct Connections ​

Every Tailscale container has a Macvlan interface for supporting direct connections.

Pluses

  • Using services that don't require internet access when the internet is down at home becomes possible.
  • Speed is improved.
  • Traffic analysis via packet capture is easier.
  • Personal data not going through derp servers as much now.

Minuses

  • Higher risk upon a compromised Tailscale container as it can reach outside it's given Docker networks. Note this is restricted by the access controls in Tailscale.
  • The work involved in adding every new Tailscale container to my Homelab network goes up dramatically.

DNS ​

Docker DNS with Dnsmasq & Upstream Pihole ​

Every docker bridge network I made has a Dnsmasq instance in it meant for providing private DNS records for the containers in that network to use.

Almost always the private records block in the Dnsmasq config will only contain a wildcard for the local Traefik address in that network which will receive and forward all traffic using its host rules for DNS handling.

All non-private-container related DNS queries will be referred upstream to a Pihole instance I am self hosting.

WARNING

In the future this Pihole instance will need to move from where it currently is as an LXC on my Proxmox machine to being in an established VM to save resources. When this change is made the only Pihole instance may be split into two with

  1. instance on Day VM for LAN and Day VLAN usage
  2. instance on Dusk VM for Dusk VLAN and Night VLAN usage

Tailscale DNS ​

All Tailscale DNS usage is Split-DNS meaning the Tailscale DNS nameservers will only be used for the specified domain(s) they are paired with.

This project has three main domains for which Tailscale DNS is used, these are subdomains that treat any further domains after as part of the given listings (wildcard behavior)

  • *.day.wiresndreams.dev
  • *.dusk.wiresndreams.dev
  • *.night.wiresndreams.dev

For each domain the associated nameservers are

  1. nginx_gate dns: this is the user gate that filter incoming client traffic headed to service endpoints
  2. network_center dns: this is the host specific dnsmasq instance in the network center that provides the traefik address in its network

There are two additional domains meant for monitor outpost usage. They are needed because Tailscale outbound containers (of which there are two in monitor outpost) use Tailscale DNS.

Already in Tailscale DNS the base .host.wiresndreams.dev is covered meaning if a container with the same name (traefik in monitor outpost) is setup with a different nameserver there will be a conflict. Eg.

  • points to different locations
    • traefik.day.wiresndreams.dev: network center nameserver
    • traefik.day.wiresndreams.dev: monitor outpost nameserver

Because of this a separate domain must be made to prevent a conflict. The separate domains include

  • *.monitor.day.wiresndreams.dev
  • *.monitor.night.wiresndreams.dev