# Server ## Hardware / OS * Dell Precision WorkStation 390 * Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz * 4GB RAM DDR 667 MHz * HDD (going to be 4TB 2x) * with Ubuntu 18.04 ## Accounts * root * jelko * luca * leo ## SSH SSH is running and available to the internet (!). Only Pubkey authentication is allowed. ## Firewall `ufw` is administering the firewall rules. Default: **Deny all.** Internal networks: `10.10.0.0/16` (HfK Inhouse) + `192.168.0.0/16` (HfK VPN) Rules (by 2019-02-04): ``` To Action From -- ------ ---- [ 1] 22/tcp ALLOW IN Anywhere [ 2] 8080/tcp ALLOW IN Anywhere [ 3] 8000/tcp ALLOW IN Anywhere [ 4] 80/tcp ALLOW IN Anywhere [ 5] 445/tcp ALLOW IN 192.168.0.0/16 [ 6] 445/tcp ALLOW IN 10.10.0.0/16 [ 7] 8001/tcp ALLOW IN Anywhere [ 8] 8002/tcp ALLOW IN Anywhere [ 9] 8088/tcp ALLOW IN 10.10.0.0/16 [10] 5060 ALLOW IN 10.0.0.0/16 [11] 5060/udp ALLOW IN 10.10.0.0/16 [12] Anywhere ALLOW IN 10.10.0.0/16/udp [13] Anywhere ALLOW IN 192.168.0.0/16/udp [14] 5060/tcp ALLOW IN 192.168.0.0/16 [15] 5038/tcp ALLOW IN 192.168.0.0/16 [16] 5038/tcp ALLOW IN 10.10.0.0/16 [17] 8088/tcp ALLOW IN 192.168.0.0/16 [18] 5039/tcp ALLOW IN 192.168.0.0/16 [19] 9090 ALLOW IN 10.10.0.0/16 [20] 2812 ALLOW IN 10.10.0.0/16 [21] 8003 ALLOW IN Anywhere [22] 2812 ALLOW IN 192.168.0.0/16 [23] 5083 ALLOW IN 192.168.0.0/16 [24] 5038 ALLOW IN 192.168.0.0/16 [25] 8010/tcp ALLOW IN Anywhere [26] 8011/tcp ALLOW IN Anywhere [27] 8012/tcp ALLOW IN Anywhere [28] 8013/tcp ALLOW IN Anywhere [29] 8014/tcp ALLOW IN Anywhere [30] 8015/tcp ALLOW IN Anywhere [31] 8016/tcp ALLOW IN Anywhere [32] 8099 ALLOW IN Anywhere [33] 2812/tcp ALLOW IN Anywhere [34] 22/tcp (v6) ALLOW IN Anywhere (v6) [35] 8080/tcp (v6) ALLOW IN Anywhere (v6) [36] 8000/tcp (v6) ALLOW IN Anywhere (v6) [37] 80/tcp (v6) ALLOW IN Anywhere (v6) [38] 8001/tcp (v6) ALLOW IN Anywhere (v6) [39] 8002/tcp (v6) ALLOW IN Anywhere (v6) [40] 8003 (v6) ALLOW IN Anywhere (v6) [41] 8010/tcp (v6) ALLOW IN Anywhere (v6) [42] 8011/tcp (v6) ALLOW IN Anywhere (v6) [43] 8012/tcp (v6) ALLOW IN Anywhere (v6) [44] 8013/tcp (v6) ALLOW IN Anywhere (v6) [45] 8014/tcp (v6) ALLOW IN Anywhere (v6) [46] 8015/tcp (v6) ALLOW IN Anywhere (v6) [47] 8016/tcp (v6) ALLOW IN Anywhere (v6) [48] 8099 (v6) ALLOW IN Anywhere (v6) [49] 2812/tcp (v6) ALLOW IN Anywhere (v6) ``` Output active numbered rules: `ufw status numbered`. You may find an [introduction to UFW at Digital Ocean](https://www.digitalocean.com/community/tutorials/ufw-essentials-common-firewall-rules-and-commands). ### Port Forwarding 8080 -> 8002 Since some organizations-network firewalls block outgoing connections to "unusual" ports (z.B. HBK BS in der Produktion mit der Schwankhalle am 23.05.2021), we have set up an alternative port at `8080`. This is forwarded to 8002 internally using NAT at `/etc/ufw/before.rules`: ``` 11 *nat 12 :PREROUTING ACCEPT [0:0] 13 -A PREROUTING -p tcp --dport 8080 -j REDIRECT --to-port 8002 14 COMMIT ``` Reload with `systemctl restart ufw` Source: or You can monitor incoming bandwith with: `sudo tcptrack -i enp4s0 port 8001`. ## Backup Backups of the are composited in three stages: 1. Daily staging of changes via rsync 2. Weekly staging of changes via rsync 3. Monthly backup to external HDD via tar For the monthly backups to happen the Backup HDD (labeled as such) needs to be plugged into the server. Now on: ### Cronjobs The cronjobs are scheduled as such.\ Update (2020-05-06): They are not scheduled in `/etc/cron*` but via monit (!) in `/etc/monit/monitrc` (Group Backup). → Disabled and moved to `/etc/cron.d/angrezi-backup` ``` # m h dom mon dow command 0 4 * * * /mnt/backup/daily_backup.sh 0 6 * * 0 /mnt/backup/weekly_backup.sh 0 0 1 * * /mnt/backup/monthly_backup.sh ``` ### Locations and Script Locations backed up: `/etc /var/angrezi /root /boot /opt /usr/local /srv /var/lib /var/mail /var/www /var/backups /var/local /var/opt /var/log`\ Excluded are: `--exclude="*.wav" --exclude=/home/*/.gvfs --exclude=/home/*/.cache --exclude=/home/*/.local/share/Trash --exclude=/media` The script for montly backup mounts the HDD, stores the last staged weekly changes to the HDD and unmounts the HDD. Do not fiddle with the HDD! If you want to initiate a manual backup, run `./mnt/backup/monthly_backup.sh` as root. ```bash # Where to backup to dest="/mnt/angrezi_backup_hdd/$(date +%Y)" # Create archive filename date=`date +%Y-%m-%d` hostname=$(hostname -s) archive_file="$hostname-$date-backup.tar.xz" Log="/var/log/backup.log" echo "" >> $Log echo "---------------------------------------------------" >> $Log echo "$date:: Full backup to $dest/$archive_file" >> $Log now=`date +"%T"` echo "$now:: starting..." >> $Log echo "$now:: mounting Backup-HDD with UUID c59e6838-79ae-4994-bf1c-729ff9fae423 to /mnt/angrezi_backup_hdd" >> $Log mount UUID="c59e6838-79ae-4994-bf1c-729ff9fae423" /mnt/angrezi_backup_hdd/ rc=$? now=`date +"%T"` echo "$now:: Mounting successful" >> $Log now=`date +"%T"` echo "$now:: starting backup..." >> $Log tar -cvpJf "$dest/$archive_file" /mnt/backup/weekly #storing permissions, using xz as compression, verbose rc=$? now=`date +"%T"` if [ $rc != 0 ]; then echo "$now:: Backup failed!" >> $Log if grep -qs '/mnt/angrezi_backup_hdd ' /proc/mounts; then echo "HDD is mounted, Backup still failed" >> $Log else echo "HDD is not mounted! Is it plugged in?" >> $Log fi exit 1 else echo "$now:: Finished." >> $Log fi now=`date +"%T"` echo "$now:: Unmounting..." >> $Log umount /mnt/angrezi_backup_hdd rc=$? if [ $rc != 0 ]; then echo "Unmounting not successful. Yikes. Exiting." >> $Log else echo "Unmounting disc successful. Exiting. Thanks for choosing our services." >> $Log fi exit 1 ``` ## Monitoring We use `monit` to monitor all "important" services at . ## Webserver Domains Mapped: * stream.radioangrezi.de * server.radioangrezi.de * studio.radioangrezi.de * voip.radioangrezi.de Webservices: * AirTime * Angrezi Controller * Fileserver (Directory Listing with styling) * Icecast (Port 8000) * ~~Cockpit (Port 9090)~~ ### Angrezi Controller Self-build controller interface. Source on [GitHub](https://github.com/radioangrezi/live-recorder). `studio.radioangrezi.de/controller` ## Services * darkice (installed via `apt` service manually placed in `/etc/systemd/system/darkice.service` and removed in `init.d`) * angrezi-master-recorder (self made) * angrezi-stream-monitor (self made) * angrezi-file-watch (self made) After changes to the services in `/etc/systemd/system/` you can reload them with `systemctl daemon-reload` + `systemctl enable ` ### angrezi-file-watch `angrezi-file-watch` uses `inotifywatch -m -r -e create` to monitor the directories of recordings. If a file has been created (aka a new show has been recorded) it automatically appends `chattr +a` to prevent deletion. ## Soundcard * onboard sound card (Intel) disabled in `/etc/modprobe.d/blacklist-angrezi.conf` * Pulse Audio disabled in `/etc/pulse/client.conf` Using ALSA with Behringer UMC202HD with following config in asound.conf ``` pcm.stream_in_32 { # the Behringer hardware we use does only support 32 bit by default. type dsnoop ipc_key 5978293 ipc_key_add_uid false hint.description "Angrezi: USB Card hw1 for Stream IN, 32 bit, dsnooped for mulitple listeners" slave { pcm "hw:1,0" channels 2 format S16_LE # this will only affect the plug pcm.stream_in_16. rate 44100 } } pcm.stream_in_16 { type plug hint.description "Angrezi: USB Card hw1 for Stream IN, 16 bit, dsnooped for mulitpl e listeners" slave.pcm "stream_in_32" } ``` two capture devices are available: * `stream_in_32` with S32\_LE (hardware default) * `stream_in_16` with S16\_LE (software conversion via `plug`) Test with (software monitor): `arecord -f S16_LE -r44100 -c2 -D stream_in_16 | aplay -D hw:1 -` ## Darkice The local studio audio in form `stream_in_16` is broadcasted to Icecast via Darkice 1.3. Darkice only supports 16 bit sampling (which makes the downconversion necessary). Recording in DarkIce was disabled after a [bug corrupted the recording on server reconnection](https://github.com/rafael2k/darkice/issues/141) occurred. This local Darkice stream functions as `master source` in AirTime / liquidsoap. ## Recording ## FM Relay to Radio Weser TV Radio Weser TV gets a special relay stream out (with a defined URL), which is made to never fail (go silent): `http://stream.radioangrezi.de:8000/live-radioweser`. The stream is produced by relaying our `live` stream in a seperate liquidsoap instance (service: `angrezi-relay-out-radioweser`) which falls back to a playback of mp3 files if the stream goes silent for more than 10 seconds. Fallback music (MP3, 44100, min. 192 kbit/s, Stereo) must be placed in `/media/storage/share/Automation/Live-Out-RWTV/Music-MP3/` and fallback Jingles in `/media/storage/share/Automation/Live-Out-RWTV/Jingles-MP3/`. Jingles and Music are randomly picked in the radio 1:5. (If all fallback is empty as well, the script will fallback to a single audio file placed at `/var/angrezi/relay-out-radioweser-fallback.mp3`.) **Notice: The service needs to be restarted if the music folder is updated.** TODO: We could add a special config on Icecast so even if liquidsoap fails, there would be a fallback. TODO: Hide the mountpoint from Icecast. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.radioangrezi.de/tech-deep-dive/server.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.