Server

Search…
Server
Hardware / OS
Dell Precision WorkStation 390
Intel(R) Core(TM)2 CPU 6600  @ 2.40GHz
4GB RAM DDR 667 MHz
HDD (going to be 4TB 2x)
with Ubuntu 18.04
Accounts
root
jelko
luca
leo
SSH
SSH is running and available to the internet (!). Only Pubkey authentication is allowed.
Firewall
ufw is administering the firewall rules. Default: Deny all. Internal networks: 10.10.0.0/16 (HfK Inhouse) + 192.168.0.0/16 (HfK VPN)
Rules (by 2019-02-04):
1
     To                         Action      From
2
     --                         ------      ----
3
[ 1] 22/tcp                     ALLOW IN    Anywhere                  
4
[ 2] 8080/tcp                   ALLOW IN    Anywhere                  
5
[ 3] 8000/tcp                   ALLOW IN    Anywhere                  
6
[ 4] 80/tcp                     ALLOW IN    Anywhere                  
7
[ 5] 445/tcp                    ALLOW IN    192.168.0.0/16            
8
[ 6] 445/tcp                    ALLOW IN    10.10.0.0/16              
9
[ 7] 8001/tcp                   ALLOW IN    Anywhere                  
10
[ 8] 8002/tcp                   ALLOW IN    Anywhere                  
11
[ 9] 8088/tcp                   ALLOW IN    10.10.0.0/16              
12
[10] 5060                       ALLOW IN    10.0.0.0/16               
13
[11] 5060/udp                   ALLOW IN    10.10.0.0/16              
14
[12] Anywhere                   ALLOW IN    10.10.0.0/16/udp          
15
[13] Anywhere                   ALLOW IN    192.168.0.0/16/udp        
16
[14] 5060/tcp                   ALLOW IN    192.168.0.0/16            
17
[15] 5038/tcp                   ALLOW IN    192.168.0.0/16            
18
[16] 5038/tcp                   ALLOW IN    10.10.0.0/16              
19
[17] 8088/tcp                   ALLOW IN    192.168.0.0/16            
20
[18] 5039/tcp                   ALLOW IN    192.168.0.0/16            
21
[19] 9090                       ALLOW IN    10.10.0.0/16              
22
[20] 2812                       ALLOW IN    10.10.0.0/16              
23
[21] 8003                       ALLOW IN    Anywhere                  
24
[22] 2812                       ALLOW IN    192.168.0.0/16            
25
[23] 5083                       ALLOW IN    192.168.0.0/16            
26
[24] 5038                       ALLOW IN    192.168.0.0/16            
27
[25] 8010/tcp                   ALLOW IN    Anywhere                  
28
[26] 8011/tcp                   ALLOW IN    Anywhere                  
29
[27] 8012/tcp                   ALLOW IN    Anywhere                  
30
[28] 8013/tcp                   ALLOW IN    Anywhere                  
31
[29] 8014/tcp                   ALLOW IN    Anywhere                  
32
[30] 8015/tcp                   ALLOW IN    Anywhere                  
33
[31] 8016/tcp                   ALLOW IN    Anywhere                  
34
[32] 8099                       ALLOW IN    Anywhere                  
35
[33] 2812/tcp                   ALLOW IN    Anywhere                  
36
[34] 22/tcp (v6)                ALLOW IN    Anywhere (v6)             
37
[35] 8080/tcp (v6)              ALLOW IN    Anywhere (v6)             
38
[36] 8000/tcp (v6)              ALLOW IN    Anywhere (v6)             
39
[37] 80/tcp (v6)                ALLOW IN    Anywhere (v6)             
40
[38] 8001/tcp (v6)              ALLOW IN    Anywhere (v6)             
41
[39] 8002/tcp (v6)              ALLOW IN    Anywhere (v6)             
42
[40] 8003 (v6)                  ALLOW IN    Anywhere (v6)             
43
[41] 8010/tcp (v6)              ALLOW IN    Anywhere (v6)             
44
[42] 8011/tcp (v6)              ALLOW IN    Anywhere (v6)             
45
[43] 8012/tcp (v6)              ALLOW IN    Anywhere (v6)             
46
[44] 8013/tcp (v6)              ALLOW IN    Anywhere (v6)             
47
[45] 8014/tcp (v6)              ALLOW IN    Anywhere (v6)             
48
[46] 8015/tcp (v6)              ALLOW IN    Anywhere (v6)             
49
[47] 8016/tcp (v6)              ALLOW IN    Anywhere (v6)             
50
[48] 8099 (v6)                  ALLOW IN    Anywhere (v6)             
51
[49] 2812/tcp (v6)              ALLOW IN    Anywhere (v6)
Copied!
Output active numbered rules: ufw status numbered. You may find an introduction to UFW at Digital Ocean.
Port Forwarding 8080 -> 8002
Since some organizations-network firewalls block outgoing connections to "unusual" ports (z.B. HBK BS in der Produktion mit der Schwankhalle am 23.05.2021), we have set up an alternative port at 8080. This is forwarded to 8002 internally using NAT at /etc/ufw/before.rules:
1
 11 *nat
2
 12 :PREROUTING ACCEPT [0:0]
3
 13 -A PREROUTING -p tcp --dport 8080 -j REDIRECT --to-port 8002
4
 14 COMMIT
Copied!
Reload with systemctl restart ufw
Source: https://www.cogini.com/blog/port-forwarding-with-iptables/ or https://serverfault.com/a/238565​
You can monitor incoming bandwith with: sudo tcptrack -i enp4s0 port 8001.
Backup
Backups of the are composited in three stages:
1.Daily staging of changes via rsync
2.Weekly staging of changes via rsync
3.Monthly backup to external HDD via tar
For the monthly backups to happen the Backup HDD (labeled as such) needs to be plugged into the server.
Now on: https://github.com/radioangrezi/angrezi-backup​
Cronjobs
The cronjobs are scheduled as such.
Update (2020-05-06): They are not scheduled in /etc/cron* but via monit (!) in /etc/monit/monitrc (Group Backup). → Disabled and moved to /etc/cron.d/angrezi-backup
1
# m h  dom mon dow   command
2
0 4 * * * /mnt/backup/daily_backup.sh
3
0 6 * * 0 /mnt/backup/weekly_backup.sh
4
0 0 1 * * /mnt/backup/monthly_backup.sh
Copied!
Locations and Script
Locations backed up: /etc /var/angrezi /root /boot /opt /usr/local /srv /var/lib /var/mail /var/www /var/backups /var/local /var/opt /var/log
Excluded are: --exclude="*.wav" --exclude=/home/*/.gvfs --exclude=/home/*/.cache --exclude=/home/*/.local/share/Trash --exclude=/media
The script for montly backup mounts the HDD, stores the last staged weekly changes to the HDD and unmounts the HDD. Do not fiddle with the HDD! If you want to initiate a manual backup, run ./mnt/backup/monthly_backup.sh as root.
1
# Where to backup to
2
dest="/mnt/angrezi_backup_hdd/$(date +%Y)"
3
​
4
# Create archive filename
5
date=`date +%Y-%m-%d`
6
hostname=$(hostname -s)
7
archive_file="$hostname-$date-backup.tar.xz"
8
Log="/var/log/backup.log"
9
​
10
echo "" >> $Log
11
echo "---------------------------------------------------" >> $Log
12
echo "$date:: Full backup to $dest/$archive_file" >> $Log
13
now=`date +"%T"`
14
echo "$now:: starting..." >> $Log
15
echo "$now:: mounting Backup-HDD with UUID c59e6838-79ae-4994-bf1c-729ff9fae423 to /mnt/angrezi_backup_hdd" >> $Log
16
    mount UUID="c59e6838-79ae-4994-bf1c-729ff9fae423" /mnt/angrezi_backup_hdd/
17
    rc=$?
18
    now=`date +"%T"`
19
​
20
        echo "$now:: Mounting successful" >> $Log
21
​
22
now=`date +"%T"`
23
echo "$now:: starting backup..." >> $Log
24
tar -cvpJf "$dest/$archive_file" /mnt/backup/weekly #storing permissions, using xz as compression, verbose
25
rc=$?
26
now=`date +"%T"`
27
if [ $rc != 0 ]; then
28
    echo "$now:: Backup failed!" >> $Log
29
    if grep -qs '/mnt/angrezi_backup_hdd ' /proc/mounts; then
30
        echo "HDD is mounted, Backup still failed" >> $Log
31
    else
32
        echo "HDD is not mounted! Is it plugged in?" >> $Log
33
    fi
34
    exit 1
35
else
36
    echo "$now:: Finished." >> $Log
37
fi
38
​
39
now=`date +"%T"`
40
echo "$now:: Unmounting..." >> $Log
41
umount /mnt/angrezi_backup_hdd
42
​
43
rc=$?
44
if [ $rc != 0 ]; then
45
    echo "Unmounting not successful. Yikes. Exiting." >> $Log
46
else
47
    echo "Unmounting disc successful. Exiting. Thanks for choosing our services." >> $Log
48
fi
49
exit 1
Copied!
Monitoring
We use monit to monitor all "important" services at http://studio.radioangrezi.de:2812/.
Webserver
Domains Mapped:
stream.radioangrezi.de 
server.radioangrezi.de
studio.radioangrezi.de
voip.radioangrezi.de
Webservices:
AirTime
Angrezi Controller
Fileserver (Directory Listing with styling)
Icecast (Port 8000)
Cockpit (Port 9090)
Angrezi Controller
Self-build controller interface. Source on GitHub.
studio.radioangrezi.de/controller
Services
darkice (installed via apt service manually placed in /etc/systemd/system/darkice.service and removed in init.d) 
angrezi-master-recorder (self made)
angrezi-stream-monitor (self made)
angrezi-file-watch (self made)
After changes to the services in /etc/systemd/system/ you can reload them with systemctl daemon-reload + systemctl enable <service name>
angrezi-file-watch
angrezi-file-watch uses inotifywatch -m -r -e create to monitor the directories of recordings. If a file has been created (aka a new show has been recorded) it automatically appends chattr +a to prevent deletion.
Soundcard
onboard sound card (Intel) disabled in /etc/modprobe.d/blacklist-angrezi.conf
Pulse Audio disabled in /etc/pulse/client.conf
Using ALSA with Behringer UMC202HD with following config in asound.conf
1
pcm.stream_in_32 { # the Behringer hardware we use does only support 32 bit by default.
2
    type dsnoop
3
    ipc_key 5978293
4
    ipc_key_add_uid false
5
    hint.description "Angrezi: USB Card hw1 for Stream IN, 32 bit, dsnooped for mulitple listeners"
6
    slave {
7
        pcm "hw:1,0" 
8
        channels 2
9
        format S16_LE # this will only affect the plug pcm.stream_in_16.
10
        rate 44100
11
    }   
12
}
13
​
14
pcm.stream_in_16 {
15
    type plug
16
    hint.description "Angrezi: USB Card hw1 for Stream IN, 16 bit, dsnooped for mulitpl    e listeners"
17
    slave.pcm "stream_in_32"
18
}
Copied!
two capture devices are available:
stream_in_32 with S32_LE (hardware default)
stream_in_16 with S16_LE (software conversion via plug)
Test with (software monitor): arecord -f S16_LE -r44100 -c2 -D stream_in_16 | aplay -D hw:1 -
Darkice
The local studio audio in form stream_in_16 is broadcasted to Icecast via Darkice 1.3. Darkice only supports 16 bit sampling (which makes the downconversion necessary). Recording in DarkIce was disabled after a bug corrupted the recording on server reconnection occurred.
This local Darkice stream functions as master source in AirTime / liquidsoap.
Recording
FM Relay to Radio Weser TV
Radio Weser TV gets a special relay stream out (with a defined URL), which is made to never fail (go silent): http://stream.radioangrezi.de:8000/live-radioweser.
The stream is produced by relaying our live stream in a seperate liquidsoap instance (service: angrezi-relay-out-radioweser) which falls back to a playback of mp3 files if the stream goes silent for more than 10 seconds.
Fallback music (MP3, 44100, min. 192 kbit/s, Stereo) must be placed in /media/storage/share/Automation/Live-Out-RWTV/Music-MP3/ and fallback Jingles in /media/storage/share/Automation/Live-Out-RWTV/Jingles-MP3/. Jingles and Music are randomly picked in the radio 1:5. (If all fallback is empty as well, the script will fallback to a single audio file placed at /var/angrezi/relay-out-radioweser-fallback.mp3.)
Notice: The service needs to be restarted if the music folder is updated.
TODO: We could add a special config on Icecast so even if liquidsoap fails, there would be a fallback. TODO: Hide the mountpoint from Icecast.