The status page in Airtime does not show a status for some services: airtime_analyzer, airtime-playout, airtime-liquidsoap. The services might be running correctly though. You can check via our own monitoring or service SERVICE-NAME status and journalctl -u SERVICE-NAME | tail
Error 500 / No PHP logs
You would suspect PHP logs in /var/log/airtime/zendlog.log. But there is none (or it is not up to date). Check /var/log/apache2/error.log: It might show:
[Mon Dec 16 16:36:41.959122 2019] [:error] [pid 26398] [client 127.0.0.1:39688] PHP Fatal error: Uncaught exception 'Zend_Log_Exception' with message '"/var/log/airtime/zendphp.log" cannot be opened with mode "a"' in /usr/share/airtime/php/vendor/zendframework/zendframework1/library/Zend/Log/Writer/Stream.php:81\nStack trace:\n#0 /usr/share/airtime/php/airtime_mvc/application/logging/Logging.php(11): Zend_Log_Writer_Stream->__construct('/var/log/airtim...')\n#1 /usr/share/airtime/php/airtime_mvc/application/logging/Logging.php(83): Logging::getLogger()\n#2 /usr/share/airtime/php/airtime_mvc/application/logging/Logging.php(166): Logging::error('Uncaught except...')\n#3 [internal function]: Logging::loggingShutdownCallback()\n#4 {main}\n thrown in /usr/share/airtime/php/vendor/zendframework/zendframework1/library/Zend/Log/Writer/Stream.php on line 81
Check the permissions on /var/log/airtime/zendlog.log. It should be 0777, probably.
DNS Problems
2019-05-20
Recently the HfK-network experienced isses with its DNS Servers (172.16.0.1 + 172.16.0.2). Local streaming services (darkice, airtime, icecast) were affecetd. Resolved by adding additional 3rd-party DNS Servers form Digitalcourage (46.182.19.48) + Chaos Computer Club (213.73.91.35) to /etc/netplan/XX.yaml.
If the Airtime logs indicate failures to connect to the RabbitMQ server, such as:
2013-10-31 08:21:11,255 ERROR - [pypomessagehandler.py : main() : line
99] - Error connecting to RabbitMQ Server. Trying again in few seconds
2013-10-31 08:21:11,255 ERROR - [pypomessagehandler.py : main() : line 99] - Error connecting to RabbitMQ Server. Trying again in few seconds - See more at: http://forum.sourcefabric.org/discussion/16050/#sthash.W8OJrNFm.dpuf
but the RabbitMQ server is running normally, this error might be due to a change in the server's hostname since Airtime installation. Directory names under /var/lib/rabbitmq/mnesia/ indicate that RabbitMQ's database files are organised according to the hostname of the server, for example:
rabbit@airtime
where the hostname is airtime.example.com. If the hostname has changed, it may be necessary to reconfigure RabbitMQ manually, as follows:
Delete the files in /var/lib/rabbitmq/mnesia/
sudo rm -r /var/lib/rabbitmq/mnesia/*
Restart RabbitMQ:
sudo invoke-rc.d rabbitmq-server restart
Enter the following commands to set up authentication and grant permissions. The rabbitmqctl add_user command requires the RabbitMQ password from the /etc/airtime/airtime.conf file as an argument. The rabbitmqctl set_permissions command should be entered on one line, with the list of Airtime services repeated three times:
Libretime logs PHP errors into /var/log/airtime/zendphp.log You can use Libretimes log controller to add custom log statements to the code. (Native echo() and print_r() will not help much.) Use Logging::error(), Logging::warn(), Logging::info() or Logging::debug() from anywhere in the Libretime codebase.
Error 500 in airtime-playout (pypo) and Invalid argument supplied for foreach() zendphp.log
2019-06-21
Phenomenon
airtime-playout (sudo -u www-data python /usr/bin/airtime-playout) is outputting 500 errors for all API requests:
for ListenerStat.php:98 and ApiController.php:1870
Not-the-solution
Checked files: ListenerStat.php and ApiController.php -> POST getParams() always empty. -> no data -> null / true object (which is not iterable).
Also checked: airtime-playout.py / listenerstat.py and api_client.py -> all not faulty, because dumped (tcpdump) http packets showed correct payload.
Notices on the side, but not related: SQLSTATE[55P03]: Lock not available: 7 ERROR: could not obtain lock on row in relation "cc_pref" (#465)
in place updated of Airtime / Libretime will not be easy. (#623)
Solution
The hostname and default apache vhost was changed. Since than the error occured. Apache was 301 forwarding requests made to anregzistudio.hfk-bremen.de to studio.radioangrezi.de. POST data is lost when forwarding!
Put correct hostname in base_url of /etc/airtime.conf. This would go for a port change as well (e.g. HTTPS)!
Traceback (most recent call last):
File "/usr/bin/pyponotify", line 4, in <module>
__import__('pkg_resources').run_script('airtime-playout==1.0', 'pyponotify')
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 664, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1444, in run_script
exec(code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/airtime_playout-1.0-py2.7.egg/EGG-INFO/scripts/pyponotify", line 6
backupCount=8)
File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__
BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__
logging.FileHandler.__init__(self, filename, mode, encoding, delay)
File "/usr/lib/python2.7/logging/__init__.py", line 920, in __init__
StreamHandler.__init__(self, self._open())
File "/usr/lib/python2.7/logging/__init__.py", line 950, in _open
stream = open(self.baseFilename, self.mode)
IOError: [Errno 2] No such file or directory: '/var/log/airtime/pypo/notify.log'
Resolution: Log files could not be written. Add log directory and give write permissions.