AARNet Network Engineer Glen Turner weighs in on NTP access control with occasional distractions
In recent months, a number of high profile Internet services have been affected by NTP amplification attacks. Here’s how and why this happens and suggestions for how to protect yourself, your systems, and others.
Network Time Protocol (NTP) is used to synchronise clocks among computers over a network like the Internet. It is one of the oldest Internet protocols, hailing from the early 1980s, and is critical to the healthy operation of many systems on the Internet. Accurate time is necessary for secure authentication as well as for good time keeping.
NTP is part of the sharing and caring ecosystem that forms the Internet which means that many NTP servers are publically accessible. Unfortunately, this opens them to abuse through what is known as an ‘amplification attack’. An amplification attack is a type of denial of service (DoS) attack in which the attacker generates a small amount of network traffic, which is then turned into a much larger volume of traffic by the amplifier (in this case, an NTP server or a series of NTP servers), which is then sent on to a victim’s computer. The victim’s network connection is overloaded to the point where it is unusable.
The time on your computer is set when it asks a NTP server for the current time. These servers use an atomic clock or GPS receivers to set their own time, and then use the NTP protocol to distribute that time to those computers.
Diagnosing issues with network time can be difficult, so as well as providing the current time the NTP servers provides access for clients to debug their time-setting issues.
client$ nptq ntp-server.example.edu.au ntpq> peers remote refid st t when poll reach delay offset jitter ============================================================================== *ntp1.example.co .GPS. 1 u 602 1024 377 26.912 -0.020 0.005 +ntp2.example.co .CDMA. 1 u 360 1024 377 168.428 0.036 0.087 +ntp3.example.co .GPS. 1 u 180 1024 377 21.650 0.018 0.015
Traditionally, access to these queries has been allowed to all users. Unfortunately this access can be readily abused, as explained in the next section. Then the remainder of this posting explains how to configure a NTP server to remove access to the query facility.
The bug du jour for denial of service attacks seems to be CVE-2013-5211. Let’s have a closer look at that.
NTP provides a time service using UDP port 123. As well as providing time using that port it also provides a query mechanism and administration. Tradition has to been to lock down the administration facilities but to allow anyone to query the status of the time server. That provides some very useful information, such as the time server’s own clock sources and their current drift.
Another way of looking at the query facility is as an amplifier: you send in a few bytes on a UDP port and you get a lot of bytes in return. Fake the source IP address of the status query (using the IP address of your victim) and a host can force a lot of packets to be sent to a victim without needing that same amount of connectivity to the victim. The ntpdc monlist command has the greatest amplification, but it’s hardly the only possibility.
The current CVE says the ntpdc‘s
monlist command is the mechanism of current Denial of Service attacks. To check that specific vulnerability on NTP server with addresses 198.51.100.1, 2001:db8:1234:5678::1 say:
$ ntpdc -n -c monlist 198.51.100.1 $ ntpdc -n -c monlist 2001:db8:1234:5678::1
If you get a list of IP addresses returned then the exploitation is viable.
remote address port local address count m ver rstr avgint lstint =============================================================================== ...
If you are forbidden then that’s good:
$ ntpdc -n -c monlist 198.51.100.1 ntpdc: read: Connection refused
You should do this remotely of the machine, as access rules are often looser for localhost than for exterior-facing interfaces. You should test this for IPv6 as well as for IPv4. If your machine has a IPv6 link-local address then that’s a “real” IPv6 address and you should check the configuration that access via IPv6 is controlled rather than left open.
The ntpdc command is in the “ntp” package in both Red Hat and Debian and their dervied products.
ntpd uses the
restrict command to control access. The
restrict command sets the access to “all”, then applies each of the listed restrictions. This gives the non-obvious behaviour that
restrict default gives everyone unrestricted access.
For the local machine we want to allow all time, query and administrative access. That is, not to list any restrictions:
restrict 127.0.0.1 restrict -6 ::1
For a typical host which acts as a NTP client and doesn’t provide a public time service, the remainder of the access control is simple: ignore all requests apart from time sychronisation with the NTP servers.
restrict default ignore restrict -6 default ignore restrict 127.0.0.1 restrict -6 ::1 server 198.51.100.1 iburst restrict 198.51.100.1 nomodify nopeer noquery notrap server -6 2001:db8:1234:5678::1 iburst restrict -6 2001:db8:1234:5678::1 nomodify nopeer noquery notrap
Let’s say we act as a NTP server for the public, getting our time from a GPS or some other high precision external source. The access control is to ignore all exterior administration and status queries:
restrict default nomodify nopeer noquery notrap limited kod restrict -6 default nomodify nopeer noquery notrap limited kod restrict 127.0.0.1 restrict -6 ::1
The phrase “
limited kod” activates rate management. You should use rate management whenever you provide NTP service beyond your administrative control; that is, on public NTP servers and on NTP servers for large communities (such as universities). The default settings are for a NTP server providing global service. The
discard command tunes the rate management algorithm and the
mru maxmem command control the maximum amount of memory the algorithm uses: this defaults to 1MB and that memory could be reduced if you provide service to a small community.
Let’s say we act as a NTP server for a site, using a number of NTP servers then acting as a NTP server to the hosts in our network interior (192.0.2.0/24, 2001:db8:1:2::/64). The access control is to ignore all but interior requests. We don’t trust interior requests enough to allow them to administer or make status queries. Obviously we shouldn’t ignore time signals from the NTP servers of whom are asking the time:
restrict default ignore restrict -6 default ignore restrict 127.0.0.1 restrict -6 ::1 restrict 192.0.2.0 netmask 255.255.255.0 nomodify nopeer noquery notrap restrict -6 2001:db8:1:2:: netmask ffff:ffff:ffff:ffff:: nomodify nopeer noquery notrap # add "limited kod" for large interior networks where administrative control is challenging server 198.51.100.1 iburst restrict 198.51.100.1 nomodify nopeer noquery notrap server -6 2001:db8:1234:5678::1 iburst restrict -6 2001:db8:1234:5678::1 nomodify nopeer noquery notrap
If you do want to allow ntpq to work then remove the
noquery from the restrictions. You might choose to do this for hosts which monitor the NTP service, such as the site’s network management host (which should be graphing the clock drift, the number of servers, the number of clients, and alarming if a server is voted as a falseticker).
# Nagios server runs ntpq restrict 192.0.2.50 nomodify nopeer notrap
The scenario where the NTP server is providing time to a network interior is the only case where NTP should also synchronise to the computer’s system clock if the better time sources are lost:
server 127.127.1.0 fudge 127.127.1.0 stratum 10
The reason for this is that maintaining the same time on all machines is important within an interior, even if the time is wrong, as that allows Kerberos and Windows Domain authentication to continue. Even in this case it is better to have three or four interior NTP servers supplying the real time: DHCP will happily supply a list of NTP servers to clients.
Do not add the system clock to ntpd if you are providing NTP service to the public. Those public clients are using many sources of time so having your clock drop out for a while is of little concern. Whereas if you start to provide bogus time from a drifting system clock, that’s not a good result.
There’s no point adding the system clock to NTP if the machine is a NTP client. All the circularity does is to make your brain hurt.
ntpd supports IPv6. Commands containing IPv4 addresses are optionally marked with “
-4“, commands containing IPv6 addresses are marked with “
This approach has one major flaw: if you are running IPv6 and you don’t know it, then there is no access restriction. Most modern operating systems run IPv6, and thus will have a link-local IPv6 address: the only question is if the network you are currently connected to gives you a global IPv6 address. In any case, you might not be told when your network starts giving you a global address. If you believe that your network does not run IPv6 then say “
restrict -6 default ignore” to prevent neighbouring hosts on IPv6 link-local addresses from gaining unrestricted access.
In the most recent ntpd releases the syntax has been unified and address type detection occurs automatically. In these releases the use of “
-4” or “
-6” causes a confusing parsing error. However this change does mean that “
restrict default ignore” now applies to all address families. It is best not to rely on that behaviour until you’ve tried to start the server with a configuration containng “
restrict -6 default ignore” and had the parser fail:
ntpd: line 10 column 34 syntax error, unexpected T_String, expecting T_EOC
server command accepts DNS names. The
restrict command does not. This becomes a problem when using NTP pool servers as per-server access controls cannot be set. As a result many distributions provide a NTP time service because they cannot override “
restrict default ignore” and so have to say “
restrict default nomodify nopeer noquery notrap“.
I’d encourage people wanting to run public NTP servers to become part of the NTP pool rather than be explicitly listed on the list of public NTP servers as then you can withdraw service and only need to wait until all users reboot until you see no more incoming queries, whereas an explicit listing can still be seeing traffic a decade after the server has been removed from the list.
If you are removed from the list of NTP servers then don’t drop incoming queries using a firewall or router, as the clients will only re-try. Configure a ntpd and use the Kiss of Death to be marked out of contention by the client; that is, “
restrict default nomodify nopeer noquery notrap limited kod” and set the rate limiting to always be active, such that the kiss of death is sent to all users. The KoD packet will mark the server permanently down and no further attempts will be made to synchronise with it during the runtime of each client’s ntpd.
Apr 7, 2020
Mar 31, 2020
Mar 25, 2020