Sync your cloud with PTP

What is PTP ?

First thing first, what is PTP. The PTP (Precision Time Protocol) is a protocol used to synchronize clocks in a network. When used in conjunction with hardware support, PTP is capable of sub-microsecond accuracy, which is far better than is normally obtainable with NTP (Network Time Protocol).

Alright, once we know what is PTP, the following article will describe how PTP is used in a cloud environment (Note, you might want to read more material about PTP related info to understand some of the concepts mentioned below, for example, PTP Boundary clock, PTP Ordinary clock etc, the PTP wiki would be good enough to know the basis of PTP,  btw, you could safely ignore the algorithm part as we’ll not go deep into the PTP protocol itself, rather to discuss how it’s used in the openstack environment.)

Basically, we will discuss two parts of work that enable the usage of PTP in the cloud:

1)  kvm virtual ptp driver introduced in linux kernel .

2) ptp configuration with TripleO (an openstack installer which use openstack to install openstack).

Before we actually go into above two items, let’s first see how PTP is normally used in Operating System.

PTP in Operating System

PTP support in Operating System is divided between the kernel and user space, for example, in Redhat linux or CentOS, the kernel includes support for PTP clocks which are provided by network drivers(since hardware PTP relies on physical NIC to provide hardware clock, you may want to visit this link for a list of NIC that supports PTP, you could also manually check if the network interface has hardware PTP capability by executing ‘ethtool -T eth0’ in terminal, make sure replace eth0 with your own interface name ). The actual implementation of the protocol is known as linuxptp, a PTPv2 implementation according to the IEEE standard 1588v2 for linux.

The linuxptp package includes ptp4l and phc2sys programs for clock synchronization. The ptp4l program implements the PTP boundary clock and ordinary clock. With hardware time stamping , it’s used to synchronize the PTP hardware clock located on physical NIC to remote master clock. The phc2sys program is needed only with hardware time stamping, for synchronizing the system clock to PTP hardware clock on the network interface card (NIC).

KVM virtual PTP driver

Ok, we have already understood how PTP is used in the host level, but in a cloud environment, we want all the guest VMs running in the host could acquire the same level accuracy from host clock, this is why KVM virtual PTP driver is introduced! so where does this goody come from ?

Recently(well, it may not that recent 🙂 ), a cool feature called KVM virtual PTP driver is introduced by Marcelo Tosatti in linux kernel which allows guest to sync its clock to the host clock with high precision. In order to use this feature, you will need to update kernels on both host and guest. Then execute the following step in guest:

1) load the kvm ptp driver

# modprobe kvm_ptp

2) add the following line to chrony config file(/etc/chrony.conf)

# refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0

3) confirm ptp device is in the list of time sources

# chronyc sources

That’s it! it looks like so easy to use, right?  but … how it works underneath?

well, let’s discuss a little bit about it via the following diagram:

ptp-blog

In above diagram:

1) Network(PTP) driver is a common linux network driver(e.g. ixgbe.ko) which support hardware PTP; use ‘ethtool -T eth0’ to check the PTP capability of hardware NIC.

2) Linuxptp is an implementation of the Precision Time Protocol (PTP) according to IEEE standard 1588v2 for Linux

3) Linuxptp:ptp4l implements Boundary Clock (BC) and Ordinary Clock (OC), it synchronizes PTP hardware clock (PHC) to remote master clock.

4) Linuxptp:phc2sys synchronizes two or more clocks in the system, typically it is used to synchronize the system clock to a PTP hardware clock (PHC).

5) System realtime clock is the system clock(CLOCK_REALTIME).

6) ptp_kvm.ko: kernel module with gettime method returning hosts realtime clock. This allows chrony to synchronize host and guest clocks with high precision.

7) With kvm virtual ptp driver, all the VMs in the same compute node can achieve equal time accuracy by using one ptp capable NIC as time source.

8) One obvious note is that linuxptp leverages network (ptp) driver to sync PHC to remote PTP master, NIC3 can NOT be used for this purpose because it’s using dpdk(vfio+pmd driver) instead of kernel network driver. (although dpdk supports getting timestamp not sure if it’s a valid request of ovs or linuxptp to leverage dpdk driver to sync PHC to remote PTP master).

Openstack/TripleO support for PTP

Well, the final missing step to enable PTP in the cloud would be to deploy and configure PTP in an openstack environment, in this article, we use TripleO to provide a configuration path of PTP. this is very much an ongoing work in the community as of writing this blog. If you are familiar with TripleO, we first need a spec to elaborate the details of ptp feature. In this spec, we’re going to consider a basic PTP usage scenario – Ordinary Clock (slave mode), but it could be extended to support Boundary Clock or Master Clock easily based on this initial work.

There are three parameters exposed to operators for ptp configuration:

1) PTP hardware interface name
# PtpInterface: ‘nic1’

2) Configure PTP clock in slave mode
# PtpSlaveMode: 1

3) Configure PTP message transport protocol
# PtpMessageTransport: ‘UDPv4’

The first parameter is ‘PtpInterface’ which is most important one as we usually configure ptp service on physical interface which gives us more accurate time compared with software timestamping.

The second parameter is ‘PtpSlaveMode’ which configures the ptp service in slave mode (Ordinary Clock).

The third parameter is ‘PtpMessageTransport’ which specifies the transport protocol of ptp messages, it supports three type of transport protocols, such as ‘L2’, ‘UDPv4’ and ‘UDPv6’.

Other Considerations

Ok, if you reach this point, congrats! we have got all the necessary elements to use ptp in the basic cloud environment, but (there is always a but)  if your have a complex cloud environment, for example, with SR-IOV, DPDK technologies deployed in the cloud, we may need more considerations on how to configure and use ptp.

1) NTP

Since most of the clouds install NTP by default, enabling ptp services means we have to consider the co-exist of NTP and PTP, whether feeds PTP source to NTP or vice versa. the latter one may not happen if you’re not going to deploy a grandmaster of PTP on one of the deployed nodes. there is also a method provided by linuxptp for leveraging NTP and PTP, that is to use timemaster which is a program (it can be configured as a service) that uses ptp4l and phc2sys in combination with chronyd or ntpd to synchronize the system clock to NTP and PTP time sources. you might also want to consider the possible impact on service or components that reply on NTP.

2) DPDK

As mentioned above, PTP uses PHC (PTP Hardware Clock) framework in kernel network driver to get/set hardware timestamp and adjust hardware PHC time. NIC with dpdk enabled doesn’t use kernel network driver which in turn cannot be used as PTP time source. although dpdk library supports basic time functionalities like get/set methods, we cannot use it without a proper user space program or OVS that interact with these dpdk methods. so whether user should pay attention to this defect during deployment or we should build an isolation on dpdk enabled card & ptp enabled card.

3) SR-IOV

With sriov enabled card, user can either choose PF passthrough to get accurate time from PTP or use kvm virtual ptp driver to sync system time to host real time clock. but if one chooses to pass ptp enabled PF to VM, then host and other VMs in the compute node may lost ptp time source if that PF is the only configured ptp interface. so the workaround would be either user is aware of the PTP configuration on sriov PF and not to attach that PF to any of VMs or we build an isolation during deployment to avoid potential break of the only PTP time source.  another defect of using sriov PF as PTP source is that VF cannot share the PHC of PF simply because there is no such hardware resource allocated to VF.

4) TC (Transparent Clock)

In real network, there are transparent clocks configured in commodity hardware switch or router to measure the residence time spent when the packet passing through the switch or router, the residence time is then added to the correction field of the PTP packet for further consumption by slave mode clock to reduce the effects of packet delay variation. but linuxptp doesn’t support Transparent Clock (TC) and there are some issues to consider if one wants to use software to implement a TC :  A TC must behave like a switch, although you can use Linux to bridge multiple ports, still the performance is not as good as real switch. So no one would be interested in a TC built on a Linux software switch.

Conclusion

With above works, I think we have introduced PTP into the cloud environment. In the future, further enhancements maybe needed to introduce timemaster configuration to combine NTP & PTP,  to support complex configuration of PTP network etc.

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s