Going Dark: measurement when the Internet hides the detail

In the final podcast for 2025, APNIC Chief Scientist Geoff Huston discusses the problem of independent measurement in an Internet which is increasingly “going dark”. Communications has always included a risk of snooping, and a matching component of work to enhance privacy, from the simplest ciphers used in ancient times, techniques of hiding and discovering messages, attempts to prevent and detect intrusion of the mail, to adoption of telegraph codes, the cutting of telegraph wires in wartime (to force messages into radio where they could be listened to) and the development of modern encryption algorithms typically using the public-private keypair model. There has always been a story of “attack” and “response” in how we communicate privately. Aside from matters of state security, banking and finance at large depend on a degree of privacy and now require it under legislation to enable use of creditcard information online. Many other contexts have an assumption of privacy, and use technology to try and preserve it. Fundamentally, individuals in their use of the Internet are entitled to expect a level of privacy where the state permits it. The publication of RFC7258 “Pervasive Monitoring Is an Attack in 2014 formalised a belief that the intrusion of third parties into a communication between two ends demanded a technology response to exclude them, where possible. Protocol designers and Internet Engineers took up the challenge. This position led over time to a marked increase in the adoption of privacy enhancing protocol features. For example, the web moved from HTTP: denoted URLs to HTTPS: where the content is protected by the Transport Layer Security (TLS) encryption protocol, which now overwhelmingly predominates in the at large. However, significant aspects of Internet communications “leak” information to third parties. Between an individual and a web service lies their provider, unknown numbers of intermediate providers, typically a content distribution system hosting the web site in a local copy, all of whom have opportunities to see and understand what is being done, and by whom. In particular the DNS typically exposes the name and address of the site being connected to across all kinds of protocols (not just the web) and exposes it to unknown intermediary systems as the DNS lookup is processed. In response to this, services are emerging which break down the DNS into dissociated queries: what is being looked for, and who is looking for it, and use intermediary services which may know one, but not both: Questions are seen to be asked, but by who is now hidden. If you know who is asking, you don’t know what they are asking for. Combined with newer network protocols like QUIC which imposes a strong end-to-end encryption model which even hides the inter-packet size and timing information (another form of leak which can be used to reconstruct what kind of traffic is flowing) it has become increasingly hard for an independent researcher to see inside the network: It’s going dark. Geoff explores the nature of privacy in the Internet at large, and how APNIC Labs gets round this problem with it’s measurement system. PING will return in January 2026 with another season of episodes. Until then, enjoy this final recording of 2025, and see you online, in the new year.