Census and Survey of the Visible Internet

Census and Survey of the Visible Internet. Heidemann, J., Pradkin, Y., Govindan, R., Papadopoulos, C., Bartlett, G., & Bannister, J. In Proceedings of the ACM Internet Measurement Conference, pages 169–182, Vouliagmeni, Greece, October, 2008. ACM.

Paper abstract bibtex

Prior measurement studies of the Internet have explored traffic and topology, but have largely ignored edge hosts. While the number of Internet hosts is very large, and many are hidden behind firewalls or in private address space, there is much to be learned from examining the population of \emphvisible hosts, those with public unicast addresses that respond to messages. In this paper we introduce two new approaches to explore the visible Internet. Applying statistical population sampling, we use \emphcensuses to walk the entire Internet address space, and \emphsurveys to probe frequently a fraction of that space. We then use these tools to evaluate address usage, where we find that only 3.6% of allocated addresses are actually occupied by visible hosts, and that occupancy is unevenly distributed, with a quarter of responsive /24 address blocks (subnets) less than 5% full, and only 9% of blocks more than half full. We show about 34%7emillion addresses are very stable and visible to our probes (about 16% of responsive addresses), and we project from this up to 60%7emillion stable Internet-accessible computers. The remainder of allocated addresses are used intermittently, with a median occupancy of 81 minutes. Finally, we show that many firewalls are visible, measuring significant diversity in the distribution of firewalled block size. To our knowledge, we are the first to take a census of edge hosts in the visible Internet since 1982, to evaluate the accuracy of active probing for address census and survey, and to quantify these aspects of the Internet.

@InProceedings{Heidemann08c,
	author = "John Heidemann and Yuri Pradkin and Ramesh
 Govindan and Christos Papadopoulos and 
 Genevieve Bartlett and Joseph Bannister",
	title = "Census and Survey of the Visible Internet",
	booktitle = 	"Proceedings of the " # "ACM Internet Measurement Conference",
	year = 		2008,
	sortdate = 		"2008-10-01", 
	project = "ant, lander, madcat, lacrend, mrnet",
	jsubject = "topology_modeling",
	publisher =	"ACM",
	address =	"Vouliagmeni, Greece",
	month =		oct,
	pages =		"169--182",
	jlocation =	"johnh: pafile",
	keywords =	"lander, final report, network monitoring",
	keywords =	"census, survey, internet address space",
	url =		"https://ant.isi.edu/%7ejohnh/PAPERS/Heidemann08c.html",
	pdfurl =	"https://ant.isi.edu/%7ejohnh/PAPERS/Heidemann08c.pdf",
	myorganization =	"USC/Information Sciences Institute",
	copyrightholder = "authors",
	abstract = "
Prior measurement studies of the Internet have explored traffic and
topology, but have largely ignored edge hosts.  While the number of
Internet hosts is very large, and many are hidden behind firewalls or
in private address space, there is much to be learned from examining
the population of \emph{visible} hosts, those with public unicast
addresses that respond to messages.  In this paper we introduce two
new approaches to explore the visible Internet.  Applying statistical
population sampling, we use \emph{censuses} to walk the entire
Internet address space, and \emph{surveys} to probe frequently a
fraction of that space.  We then use these tools to evaluate address
usage, where we find that only 3.6\% of allocated addresses are
actually occupied by visible hosts, and that occupancy is unevenly
distributed, with a quarter of responsive /24 address blocks (subnets)
less than 5\% full, and only 9\% of blocks more than half full.  We
show about 34%7emillion addresses are very stable and visible to our
probes (about 16\% of responsive addresses), and we project from this
up to 60%7emillion stable Internet-accessible computers.  The remainder
of allocated addresses are used intermittently, with a median
occupancy of 81 minutes.  Finally, we show that many firewalls are
visible, measuring significant diversity in the distribution of
firewalled block size.  To our knowledge, we are the first to take a
census of edge hosts in the visible Internet since 1982, to evaluate
the accuracy of active probing for address census and survey, and to
quantify these aspects of the Internet.
",
}

Downloads: 0

{"_id":"rJWG7GW4avrq5HCPQ","bibbaseid":"heidemann-pradkin-govindan-papadopoulos-bartlett-bannister-censusandsurveyofthevisibleinternet-2008","author_short":["Heidemann, J.","Pradkin, Y.","Govindan, R.","Papadopoulos, C.","Bartlett, G.","Bannister, J."],"bibdata":{"bibtype":"inproceedings","type":"inproceedings","author":[{"firstnames":["John"],"propositions":[],"lastnames":["Heidemann"],"suffixes":[]},{"firstnames":["Yuri"],"propositions":[],"lastnames":["Pradkin"],"suffixes":[]},{"firstnames":["Ramesh"],"propositions":[],"lastnames":["Govindan"],"suffixes":[]},{"firstnames":["Christos"],"propositions":[],"lastnames":["Papadopoulos"],"suffixes":[]},{"firstnames":["Genevieve"],"propositions":[],"lastnames":["Bartlett"],"suffixes":[]},{"firstnames":["Joseph"],"propositions":[],"lastnames":["Bannister"],"suffixes":[]}],"title":"Census and Survey of the Visible Internet","booktitle":"Proceedings of the ACM Internet Measurement Conference","year":"2008","sortdate":"2008-10-01","project":"ant, lander, madcat, lacrend, mrnet","jsubject":"topology_modeling","publisher":"ACM","address":"Vouliagmeni, Greece","month":"October","pages":"169–182","jlocation":"johnh: pafile","keywords":"census, survey, internet address space","url":"https://ant.isi.edu/%7ejohnh/PAPERS/Heidemann08c.html","pdfurl":"https://ant.isi.edu/%7ejohnh/PAPERS/Heidemann08c.pdf","myorganization":"USC/Information Sciences Institute","copyrightholder":"authors","abstract":"Prior measurement studies of the Internet have explored traffic and topology, but have largely ignored edge hosts. While the number of Internet hosts is very large, and many are hidden behind firewalls or in private address space, there is much to be learned from examining the population of \\emphvisible hosts, those with public unicast addresses that respond to messages. In this paper we introduce two new approaches to explore the visible Internet. Applying statistical population sampling, we use \\emphcensuses to walk the entire Internet address space, and \\emphsurveys to probe frequently a fraction of that space. We then use these tools to evaluate address usage, where we find that only 3.6% of allocated addresses are actually occupied by visible hosts, and that occupancy is unevenly distributed, with a quarter of responsive /24 address blocks (subnets) less than 5% full, and only 9% of blocks more than half full. We show about 34%7emillion addresses are very stable and visible to our probes (about 16% of responsive addresses), and we project from this up to 60%7emillion stable Internet-accessible computers. The remainder of allocated addresses are used intermittently, with a median occupancy of 81 minutes. Finally, we show that many firewalls are visible, measuring significant diversity in the distribution of firewalled block size. To our knowledge, we are the first to take a census of edge hosts in the visible Internet since 1982, to evaluate the accuracy of active probing for address census and survey, and to quantify these aspects of the Internet. ","bibtex":"@InProceedings{Heidemann08c,\n\tauthor = \"John Heidemann and Yuri Pradkin and Ramesh\n Govindan and Christos Papadopoulos and \n Genevieve Bartlett and Joseph Bannister\",\n\ttitle = \"Census and Survey of the Visible Internet\",\n\tbooktitle = \t\"Proceedings of the \" # \"ACM Internet Measurement Conference\",\n\tyear = \t\t2008,\n\tsortdate = \t\t\"2008-10-01\", \n\tproject = \"ant, lander, madcat, lacrend, mrnet\",\n\tjsubject = \"topology_modeling\",\n\tpublisher =\t\"ACM\",\n\taddress =\t\"Vouliagmeni, Greece\",\n\tmonth =\t\toct,\n\tpages =\t\t\"169--182\",\n\tjlocation =\t\"johnh: pafile\",\n\tkeywords =\t\"lander, final report, network monitoring\",\n\tkeywords =\t\"census, survey, internet address space\",\n\turl =\t\t\"https://ant.isi.edu/%7ejohnh/PAPERS/Heidemann08c.html\",\n\tpdfurl =\t\"https://ant.isi.edu/%7ejohnh/PAPERS/Heidemann08c.pdf\",\n\tmyorganization =\t\"USC/Information Sciences Institute\",\n\tcopyrightholder = \"authors\",\n\tabstract = \"\nPrior measurement studies of the Internet have explored traffic and\ntopology, but have largely ignored edge hosts. While the number of\nInternet hosts is very large, and many are hidden behind firewalls or\nin private address space, there is much to be learned from examining\nthe population of \\emph{visible} hosts, those with public unicast\naddresses that respond to messages. In this paper we introduce two\nnew approaches to explore the visible Internet. Applying statistical\npopulation sampling, we use \\emph{censuses} to walk the entire\nInternet address space, and \\emph{surveys} to probe frequently a\nfraction of that space. We then use these tools to evaluate address\nusage, where we find that only 3.6\\% of allocated addresses are\nactually occupied by visible hosts, and that occupancy is unevenly\ndistributed, with a quarter of responsive /24 address blocks (subnets)\nless than 5\\% full, and only 9\\% of blocks more than half full. We\nshow about 34%7emillion addresses are very stable and visible to our\nprobes (about 16\\% of responsive addresses), and we project from this\nup to 60%7emillion stable Internet-accessible computers. The remainder\nof allocated addresses are used intermittently, with a median\noccupancy of 81 minutes. Finally, we show that many firewalls are\nvisible, measuring significant diversity in the distribution of\nfirewalled block size. To our knowledge, we are the first to take a\ncensus of edge hosts in the visible Internet since 1982, to evaluate\nthe accuracy of active probing for address census and survey, and to\nquantify these aspects of the Internet.\n\",\n}\n\n","author_short":["Heidemann, J.","Pradkin, Y.","Govindan, R.","Papadopoulos, C.","Bartlett, G.","Bannister, J."],"bibbaseid":"heidemann-pradkin-govindan-papadopoulos-bartlett-bannister-censusandsurveyofthevisibleinternet-2008","role":"author","urls":{"Paper":"https://ant.isi.edu/%7ejohnh/PAPERS/Heidemann08c.html"},"keyword":["census","survey","internet address space"],"metadata":{"authorlinks":{}}},"bibtype":"inproceedings","biburl":"https://bibbase.org/f/dHevizJoWEhWowz8q/johnh-2023-2.bib","dataSources":["YLyu3mj3xsBeoqiHK","fLZcDgNSoSuatv6aX","fxEParwu2ZfurScPY","7nuQvtHTqKrLmgu99"],"keywords":["census","survey","internet address space"],"search_terms":["census","survey","visible","internet","heidemann","pradkin","govindan","papadopoulos","bartlett","bannister"],"title":"Census and Survey of the Visible Internet","year":2008}