On near-uniform URL sampling. Henzinger, M. R., Heydon, A., Mitzenmacher, M., & Najork, M. Comput. Networks, 33(1-6):295–308, June, 2000.
doi  abstract   bibtex   
We consider the problem of sampling URLs uniformly at random from the Web. A tool for sampling URLs uniformly can be used to estimate various properties of Web pages, such as the fraction of pages in various Internet domains or written in various languages. Moreover, uniform URL sampling can be used to determine the sizes of various search engines relative to the entire Web. In this paper, we consider sampling approaches based on random walks of the Web graph. In particular, we suggest ways of improving sampling based on random walks to make the samples closer to uniform. We suggest a natural test bed based on random graphs for testing the effectiveness of our procedures. We then use our sampling approach to estimate the distribution of pages over various Internet domains and to estimate the coverage of various search engine indexes.
@article{Henzinger2000,
abstract = {We consider the problem of sampling URLs uniformly at random from the Web. A tool for sampling URLs uniformly can be used to estimate various properties of Web pages, such as the fraction of pages in various Internet domains or written in various languages. Moreover, uniform URL sampling can be used to determine the sizes of various search engines relative to the entire Web. In this paper, we consider sampling approaches based on random walks of the Web graph. In particular, we suggest ways of improving sampling based on random walks to make the samples closer to uniform. We suggest a natural test bed based on random graphs for testing the effectiveness of our procedures. We then use our sampling approach to estimate the distribution of pages over various Internet domains and to estimate the coverage of various search engine indexes.},
author = {Henzinger, Monika R. and Heydon, Allan and Mitzenmacher, Michael and Najork, Marc},
doi = {10.1016/S1389-1286(00)00055-4},
file = {:home/ecem/Dropbox/mendeley\_sampling\_references/Henzinger et al/2000\_Henzinger et al.\_On near-uniform URL sampling.pdf:pdf},
issn = {13891286},
journal = {Comput. Networks},
keywords = {internet domain distribution,random walks,search engine size,uniform sampling,url sampling,www},
mendeley-tags = {uniform sampling,www},
month = jun,
number = {1-6},
pages = {295--308},
title = {{On near-uniform URL sampling}},
volume = {33},
year = {2000}
}

Downloads: 0