Unsupervised Anomaly Detection in Large Databases Using Bayesian Networks. Cansado, A. & Soto, A. Applied Artificial Intelligence, 22(4):309-330, 2008.
Unsupervised Anomaly Detection in Large Databases Using Bayesian Networks [pdf]Paper  abstract   bibtex   2 downloads  
Today, there has been a massive proliferation of huge databases storing valuable information. The opportunities of an effective use of these new data sources are enormous, however, the huge size and dimensionality of current large databases call for new ideas to scale up current statistical and computational approaches. This paper presents an application of Ar- tificial Intelligence technology to the problem of automatic detection of candidate anomalous records in a large database. We build our approach with three main goals in mind: 1)An effective detection of the records that are potentially anomalous, 2)A suitable selection of the subset of at- tributes that explains what makes a record anomalous, and 3)An efficient implementation that allows us to scale the approach to large databases. Our algorithm, called Bayesian Network Anomaly Detector (BNAD), uses the joint probability density function (pdf) provided by a Bayesian Net- work (BN) to achieve these goals. By using appropriate data structures, advanced caching techniques, the flexibility of Gaussian Mixture models, and the efficiency of BNs to model joint pdfs, BNAD manages to effi- ciently learn a suitable BN from a large dataset. We test BNAD using synthetic and real databases, the latter from the fields of manufacturing and astronomy, obtaining encouraging results.

Downloads: 2