var bibbase_data = {"data":"\"Loading..\"\n\n
\n\n \n\n \n\n \n \n\n \n\n \n \n\n \n\n \n
\n generated by\n \n \"bibbase.org\"\n\n \n
\n \n\n
\n\n \n\n\n
\n\n Excellent! Next you can\n create a new website with this list, or\n embed it in an existing web page by copying & pasting\n any of the following snippets.\n\n
\n JavaScript\n (easiest)\n
\n \n <script src=\"https://bibbase.org/show?bib=https%3A%2F%2Fleelabcnbc.github.io%2Flab-wiki-builder%2Fdemo_ref%2Fcomputer_vision%2Fbib.bib&jsonp=1&jsonp=1\"></script>\n \n
\n\n PHP\n
\n \n <?php\n $contents = file_get_contents(\"https://bibbase.org/show?bib=https%3A%2F%2Fleelabcnbc.github.io%2Flab-wiki-builder%2Fdemo_ref%2Fcomputer_vision%2Fbib.bib&jsonp=1\");\n print_r($contents);\n ?>\n \n
\n\n iFrame\n (not recommended)\n
\n \n <iframe src=\"https://bibbase.org/show?bib=https%3A%2F%2Fleelabcnbc.github.io%2Flab-wiki-builder%2Fdemo_ref%2Fcomputer_vision%2Fbib.bib&jsonp=1\"></iframe>\n \n
\n\n

\n For more details see the documention.\n

\n
\n
\n\n
\n\n This is a preview! To use this list on your own web site\n or create a new web site from it,\n create a free account. The file will be added\n and you will be able to edit it in the File Manager.\n We will show you instructions once you've created your account.\n
\n\n
\n\n

To the site owner:

\n\n

Action required! Mendeley is changing its\n API. In order to keep using Mendeley with BibBase past April\n 14th, you need to:\n

    \n
  1. renew the authorization for BibBase on Mendeley, and
    2. \n
  2. update the BibBase URL\n in your page the same way you did when you initially set up\n this page.\n
    3. \n
\n

\n\n

\n \n \n Fix it now\n

\n
\n\n
\n\n\n
\n \n \n
\n
\n  \n 2015\n \n \n (1)\n \n \n
\n
\n \n \n
\n \n\n \n \n \n \n \n \n Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.\n \n \n \n \n\n\n \n Xu, K.; Ba, J. L.; Kiros, R.; Cho, K.; Courville, A.; Salakhutdinov, R.; Zemel, R. S; and Bengio, Y.\n\n\n \n\n\n\n ArXiv e-prints, cs.LG. February 2015.\n \n\n\n\n
\n\n\n\n \n \n \"Show,Paper\n  \n \n\n \n\n \n link\n  \n \n\n bibtex\n \n\n \n\n \n  \n \n 1 download\n \n \n\n \n \n \n \n \n \n \n\n  \n \n \n \n \n \n \n\n\n\n
\n 
@article{Xu:2015ut,\nauthor = {Xu, Kevin and Ba, Jimmy Lei and Kiros, Ryan and Cho, Kyunghyun and Courville, Aaron and Salakhutdinov, Ruslan and Zemel, Richard S and Bengio, Yoshua},\ntitle = {{Show, Attend and Tell: Neural Image Caption Generation with Visual Attention}},\njournal = {ArXiv e-prints},\nyear = {2015},\nvolume = {cs.LG},\nmonth = feb,\nannote = {Great application of "attention" to image captioning.\n\nNotice that where to attend is not supervised, just like the case in CTC.\n\nCS231n says that one problem with this work is the resolution of attention. Maybe can be addressed using per pixel hypercolumn features, after upsampling.\n\npp. 4.\n\nCheck CS231n for the notation, notice that I think here T should be $T_{D+m+n, 4n}$.\n\nSection 4.1 Hard attention.\n\nThis version is learned by maximizing lower bound. In Eq. (11), s is a vector of all choices along time. and to learn this, we need sampling.\n\nin pp.5, there are some further refinements on optimization. Maybe these are just some tricks in MCMC. anyway.\n\nSection 4.2 soft attention.\n\npp. 6\n\nmost of left column is about connection between soft and hard versions. Ignored.\n\n\nSection 4.2.1 Doubly Stochastic Attention.\n\nNo idea why across time, the alpha should sum to 1 for each location. See <https://github.com/kelvinxu/arctic-captions/issues/38> and <https://www.reddit.com/r/MachineLearning/comments/2vjdcw/show_attend_and_tell_neural_image_caption/cok4yc9/>. Maybe it just worked, whatsoever.\n\nIn addition, that \\beta stuff might be some hack that emprically helps.\n\n},\nkeywords = {deep learning, recurrent},\nread = {Yes},\nrating = {5},\ndate-added = {2017-02-21T16:11:00GMT},\ndate-modified = {2017-02-23T19:57:09GMT},\nurl = {http://arxiv.org/abs/1502.03044},\nlocal-url = {file://localhost/Users/yimengzh/Documents/Papers3_revised/Library.papers3/Articles/2015/Xu/arXiv%202015%20Xu.pdf},\nfile = {{arXiv 2015 Xu.pdf:/Users/yimengzh/Documents/Papers3_revised/Library.papers3/Articles/2015/Xu/arXiv 2015 Xu.pdf:application/pdf}},\nuri = {\\url{papers3://publication/uuid/881D0A60-2E59-477D-814F-5E8908F34D89}}\n}\n
\n
\n\n\n\n
\n\n\n\n\n\n
\n
\n\n
\n
\n  \n 2014\n \n \n (1)\n \n \n
\n
\n \n \n
\n \n\n \n \n \n \n \n \n Visualizing and Understanding Convolutional Networks.\n \n \n \n \n\n\n \n Zeiler, M. D; and Fergus, R.\n\n\n \n\n\n\n In Fleet, D. J; Pajdla, T.; Schiele, B.; and Tuytelaars, T., editor(s), Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I, pages 818–833, 2014. Springer\n \n\n\n\n
\n\n\n\n \n \n \"VisualizingPaper\n  \n \n\n \n \n doi\n  \n \n\n \n link\n  \n \n\n bibtex\n \n\n \n  \n \n abstract \n \n\n \n\n \n \n \n \n \n \n \n\n  \n \n \n \n \n \n \n\n\n\n
\n 
@inproceedings{Zeiler:2014fr,\nauthor = {Zeiler, Matthew D and Fergus, Rob},\ntitle = {{Visualizing and Understanding Convolutional Networks}},\nbooktitle = {Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I},\nyear = {2014},\neditor = {Fleet, David J and Pajdla, Tom{\\'a}s and Schiele, Bernt and Tuytelaars, Tinne},\npages = {818--833},\npublisher = {Springer},\nannote = {a very popular and useful method for examining what a CNN learns.\n\nI think the method of deconvolution is better described in CS231n <http://cs231n.stanford.edu/slides/2016/winter1516_lecture9.pdf>. Essentially, we do backprop for all layers, except for ReLU, where negative diff is relu'ed (zeroed out).\n\n\nHowever, if you check the caffe for deep visualization toolbox, to see how deconv is done. <https://github.com/BVLC/caffe/compare/master...yosinski:deconv-deep-vis-toolbox>\n\nFor LRN, it's ignored; for ReLU, it's another ReLU.\n\nIn addition, it sets an example using visualization to guide architecture design. See Section 4.1. The "aliasing artifacts caused by the large stride..." actually means those blocky patterns in third panel of Figure 5 (it can be c or d depending whether you read label in the figure, or in the caption). Here, ther's only one (first row, second column). But in author's slides, there are more. It's from <http://videolectures.net/eccv2014_zeiler_convolutional_networks/>\n\n![comparison with alexnet](./deconv/comparison_with_alexnet.png)\n\nFig. 5 is wrong. Check arxiv version (v3, Fig 6 there) <https://arxiv.org/pdf/1311.2901.pdf>.\n\n![correct fig 5](./deconv/arxiv_v3_fig6.png)\n\nSome details.\n\nEnd of Section 3, they say they normalize filter weight size. I don't think this is needed in general.\n},\nkeywords = {classics, deep learning},\ndoi = {10.1007/978-3-319-10590-1_53},\nlanguage = {English},\nread = {Yes},\nrating = {5},\ndate-added = {2017-02-16T19:50:02GMT},\ndate-modified = {2017-03-27T00:55:38GMT},\nabstract = {Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark Krizhevsky et al. [18]. However there is no clear understanding of why the},\nurl = {http://dx.doi.org/10.1007/978-3-319-10590-1_53},\nlocal-url = {file://localhost/Users/yimengzh/Documents/Papers3_revised/Library.papers3/Articles/2014/Zeiler/ECCV%202014%20Part%20I%202014%20Zeiler.pdf},\nfile = {{ECCV 2014 Part I 2014 Zeiler.pdf:/Users/yimengzh/Documents/Papers3_revised/Library.papers3/Articles/2014/Zeiler/ECCV 2014 Part I 2014 Zeiler.pdf:application/pdf}},\nuri = {\\url{papers3://publication/doi/10.1007/978-3-319-10590-1_53}}\n}\n\n\n\n
\n
\n\n\n
\n Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark Krizhevsky et al. [18]. However there is no clear understanding of why the\n
\n\n\n
\n\n\n\n\n\n
\n
\n\n
\n
\n  \n 2013\n \n \n (1)\n \n \n
\n
\n \n \n
\n \n\n \n \n \n \n \n \n Intriguing properties of neural networks.\n \n \n \n \n\n\n \n Szegedy, C; Zaremba, W; Sutskever, I; Bruna, J; Erhan, D; Goodfellow, I; and Fergus, R.\n\n\n \n\n\n\n ArXiv e-prints, cs.CV. December 2013.\n \n\n\n\n
\n\n\n\n \n \n \"IntriguingPaper\n  \n \n\n \n\n \n link\n  \n \n\n bibtex\n \n\n \n\n \n\n \n \n \n \n \n \n \n\n  \n \n \n \n \n \n \n\n\n\n
\n 
@article{Szegedy:2013vw,\nauthor = {Szegedy, C and Zaremba, W and Sutskever, I and Bruna, J and Erhan, D and Goodfellow, I and Fergus, Rob},\ntitle = {{Intriguing properties of neural networks}},\njournal = {ArXiv e-prints},\nyear = {2013},\nvolume = {cs.CV},\nmonth = dec,\nannote = {Good analysis paper, showing that 1) at higher layers, units or mixture of units are roughtly the same. BUT, I'm not sure if this is still the case for networks with only one FC layer (only fc8, in AlexNet's term) 2) adversarial examples. Their root reason might be that each layer is not stable (Table 5).\n\n\n\nSection 4\n\n> a non-local generalization prior over the input space. In other words, it is assumed that is possible for the output unit to assign non- significant (and, presumably, non-epsilon) probabilities to regions of the input space that contain no training examples in their vicinity.\n\nthis is also mentioned in DL book. DL models assumes some non-local prior.\n\n\n\n\n\nSection 4.1 here they assume images pixels have range 0 to 1, not 0-255.\n\nHere D(x,f(x)) should be x, not f(x). clearly f(x) doesn't makes dimension match.\n\n\nHere how they actually find an adversarial example. is really vague. It's explained more clearly in [Exploring the Space of Adversarial Images](https://arxiv.org/pdf/1510.05328.pdf)  (version 5 on ArXiv) with code at <https://github.com/tabacof/adversarial>. Later referred to "the 2015 paper". I copied relevant page in the notes.\n\nEssentially, you need to first select an wrong label l to fool the network, and then you start from a small C (this paper), or big C (the 2015 paper), and then surely you will find a somewhat big r that will satisfy that f(x+r)=l. Then you decrease C little by little, until that f(x+r) != l. This process will intuitively decrease r, yet biasing the r to those preserving f(x+r)=l. Intuitively makes sense. Not sure how good it is in theory.\n\n\n> This penalty function method would yield the exact solution for D(X,l) in the case of convex losses, however neural networks are non-convex in general, so we end up with an approximation in this case.\n\nI don't think this is right. At least I can't derive it using my RUBBISH note.\n\n\n\n======RUBBISH START======\n\nI don't understand the math of box-constrained L-BFGS. I think here it's using the correspondence between Constrained and Lagrange forms. See text around Eq. 5.7 and 5.8 of [Statistical Learning with Sparsity](https://trevorhastie.github.io/): "For convex programs, the Lagrangian allows for the constrained problem (5.5) to be solved by reduction to an equivalent unconstrained problem.", or page 16-17 of <http://www.stat.cmu.edu/{\\textasciitilde}ryantibs/convexopt-S15/lectures/12-kkt.pdf>\n\nHere I would say the correspondence is loose. to make it more precise, we should modify the constraint f(x+r)=l in the first (original) problem to be something like loss(x+r,l)<0.001.\n\nTwo problems, written as constrained form, and lagrange form, are\n\n1) constrained form.\n\nmin |r|, s.t. loss(x+r,l) < \\eps\n\n2) lagrange form\n\nmin |r| + \\lambda loss(x+r,l).\n\nFirst, we replace hard constraint f(x+r)=l with a more soft constraint using loss (I would say the paper is poorly written; no relationship between f and loss_f is mentioned).\n\nThen they also assume that, big eps correspond to small lambda, and small eps correspond to big lambda.    This assumption is also kind of assumed in Section 7.2 of Deep Learning book. Check my notes on that.\n\n======RUBBISH END======\n\n\n\n\nSection 4.2\n\n> A subtle, but essential detail is that we only got improvements by generating adversarial examples for each layer outputs which were used to train all the layers above. The network was trained in an alternating fashion, maintain- ing and updating a pool of adversarial examples for each layer separately in addition to the original training set.\n\nFor adversarial example to be useful, you can't use input-level adversarial examples. Instead, you need intermediate adversarial examples. But this would really make training complicated.\n\nSection 4.3\n\nNot sure about the math about deriving operator norm of W when W is convolution. But whatever... Conclusion is important.},\nkeywords = {deep learning, To Read},\nread = {Yes},\nrating = {4},\ndate-added = {2017-03-29T19:52:30GMT},\ndate-modified = {2017-03-30T14:48:37GMT},\nurl = {http://arxiv.org/abs/1312.6199},\nlocal-url = {file://localhost/Users/yimengzh/Documents/Papers3_revised/Library.papers3/Articles/2013/Szegedy/arXiv%202013%20Szegedy.pdf},\nfile = {{arXiv 2013 Szegedy.pdf:/Users/yimengzh/Documents/Papers3_revised/Library.papers3/Articles/2013/Szegedy/arXiv 2013 Szegedy.pdf:application/pdf}},\nuri = {\\url{papers3://publication/uuid/C6374003-66D2-4059-941E-D3B1E4F50BBB}}\n}\n\n\n\n
\n
\n\n\n\n
\n\n\n\n\n\n
\n
\n\n
\n
\n  \n 2006\n \n \n (1)\n \n \n
\n
\n \n \n
\n \n\n \n \n \n \n \n \n On the Analysis and Interpretation of Inhomogeneous Quadratic Forms as Receptive Fields.\n \n \n \n \n\n\n \n Berkes, P.; and Wiskott, L.\n\n\n \n\n\n\n Neural Computation, 18(8): 1868–1895. August 2006.\n \n\n\n\n
\n\n\n\n \n \n \"OnPaper\n  \n \n\n \n \n doi\n  \n \n\n \n link\n  \n \n\n bibtex\n \n\n \n  \n \n abstract \n \n\n \n\n \n \n \n \n \n \n \n\n  \n \n \n \n \n\n\n\n
\n 
@article{Berkes:2006el,\nauthor = {Berkes, Pietro and Wiskott, Laurenz},\ntitle = {{On the Analysis and Interpretation of Inhomogeneous Quadratic Forms as Receptive Fields}},\njournal = {Neural Computation},\nyear = {2006},\nvolume = {18},\nnumber = {8},\npages = {1868--1895},\nmonth = aug,\nannote = {I only read it upt to Section 5 (inlcuded), as I care more about visualization, not about statistical test.\n\nessential contributions.\n\n1. General methods to solve Eq. (4.1). That is, maximizing/minimizing a inhomogeneous quadratic form (which can model complex V1 cells), under a norm constraint.\n\n2. Eq. (5.5). given optimal stimulus, find the change of firing rate along all directions orthgonal to the direction of optimal stimulus (Eq. (5.10)). For a demo of this, see\n    * [Visualization of optimal stimuli and invariances \nfor Tiled Convolutional Neural Networks](http://ai.stanford.edu/{\\textasciitilde}quocle/TCNNweb/)\n    * [original code](http://people.brandeis.edu/~berkes/software/qforms-tk/index.html)\n    * [Yimeng's implementation](https://github.com/leelabcnbc/tang-paper-2017/blob/master/neuron_fitting/debug/debug_hessian_visualization_complex_cell.ipynb)\n\nThere are some errors and points note taking in the paper about the method.\n\n1. Above (5.8), we actually need to compute the null space of $x^+$ (N-1 vectors orthogonal to $x^+$). Check Quoc Le and my implementation on how to do this. Doing Gram-Schmidt on $x^+$ plus $e_1$ through $e_{N-1}$ doesn't necessarily give the correct result.\n2. As $x^+$ is local maxima, then you would expect along all directions, (5.5) gives you negative values. This is true, but eigenvalues of Eq. (5.8) are not all negative. Instead, you need to add back the offset term in Eq. (5.5). Then all offsetted eigenvalues are negative. Least negative ones are most invariant, and vice versa. Check original implmentation and Yimeng's implementation (`best_variance_and_invariance_directions `).\n3. Eq. (5.8) is simply computing the eigenvalues of H, along directions othorgonal to $x^+$. That is, they paramterize $w$ with $Bx$, $B$ being a $N$ by $N-1$ basis matrix, and $x$ being a $N-1$ vector. This justifies why using Eq. (5.9) to recover the original space.\n\n\nSome caveats\n\nFor this to work, you need to 1) really find the optimal stimulus (at least some local maxima), and 2) compute the hessian correctly. Both are pretty difficult for a really complex neuron, say CNN.\n\n1) is either not very accurate, or it gives noise like input (from my experience, images that excite a aritifical neuron most are mostly noise like).\n2) is either theoretically impossbie (see <https://github.com/leelabcnbc/tang-paper-2017/blob/master/neuron_fitting/debug/debug_hessian_old_plus_adam_vs_lbfgs.ipynb>), or it takes so long to compute (say, using TensorFlow).\n\nYimeng found that this method doesn't work well with really complex neurons. See <https://github.com/leelabcnbc/tang-paper-2017/blob/master/neuron_fitting/debug/cnn_fitting_debug_nonOT_visualize.ipynb>\n\nIn the paper, they are not analyzing some fitted V1 cells. Instead, they are analzying some artificial units learned by Slow Feature Analysis. Check Section 2. According to Deep Learning book, SFA has closed form solution, so it can't be too complicated.\n\n\nLast paragraph of Section 4.  optimal x might not make sense or be relevant.\n\n> Note that although $x^+$ is the stimulus that elicits the strongest response in the function, it does not necessarily mean that it is representative of the class of stimuli that give the most important contribution to its output. This depends on the distribution of the input vectors.\n\n},\npublisher = {MIT Press 238 Main St., Suite 500, Cambridge, MA 02142-1046 USA journals-info@mit.edu},\nkeywords = {V1},\ndoi = {10.1162/neco.2006.18.8.1868},\nlanguage = {English},\nread = {Yes},\nrating = {4},\ndate-added = {2017-05-29T17:19:56GMT},\ndate-modified = {2017-06-12T15:26:03GMT},\nabstract = {In this letter, we introduce some mathematical and numerical tools to analyze and interpret inhomogeneous quadratic forms. The resulting characterization is in some aspects similar to that given by experimental studies of cortical cells, making it particularly suitable for application to second-order approximations and theoretical models of physiological receptive fields. We first discuss two ways of analyzing a quadratic form by visualizing the coefficients of its quadratic and linear term directly and by considering the eigenvectors of its quadratic term. We then present an algorithm to compute the optimal excitatory and inhibitory stimuli{\\textemdash}those that maximize and minimize the considered quadratic form, respectively, given a fixed energy constraint. The analysis of the optimal stimuli is completed by considering their invariances, which are the transformations to which the quadratic form is most insensitive, and by introducing a test to determine which of these are statistically significant. Next we prop...},\nurl = {http://www.mitpressjournals.org/doi/10.1162/neco.2006.18.8.1868},\nlocal-url = {file://localhost/Users/yimengzh/Documents/Papers3_revised/Library.papers3/Articles/2006/Berkes/Neural%20Computation%202006%20Berkes.pdf},\nfile = {{Neural Computation 2006 Berkes.pdf:/Users/yimengzh/Documents/Papers3_revised/Library.papers3/Articles/2006/Berkes/Neural Computation 2006 Berkes.pdf:application/pdf}},\nuri = {\\url{papers3://publication/doi/10.1162/neco.2006.18.8.1868}}\n}\n\n\n\n
\n
\n\n\n
\n In this letter, we introduce some mathematical and numerical tools to analyze and interpret inhomogeneous quadratic forms. The resulting characterization is in some aspects similar to that given by experimental studies of cortical cells, making it particularly suitable for application to second-order approximations and theoretical models of physiological receptive fields. We first discuss two ways of analyzing a quadratic form by visualizing the coefficients of its quadratic and linear term directly and by considering the eigenvectors of its quadratic term. We then present an algorithm to compute the optimal excitatory and inhibitory stimuli—those that maximize and minimize the considered quadratic form, respectively, given a fixed energy constraint. The analysis of the optimal stimuli is completed by considering their invariances, which are the transformations to which the quadratic form is most insensitive, and by introducing a test to determine which of these are statistically significant. Next we prop...\n
\n\n\n
\n\n\n\n\n\n
\n
\n\n\n\n\n
\n\n\n \n\n \n \n \n \n\n
\n"}; document.write(bibbase_data.data);