Enabling Visual Action Planning for Object Manipulation Through Latent Space Roadmap. Lippi, M., Poklukar, P., Welle, M. C., Varava, A., Yin, H., Marino, A., & Kragic, D. IEEE TRANSACTIONS ON ROBOTICS, 2022. ![Enabling Visual Action Planning for Object Manipulation Through Latent Space Roadmap [link]](https://bibbase.org/img/filetypes/link.svg "link")
Paper doi abstract bibtex In this article, we present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a latent space roadmap (LSR) for task planning, which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of the following three parts. First, a mapping module (MM) that maps observations is given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states. Second, the LSR, which builds and connects clusters containing similar states in order to find the latent plans between start and goal states, extracted by MM. Third, the action proposal module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.
@article{
11580_93946,
author = {Lippi, M. and Poklukar, P. and Welle, M. C. and Varava, A. and Yin, H. and Marino, A. and Kragic, D.},
title = {Enabling Visual Action Planning for Object Manipulation Through Latent Space Roadmap},
year = {2022},
journal = {IEEE TRANSACTIONS ON ROBOTICS},
abstract = {In this article, we present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a latent space roadmap (LSR) for task planning, which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of the following three parts. First, a mapping module (MM) that maps observations is given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states. Second, the LSR, which builds and connects clusters containing similar states in order to find the latent plans between start and goal states, extracted by MM. Third, the action proposal module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.},
keywords = {Deep Learning in Robotics and Automation; Heuristic algorithms; Latent Space Planning; Manipulation Planning; Planning; Robots; Stacking; Task analysis; Trajectory; Visual Learning; Visualization},
url = {https://ieeexplore.ieee.org/document/9833914},
doi = {10.1109/TRO.2022.3188163},
pages = {1--19},
number = {1}
}
Downloads: 0
{"_id":"8AsSWRQB6M7at5hgf","bibbaseid":"lippi-poklukar-welle-varava-yin-marino-kragic-enablingvisualactionplanningforobjectmanipulationthroughlatentspaceroadmap-2022","author_short":["Lippi, M.","Poklukar, P.","Welle, M. C.","Varava, A.","Yin, H.","Marino, A.","Kragic, D."],"bibdata":{"bibtype":"article","type":"article","author":[{"propositions":[],"lastnames":["Lippi"],"firstnames":["M."],"suffixes":[]},{"propositions":[],"lastnames":["Poklukar"],"firstnames":["P."],"suffixes":[]},{"propositions":[],"lastnames":["Welle"],"firstnames":["M.","C."],"suffixes":[]},{"propositions":[],"lastnames":["Varava"],"firstnames":["A."],"suffixes":[]},{"propositions":[],"lastnames":["Yin"],"firstnames":["H."],"suffixes":[]},{"propositions":[],"lastnames":["Marino"],"firstnames":["A."],"suffixes":[]},{"propositions":[],"lastnames":["Kragic"],"firstnames":["D."],"suffixes":[]}],"title":"Enabling Visual Action Planning for Object Manipulation Through Latent Space Roadmap","year":"2022","journal":"IEEE TRANSACTIONS ON ROBOTICS","abstract":"In this article, we present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a latent space roadmap (LSR) for task planning, which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of the following three parts. First, a mapping module (MM) that maps observations is given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states. Second, the LSR, which builds and connects clusters containing similar states in order to find the latent plans between start and goal states, extracted by MM. Third, the action proposal module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.","keywords":"Deep Learning in Robotics and Automation; Heuristic algorithms; Latent Space Planning; Manipulation Planning; Planning; Robots; Stacking; Task analysis; Trajectory; Visual Learning; Visualization","url":"https://ieeexplore.ieee.org/document/9833914","doi":"10.1109/TRO.2022.3188163","pages":"1–19","number":"1","bibtex":"@article{\n\t11580_93946,\n\tauthor = {Lippi, M. and Poklukar, P. and Welle, M. C. and Varava, A. and Yin, H. and Marino, A. and Kragic, D.},\n\ttitle = {Enabling Visual Action Planning for Object Manipulation Through Latent Space Roadmap},\n\tyear = {2022},\n\tjournal = {IEEE TRANSACTIONS ON ROBOTICS},\n\tabstract = {In this article, we present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a latent space roadmap (LSR) for task planning, which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of the following three parts. First, a mapping module (MM) that maps observations is given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states. Second, the LSR, which builds and connects clusters containing similar states in order to find the latent plans between start and goal states, extracted by MM. Third, the action proposal module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.},\n\tkeywords = {Deep Learning in Robotics and Automation; Heuristic algorithms; Latent Space Planning; Manipulation Planning; Planning; Robots; Stacking; Task analysis; Trajectory; Visual Learning; Visualization},\n\turl = {https://ieeexplore.ieee.org/document/9833914},\n\tdoi = {10.1109/TRO.2022.3188163},\n\tpages = {1--19},\n\tnumber = {1}\n}\n","author_short":["Lippi, M.","Poklukar, P.","Welle, M. C.","Varava, A.","Yin, H.","Marino, A.","Kragic, D."],"key":"11580_93946","id":"11580_93946","bibbaseid":"lippi-poklukar-welle-varava-yin-marino-kragic-enablingvisualactionplanningforobjectmanipulationthroughlatentspaceroadmap-2022","role":"author","urls":{"Paper":"https://ieeexplore.ieee.org/document/9833914"},"keyword":["Deep Learning in Robotics and Automation; Heuristic algorithms; Latent Space Planning; Manipulation Planning; Planning; Robots; Stacking; Task analysis; Trajectory; Visual Learning; Visualization"],"metadata":{"authorlinks":{}}},"bibtype":"article","biburl":"https://bibbase.org/network/files/LhpEBBMAufCZxdRLG","dataSources":["YYA4jfiKR2BsiLkFH","a9jNc2NyyC7d58MTq","YjpAZwNicNHANadbW","9HNP8ouNgGHQfHxfN"],"keywords":["deep learning in robotics and automation; heuristic algorithms; latent space planning; manipulation planning; planning; robots; stacking; task analysis; trajectory; visual learning; visualization"],"search_terms":["enabling","visual","action","planning","object","manipulation","through","latent","space","roadmap","lippi","poklukar","welle","varava","yin","marino","kragic"],"title":"Enabling Visual Action Planning for Object Manipulation Through Latent Space Roadmap","year":2022}