PokeFlex: A Real-World Dataset of Deformable Objects for Robotics

Jan Obrist*,1, Miguel Zamora*,1, Hehui Zheng*,2,3, Ronan Hinchet2, Firat Ozdemir4, Juan Zarate5, Robert K. Katzschmann2,3, Stelian Coros1,
*Equal contribution.
1Computational Robotics Lab, Department of Computer Science, ETH Zurich
2Soft Robotics Lab, Department of Mechanical Engineering, ETH Zurich
3ETH AI Center, ETH Zurich
4Swiss Data Science Center, ETH Zurich and EPFL
5Advanced Interactive Technologies Lab, Department of Computer Science, ETH Zurich
Preprint

+ Website might take some time to load for the first time.

PokeFlex is a dataset containing real-world paired and annotated multimodal data of deformable objects
that includes 3D textured meshes, point clouds, RGB images, and depth maps.

Abstract

Data-driven methods have shown great potential in solving challenging manipulation tasks, however, their application in the domain of deformable objects has been constrained, in part, by the lack of data. To address this, we propose PokeFlex, a dataset featuring real-world paired and annotated multimodal data that includes 3D textured meshes, point clouds, RGB images, and depth maps. Such data can be leveraged for several downstream tasks such as online 3D mesh reconstruction and it can potentially enable underexplored applications such as the real-world deployment of traditional control methods based on mesh simulations. To deal with the challenges posed by real-world 3D mesh reconstruction, we leverage a professional volumetric capture system that allows complete 360° reconstruction. PokeFlex consists of 18 deformable objects with varying stiffness and shapes. Deformations are generated by dropping objects onto a flat surface or by poking the objects with a robot arm. Interaction forces and torques are also reported for the latter case. Using different data modalities, we demonstrated a use case for the PokeFlex dataset in online 3D mesh reconstruction.

Overview of objects

Deformation sequences

Top: Mesh reconstructions of foam dice for a poking sequence shown in every third frame.
Bottom: Mesh reconstructions of plush octopus for a dropping sequence.

Data modalities

Interactive Visualization

  • The plot on the left illustrates the 3D mesh reconstructed using a professional volumetric capture system for a toilet paper roll. The dotted red line donotes the trajectory of the robot's end-effector executing a poking strategy.
  • The plot on the right shows the magnitud of the force applied by the end effector on the toilet paper roll.

Notes:

  • Feel free to rotate the 3D mesh on the left.
  • The visualization might take some time load.
  • Consider pressing the Stop button and then pressing the Play button again, if playback is slow.
  • On mobile devices consider using landscape mode (Rotate your device).

Learning-based online 3D mesh reconstruction

Superimposed representation of the proposed network architectures for ingesting the multi-modal PokeFlex data to predict deformed mesh reconstruction. Inference rates across different data modalities range from 106 Hz to 215 Hz for dense point clouds and forces, respectively. (See paper for more details).

BibTeX Preprint

@article{obrist2024pokeflex,
      author    = {Obrist, Jan and Zamora, Miguel and Zheng, Hehui and Hinchet, Ronan and Ozdemir, Firat and Zarate, Juan and Katzschmann, Robert K. and Coros, Stelian},
      title     = {PokeFlex: A Real-World Dataset of Deformable Objects for Robotics},
      journal   = {Under review},
      year      = {2024}
      url       = {https://arxiv.org/pdf/2410.07688}
      }