Dualtip ml objectives

[sinaaghli]

Merging ML recognition, grasping and dualtip flipping action with @bkanator !

[nbbrooks]

Do all of the _hw objectives need to live in _hw ? Any reason they could not be in _sim?

Can you rename the Objectives like "GetLeftRightPickPoseFromBottlePose" to be "Get Left Right Pick Pose From Bottle Pose" for readability?

[sinaaghli]

Do all of the _hw objectives need to live in _hw ? Any reason they could not be in _sim?

Can you rename the Objectives like "GetLeftRightPickPoseFromBottlePose" to be "Get Left Right Pick Pose From Bottle Pose" for readability?

There are some device names that are hardcoded in the behavior atm, I was mostly focusing to get things running for the friday deadline! but as you mentioned we should define those device names as behavior IO and then move the generic one into the SIM, this is in my to do list after I finish detecting the rack with QR code!

[nbbrooks]

I suspect a lot of these objectives are duplicates of core perception Objectives (or if not, should instead live in the core perception objectives)

Similar for some of the motion planning ones - may be duplicative of items from #6

[bkanator]

I suspect a lot of these objectives are duplicates of core perception Objectives (or if not, should instead live in the core perception objectives)

Similar for some of the motion planning ones - may be duplicative of items from #6

so there are hardly any core perception objectives. Many live in lab_sim or other places, so i don't want to start putting things in core when I don't understand what qualifies to get it there.

[bkanator]

ok @nbbrooks take another look. I removed a lot of objectives that I believe were just being used as examples. I am not sure what hardcoded hw things there were for the ones under _hw.... but kept those there so it was obvious what sina ran in hardware. So, I kind of like the separation since it was obvious then what he used for the hardware demo.

[Copilot]

Pull request overview

This PR integrates ML-based object recognition, dual-arm grasping, and multi-tip manipulation capabilities into the dual-arm mobile robot system. The changes enable the robot to locate bottles using vision ML models, grasp them with coordinated dual-arm movements, and perform complex manipulation tasks.

Changes:

• Added ML model files and CLIPSeg dependency for vision-based object segmentation
• Implemented bottle detection, grasping, and placement workflows using behavior trees
• Updated robot waypoint configurations for new manipulation poses
• Disabled unavailable scene camera in hardware launch configuration

Reviewed changes

Copilot reviewed 26 out of 27 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/dual_arm_mobile_sim/models/*.onnx	Added SAM2 encoder, decoder, and L2G ML model files for vision processing
src/dual_arm_mobile_sim/description/assets/bottle.*	Added bottle 3D mesh assets (STL, OBJ, MTL)
src/dual_arm_mobile_sim/objectives/*.xml	Created behavior tree objectives for wrist camera snapshots, image segmentation, and multi-tip path planning
src/dual_arm_mobile_hw/objectives/*.xml	Implemented complete pick-and-place workflow with bottle localization, dual-arm grasping, and rack placement
src/dual_arm_mobile_sim/waypoints/waypoints.yaml	Added "ApproachBottle" and "AwayTable" waypoints with updated joint configurations
src/dual_arm_mobile_hw/waypoints/waypoints.yaml	Simplified waypoint configuration, added "Look For Task Trainer" and "side look at table" poses
src/dual_arm_mobile_hw/launch/*.xml	Commented out unavailable scene camera configuration
.gitmodules	Added moveit_pro_clipseg submodule for CLIPSeg-based segmentation
src/dual_arm_mobile_sim/CMakeLists.txt	Added models directory to installation targets

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bkanator]

in porting phoebe_ws to lunar_sim, I need to remove some sim dependencies on things that are not actually used (only needed in hw). So I am also using this branch to do some updates for the package.xml where these are hardware-only and not referenced by any sim file:

- robotiq_driver
- ur_robot_driver
- moveit_studio_ur_pstop_manager
- clearpath_controller_interface

and removing the clearpath_controller_interface dependency, since phoebe_sim already uses the MuJoCo plugin. HW urdf is separate so it should continue using that.

also removing the dependency from phoebe_description since this sim description should not depend on hardware src/phoebe_bridgeback_description/package.xml

[bkanator]

as part of port to lunar_sim also need to make mujoco_model_package a xacro arg, rather than hardcoded

[nbbrooks]

Approving for now so that v9 controller migration can be tested against this config, but I am making a followup PR to move more of the _hw objectives into _sim and make more reusable subtree objectives (e.g. less use of waypoints)

[nbbrooks]

Removing approval to address hw dependencies strategy

[bkanator]

added back clearpath_controller_interface