template : moondream vision + groq orchestration #104
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
| } catch (err) { | ||
| error = err instanceof Error ? err.message : String(err); | ||
| raw = '{"actions":[]}'; | ||
| } |
There was a problem hiding this comment.
Variable shadowing loses error message when retry fails
Medium Severity
The catch (error) clause at line 205 shadows the outer error variable declared at line 197. When the inner retry also fails at line 213, error = err instanceof Error ? err.message : String(err) assigns to the catch block's error parameter, not the outer tracking variable. This causes the error message to be lost, and the function returns with error as undefined even when both API calls failed. The Python equivalent correctly avoids this by using exc and exc2 as catch variable names.
Additional Locations (1)
| const result = await computer.executeAction(ComputerAction.SCROLL_DOCUMENT, args); | ||
| lastScreenshot = updateScreenshot(result.base64Image, lastScreenshot); | ||
| } | ||
| logs.push({ step: stepNumber, action, status: 'success', detail: `Scrolled ${direction}` }); |
There was a problem hiding this comment.
Scroll action always reports success ignoring actual result
Medium Severity
The scroll action at line 147 hardcodes status: 'success' regardless of whether the underlying action succeeded or failed. Unlike all other actions in this file (e.g., navigate, go_back, key) which use status(result.error) to check the actual result, the scroll action's log is placed outside the if-else branches where result is scoped, making result.error inaccessible. The Python equivalent correctly uses _status(result) at line 141. This causes scroll failures to be incorrectly reported as successful.
1 participant
new template
overview
uses opensource models (via api) that can be ported locally.
emphasizes screenshot-based interactions (similar to the gemini template), playwright usage limited to getting tab title|url and handling loading waits.
demo
task
ebay : search "Dreamcast", open a listing under $500, return JSON {product_url,price,currency}.replay
groq-moondream-cua.mp4
logs
{ "summary": "Completed 28/28 steps", "final_response": "{\"product_url\":\"https://www.ebay.com/itm/157653229097?_skw=Dreamcast&itmmeta=01KGG3FJ7H45PC8E84R92SSCNS&hash=item24b4dd5e29:g:DeEAAeSwwwtpf-mK&itmprp=enc%3AAQALAAAA8O7PUuNWmJ%2B%2BUShgI9tQz%2FoqO9upPS28NnONv2gVIoF1CYmpPAnniYiiA2wCDdcXzzG1H1Z8w%2FDyq2JE9dbo76w%2FpDrHxhnnGsstpns55AnQD2HkQZOH62ZWrm5M1tmCsc48u58%2FqZW%2B7Q%2BEQxEVQKmbJsrIbZBKYasTsjPnL594JiHI%2BQ28aDgnueDXpGB5HoA9RT7tCUbRnkwELToWmn354C%2B16qcjqimEnC%2Flkc7E8zT3M2EJ7P408vH37VYz6rNcd4LnjHnw9P4siZTlsHcwurwaDQ8whwyFFxL0mfCsUULgQdySV4sRj9TpkHd5lQ%3D%3D%7Ctkp%3ABk9SR_qjvoOEZw\",\"price\":\"159.99\",\"currency\":\"USD\"}", "steps": [ { "step": 1, "action": "navigate", "status": "success", "detail": "Navigated to https://www.ebay.com" }, { "step": 1, "action": "moondream_point", "status": "success", "detail": "Point found", "output": "{\"x\":0.41796875,\"y\":0.197265625,\"x_norm\":418,\"y_norm\":197,\"x_px\":502,\"y_px\":158,\"screen\":{\"width\":1200,\"height\":800}}" }, { "step": 2, "action": "click_at", "status": "success", "detail": "Clicked at coordinates" }, { "step": 2, "action": "type_text_at", "status": "success", "detail": "Typed text" }, { "step": 2, "action": "wait", "status": "success", "detail": "Waited 1.00s" }, { "step": 2, "action": "moondream_point", "status": "success", "detail": "Point found", "output": "{\"x\":0.3828125,\"y\":0.1982421875,\"x_norm\":383,\"y_norm\":198,\"x_px\":459,\"y_px\":159,\"screen\":{\"width\":1200,\"height\":800}}" }, { "step": 2, "action": "click_at", "status": "success", "detail": "Clicked at coordinates" }, { "step": 2, "action": "wait", "status": "success", "detail": "Waited 1.00s" }, { "step": 2, "action": "moondream_query", "status": "success", "detail": "Answered question", "output": "The image does not display any price or currency information." }, { "step": 2, "action": "page_info", "status": "success", "detail": "Page info", "output": "{\"url\":\"https://www.ebay.com/sch/i.html?_nkw=Dreamcast&_sacat=0&_from=R40&_trksid=p4624852.m570.l1313\",\"title\":\"Dreamcast for sale | eBay\",\"state_changed\":true}" }, { "step": 3, "action": "moondream_point", "status": "success", "detail": "Point found", "output": "{\"x\":0.26953125,\"y\":0.2490234375,\"x_norm\":270,\"y_norm\":249,\"x_px\":323,\"y_px\":199,\"screen\":{\"width\":1200,\"height\":800}}" }, { "step": 4, "action": "click_at", "status": "success", "detail": "Clicked at coordinates" }, { "step": 4, "action": "wait", "status": "success", "detail": "Waited 1.00s" }, { "step": 4, "action": "moondream_query", "status": "success", "detail": "Answered question", "output": "The image does not show any price or currency information." }, { "step": 4, "action": "page_info", "status": "success", "detail": "Page info", "output": "{\"url\":\"https://www.ebay.com/sch/i.html?_nkw=Dreamcast&_sacat=0&_from=R40&_trksid=p2334524.m570.l1312&_odkw=Dreamcast&_osacat=0\",\"title\":\"Dreamcast for sale | eBay\",\"state_changed\":true}" }, { "step": 5, "action": "moondream_point", "status": "success", "detail": "Point found", "output": "{\"x\":0.71484375,\"y\":0.197265625,\"x_norm\":715,\"y_norm\":197,\"x_px\":858,\"y_px\":158,\"screen\":{\"width\":1200,\"height\":800}}" }, { "step": 5, "action": "click_at", "status": "success", "detail": "Clicked at coordinates" }, { "step": 5, "action": "wait", "status": "success", "detail": "Waited 1.00s" }, { "step": 5, "action": "moondream_query", "status": "success", "detail": "Answered question", "output": "No, this is a multi-item product page on Dreamcast consoles and related items." }, { "step": 5, "action": "moondream_query", "status": "success", "detail": "Answered question", "output": "$159.99 USD" }, { "step": 5, "action": "page_info", "status": "success", "detail": "Page info", "output": "{\"url\":\"https://www.ebay.com/sch/i.html?_nkw=Dreamcast&_sacat=0&_from=R40&_trksid=p2334524.m570.l1312&_odkw=Dreamcast&_osacat=0\",\"title\":\"Dreamcast for sale | eBay\",\"state_changed\":false}" }, { "step": 6, "action": "go_back", "status": "success", "detail": "Went back" }, { "step": 6, "action": "moondream_point", "status": "success", "detail": "Point found", "output": "{\"x\":0.6953125,\"y\":0.58203125,\"x_norm\":695,\"y_norm\":582,\"x_px\":834,\"y_px\":466,\"screen\":{\"width\":1200,\"height\":800}}" }, { "step": 7, "action": "click_at", "status": "success", "detail": "Clicked at coordinates" }, { "step": 7, "action": "wait", "status": "success", "detail": "Waited 1.00s" }, { "step": 7, "action": "moondream_query", "status": "success", "detail": "Answered question", "output": "No, this is a product page showing a video of the console." }, { "step": 7, "action": "moondream_query", "status": "success", "detail": "Answered question", "output": "US $159.99" }, { "step": 7, "action": "page_info", "status": "success", "detail": "Page info", "output": "{\"url\":\"https://www.ebay.com/itm/157653229097?_skw=Dreamcast&itmmeta=01KGG3FJ7H45PC8E84R92SSCNS&hash=item24b4dd5e29:g:DeEAAeSwwwtpf-mK&itmprp=enc%3AAQALAAAA8O7PUuNWmJ%2B%2BUShgI9tQz%2FoqO9upPS28NnONv2gVIoF1CYmpPAnniYiiA2wCDdcXzzG1H1Z8w%2FDyq2JE9dbo76w%2FpDrHxhnnGsstpns55AnQD2HkQZOH62ZWrm5M1tmCsc48u58%2FqZW%2B7Q%2BEQxEVQKmbJsrIbZBKYasTsjPnL594JiHI%2BQ28aDgnueDXpGB5HoA9RT7tCUbRnkwELToWmn354C%2B16qcjqimEnC%2Flkc7E8zT3M2EJ7P408vH37VYz6rNcd4LnjHnw9P4siZTlsHcwurwaDQ8whwyFFxL0mfCsUULgQdySV4sRj9TpkHd5lQ%3D%3D%7Ctkp%3ABk9SR_qjvoOEZw\",\"title\":\"Sega Dreamcast HKT-3020 White Console - Cleaned & Tested - Can Play Copies | eBay\",\"state_changed\":true}" } ], "answers": [ "The image does not display any price or currency information.", "The image does not show any price or currency information.", "No, this is a multi-item product page on Dreamcast consoles and related items.", "$159.99 USD", "No, this is a product page showing a video of the console.", "US $159.99" ] }Note
Medium Risk
Large addition of new template code and dependencies (Groq, Moondream API calls, Playwright CDP) that may introduce runtime/config issues, but it is isolated to new template directories and does not alter existing CLI logic.
Overview
Introduces a new
moondream-groq-computer-useapp template for both Python and TypeScript, enabling computer-use/browser automation where Groq orchestrates a JSON action loop and Moondream provides vision primitives (query,caption,point,detect) against Kernel screenshots.The templates include a deterministic step-runner plus an LLM-driven loop with JSON repair/validation, coordinate normalization, optional replay recording, and a lightweight
page_infoimplementation via Playwright CDP to fetch current tab title/URL;README.mdis updated to list the template and showkernel createusage, and new.env.examplefiles document requiredMOONDREAM_API_KEY/GROQ_API_KEY.Written by Cursor Bugbot for commit 5779762. This will update automatically on new commits. Configure here.