agentic_mouse_action

Perform mouse actions at recorded coordinates with Operator agent fallback

The agentic_mouse_action method performs mouse actions at specific screen coordinates captured during a recording. If the action fails at those coordinates (e.g., due to page layout changes), it falls back to the Operator agent using a natural language query.

This method is primarily used by agents generated through Imitation Learning, where click coordinates are captured during recording. For most manual automations, prefer agentic_selector() which targets elements by CSS selectors.

Method Signature

async def agentic_mouse_action(
    self,
    *,
    action: AgenticMouseAction,
    recorded_click: RecordedClick,
    fallback_operator_query: str,
    resize_window: bool = True,
    timeout: int | None = 60,
) -> None

Parameters

action · AgenticMouseAction · required

The mouse action to perform. Supported actions:

{"type": "click"}                                          # Left click
{"type": "right_click"}                                    # Right click
{"type": "double_click"}                                   # Double click
{"type": "fill", "text": "hello", "press_enter": False}    # Type text at the location
{"type": "scroll", "horizontal": 0, "vertical": 300}      # Scroll by pixel offset

recorded_click · RecordedClick · required

The coordinates and viewport dimensions from when the action was originally recorded.

{
    "x": 450,          # X coordinate in pixels
    "y": 320,          # Y coordinate in pixels
    "viewport": {
        "width": 1920,  # Viewport width when recorded
        "height": 1080  # Viewport height when recorded
    }
}

fallback_operator_query · str · required

Natural language instruction for the Operator agent to use if the coordinate-based action fails.

fallback_operator_query="click the Submit button at the bottom of the form"

resize_window · bool · default: "True"

Whether to resize the browser window to match the recorded viewport dimensions before performing the action. This improves accuracy when the page layout is responsive.

timeout · int | None · default: "60"

Maximum time in seconds to wait for the operation to complete.

Return Value

This method returns None and completes when the action has been performed successfully.

Example

import asyncio

from narada import Agent, BrowserEnvironment

async def main():
    env = BrowserEnvironment()
    agent = Agent(environment=env)

    try:
        await agent.go_to_url(url="https://www.google.com")

        # Click at recorded coordinates, with Operator fallback
        await agent.agentic_mouse_action(
            action={"type": "click"},
            recorded_click={
                "x": 600,
                "y": 450,
                "viewport": {"width": 1440, "height": 900}
            },
            fallback_operator_query='click the "Google Search" button'
        )

        # Type into a field at recorded coordinates
        await agent.agentic_mouse_action(
            action={"type": "fill", "text": "Narada AI", "press_enter": True},
            recorded_click={
                "x": 600,
                "y": 350,
                "viewport": {"width": 1440, "height": 900}
            },
            fallback_operator_query='type "Narada AI" in the search box and press Enter'
        )
    finally:
        await env.close()

if __name__ == "__main__":
    asyncio.run(main())

When to Use This vs agentic_selector

	agentic_mouse_action	agentic_selector
Targets by	Screen coordinates	CSS selectors / element attributes
Best for	Replaying recorded actions	Programmatic element targeting
Resilience	Depends on page layout stability	Depends on element attribute stability
Typical use	Generated by Imitation Learning	Written manually or in custom agents

Both methods fall back to the Operator agent when their primary targeting strategy fails, ensuring robust automation even when pages change.