Getting began with laptop use in Amazon Bedrock Brokers

Computer use is a breakthrough functionality from Anthropic that permits basis fashions (FMs) to visually understand and interpret digital interfaces. This functionality permits Anthropic’s Claude fashions to determine what’s on a display, perceive the context of UI components, and acknowledge actions that needs to be carried out comparable to clicking buttons, typing textual content, scrolling, and navigating between functions. Nonetheless, the mannequin itself doesn’t execute these actions—it requires an orchestration layer to securely implement the supported actions.

At the moment, we’re asserting computer use assist inside Amazon Bedrock Agents utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude Sonnet 3.7 fashions on Amazon Bedrock. This integration brings Anthropic’s visible notion capabilities as a managed instrument inside Amazon Bedrock Brokers, offering you with a safe, traceable, and managed option to implement laptop use automation in your workflows.

Organizations throughout industries battle with automating repetitive duties that span a number of functions and methods of file. Whether or not processing invoices, updating buyer information, or managing human useful resource (HR) paperwork, these workflows typically require workers to manually switch info between completely different methods – a course of that’s time-consuming, error-prone, and troublesome to scale.

Conventional automation approaches require customized API integrations for every software, creating important improvement overhead. Pc use capabilities change this paradigm by permitting machines to understand present interfaces simply as people.

On this submit, we create a pc use agent demo that gives the essential orchestration layer that transforms laptop use from a notion functionality into actionable automation. With out this orchestration layer, laptop use would solely determine potential actions with out executing them. The pc use agent demo powered by Amazon Bedrock Brokers supplies the next advantages:

• Safe execution surroundings – Execution of laptop use instruments in a sandbox surroundings with restricted entry to the AWS ecosystem and the online. It’s essential to notice that at the moment Amazon Bedrock Agent doesn’t present a sandbox surroundings
• Complete logging – Potential to trace every motion and interplay for auditing and debugging
• Detailed tracing capabilities – Visibility into every step of the automated workflow
• Simplified testing and experimentation – Decreased threat when working with this experimental functionality by managed controls
• Seamless orchestration – Coordination of advanced workflows throughout a number of methods with out customized code

This integration combines Anthropic’s perceptual understanding of digital interfaces with the orchestration capabilities of Amazon Bedrock Brokers, creating a strong agent for automating advanced workflows throughout functions. Slightly than construct customized integrations for every system, builders can now create brokers that understand and work together with present interfaces in a managed, safe method.

With laptop use, Amazon Bedrock Brokers can automate duties by fundamental GUI actions and built-in Linux instructions. For instance, your agent may take screenshots, create and edit textual content recordsdata, and run built-in Linux instructions. Utilizing Amazon Bedrock Brokers and suitable Anthropic’s Claude fashions, you need to use the next motion teams:

• Pc instrument – Permits interactions with person interfaces (clicking, typing, scrolling)
• Textual content editor instrument – Offers capabilities to edit and manipulate recordsdata
• Bash – Permits execution of built-in Linux instructions

Resolution overview

An instance laptop use workflow consists of the next steps:

1. Create an Amazon Bedrock agent and use pure language to explain what the agent ought to do and the way it ought to work together with customers, for instance: “You might be laptop use agent able to utilizing Firefox internet browser for internet search.”
2. Add the Amazon Bedrock Brokers supported laptop use motion teams to your agent utilizing CreateAgentActionGroup API.
3. Invoke the agent with a person question that requires laptop use instruments, for instance, “What’s Amazon Bedrock, are you able to search the online?”
4. The Amazon Bedrock agent makes use of the instrument definitions at its disposal and decides to make use of the pc motion group to click on a screenshot of the surroundings. Utilizing the return control functionality of Amazon Bedrock Brokers, the agent the responds with the instrument or instruments that it desires to execute. The return management functionality is required for utilizing laptop use with Amazon Bedrock Brokers.
5. The workflow parses the agent response and executes the instrument returned in a sandbox surroundings. The output is given again to the Amazon Bedrock agent for additional processing.
6. The Amazon Bedrock agent continues to reply with instruments at its disposal till the duty is full.

You possibly can recreate this instance within the us-west-2 AWS Area with the AWS Cloud Development Kit (AWS CDK) by following the directions within the GitHub repository. This demo deploys a containerized software utilizing AWS Fargate throughout two Availability Zones within the us-west-2 Area. The infrastructure operates inside a digital personal cloud (VPC) containing public subnets in every Availability Zone, with an web gateway offering exterior connectivity. The structure is complemented by important supporting providers, together with AWS Key Management Service (AWS KMS) for safety and Amazon CloudWatch for monitoring, making a resilient, serverless container surroundings that alleviates the necessity to handle underlying infrastructure whereas sustaining strong safety and excessive availability.

The next diagram illustrates the answer structure.

On the core of our resolution are two Fargate containers managed by Amazon Elastic Container Service (Amazon ECS), every protected by its personal safety group. The primary is our orchestration container, which not solely handles the communication between Amazon Bedrock Brokers and finish customers, but in addition orchestrates the workflow that allows instrument execution. The second is our surroundings container, which serves as a safe sandbox the place the Amazon Bedrock agent can safely run its laptop use instruments. The surroundings container has restricted entry to the remainder of the ecosystem and the web. We make the most of service discovery to attach Amazon ECS providers with DNS names.

The orchestration container consists of the next elements:

• Streamlit UI – The Streamlit UI that facilitates interplay between the tip person and laptop use agent
• Return management loop – The workflow liable for parsing the instruments that the agent desires to execute and returning the output of those instruments

The surroundings container consists of the next elements:

• UI and pre-installed functions – A light-weight UI and pre-installed Linux functions like Firefox that can be utilized to finish the person’s duties
• Software implementation – Code that may execute laptop use instrument within the surroundings like “screenshot” or “double-click”
• Quart (RESTful) JSON API – An orchestration container that makes use of Quart to execute instruments in a sandbox surroundings

The next diagram illustrates these elements.

Conditions

1. AWS Command Line Interface (CLI), observe directions here. Be sure to setup credentials, observe directions here.
2. Require Python 3.11 or later.
3. Require Node.js 14.15.0 or later.
4. AWS CDK CLI, observe directions here.
5. Allow model access for Anthropic’s Claude Sonnet 3.5 V2 and for Anthropic’s Claude Sonnet 3.7.
6. Boto3 model >= 1.37.10.

Create an Amazon Bedrock agent with laptop use

You should use the next code pattern to create a easy Amazon Bedrock agent with laptop, bash, and textual content editor motion teams. It’s essential to offer a suitable motion group signature when utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude 3.7 Sonnet as highlighted here.

import boto3
import time

# Step 1: Create the bedrock agent consumer

bedrock_agent = boto3.consumer("bedrock-agent", region_name="us-west-2")

# Step 2: Create an agent

create_agent_response = create_agent_response = bedrock_agent.create_agent(
        agentResourceRoleArn=agent_role_arn, # Amazon Bedrock Agent execution position
        agentName="computeruse",
        description="""Instance agent for laptop use. 
				This agent ought to solely function on 
				Sandbox environments with restricted privileges.""",
        foundationModel="us.anthropic.claude-3-7-sonnet-20250219-v1:0",      
		instruction="""You might be laptop use agent able to utilizing Firefox 
                 internet browser for internet search.""",
)

time.sleep(30) # watch for agent to be created

# Step 3.1: Create and fix laptop motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="ComputerActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Pc",
    parentActionGroupSignatureParams={
        "kind": "computer_20250124",
        "display_height_px": "768",
        "display_width_px": "1024",
        "display_number": "1",
    },
)

# Step 3.2: Create and fix bash motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="BashActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Bash",
    parentActionGroupSignatureParams={
        "kind": "bash_20250124",
    },
)

# Step 3.3: Create and fix textual content editor motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="TextEditorActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.TextEditor",
    parentActionGroupSignatureParams={
        "kind": "text_editor_20250124",
    },
)

# Step 3.4 Create Climate Motion Group

bedrock_agent.create_agent_action_group(
        actionGroupName="WeatherActionGroup",
        agentId=create_agent_response["agent"]["agentId"],
        agentVersion="DRAFT",
        actionGroupExecutor = {
            'customControl': 'RETURN_CONTROL',
        },
        functionSchema = {
            'capabilities': [
                {
                    "name": "get_current_weather",
                    "description": "Get the current weather in a given location.",
                    "parameters": {
                        "location": {
                            "type": "string",
                            "description": "The city, e.g., San Francisco",
                            "required": True,
                        },
                        "unit": {
                            "type": "string",
                            "description": 'The unit to use, e.g., 
									fahrenheit or celsius. Defaults to "fahrenheit"',
                            "required": False,
                        },
                    },
                    "requireConfirmation": "DISABLED",
                }
            ]
        },
)
time.sleep(10)
# Step 4: Put together agent

bedrock_agent.prepare_agent(agentId=create_agent_response["agent"]["agentId"])

Instance use case

On this submit, we reveal an instance the place we use Amazon Bedrock Brokers with the pc use functionality to finish an internet type. Within the instance, the pc use agent may also swap Firefox tabs to work together with a customer relationship management (CRM) agent to get the required info to finish the shape. Though this instance makes use of a pattern CRM software because the system of file, the identical strategy works with Salesforce, SAP, Workday, or different methods of file with the suitable authentication frameworks in place.

Within the demonstrated use case, you may observe how nicely the Amazon Bedrock agent carried out with laptop use instruments. Our implementation accomplished the shopper ID, buyer title, and e mail by visually analyzing the excel knowledge. Nonetheless, for the overview, it determined to pick out the cell and duplicate the info, as a result of the knowledge wasn’t fully seen on the display. Lastly, the CRM agent was used to get extra info on the shopper.

Finest practices

The next are some methods you may enhance the efficiency to your use case:

Issues

The pc use function is made accessible to you as a beta service as outlined within the AWS Service Phrases. It’s topic to your settlement with AWS and the AWS Service Phrases, and the relevant mannequin EULA. Pc use poses distinctive dangers which can be distinct from commonplace API options or chat interfaces. These dangers are heightened when utilizing the pc use function to work together with the web. To attenuate dangers, take into account taking precautions comparable to:

• Function laptop use performance in a devoted digital machine or container with minimal privileges to attenuate direct system exploits or accidents
• To assist forestall info theft, keep away from giving the pc use API entry to delicate accounts or knowledge
• Restrict the pc use API’s web entry to required domains to scale back publicity to malicious content material
• To implement correct oversight, preserve a human within the loop for delicate duties (comparable to making selections that would have significant real-world penalties) and for something requiring affirmative consent (comparable to accepting cookies, executing monetary transactions, or agreeing to phrases of service)

Any content material that you simply allow Anthropic’s Claude to see or entry can doubtlessly override directions or trigger the mannequin to make errors or carry out unintended actions. Taking correct precautions, comparable to isolating Anthropic’s Claude from delicate surfaces, is crucial – together with to keep away from dangers associated to immediate injection. Earlier than enabling or requesting permissions essential to allow laptop use options in your personal merchandise, inform finish customers of any related dangers, and procure their consent as applicable.

Clear up

When you find yourself finished utilizing this resolution, make certain to wash up all of the sources. Observe the directions within the supplied GitHub repository.

Conclusion

Organizations throughout industries face important challenges with cross-application workflows that historically require handbook knowledge entry or advanced customized integrations. The combination of Anthropic’s laptop use functionality with Amazon Bedrock Brokers represents a transformative strategy to those challenges.

Through the use of Amazon Bedrock Brokers because the orchestration layer, organizations can alleviate the necessity for customized API improvement for every software, profit from complete logging and tracing capabilities important for enterprise deployment, and implement automation options shortly.

As you start exploring laptop use with Amazon Bedrock Brokers, take into account workflows in your group that would profit from this strategy. From bill processing to buyer onboarding, HR documentation to compliance reporting, the potential functions are huge and transformative.

We’re excited to see how you’ll use Amazon Bedrock Brokers with the pc use functionality to securely streamline operations and reimagine enterprise processes by AI-driven automation.

Assets

To be taught extra, consult with the next sources:

Concerning the Authors

Eashan Kaushik is a Specialist Options Architect AI/ML at Amazon Internet Providers. He’s pushed by creating cutting-edge generative AI options whereas prioritizing a customer-centric strategy to his work. Earlier than this position, he obtained an MS in Pc Science from NYU Tandon Faculty of Engineering. Outdoors of labor, he enjoys sports activities, lifting, and operating marathons.

Maira Ladeira Tanke is a tech Lead for Agentic workloads in Amazon Bedrock at AWS, the place she permits clients on their journey to develop autonomous AI methods. With over 10 years of expertise in AI/ML. At AWS, Maira companions with enterprise clients to speed up the adoption of agentic functions utilizing Amazon Bedrock, serving to organizations harness the ability of basis fashions to drive innovation and enterprise transformation. In her free time, Maira enjoys touring, taking part in along with her cat, and spending time along with her household someplace heat.

Raj Pathak is a Principal Options Architect and Technical advisor to Fortune 50 and Mid-Sized FSI (Banking, Insurance coverage, Capital Markets) clients throughout Canada and the USA. Raj makes a speciality of Machine Studying with functions in Generative AI, Pure Language Processing, Clever Doc Processing, and MLOps.

Adarsh Srikanth is a Software program Improvement Engineer at Amazon Bedrock, the place he develops AI agent providers. He holds a grasp’s diploma in laptop science from USC and brings three years of business expertise to his position. He spends his free time exploring nationwide parks, discovering new mountaineering trails, and taking part in varied racquet sports activities.

Abishek Kumar is a Senior Software program Engineer at Amazon, bringing over 6 years of helpful expertise throughout each retail and AWS organizations. He has demonstrated experience in growing generative AI and machine studying options, particularly contributing to key AWS providers together with SageMaker Autopilot, SageMaker Canvas, and AWS Bedrock Brokers. All through his profession, Abishek has proven ardour for fixing advanced issues and architecting large-scale methods that serve tens of millions of shoppers worldwide. When not immersed in know-how, he enjoys exploring nature by mountaineering and touring adventures along with his spouse.

Krishna Gourishetti is a Senior Software program Engineer for the Bedrock Brokers workforce in AWS. He’s keen about constructing scalable software program options that clear up buyer issues. In his free time, Krishna likes to go on hikes.