Browsing the Web with LLMs - Browser-use + Craw4AI Example

This is an example how to use browser-use in conjunction with crawl4ai to accomplish a task on the web using an LLM. It is mostly work by my colleague Jonathan Decker: https://gitlab-ce.gwdg.de/-/snippets/632.

To run it create a virtual environment in the directory where you put these files and activate it.

python -m venv .venv
source .venv/bin/activate

Install the required dependencies in the venv:

pip install browser-use crawl4ai python-dotenv

browser_use uses playwright for automating browsers. Install the browsers (in ~/.cache/ms-playwright in my case) required by playwright:

playwright install

Put your Chat-AI API Key (see here how to get one: https://docs.hpc.gwdg.de/services/saia/index.html) into the .env file. Now you can run the example:

python main.py

The example has roughly three stages:

1. Discover the URL of the GWDG events page using browser_use
2. Extract the page content in Markdown format using crawl4ai
3. Summarize the courses using the OpenAI client.

The example is a bit unreliable. The agent sometimes yields https://academy.gwdg.de/index.xhtml instead of the desired https://academy.gwdg.de/p/events.xhtml as the events page. This happens randomly due to the default temperature of the model.