A Robot Opens a Word Document in My Nextcloud Twice a Day: End-to-End Tests with Playwright and Zabbix

All my checks can be green while nobody can open a document. So a real browser now does what a person does: log in to Nextcloud, open a Word file in Collabora, log out, and search my blog. Twice a day, in a rootless Podman container, with the results in Zabbix. Why end-to-end tests, which tools exist, how it is built, a video of a run and a public repository.

Comic: in a data center corridor with blue light a white robot in an “E2E Tests” shirt works at a desk, the screen shows passed test steps; a man in a white T-shirt gives it a thumbs up with a coffee mug

🌐 Also available in: Deutsch · Français · Español · Italiano · Português · Nederlands · Polski

In short: My monitoring checks whether services run. It does not check whether anyone can actually work with them. Now a real browser does that twice a day: it logs in to our two Nextclouds, opens a Word document in Collabora Online, waits until the document server has really rendered it, closes it and logs out. A third test searches my blog. The tests run with Playwright in a rootless Podman container, and the results go to Zabbix, which draws the timings and sends the alarm. Scripts and instructions are public in a GitHub repository.

This is what one run looks like, recorded from a real browser:

One run of the check (47 s, slowed down so it can be followed): type the address, log in as the test user, click the Word document, wait for Collabora, close, log out. The regular runs take about 12 seconds. Music: “Ambient – Ambient Music” by JonasBlakewood via Pixabay (Pixabay Content License).
Table of Contents

🎯 Why end-to-end tests say the most

Almost everything I monitor is a state. The container is running. Port 443 answers. status.php says installed: true. The certificate is valid for another 60 days. Collabora’s discovery URL returns XML. Each of these checks is cheap, fast and useful – and all of them can be green while nobody can open a single document.

Between “the service runs” and “a person can work” lies a chain: the login page has to render its JavaScript, the password has to be accepted, the Files app has to list the files, Nextcloud has to hand the file to Collabora through WOPI, Collabora has to call back, fetch it, convert it and send the rendered pages to the browser. A missing proxy header, an app update that breaks the viewer, a full disk in the wrong place, a bundle that 404s after an upgrade – each of these breaks the chain, and none of them turns a single status check red.

An end-to-end test checks the effect, not the state. It does what a person does and reports success only if the person would have succeeded. That makes it the most meaningful check there is. It is also the slowest and the most fragile one, so it belongs in addition to the simple checks, not instead of them, and it should run rarely and only along the paths that really matter.

What I hope to gain is mostly confidence after updates. Nextcloud, Collabora, K3s, the Helm charts and the operating system are updated regularly on both servers. Until now I clicked through by hand afterwards – or, more honestly, sometimes I did not. Now the next run tells me whether the whole chain still works, not only each of its parts.

The takeaway: State checks tell you whether the parts are alive. Only an end-to-end test tells you whether they work together.

🧰 The well-known tools – and why Playwright

Browser automation is an old discipline, and there are many tools for it:

ToolWhat it isFit for this job
Seleniumthe veteran (since 2004), W3C WebDriver standard, every languageworks, but waiting, frames and video need extra code or extra containers
PlaywrightMicrosoft, since 2020; Chromium, Firefox, WebKit; Python, JS, Java, .NETwaits on its own, handles nested iframes, records video – my choice
PuppeteerGoogle, Chrome/Chromium only, JavaScriptgood, but only JavaScript and one browser family
Cypresstest runner for web developers, runs inside the browserbuilt for testing your own app during development, awkward across domains and iframes
Robot Frameworkkeyword-driven tests; Checkmk uses it as “Robotmk” for synthetic monitoringreadable, but another layer on top of Selenium or Playwright
Grafana k6 browserload testing tool with a browser modulestrong for load, overkill for a login twice a day
Elastic SyntheticsPlaywright scripts inside the Elastic stackonly makes sense if you run Elastic anyway
Zabbix browser itemssince Zabbix 7.0: JavaScript tests against a Selenium WebDriverthe most integrated way, but a separate Selenium container and less comfortable with frames
PyAutoGUImoves the real mouse by screen coordinatesthe counter-example: breaks with every pixel of layout change

I chose Playwright with Python. Three properties decided it. It waits by itself: a click waits until the element exists, is visible and accepts input, so the script needs no sleep(). It handles nested iframes cleanly, which matters because Collabora runs in a frame inside Nextcloud. And it comes as an official container image with all browsers, so nothing has to be installed on the host. On top of that, my Zabbix setup already used the same image to take its screenshots.

The takeaway: For monitoring, a tool that waits reliably and copes with frames beats the most famous name.

🔍 What the tests do

The Nextcloud test runs six steps and times each of them:

StepPasses when
login pagethe login form has been rendered – so the page’s JavaScript works
loginthe test user’s password is accepted and the browser leaves /login
files listthe Files app shows the test document
document loadedCollabora’s status bar shows “11 words”
document closedthe Collabora frame is gone and the file list is back
logoutthe browser is back on the login page

The interesting step is the fourth. Collabora’s iframe appears seconds before the document is usable, and the document itself is painted onto a canvas, so its text never appears in the page. What does appear is the word count in the status bar, and only the document server can compute it – from the real file, after Nextcloud has handed it over and Collabora has opened it. If the status bar shows exactly the expected number, the whole chain has worked:

frame = page.frame_locator("iframe[name^=collaboraframe]").first
frame.locator("#StateWordCount").filter(has_text="11 words").wait_for(timeout=90000)

The test document is generated by a small script: a heading, one sentence and a bar chart, so Collabora also has to render an image. On the first run on a new cloud, the test uploads it through the browser itself – “New”, “Upload files”, exactly like a person.

The test user monitor-check exists only for this: no admin rights, no groups, 100 MB quota, nothing in it but the test document.

The blog test has three steps: open the home page, open the search, search for “Zabbix”, wait for hits that contain the word, open the first hit and check that title, text and title image are there. That is exactly the part a simple HTTP check cannot see: the search on this blog runs entirely in the visitor’s browser (Pagefind). If its index were missing after a build, every page would still answer with 200, and the search would silently find nothing.

The takeaway: Wait for something only the real system can produce – not for the frame, the spinner or the HTTP status.

🧑‍💻 What the test looks like in code

The six steps are not in a configuration file but in a small Python program that drives a real browser. Every step is one line of Playwright. Shortened, check_nextcloud.py looks like this:

page.goto(f"{URL}/login")                     # 1. open the login page
page.locator("#user").fill(USER)              # 2. type the user name
page.locator("#password").fill(PASSWORD)      #    type the password
page.locator("button[type=submit]").click()   #    click "Log in"
page.goto(f"{URL}/index.php/apps/files/")     # 3. open the Files app
row.click()                                   # 4. click the Word document …
frame.locator("#StateWordCount") \
     .filter(has_text="11 words").wait_for()  #    … wait until Collabora shows the word count
frame.locator("#closebutton").click()         # 5. close the document
page.get_by_text("Log out").click()            # 6. log out

Three commands cover almost everything: fill types something, click clicks, wait_for waits until something appears on the page. If it does not appear in time, the test has failed, and the name of the step goes to Zabbix as the error.

The configuration file targets.ini only defines which targets are checked, not how: one section per target with the kind of test (nextcloud or site), the host’s name in Zabbix and a separate file with address, user and password (mode 600):

# a Nextcloud: credentials in a separate file (mode 600)
[cloud]
kind = nextcloud
credentials = .credentials-cloud
zabbix_host = Nextcloud
# file .credentials-cloud: NC_URL=…, NC_USER=…, NC_PASS=…

To check another Nextcloud you only add a section and a credentials file. To check something else – a webmail, a wiki, a shop – you write a short script of your own following the same pattern.

🐳 How it runs

Everything lives on my home server HP3, next to Zabbix, as an ordinary user without root:

  • A cron job starts run.py at 06:57 and 18:57. Twice a day is enough for this kind of check, and each run opens a real Collabora session on the servers, which I don’t want every five minutes.
  • For every target, run.py starts the Playwright container with podman run. The credentials come from a file with mode 600 via --env-file, never on a command line.
  • If a run fails, it waits two minutes and tries once more. Only the second result counts, so a single hiccup does not wake anyone.
  • On failure, a screenshot and the page text are saved, so I can see afterwards where it got stuck.

A normal run takes 11 to 14 seconds per Nextcloud and 2 seconds for the blog. Collabora shows the document after about 3 to 4 seconds.

The takeaway: A rare, reliable check is worth more than a frequent, flaky one.

📊 Into Zabbix

At first the script mailed me itself. That worked, but it was a second alarm system next to Zabbix, with its own logic and no history. So the results now go to Zabbix, and these are the changes there:

Five trapper items per host. A trapper item does not ask anything; it accepts values someone sends. Each of the three hosts (Nextcloud cvjm, Nextcloud sofie, blog) got:

ItemContent
e2e.ok1 passed, 0 failed
e2e.secondsduration of the whole run
e2e.loadthe step that matters: document rendered, or search answered
e2e.stepthe failed step
e2e.errorthe error message

Sent without extra software. The Zabbix sender protocol is simple – a fixed header, the length, then JSON – so run.py speaks it directly with Python’s standard library, and the zabbix_sender package is not needed. The items only accept values from localhost.

Three triggers per host:

  • Test failed (High). It depends on the trigger “internet unreachable” and on the host’s existing High triggers. If the server is simply down, the existing alert already speaks, and the E2E trigger stays quiet. It only fires when every simple check is green but the way through the application is broken – exactly the case it was built for.
  • No result for 13 hours (Average). A check that silently stops is worse than none. If cron, the image or the credentials break, Zabbix notices the missing values.
  • Slow (Warning) if the key step takes longer than 30 seconds. Creeping degradation shows up long before an outage, and the graph of e2e.load makes it visible.

Tested like an outage. I sent an artificial failure (“TEST”) to Zabbix: the problem appeared, the mail arrived, and a manual “ok” afterwards closed it again. Only if Zabbix itself cannot be reached does run.py still send a mail of its own.

The takeaway: One place for all alarms. The dependency on the existing triggers keeps the new check from shouting about outages that are already reported.

🪤 Where it got stuck

  • HTTP 401 on the upload. On both clouds token_auth_enforced is switched on: clients such as WebDAV need app passwords, the normal password is rejected. That is a deliberate hardening, so instead of weakening it, the test uploads the document through the browser.
  • The login form never appeared. In the default rootless network, Chromium saw a network change right after the container started (ERR_NETWORK_CHANGED) and dropped the page’s JavaScript. With --network=host it is gone.
  • “Browser not supported.” Nextcloud sends unknown browsers to /unsupported. My own user agent was too unusual; now it is the normal Chrome string with “e2e-check” appended, so the runs remain recognizable in the logs.
  • Container start hung for minutes. --userns=keep-id made Podman copy the 2.6 GB image with rewritten owners. In rootless Podman it is not needed at all: root in the container already is my user.
  • /apps/files/ returned 403 behind the nginx of one setup; /index.php/apps/files/ works everywhere.
  • The welcome tour. On the first opening Collabora shows a tour over the document; Escape closes it.

The takeaway: Most problems were not in the test but in the environment around it – network, user agent, security settings. Every one of them is now a comment in the code.

🎬 How the video was made

A headless browser has no address bar and no mouse pointer, so the regular checks cannot be filmed in a way that looks like a person. For the video, a separate demo script runs in a container with a virtual screen (Xvfb): a visible Chrome window opens, the address is typed into the address bar, and the real mouse pointer moves to each element and clicks (xdotool). ffmpeg records the screen and adds a caption for each step. Two details mattered: Chrome’s “save password?” bubble covered the user menu and had to be switched off in the profile, and --test-type removes the warning bar about --no-sandbox. The music is “Ambient – Ambient Music” by JonasBlakewood from Pixabay.

📦 The repository

Everything is public as e2e-browser-check, with placeholders instead of my servers: the two checks, run.py with the retry and the Zabbix sender, zbx_setup.py for items and triggers, the test document generator, cron and systemd timer, the demo recorder and a README that explains setup, Zabbix side and pitfalls step by step. The Zabbix API helper comes from my earlier repository zabbix-podman-homelab.

🎯 The lessons

  1. A green status is not a working service. Only a test that does what a user does shows whether the chain holds.
  2. Wait for proof, not for the frame. The word count in Collabora’s status bar can only come from the real document.
  3. Rarely and reliably. Twice a day with one retry is enough for an end-to-end test; the simple checks cover the minutes in between.
  4. One alarm system. Results into Zabbix, with dependencies, so a known outage does not ring twice – and with a trigger for “the check itself has stopped”.
  5. Respect your own hardening. When the test ran into token_auth_enforced, the right fix was to change the test, not the server.
Some text and images on this site are AI-assisted or AI-generated. Details