{ "version": "https://jsonfeed.org/version/1", "title": "George Mandis", "description": "This is my blog where I write about web development and travel.", "home_page_url": "https://george.mand.is", "feed_url": "https://george.mand.is/feed.plain.json", "favicon": "https://s3-us-west-2.amazonaws.com/george.mand.is/me-sketch-avatar.jpg", "author": { "name": "George Mandis" }, "items": [ { "id": "https://george.mand.is/2026/05/tezcatl-a-2mb-alternative-to-puppeteer-for-scraping-on-macos/", "url": "https://george.mand.is/2026/05/tezcatl-a-2mb-alternative-to-puppeteer-for-scraping-on-macos.txt", "title": "tezcatl: a 2MB alternative to Puppeteer for scraping on macOS", "content_html": "I've been working on a small project that uses[whereami](https://less.software/whereami) and[nearme](https://less.software/nearme) to do some local scraping andexperimentation with LLMs to build enriched datasets. The details of thatproject are for a future post, but the short version is I needed to fetch webpages, extract some data and move on.> [!tip] A quick cut to the chase:>> If you're on a Mac and find yourself needing to do light scraping work on> websites as they are rendered—not as they are sent over the wire—please> consider trying `tezcatl`:>> - [https://github.com/georgemandis/tezcatl](https://github.com/georgemandis/tezcatl)>> Some selling points:>> - A little like Puppeteer but only ~2MB instead of nearly 300MB.> - Built around WebKit, which is already on your Mac, and only cares about> returning an accurate snapshot of the DOM.> - About as lightweight as a tool like this can be; has no dependencies or> build steps you'll ever have to think about.> - Can become a tool in your scraping + parsing toolkit right alongside `jq`,> `curl` and other CLI affordances you reach for daily.>> For those that want more context and story, read on!## The problem`curl` handles about 70% of what I need. Most pages serve a usable version oftheir content in the initial HTML response and you can pipe `curl` into whateveryou want—namely Simon Willison's [llm](https://github.com/simonw/llm) tool inthis case, which I adore.But some sites render everything client-side with JavaScript and come back assoulless husks. A `<div id=\"root\"></div>` and a pile of `<script>` tags—anillegible affront to a medium inherently centered around reading and the writtenword. Oh, the horror. The horror.The standard answer to this reality is [Puppeteer](https://pptr.dev) or[Playwright](https://playwright.dev/). Spin up a headless Chromium, wait for thepage to render, grab the DOM. It works, but it's a lot of overhead for what Iactually needed, which was just to load a URL, wait until the page loaded andgrab the HTML.I didn't need cross-platform, cross-browser QA automations, screenshots or HARdumps. I just needed the rendered DOM so I could strip the tags and ask anLLM to do some things with the actual words.Every Mac ships with WebKit. It's the same engine Safari uses. Apple exposes itthrough `WKWebView`, which is how native apps embed web content. You can use itfrom a CLI tool.So I made [tezcatl](https://github.com/georgemandis/tezcatl). It's not forautomating browsers or running tests. It's just for scraping web pages thatdon't render without JavaScript.```bash$ tezcatl https://example.com<html lang=\"en\"><head><title>Example Domain</title>...$ tezcatl https://spa-site.com --wait=2000# waits 2s after load for JS to render```It creates an offscreen `WKWebView`, loads the URL, waits for the navigationdelegate to fire, optionally pauses for additional JS settling time, thenevaluates JavaScript against the page (if specified) and writes the result tostdout. By default it returns the full rendered DOM. With `--eval` you can runarbitrary JS instead:```bash$ tezcatl https://example.com --eval=\"document.title\"Example Domain$ tezcatl https://example.com --eval=\"document.querySelectorAll('a').length\"1```## A real exampleApple's own[developer documentation](https://developer.apple.com/documentation/) (at thetime of this writing) renders in the client and seems to be a Vue app. `curl`gives you this:```This page requires JavaScript.```With `tezcatl` you can render the page and pull structured data out of it:```bash$ tezcatl https://developer.apple.com/documentation/ --wait=3000 \\ --eval=\"JSON.stringify([...document.querySelectorAll('a.card')].slice(0,5).map(c => ({ title: c.querySelector('.title')?.textContent?.trim(), description: c.querySelector('.card-content .content')?.textContent?.trim(), url: c.href })), null, 2)\"```You'll get something like this:```json[ { \"title\": \"Explore the new design principles\", \"description\": \"Learn how to design and develop beautiful interfaces that leverage Liquid Glass.\", \"url\": \"https://developer.apple.com/documentation/TechnologyOverviews/liquid-glass\" }, { \"title\": \"Adopting Liquid Glass\", \"description\": \"Find out how to bring the new material to your app.\", \"url\": \"https://developer.apple.com/documentation/TechnologyOverviews/adopting-liquid-glass\" }]```It pipes with other tools, which was another reason I rolled my own solution. Iwanted something that fit into the CLI workflows I was already buildingwith [whereami](https://less.software/whereami),[nearme](https://less.software/nearme), [lingua](https://less.software/lingua)and [loupe](https://less.software/loupe).```bash# Get the rendered DOM and extract text with linguatezcatl https://example.com | lingua detect# Scrape a title for use in a scriptTITLE=$(tezcatl https://example.com --eval=\"document.title\")# Find the nearest pizza place, grab its website, render itwhereami --json | nearme \"pizza\" --json | jq -r '.[0].url' | xargs tezcatl```## How it's builtLike the rest of[my Zig tools](https://github.com/georgemandis?tab=repositories&q=&type=&language=zig&sort=),`tezcatl` talks to the Objective-C runtime directly. No Swift or Objective-Csource files. Zig calls `objc_msgSend` and friends to create a `WKWebView`,register a navigation delegate class at runtime with `objc_allocateClassPair`,and wire up the completion handler using the ObjC block ABI.WebKit's `evaluateJavaScript:completionHandler:` expects the completion handlerto be an Objective-C block—not a function pointer. Blocks have a[documented C ABI](https://clang.llvm.org/docs/Block-ABI-Apple.html), so you canbuild a struct that pretends to be one and WebKit will call your function. InZig:```zigconst JSBlockLiteral = extern struct { isa: *anyopaque, flags: c_int, reserved: c_int, invoke: *const fn (*JSBlockLiteral, ?objc.id, ?objc.id) callconv(.c) void, descriptor: *const BlockDescriptor,};````isa` tells the ObjC runtime \"I am a block\" (you point it at`_NSConcreteStackBlock`). `flags` and `reserved` are bookkeeping the runtimeexpects; you set them to 0. `invoke` is the actual function pointer — whenWebKit finishes evaluating JS, it calls this with the result and error.`descriptor` points to a tiny struct that just says how big the block is. Thelayout has to match what the ObjC compiler would emit for a`^void(id result, id error)` block, but as long as the fields are in the rightorder with the right sizes, the runtime doesn't care what language built it.Zig is honestly kind of a weird choice for a tool that's designed to be sodeeply macOS native. The reason it's written this way is that I've been buildinga bunch of cross-platform CLI tools in Zig and I have an `objc.zig` module thatI copy between projects — it handles `objc_msgSend`, class creation, blockconstruction, all the runtime bridging. For tools like[whereami](https://less.software/whereami) and[loupe](https://less.software/loupe) that also build as C-compatible sharedlibraries, having the bridge in pure Zig means the library is self-containedwith no ObjC compilation step. For `tezcatl` that doesn't matter much, but I'min the habit and the pattern works.WebKit is a GUI framework underneath and assumes it's running inside anapplication with an event loop. In a normal macOS app, `NSApplicationMain` spinsthat up and everything works. In a CLI there's no app, no window, no event loop.WebKit will accept your `loadRequest:` call and then do nothing, because nobodyis pumping the events that drive the network and rendering work.The fix is straightforward: pump it yourself. After telling the `WKWebView` toload a URL, I call `CFRunLoopRunInMode(kCFRunLoopDefaultMode, timeout, false)`,which hands control to the system run loop until either my navigation delegatecallback fires (and calls `CFRunLoopStop`) or the timeout expires. Same thingafter `evaluateJavaScript:completionHandler:`; pump the loop, wait for the blockcallback. The Dock icon is suppressed with`NSApplicationActivationPolicyAccessory` so the whole thing stays invisible.## When it's the wrong toolIf you need to crawl thousands of pages or run on Linux, use Puppeteer orPlaywright. They exist for good reason. _This is not a replacement!_`tezcatl` is for when you're on a Mac, you need a handful of pages to rendertheir JS and you want to stay in the terminal. The kind of thing where spinningup a headless Chromium feels like driving a semi truck to the corner store.## On namingI wanted a name that suggested \"seeing the true form of something,\" since that'swhat the tool does; strips away the loading spinners and empty `<div>`s and showsyou the page left standing after all the little JavaScript cycles have finished.It also wouldn't hurt if it was a little bit metal.`tezcatl` is the [Nahuatl](https://en.wikipedia.org/wiki/Nahuatl) word formirror—specifically an obsidian mirror. It's hard to look it up without runninginto [Tezcatlipoca](https://en.wikipedia.org/wiki/Tezcatlipoca), the Aztec deityassociated with an obsidian mirror that could see through illusions. The Gettyhas a[great writeup](https://www.getty.edu/research/exhibitions_events/exhibitions/obsidian_mirror/through_the_mirror.html)on the mirror and its history, and, man,[if ever there were to be an icon](https://en.wikipedia.org/wiki/Tezcatlipoca#/media/File:TurquoiseAztecMask2.jpg)for a CLI tool... Apologies for mild appropriation, but the word is great andchecked all the boxes for me.It's on [GitHub](https://github.com/georgemandis/tezcatl) and in my Homebrewtap:```bashbrew install georgemandis/tap/tezcatl```", "date_published": "2026-05-29T00:00:00.000Z" }, { "id": "https://george.mand.is/2026/05/tracking-homebrew-downloads-with-githubs-api/", "url": "https://george.mand.is/2026/05/tracking-homebrew-downloads-with-githubs-api.txt", "title": "Tracking Homebrew downloads with GitHub's API", "content_html": "If you distribute software through [Homebrew](https://brew.sh/) or[Scoop](https://scoop.sh/), you've probably wondered if anyone is actuallyinstaslling your tools. Neither package manager gives you install analytics outof the box. But if your release pipeline flows through GitHub and producesplatform-specific artifacts—tarballs, zips, binaries per operating-system andarchitecture combo—their API quietly tracks download counts on each releaseasset. This turns out to be a decent-ish heuristic depending on how you've setupyour taps and buckets.![Screenshot of the release page for Less Software which uses this technique](https://georgemandis.s3-us-west-1.amazonaws.com/media/less-software-release-page-preview.png)When you set up a Homebrew tap or Scoop bucket, the formula/manifest typicallypoints to a specific release asset. These could be hosted anywhere, technically.If you're hosting them yoursef on an S3 or R2 bucket for example you canprobably build better tools for tracking installs from package manangers—maybeeven get [[who-reads-my-rss-feed|interesting user-agent data]] to tell you more about how people are consuming them. On GitHub they track requests to your release assets for you! They don't publishthem anywhere on the dedicated Releases page for your repo, but when someoneruns `brew install georgemandis/tap/copycat` (for example), Homebrew downloadsthe tarball from the GitHub release for that project and GitHub increments thedownload count on that asset. This also happens if you click on an asset on theReleases page for your repo and download it from the browser.The GitHub API exposes this per-asset:```GET /repos/{owner}/{repo}/releases```Give it a look here for my CLI clipboard introspection tool[copycat](https://github.com/georgemandis/copycat):- [https://api.github.com/repos/georgemandis/copycat/releases](https://api.github.com/repos/georgemandis/copycat/releases)Each release object includes an `assets` array, and each asset has a`download_count` field. Sum those up across releases and you have a rough total.![A screenshow showing the raw output of the API URL which you can pull up in the browser. 6 downloads. Voila](https://georgemandis.s3-us-west-1.amazonaws.com/media/github-releases-download-count.png)We can't distinguish between someone installing via Homebrew, downloadingdirectly from the releases page, or `curl`-ing the URL from a script, but if themajority of your distribution is through package managers—and your releases haveper-platform artifacts that only make sense to download through a tap orbucket—it's probably close enough to be useful.I'm using this right now over at [less.software](https://less.software), where Imaintain brief pages for the tools I've been building. I wanted a[releases page](https://less.software/releases) that automatically pullsdownload stats, clone traffic, and version history for every project in myHomebrew tap and Scoop bucket.The setup is a bash script called `releases.sh` that runs during the site'sGitHub Actions build. It lists the contents of my[`homebrew-tap`](https://github.com/georgemandis/homebrew-tap) and[`scoop-bucket`](https://github.com/georgemandis/scoop-bucket) repos via theAPI, takes the union of formula/manifest names, fetches release data for eachproject (versions, dates, per-asset download counts), pulls clone traffic fromthe [traffic API](https://docs.github.com/en/rest/metrics/traffic) (14-dayrolling window, merged into a persistent `clone-history.json` so all-timenumbers accumulate), and generates a Markdown page with the results.The workflow is simple:```yaml- name: Fetch release data run: bash releases.sh env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}- name: Build site run: bash less.sh```The discovery step is convenient. When I add a new project to the tap or bucketit appears on the releases page automatically. If I'm on top of providing a nicedescription for my projects it even looks kind-of nice!Here's what the project discovery looks like in the script:```bash# Discover projects from homebrew-taphomebrew_projects=$(cached_fetch \\ \"$API_BASE/repos/$GITHUB_OWNER/homebrew-tap/contents/Formula\" \\ | jq -r '.[].name | select(endswith(\".rb\")) | rtrimstr(\".rb\")')# Discover projects from scoop-bucketscoop_projects=$(cached_fetch \\ \"$API_BASE/repos/$GITHUB_OWNER/scoop-bucket/contents/\" \\ | jq -r '.[].name | select(endswith(\".json\")) | rtrimstr(\".json\")')# Union and deduplicateprojects=$(printf '%s\\n%s' \"$homebrew_projects\" \"$scoop_projects\" | sort -u)```And the per-release download counting:```bashwhile IFS=$'\\t' read -r tag_name published_at downloads; do total_downloads=$((total_downloads + downloads))done < <(echo \"$releases_json\" | jq -r ' .[] | [ .tag_name, .published_at, ([.assets[].download_count] | add // 0) ] | @tsv')```The result is a page like this, rebuilt daily:| Project | Versions | Downloads | Clones || --------- | -------- | --------- | ------- || copycat | 5 | 61 | 147 || loupe | 5 | 35 | 81 || whereami | 6 | 103 | 40 || **Total** | ... | **232** | **442** |You can see the live version at[less.software/releases](https://less.software/releases).I extracted the core of this into a standalone script called[package-release-tracker.sh](https://github.com/georgemandis/package-release-tracker.sh).It takes a GitHub username, auto-discovers all `homebrew-*` and `scoop-*` reposfor that user or org, and spits out a self-contained HTML page. No dependenciesbeyond `bash`, `curl`, and `jq`. Some fun ones to try it against:- [aws](https://github.com/aws) — SAM CLI, copilot, eksctl, and ~20 other tools in a centralized tap- [charmbracelet](https://github.com/charmbracelet) — gum (3.1M downloads), glow, vhs, soft-serve, and more- [goreleaser](https://github.com/goreleaser) — the tool many people use to build these release pipelines in the first place- [derailed](https://github.com/derailed) — k9s and popeye, distributed through separate per-project tapsOne important caveat: this only works for projects distributed through your owntaps and buckets. Popular projects that graduate to[homebrew-core](https://github.com/Homebrew/homebrew-core) or Scoop's[main bucket](https://github.com/ScoopInstaller/Main) get installed from thosecentral repos instead — their downloads no longer hit your GitHub releaseassets. If you look at a tap and find nothing but a `tap_migrations.json`,that's a sign the project has moved on. Something like `gcloud-cli`, forexample, lives in `homebrew/cask` and downloads from Google's own CDN —completely invisible to this approach.The download count is cumulative and never resets. There's no time-series datafor release assets. You know how many but not when. The clone traffic API givesyou daily granularity, but only a 14-day rolling window, which is why I'mpersisting and merging the data across builds on less.software.Like I mentioned earlier, there's no way to attribute downloads to a source. A`brew install` and a browser click on the `.tar.gz` link look identical. This isprobably fine for 99% of projects. If you really care about a more accuratemeasurement you could explore hosting the assets yourself.Separate trick while I'm on the subject of GitHub Actions: my[GitHub profile](https://github.com/georgemandis) has a README thatautomatically updates daily with my latest blog posts and recent projects.I <s>stole</s> borrowed the idea from[Frank Chiarulli Jr.](https://github.com/fcjr), whose profile does the samething with a Go script. Mine uses a short TypeScript file that runs with[Bun](https://bun.sh/):```yamlname: Update READMEon: schedule: - cron: \"0 0 * * *\" push: branches: [main] workflow_dispatch:jobs: update-readme: runs-on: ubuntu-latest if: github.ref == 'refs/heads/main' steps: - uses: actions/checkout@v4 - uses: oven-sh/setup-bun@v2 - run: bun run generate.ts - name: Commit and push run: | git diff --quiet README.md && exit 0 git config user.name \"github-actions[bot]\" git config user.email \"github-actions[bot]@users.noreply.github.com\" git add README.md git commit -m \"Update README with latest blog posts and projects\" git push```The `generate.ts` script does two things:1. Fetches my blog's [JSON feed](https://george.mand.is/feed.json) and extracts the five most recent posts2. Hits the GitHub API for my most recently pushed repos, filters out forks and infrastructure repos (taps, buckets, the profile repo itself), and takes the top tenThen it writes a Markdown file with tables for each section. The workflow shouldcommit and push only if the README actually changed.You need an RSS or JSON feed for the blog post part. If your blog alreadypublishes one, you're most of the way there. The profile README repo is just arepo named after your GitHub username, and GitHub renders its `README.md` onyour profile page.You can see the [workflow](https://github.com/georgemandis/georgemandis/blob/main/.github/workflows/update-readme.yml) and [generator script](https://github.com/georgemandis/georgemandis/blob/main/generate.ts) directly.The download tracking gives me a rough sense of whether anyone is using mytools. The README keeps my GitHub profile current without me thinking about it.Both run on free GitHub Actions minutes, which I'm keeping an eye on and mayreduce to weekly if I start breaking out of the free threshold.", "date_published": "2026-05-26T00:00:00.000Z" }, { "id": "https://george.mand.is/2026/05/who-reads-my-rss-feed/", "url": "https://george.mand.is/2026/05/who-reads-my-rss-feed.txt", "title": "Who Reads My RSS Feed?", "content_html": "![](https://georgemandis.s3-us-west-1.amazonaws.com/rss-feeds/visidata-rss-feed-analysis-1.png)I recently[moved my site from Netlify to Cloudflare Pages](/2026/05/migrating-from-netlify-to-cloudflare-pages/),partly to consolidate the services I'm using and partly to finally have properserver logs. [Plausible](https://plausible.io/sites) is great for getting thegist of who's visiting and where they're coming from, but sometimes I'm curiousabout things like:- Who is actually using my RSS feeds?- What weird bots are sniffing around?- Which home-rolled projects are people running against my site?I have an [RSS feed](/feed.xml), a [JSON feed](/feed.json), and an experimental[plain-text feed](/feed.plain.xml). I wanted to see who or what was actuallypinging them.I've been able to collect these logs for about a week now—a week that happenedto include one of my posts [[2025-06-24-a-stupid-trick-for-summarizing-long-audio|hitting the front page of Hacker News]]. This was great timing! Hacker News is good for bursts of interesting traffic toexplore.Across all feeds and the broader site logs I found over 700 unique user agentstrings representing tens of thousands of requests. Some were familiar names tome, many were not. A few were genuinely bizarre. Here's a tour of the ones thatcaught my attention from the feed logs, plus a few highlights from the widersite traffic.**Note**: this turned out to be a **long** write-up with a ton of links. There'sa CSV at the bottom if all you want is to joylelssly ingest a list ofuser-agents with references. To each their own!## Readers![](https://georgemandis.s3-us-west-1.amazonaws.com/rss-feeds/visidata-rss-feed-analysis-2.png)The core of the traffic is what you'd hope to see: people reading RSS withdedicated feed readers. But the _variety_ is striking. There is a lot ofhomogeneity in the web browsing world, but the diversity in the feed readerspace seems surprisingly strong. I have a hunch this is because the reader spaceis already quite niche, but it's also pretty easy to roll your own readercompared to your own browser.I counted over 20 distinct reader products, and a significant self-hostedcontingent.**Big hosted services** I've heard of were all present:[Feedly](https://feedly.com), [Feedbin](https://feedbin.com),[Inoreader](https://www.inoreader.com), and [Feeder](https://feeder.co). Severalof them helpfully report subscriber counts right in their user agent strings.Feedbin told me I have 16 subscribers! Feedly reported 14 on my RSS feed and 3on my JSON feed. Inoreader said 5. That's a nice touch I didn't know about orexpect. A small bit of transparency in an exchange that otherwise givespublishers almost no feedback.**The self-hosted readers** were the really fun ones.[Miniflux](https://miniflux.app) appeared in _nine_ different version strings,which I think means at least nine separate installations are polling my feed.That's nine people running their own Go-based feed reader on their own servers.[FreshRSS](https://freshrss.org) showed up with four versions.[CommaFeed](https://github.com/Athou/commafeed), the Java-based Google Readerreplacement, was there. So was [Bubo Reader](https://buboreader.com), an\"irrationally minimal\" reader that generates a static HTML page from your feedlist—no account, no descriptions, no server-side state—just a webpage full oflinks. Full disclosure: Bubo Reader is[my project](https://github.com/georgemandis/bubo-rss). It's open-source and myblog is in the default feed list, so I suspect some of these hits are people whoforked the repo, deployed their own instance, and never removed my blog from thedefaults. I thank you all for your patronage 🫡.**Native apps** are still out there too. [NetNewsWire](https://netnewswire.com),the long-running open-source Mac/iOS reader originally by Brent Simmons, had astrong showing. [Unread](https://www.goldenhillsoftware.com/unread/) was a newone to me and completely dominated my JSON feed, accounting for over 40% of allJSON feed traffic (the JSON feed generally is much lower traffic than mytraditional RSS feed). It barely appeared in the RSS logs at all.[feeeed](https://feeeed.nateparrott.com), an iOS app by Nate Parrott that mixesRSS with Reddit, YouTube, weather, and step counts into a single scroll, showedup too. And [AntennaPod](https://antennapod.org), an open-source Android podcastapp, was polling my blog feed. Someone has apparently subscribed to my blog asif it were a podcast? Please leave 5 stars.On Android, [ReadYou](https://github.com/Ashinch/ReadYou) (an open-sourceMaterial Design reader) appeared in the wider site logs alongside the charminglynamed[SpaceCowboys Android RSS Reader](https://play.google.com/store/apps/details?id=com.spacecowboys.feeder).[BazQux](https://bazqux.com) checked in with exactly 1 subscriber.[GoodLinks](https://goodlinks.app), a reading-list app for Apple platforms,showed up too. And [ReaderDesktop](https://readerdotone.app), a native macOSreader, made a single appearance.**The terminal readers** represent a particular type of person—people after myown heart (I'm actually considering taking Bubo this direction).[Newsboat](https://newsboat.org) was there, a terminal-based reader popular withthe CLI crowd, but the one that really got me was[Elfeed](https://github.com/skeeto/elfeed)—`Emacs Elfeed 3.4.2`. Someone isreading my blog inside Emacs. Christopher Wellons' tag-based feed reader withOrg-mode integration.And then there's [Syndicator](https://app.syndicator.one), a personalizedreading app that learns from your behavior and aggregates blogs, news, YouTube,and Substacks. It accounted for 37% of my JSON feed traffic. Between Unread andSyndicator, two apps I'd never heard of make up nearly 80% of all JSON feedrequests.## The FediverseMultiple Fediverse platforms showed up in the logs. At least five differentMisskey instances and six different Friendica instances were fetching mycontent—I won't link to any of them individually, but the pattern is clear: whensomeone shares a link on a Fediverse instance, the server fetches it for linkpreview generation. Tools like [rsskey](https://github.com/pjeby/rsskey) canalso mirror an RSS feed directly into the Fediverse as posts, which mightexplain some of the traffic.[Micro.blog](https://micro.blog), Manton Reece's indie microblogging platform,was crawling too. As was [Flipboard](https://about.flipboard.com), fetchingcontent for its magazine-style card previews.## Indie search enginesThis category might've been the most interesting to me.[Marginalia Search](https://www.marginalia.nu/) (`search.marginalia.nu`) is aone-person search engine built by Viktor Lofgren that deliberately indexes thesmall, personal, non-commercial web. It deprioritizes SEO-optimized content andelevates human-scale sites. The whole thing is[open source](https://github.com/MarginaliaSearch/MarginaliaSearch) and designedto run on affordable hardware. Being crawled by Marginalia feels like acompliment.[Kagi](https://kagi.com) (`kagibot`) was there too! It's a paid, ad-free,privacy-focused search engine that I've[written about before](/2024/12/kagi-orion-and-gopher/). Kagi has a cleverpolicy: if your `robots.txt` doesn't mention `kagibot` specifically but doeshave rules for `Googlebot`, it[follows the Googlebot rules as a fallback](https://kagi.com/bot). Unclear iftheir [Orion browser](https://orionbrowser.com) was part of the traffic since ittries to obfuscate by design.[rssanyway](https://rssanyway.com/) (v0.1—very early) is a new service by RyanX. Charles that generates RSS feeds for sites that don't have them and rankstrending content across all feeds in its index. It's bootstrapped from HackerNews and any URL that reaches some threshold of popularity gets added.And [GeistHaus](https://geist.haus) is building an RSS meta-layer that ranksarticles by how many feeds link to them—a kind of Techmeme built entirely on RSSdata, with echoes of [PageRank](https://en.wikipedia.org/wiki/PageRank) to myears.In the wider site logs, [Smallwebindexbot](https://smallwebindex.com/bot.html)appeared—yet another indie search engine specifically for the small web.[rawweb-bot](https://github.com/0x2E/RawWeb.org) is a search engine forindependent personal websites and blogs, built by someone who misses \"the goldenage of the internet when people thought, wrote, and shared on their personalwebsites and blogs.\" And[YandoriRSSBot](https://social.emucafe.org/naferrell/what-is-the-yandorirssbot-02-25-26/),built by a developer called antiochIst, monitors around 200,000 RSS feeds innear real-time and clusters related articles to track how stories spread acrossthe web.`HN-serendipity-research/1.0` identifies itself as being from`news.ycombinator.com`—a research bot exploring serendipitous content discoveryacross Hacker News links. Whatever \"serendipitous content\" means, exactly.## The AI Crawlers![Claude very cutely recognizing itself in the data.](https://georgemandis.s3-us-west-1.amazonaws.com/rss-feeds/visidata-rss-feed-analysis-3.png)Unsurprisingly, the AI companies have arrived, and they are <s>harvesting yourtokens</s> reading your feeds. The feed logs alone only showed a handful, butthe full site logs revealed the true scale: AI crawlers are now one of thedominant traffic categories.**OpenAI** alone sent three different bots which I learned more about in thecourse of researching and writing this:- [GPTBot](https://platform.openai.com/docs/bots) is their training data crawler. It's also [one of the most-blocked bots on the web](https://darkvisitors.com/).- [OAI-SearchBot](https://platform.openai.com/docs/bots) powers SearchGPT results. I'm not sure, but I wonder if this is the agent you'll see when a model with [web search capabilities](https://developers.openai.com/api/docs/guides/tools-web-search) checks out your site.- And [ChatGPT-User](https://platform.openai.com/docs/bots) is the one that fires when a ChatGPT user asks it to browse a specific URL. That last one showed over 200 hits in the full site logs—though that number is probably misleading. A single \"go read my blog\" prompt can send ChatGPT on a crawling spree across multiple pages, so this doesn't mean 200 people pasted my URL—some of those hits are definitely me, when I asked it to go read my site and find themes.[Applebot](https://support.apple.com/en-us/119829) was there, powering Siri,Spotlight, and Apple Intelligence.[Amazonbot](https://developer.amazon.com/amazonbot) too, feeding Alexa's answersand the Rufus AI shopping assistant.[ClaudeBot](https://docs.anthropic.com/en/docs/about-claude/models) (Anthropic)showed up.[PerplexityBot and Perplexity-User](https://perplexity.ai/perplexitybot) (thesame training/real-time split as OpenAI).[Bytespider](https://knownagents.com/agents/bytespider) (ByteDance/TikTok) wasone of the noisier ones, with a separate[TikTokSpider](https://knownagents.com/agents/tiktokspider) for good measure.[Bravebot](https://search.brave.com/help/brave-search-crawler) for Brave Search.[DuckDuckBot](http://duckduckgo.com/duckduckbot.html) for traditional search and[DuckAssistBot](http://duckduckgo.com/duckassistbot.html) for DuckDuckGo's AIassistant features.Beyond the big names, there's a newer class of AI-native crawlers. AIWebIndex(from a company called Lyrenth) implements an open standard for AI-readable webcrawling, converting URLs to structured JSON optimized for AI consumption. Theirbot info page says \"You probably landed here from a server log.\" Accurate!LinkupBot (from linkup.so) is building a search API for RAG applications, thekind of thing you'd plug into an AI agent that needs to search the web. And`sauce.ai-news/1.0 (+discovery)` appeared once with no documentation, nowebsite, nothing. Just a name and the word \"discovery.\"## Prediction markets (!!)![Polymarket-Bot showing up in my logs](https://georgemandis.s3-us-west-1.amazonaws.com/rss-feeds/visidata-rss-feed-analysis-4.png)Okay, this one got me thinking.[Polymarket-Bot/1.0](https://polymarket.com)—as in Polymarket, thecryptocurrency prediction market—made 24 requests to my RSS feed. Not otherpages, curiously—just the feed. I know prediction market bots scrape newssources to detect events that could move markets. My blog, apparently, is intheir index of potential signal sources.God, I hope[writing about PaTUI](https://george.mand.is/2026/05/meet-patui-ms-paint-for-the-terminal-with-vim-controls/)shook markets.This sent me down a brief daydream about a world where a bunch of us coordinateto publish increasingly unhinged content in our RSS feeds specifically toconfuse Polymarket's bots. A sort-of distributed attack on prediction marketsvia RSS feed poisoning?Maybe skim the latest markets and use a little LLM magic to conjure believablyridiculous stories that might drive things in a direction where they lose themost money? \"Elon considers buying the moon.\" Maybe that's not unhinged enough.**Note**: if you are noticing Polymarket sniffing around your blog and havesimilar aspirations, get in touch! I love a good scheme.## Preservation Layer[archive.org_bot](https://archive.org/details/archive.org_bot)—the InternetArchive's Wayback Machine crawler—appeared in the logs. My posts are beingarchived for posterity. I can still find websites I made in high school (!!) onthere. Like an old friend you are always happy to see.[Shiori](https://github.com/go-shiori/shiori) showed up twice, in two differentversions. It's a self-hosted bookmark manager written in Go, an open-sourcealternative to Pocket. Someone bookmarked something from my site and Shiorifetched it to create an offline archive.And someone is monitoring whether my site stays up.[Uptime-Kuma](https://github.com/louislam/uptime-kuma), an open-sourceself-hosted monitoring tool, was the fourth most active agent in the entire sitelogs. Okay, one of those is me—but the other might not be! Someone out therecares enough about my uptime to run a health check against it.## Pipelines to email, Slack and newslettersRSS isn't just about reading. It's also about syndication. It's _piped_. Asignificant chunk of traffic came from services that take feed content and pushit somewhere else.[Slackbot](https://api.slack.com/robots) was the second most active agentoverall. This is just Slack's link unfurling, not a subscriber. Every timesomeone pastes a post URL in a Slack workspace, Slackbot fetches it to generatea preview card. Hundreds of hits in a week means people are actively sharing myposts in Slack conversations. That's arguably more interesting signal thansubscription counts and pings, since a human may very well have had to activelyshare it![Blogtrottr](https://blogtrottr.com) delivers RSS to email inboxes. Someone isgetting my posts as emails. [FeedBurner](https://feedburner.google.com) issomehow still alive, the Google-acquired feed proxy from 2007 that Google hasbeen threatening to kill for a decade. If it's hitting my feed, someone stillhas a FeedBurner URL pointed at me from the old days.[WordPress.com's Feedbot](https://wordpress.com) powers the WordPress.comReader. [Kingfisher](https://github.com/tldrmedia/kingfisher) is the ingestionbot for the [TLDR Newsletter](https://tldr.tech). Cool!## Runtime fingerprintsSome of the most interesting user agent strings are the ones that (presumably)reveal what someone's homebrewed script is written in.[Hackney](https://github.com/benoitc/hackney), an Elixir/Erlang HTTP client, wasone of the most active agents overall. Some Elixir application is veryinterested in my feed and I have no idea what it is.[Apache HttpClient](https://hc.apache.org) reveals a Java application. The barestring `node`—just \"node,\" nothing else—means someone's Node.js script didn'tbother setting a custom user agent. [`undici`](https://github.com/nodejs/undici)is what you get when you use `fetch()` in Node.js 18+ without customizingheaders—it's the name of the HTTP client library that powers Node's native fetchimplementation. [Deno/2.7.5](https://deno.com) means someone wrote a Denoscript—created by Ryan Dahl, the same person who created Node.js.[Bun/1.3.14](https://bun.sh) rounds out the JavaScript runtime trifecta on theJSON feed side.[SimplePie](https://simplepie.org/) is the PHP feed parsing library that powersWordPress's RSS widget—so a WordPress site somewhere has my feed in a sidebarwidget. [feedparser](https://github.com/kurtmckee/feedparser) and[newspaper](https://github.com/codelucas/newspaper) are Python libraries forparsing feeds and extracting article content respectively.[feed2exec](https://feed2exec.readthedocs.io) is a Python CLI tool that runsarbitrary commands when new feed items appear—someone has a little pipeline thattriggers _something_ every time I post.The wider site logs add more to the collection.[python-requests](https://docs.python-requests.org/) appeared in at least fourdifferent versions—the most common Python HTTP library, each version probably adifferent script or project.[trafilatura](https://github.com/adbar/trafilatura), a Python libraryspecifically designed for web text extraction and corpus building, showed upwith notable volume. [colly](https://github.com/gocolly/colly), a popular Goscraping framework. [axios](https://axios-http.com/), the ubiquitous, and nowinfamously supply-chain-attacked, JavaScript HTTP client. And one of myfavorites: `gen_candidates.py (tech-news-daily)`—someone left their literalPython filename as the user agent. A tech news aggregation pipeline, running ascript called `gen_candidates.py`, apparently evaluating my blog for inclusion.Hey, it is an honor to be nominated.And then there's `Embarcadero URI Client/1.0`. This is the default HTTP useragent from[Embarcadero RAD Studio](https://www.embarcadero.com/products/rad-studio)—theDelphi IDE. Someone built a _Delphi_ application that fetches my feed. In 2026.Or at least, something out there is pretending to do that. It appeared in bothmy RSS and JSON feed logs. Respect.## Lies, damn lies and browser stringsA good chunk of the logs are standard browser user agent strings. A few might bereal humans who opened the feed URL in a browser tab, but most are certainlynot.Chrome 84 from 2020. Chrome 30 from 2013. Internet Explorer 9 from 2011. IE 6 onWindows 2000. And my personal favorite from the JSON feed: someone claiming tobe **Firefox 35 on Windows 98**. I'm not sure Windows 98 could run Firefox 35even when Firefox 35 was current in 2015, let alone in 2026.The wider site logs turn this into a full museum. Highlights from thecollection:- **Opera on a Nintendo Wii**—`Opera/9.30 (Nintendo Wii; U; ; 2047-7; en)`. The Wii's Opera-based Internet Channel, vintage 2007.- **SeaMonkey on BeOS**—`Mozilla/5.0 (BeOS; U; BeOS BePC)... SeaMonkey/1.5a`. BeOS hasn't been a going concern since 2001.- **Nokia N97 on Symbian**—`SymbianOS/9.4; Series60/5.0 NokiaN97-1`. The last gasp of Nokia's pre-Windows Phone era.- **Nokia N9 on MeeGo**—`NokiaBrowser/8.5.0`. Nokia's beautiful, doomed Linux phone from 2011.- **Sony Ericsson K800i**—A feature phone from 2006, identifying via WAP headers.- **Konqueror on Linux**—KDE's browser. Technically still maintained, but not commonly seen in the wild.- **Links on FreeBSD**—`Links (2.1pre15; FreeBSD 5.3-RELEASE i386; 196x84)`. A text-mode browser at 196x84 character resolution.- **Namoroka on NetBSD**—`Namoroka/3.6.15`. This was Firefox's internal codename before it was released as Firefox 3.6.- **AOL Browser**—`AOLBUILD/11.0.1839`. Yes, really.- **Opera on PPC Mac**—`Opera/9.0 (Macintosh; PPC Mac OS X; U; en)`. PowerPC Macs haven't been manufactured since 2006.These are bots or scrapers doing browser cosplay from years they weren't evenplausible. This really underscores how user agent strings are a loose socialcontract and always have been—but these aren't even trying.I look forward to updating my user agent string to let sites know I'm runningLynx on my Atari 2600.## SEO crawlers and international search enginesThe SEO usual suspects were all present: [AhrefsBot](https://ahrefs.com/robot)(one of the most active crawlers on the web), [MJ12bot](https://mj12bot.com/)(Majestic SEO, doing backlink analysis since 2004),[DataForSeoBot](https://dataforseo.com/dataforseo-bot), and[SemrushBot](http://www.semrush.com/bot.html) in multiple flavors.More interesting was the international search engine contingent.[PetalBot](https://knownagents.com/agents/petalbot) (Huawei's search crawler forPetal Search) was among the more active crawlers overall.[Sogou](https://knownagents.com/agents/sogou) (Chinese searchengine—[more info](http://www.sogou.com/docs/help/webmasters.htm#07)?),[Baiduspider](http://www.baidu.com/search/spider.html) (Baidu),[CocCocBot](https://knownagents.com/agents/coccocbot) (Vietnamese search enginefor Coc Coc—[more info](http://help.coccoc.com/searchengine)?),[YandexBot](http://yandex.com/bots) (Russian search),[SeznamBot](https://o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/)(Czech search engine Seznam.cz),[Yeti/Naver](https://knownagents.com/agents/yeti-by-naver) (Korean search for[Naver](https://naver.me/spd)), and [Qwantbot](https://help.qwant.com/bot/)(French privacy-focused search). My little personal blog, being indexed forsearch engines in at least eight countries. The web really is worldwide, even ifthat's easy to forget.[Xobaque](https://alexschroeder.ch/view/Xobaque) also showed up. It seems like adelightfully personal project from Alex Schroeder trying to create an opt-insearch engine.**As a fun aside**: I used AI to give a first pass at the logs and help meresearch the different user agents and provide links so I could learn more aboutthem. Then I went through all of the links to see what they were about. Withthis one it said \"No web presence, no documentation, no results for the nameanywhere. A ghost.\" I did a search in GitHub and found one reference on[this AntennaApp issue](https://github.com/rsdoiel/antennaApp/issues/19) whichled me to Alex's blog, where I promptly asked to confirm my humanity. I guessit's working, since AI couldn't find it?Also lurking in the wider logs: something called `go_revenue_model`, presumablya Go tool by Florin Badita that apparently analyzes website revenue models, made33 requests. The GitHub repo seems to be private, so I can't tell you much moreabout it.Maybe someone is studying how I make money from this blog? Spoiler: I don't.Unless you'd like to [sponsor me](https://github.com/sponsors/georgemandis).## The Mysteries![The award for the politest bot goes to Thinkbot/0.5.8](https://georgemandis.s3-us-west-1.amazonaws.com/rss-feeds/visidata-rss-feed-analysis-5.png)We are hitting the long-tail of user agents and trodding firmly in the[Low Information Zone](https://www.metabunk.org/threads/ufo-acronyms-what-is-the-liz.11742/)(Yes, I'm outing my hobby of reading about debunking conspiracy theories onsites like Metabunk and Skeptoid). Here are some weird ones:- **bushbaby/2026.5.1**—This one looked weird. Turns out it's a [Cloudflare internal bot](https://radar.cloudflare.com/bots/directory/bushbaby) used for SSL certificate renewal checks.- **ED309134-1C93-41BB-A10D-3278DF6BCF72/310**—An iOS app that forgot to set its display name in the app bundle! It shows up as a raw UUID. The `CFNetwork/Darwin` suffix confirms it's a native iOS app. Someone's homebrew feed reader, still in development, already subscribed to my blog? Charmed. I hope they ship it.- **br-crawler/0.5**—Listed in [bot directories](https://knownagents.com/agents/br-crawler) but categorized as \"uncategorized.\" I couldn't find anything.- **Thinkbot/0.5.8**—Gets honorable mention for the most polite user agent string I've ever seen: `\"In_the_test_phase,_if_the_Thinkbot_brings_you_trouble,_please_block_its_IP_address._Thank_you.\"`## The JSON readersMy [JSON Feed](https://www.jsonfeed.org/) had far fewer unique agents but acompletely different audience.[Unread](https://www.goldenhillsoftware.com/unread/) and[Syndicator](https://app.syndicator.one) together account for nearly 80% of allJSON feed traffic—yet they barely appear in the RSS logs. If you publish bothformats, you might be reaching different tools and potentially different people.Maybe I should check-in on my Gopher site.The JSON feed also attracted [GPTBot](https://platform.openai.com/docs/bots),[Shiori](https://github.com/go-shiori/shiori), and the Embarcadero Delphiclient. But two apps seem to generally own the JSON feed, while the RSS feed isa cornucopia of crawlers and clients.## What did we learn?**RSS is inspiringly diverse!** Way more diverse than browsers feel to me atleast. Syndicating content on the web is and _isn't_ as solved as you'd think.That there is this much variety gives me hope for something. In one week I sawover 50 distinct, identifiable products touching my feeds: commercial readers,self-hosted installations, newsletter bots, a podcast app, Fediverse instancesfrom two different platforms, search engines from eight countries, AI crawlersfrom every major AI company, a prediction market, preservation services, emailrelays, bookmark managers, SEO crawlers, terminal readers, an Emacs package, andsomeone's Delphi app.**The self-hosted contingent is bigger than I thought.** Miniflux, FreshRSS,CommaFeed, Bubo Reader, Newsboat, Elfeed—people running their own infrastructureto read feeds. Says something about the audience of a personal tech blog, maybe,but I think it also says something about the health of the self-hosted RSSecosystem.**Everyone wants to read your feed, and not all of them are readers.**Prediction markets. AI training pipelines. SEO crawlers. Newsletter ingestionbots. Link preview services. Bookmark archivers. The Internet Archive. RSS isn'tjust a reading protocol—it's infrastructure. A structured, machine-readable datafeed with no authentication required that anything can consume. The humanreaders are almost certainly the minority, though I hope they exist somewhere atthe end of these aggregation chains.**User agents are a beautiful mess.** Between Elixir libraries, bare runtimestrings, a UUID that should be an app name, fake browsers from 2011, a NintendoWii, a Sony Ericsson feature phone, a BeOS installation, someone's literalPython filename, and at least five agents I simply cannot identify, the useragent string remains the web's most charmingly unreliable metadata field. It's[Knights and Knaves](https://en.wikipedia.org/wiki/Knights_and_Knaves), butreally mostly knaves.**Publishing both RSS and JSON feeds feels worth!** They reach differentaudiences and are consumed by different tools. This was the most concretetakeaway—if I only had RSS, I wonder if I'd be invisible to Unread andSyndicator's users at all?---## Full Reference TableMost of the identifiable user agents I observed, linked to their source as bestI could find it (some omitted at my discretion).[Download as CSV](/media/rss-feed-readers-reference.csv).| Name | Type | Description | Link || -------------------------- | ------------------ | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- || **Feedly** | RSS Reader | Cloud-based reader. Reports subscriber count in UA | [feedly.com](https://feedly.com) || **Feedbin** | RSS Reader | Hosted reader with sync. Reports subscribers and feed ID | [feedbin.com](https://feedbin.com) || **Feeder** | RSS Reader | Browser-extension reader (Chrome, Firefox, Edge) | [feeder.co](https://feeder.co) || **Inoreader** | RSS Reader | Web-based reader with filtering and automation | [inoreader.com](https://www.inoreader.com) || **NetNewsWire** | RSS Reader | Free, open-source Mac/iOS reader by Brent Simmons | [netnewswire.com](https://netnewswire.com) || **Unread** | RSS Reader | Beautiful iOS/Mac reader. Dominates JSON feed traffic | [goldenhillsoftware.com/unread](https://www.goldenhillsoftware.com/unread/) || **feeeed** | RSS Reader | iOS app mixing RSS with Reddit, YouTube, weather, and more | [feeeed.nateparrott.com](https://feeeed.nateparrott.com) || **AntennaPod** | Podcast App | Open-source Android podcast manager | [antennapod.org](https://antennapod.org) || **Miniflux** | Self-Hosted Reader | Minimalist self-hosted reader in Go | [miniflux.app](https://miniflux.app) || **FreshRSS** | Self-Hosted Reader | PHP-based self-hosted RSS aggregator | [freshrss.org](https://freshrss.org) || **CommaFeed** | Self-Hosted Reader | Java/React Google Reader replacement | [github.com/Athou/commafeed](https://github.com/Athou/commafeed) || **Bubo Reader** | Self-Hosted Reader | Generates a static HTML page from your feed list | [buboreader.com](https://buboreader.com) || **Newsboat** | Terminal Reader | Terminal-based reader for Linux/macOS | [newsboat.org](https://newsboat.org) || **Elfeed** | Emacs Reader | Tag-based feed reader for Emacs | [github.com/skeeto/elfeed](https://github.com/skeeto/elfeed) || **Syndicator** | Reader/Aggregator | Personalized reading app that learns from behavior | [app.syndicator.one](https://app.syndicator.one) || **Feedstand** | Reader | RSS reader service with feed IDs | [feedstand.com](https://feedstand.com) || **ReadYou** | RSS Reader | Open-source Material Design reader for Android | [github.com/Ashinch/ReadYou](https://github.com/Ashinch/ReadYou) || **SpaceCowboys** | RSS Reader | Android RSS reader | [play.google.com](https://play.google.com/store/apps/details?id=com.spacecowboys.feeder) || **BazQux** | RSS Reader | Web-based reader. Reports subscriber count | [bazqux.com](https://bazqux.com) || **GoodLinks** | Reading List | Bookmarking/reading-list app for Apple platforms | [goodlinks.app](https://goodlinks.app) || **ReaderDesktop** | RSS Reader | Native macOS RSS reader | [readerdotone.app](https://readerdotone.app) || **Misskey** | Fediverse | Decentralized Fediverse social platform | [misskey-hub.net](https://misskey-hub.net/en/) || **Friendica** | Fediverse | Facebook-like Fediverse platform | [friendi.ca](https://friendi.ca/) || **Micro.blog** | Fediverse/Platform | Indie microblogging platform's feed crawler | [micro.blog](https://micro.blog) || **Flipboard** | Content Platform | Digital magazine app's content proxy | [flipboard.com](https://about.flipboard.com) || **Slackbot** | Link Unfurling | Slack's preview bot. Fires when URLs are shared | [api.slack.com/robots](https://api.slack.com/robots) || **Marginalia** | Indie Search | One-person search engine for the small/personal web | [marginalia.nu](https://www.marginalia.nu/) || **Kagi** | Indie Search | Paid, ad-free, privacy-focused search engine | [kagi.com](https://kagi.com) || **rssanyway** | Content Discovery | Generates feeds for sites without them. Ranks trending content | [rssanyway.com](https://rssanyway.com/) || **GeistHaus** | Content Discovery | RSS meta-layer ranking articles by cross-feed links | [geist.haus](https://geist.haus) || **GPTBot** | AI Crawler | OpenAI's training data crawler | [platform.openai.com/docs/bots](https://platform.openai.com/docs/bots) || **ChatGPT-User** | AI Crawler | OpenAI's real-time browsing crawler | [platform.openai.com/docs/bots](https://platform.openai.com/docs/bots) || **OAI-SearchBot** | AI Crawler | OpenAI's SearchGPT crawler | [platform.openai.com/docs/bots](https://platform.openai.com/docs/bots) || **ClaudeBot** | AI Crawler | Anthropic's web crawler | [docs.anthropic.com](https://docs.anthropic.com) || **PerplexityBot** | AI Crawler | Perplexity's indexing crawler | [perplexity.ai](https://perplexity.ai/perplexitybot) || **Bytespider** | AI Crawler | ByteDance/TikTok's web crawler | [zhanzhang.toutiao.com](https://zhanzhang.toutiao.com/) || **Applebot** | AI/Search Crawler | Powers Siri, Spotlight, and Apple Intelligence | [support.apple.com](https://support.apple.com/en-us/119829) || **Amazonbot** | AI/Search Crawler | Feeds Alexa and the Rufus AI assistant | [developer.amazon.com/amazonbot](https://developer.amazon.com/amazonbot) || **Bravebot** | Search Crawler | Brave Search's web crawler | [search.brave.com](https://search.brave.com/help/brave-search-crawler) || **DuckDuckBot** | Search Crawler | DuckDuckGo's search crawler | [duckduckgo.com](http://duckduckgo.com/duckduckbot.html) || **DuckAssistBot** | AI Crawler | DuckDuckGo's AI assistant crawler | [duckduckgo.com](http://duckduckgo.com/duckassistbot.html) || **Polymarket-Bot** | Prediction Market | Crypto prediction market's news scraper | [polymarket.com](https://polymarket.com) || **archive.org_bot** | Preservation | Internet Archive's Wayback Machine crawler | [archive.org](https://archive.org/details/archive.org_bot) || **Shiori** | Bookmarking | Self-hosted bookmark manager in Go (Pocket alternative) | [github.com/go-shiori/shiori](https://github.com/go-shiori/shiori) || **Blogtrottr** | Feed-to-Email | Delivers RSS to email inboxes | [blogtrottr.com](https://blogtrottr.com) || **FeedBurner** | Feed Relay | Google's deprecated (but undead) feed proxy from 2007 | [feedburner.google.com](https://feedburner.google.com) || **WordPress.com Feedbot** | Feed Relay | Automattic's crawler for WP.com Reader | [wordpress.com](https://wordpress.com) || **Kingfisher (TLDR)** | Newsletter Bot | Feed ingestion for the TLDR Newsletter | [github.com/tldrmedia/kingfisher](https://github.com/tldrmedia/kingfisher) || **RSS.Social** | Social Platform | Social platform built around RSS feeds | [rss.social](https://rss.social) || **Lighthouse** | Curation App | RSS curation with AI summaries (not Google Lighthouse) | [lighthouseapp.io](https://lighthouseapp.io) || **Hackney** | Library (Elixir) | Elixir/Erlang HTTP client. Product unknown | [github.com/benoitc/hackney](https://github.com/benoitc/hackney) || **Apache HttpClient** | Library (Java) | Java HTTP library. Product unknown | [hc.apache.org](https://hc.apache.org) || **feedparser** | Library (Python) | Canonical Python RSS parsing library | [github.com/kurtmckee/feedparser](https://github.com/kurtmckee/feedparser) || **newspaper** | Library (Python) | Python article extraction library | [github.com/codelucas/newspaper](https://github.com/codelucas/newspaper) || **SimplePie** | Library (PHP) | PHP feed parser. Powers WordPress's RSS widget | [simplepie.org](https://simplepie.org/) || **rss-parser** | Library (Node.js) | Popular npm package for parsing feeds | [github.com/rbren/rss-parser](https://github.com/rbren/rss-parser) || **PicoFeed** | Library (PHP) | PHP library originally built for Miniflux | [github.com/miniflux/picoFeed](https://github.com/miniflux/picoFeed) || **feed2exec** | CLI Tool | Python CLI that runs commands on new feed items | [feed2exec.readthedocs.io](https://feed2exec.readthedocs.io) || **Embarcadero URI Client** | Runtime (Delphi) | Default UA from Delphi/RAD Studio apps | [embarcadero.com](https://www.embarcadero.com/products/rad-studio) || **Deno** | Runtime | Default UA from Deno's fetch() | [deno.com](https://deno.com) || **Bun** | Runtime | Default UA from Bun JavaScript runtime | [bun.sh](https://bun.sh) || **undici** | Runtime (Node.js) | Default UA from Node.js 18+ native fetch() | [github.com/nodejs/undici](https://github.com/nodejs/undici) || **node** | Runtime (Node.js) | Bare Node.js string. No custom UA set | [nodejs.org](https://nodejs.org) || **AhrefsBot** | SEO Crawler | One of the most active backlink crawlers | [ahrefs.com/robot](https://ahrefs.com/robot) || **MJ12bot** | SEO Crawler | Majestic SEO's backlink crawler. Active since 2004 | [mj12bot.com](https://mj12bot.com/) || **DataForSeoBot** | SEO Crawler | SEO data API for other companies' tools | [dataforseo.com](https://dataforseo.com/dataforseo-bot) || **SemrushBot** | SEO Crawler | SEO analytics and competitive research | [semrush.com](http://www.semrush.com/bot.html) || **PetalBot** | Search Crawler | Huawei's Petal Search engine crawler | [webmaster.petalsearch.com](https://webmaster.petalsearch.com/site/petalbot) || **Qwantbot** | Search Crawler | French privacy-focused search engine | [help.qwant.com](https://help.qwant.com/bot/) || **SeznamBot** | Search Crawler | Czech search engine Seznam.cz | [o-seznam.cz](https://o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/) || **Smallwebindexbot** | Indie Search | Search engine for the small web | [smallwebindex.com](https://smallwebindex.com/bot.html) || **rawweb-bot** | Indie Search | Search engine for independent personal websites and blogs | [github.com/0x2E/RawWeb.org](https://github.com/0x2E/RawWeb.org) || **YandoriRSSBot** | News Monitoring | Monitors ~200K RSS feeds, clusters related articles | [writeup by N.A. Ferrell](https://social.emucafe.org/naferrell/what-is-the-yandorirssbot-02-25-26/) || **Uptime-Kuma** | Monitoring | Open-source self-hosted uptime monitor | [github.com/louislam/uptime-kuma](https://github.com/louislam/uptime-kuma) || **trafilatura** | Library (Python) | Web text extraction for NLP/corpus building | [github.com/adbar/trafilatura](https://github.com/adbar/trafilatura) || **colly** | Library (Go) | Popular Go web scraping framework | [github.com/gocolly/colly](https://github.com/gocolly/colly) || **Meta External Agent** | Platform Crawler | Facebook/Instagram Open Graph preview crawler | [developers.facebook.com](https://developers.facebook.com/docs/sharing/webmasters/crawler) |", "date_published": "2026-05-18T00:00:00.000Z" }, { "id": "https://george.mand.is/2026/05/my-favorite-bugs-invalid-surrogate-pairs/", "url": "https://george.mand.is/2026/05/my-favorite-bugs-invalid-surrogate-pairs.txt", "title": "My Favorite Bugs: Invalid Surrogate Pairs", "content_html": "If you're in the business of building things that run on computers long enough,I think you will eventually acquire a favorite bug story. This is a short storyabout mine. I've also built an [interactive tool](invalid-surrogate-pairs/)where you can explore the concepts underpinning the heart of this bug.## The bug: two emoji enter, none leaveI was working on migrating a legacy editor to a more collaborative experiencewith my team. [TipTap](https://tiptap.dev/) on top (itself a wrapper around[ProseMirror](https://prosemirror.net)), [Yjs](https://yjs.dev/) underneathhandling the CRDT magic for real-time syncing. It worked well! Mostly.In our alpha/early release days, when it was still mostly internal and/or earlyrollout users, sometimes the editor would just stop saving your content.Silently. You'd keep typing and everything looked fine, but your edits stoppedsyncing to the Yjs document. The next time you opened the page, everything you'dwritten after the failure point was gone.It was utterly terrifying, very rare and almost impossible to diagnose becausewe could never recreate it. We really tried! My early suspicions generallyrevolved around shaky wifi connections and wonky websocket behaviors, but noamount of throttling or turning my wifi on and off seemed to recreate the issue.The experience was surprisingly resilient in those scenarios, in my memory. Itfelt like it happened randomly, never when anyone was looking. No obvious errorspicked up in the console, no stack trace, no crash. Just... \"Hey, I think mychanges didn't save.\"Then one day our product manager cracked it. This was not a trivial thing tofind. He'd been experiencing it more than anyone else (probably because he wasthe best at dogfooding our product) and had been methodically narrowing it down._\"I feel like I'm going crazy, but I think it's when I type specific characterstogether, go back and insert a character between them...\"_He'd been using 🟢 and 🔴 in his weekly project status emails to communicategeneral health. Green for on-track, red for at-risk. Every week the template hewas using had both characters already present and he would simply remove the onehe didn't need (Generally the red one, I am happy to say!).On this occasion he'd copied the green circle and pasted it in front of the redone at some point, or maybe vice versa. That specific operation— inserting onemulti-byte emoji adjacent to another— was triggering a splice in the underlyingCRDT library, which split a surrogate pair down the middle.I remember being on the call when he showed this to me and one of my directreports who'd been toiling away at the collaborative editing transition. Imust've gotten a little too excited—I live for esoteric bugs—\"I feel like yougot _energized_ by this,\" he said. He wasn't wrong.Adding to the fun, not every emoji triggered it. Only the ones above `U+FFFF`that required surrogate pairs. And not all edits resulted in the problemeither—only the ones that caused a **splice** at exactly the wrong byte offset.It was a wild one to debug before we knew what was going on.## Code units, code points, and grapheme clustersSo what was going on? What does \"ones above `U+FFFF`\" in that last paragrapheven mean? What byte offsets?To understand this bug we need to introduce three pieces of vocabulary:```cssCode Units → Code Points → Grapheme Clusters```**Code units** are the raw 16-bit values that JavaScript uses to store stringsinternally (UTF-16). This is what `.length` counts. This is what `.slice()` and`.charCodeAt()` operate on as well. JavaScript operates at the code unit levelby default**Code points** are what Unicode actually defines as a single character. A codepoint like U+1F920 (🤠) is one character in Unicode's view, but it's too big tofit in a single 16-bit code unit. So UTF-16 splits it into two code units calleda **surrogate pair**: a high surrogate and a low surrogate. Simple ASCIIcharacters and a lot of common symbols fit in one code unit, so the distinctiondoesn't matter for them. Emoji, though? Almost always two.**Grapheme clusters** are what a human perceives as \"one character.\" The femaleastronaut 👩‍🚀 looks like one character but is actually three code points gluedtogether: 👩 (woman) + a zero-width joiner + 🚀 (rocket). Five code units, threecode points, one grapheme. The deceptively simple 👨‍👨‍👧‍👧 (Family: Man, Man, Girl,Girl) emoji is an impressive eleven! The enigmatic ☃ is 1.Here's how those numbers diverge:| | Code units | Code points | Graphemes || -------- | ---------- | ----------- | --------- || A | 1 | 1 | 1 || 🤠 | 2 | 1 | 1 || 👩‍🚀 | 5 | 3 | 1 || 👨‍👨‍👧‍👧 | 11 | 7 | 1 |I will pause to once again plugin the[interactive surrogate explorer](/invalid-surrogate-pairs/) I alluded to at thetop. You can type any emoji and see this breakdown yourself![![Screenshot from my surrogate pair explorer](https://georgemandis.s3-us-west-1.amazonaws.com/surrogate-pair-explorer-1-2.png)](/invalid-surrogate-pairs)## How `.slice()` breaks thingsThe cowboy 🤠 is one code point stored as two code units (a surrogate pair). Ifyou slice between them:```js\"🤠\".slice(0, 1); // → '\\uD83E' (lone high surrogate)\"🤠\".slice(1, 2); // → '\\uDD20' (lone low surrogate)```Those fragments aren't valid characters. They're half a pair with no partner. Ontheir own they render as replacement characters (�) or get silently swallowed.But the real problem comes when you try to encode one:```jsencodeURIComponent(\"🤠\".slice(0, 1));// URIError: URI malformed```That's what was crashing our tool.## What was actually happeningYjs depends on a utility library called lib0. The lib0 `splice` method usedJavaScript's `.slice()` internally. When a CRDT operation happened to landbetween the two halves of an emoji's surrogate pair, lib0 would produce a stringwith an orphaned surrogate. That string would eventually get passed to`encodeURIComponent` during sync, which threw an uncaught `URIError`.The error was uncaught. Nothing in the Yjs or TipTap error handling caught it.So sync just... stopped. The editor kept working locally, giving you everyindication that things were fine, while your changes silently went nowhere.It only showed up on pathological edits: replacing one emoji with another, orinserting a character right between two emoji.## The hack we shippedWe couldn't fix lib0—though I'm happy to report it did[eventually get fixed](https://github.com/dmonad/lib0/commit/51ab65b46da8110d85384ccec631647de3248c96)!We couldn't patch Yjs. We needed to ship something.So we did two things:- Although we didn't initially care about [offline support](https://tiptap.dev/docs/guides/offline-support) for our product, adding it was pretty trivial. Our thinking was it could save us in a future situation should the user get disconnected and keep typing. We would continue to update the CRDT locally, and the _next_ time they came back to the document their changes would be updated and merged with the current state of things. This was a hedge and leaned into what CRDTs are actually good at and designed for.- An embarrassingly nuclear option (my call, with my fingerprints all over): we attached a global `window.addEventListener(\"error\", ...)` listener that regex-matched for `URIError: URI malformed`. When it caught one, it logged the event for tracking and set a piece of state that our editor would check. If we saw the error, we'd throw up a modal telling the user something went wrong and asked them to reload the page. I watched this metric like a hawk and was relieved with how rare it ended up being.We weren't the only ones. The upstream issues([yjs#303](https://github.com/yjs/yjs/issues/303),[tiptap#3020](https://github.com/ueberdosis/tiptap/issues/3020)) had othereditors reporting the same problem with similar workarounds.## The real fixTwo things eventually fixed it for real:**lib0 got patched.** The upstream fix was to detect if the first character of asliced string was a high surrogate without a matching low surrogate, and replaceit with U+FFFD (the Unicode replacement character, �). Not perfect, but itstopped the `URIError` from happening and prevented sync from dying.**We made emoji an atomic node type.** In ProseMirror (and by extension TipTap),you can define custom node types. We setup an extension that made emoji theirown node, which meant the editor treated each one as an indivisible unit. Cursormovements and editing operations couldn't split an emoji in half. This didn'tfix the lib0 bug, and there were some other side-effects here that werechallenging, but it eliminated most of the editing patterns that triggered it.I'm happy to report that the bug popped up _very_ rarely during this hackyinterim phase... but I was pretty happy when the patched version of lib0 finallylanded.## The modern answerIf you're doing string manipulation in JavaScript and you care about notcorrupting characters, use `Intl.Segmenter`:```jsconst seg = new Intl.Segmenter(undefined, { granularity: \"grapheme\" });const segments = [...seg.segment(\"👩‍🚀A👍\")].map((s) => s.segment);// → ['👩‍🚀', 'A', '👍']```This splits by grapheme clusters rather than code units. No orphaned surrogates,no split emoji. It's what `.slice()` should have been doing all along, but ofcourse UTF-16 predates emoji by decades.## InfamyAfter we shipped the fix I wrote about it in my internal newsletter.![Newsletter screenshot from a later update in which I recant the story](https://georgemandis.s3-us-west-1.amazonaws.com/my-newsletter1-2.png)The bug became a bit of an inside joke. Coworkers would ping me with 🟢🔴—theemoji combo that broke everything.Years later, I still get memes and messages from former coworkers out of theblue about this. Some bugs you fix and some bugs fix... you?![Slack message: \"I no longer see words, only graphemes and invalid surrogate pairs waiting to be split\"](https://georgemandis.s3-us-west-1.amazonaws.com/my-emoji-response.png)## Hard to unsee unicode problemsOnce you know about it, you start seeing it in the wild. Any code that does`str.slice(0, 1)` or `str[0]` to get \"the first character\" is potentiallybroken. The most common offender: tools that generate initials from a user'sname. Try putting an emoji as the first character of your first or last name inany app that displays your avatar as initials. Most of them will do somethinglike `firstName[0] + lastName[0]` and end up with half a surrogate pair. Somerender garbage. Some crash.It's the same class of bug every time. JavaScript gives you code units when youwanted characters, and nobody notices until someone types something outside the[Basic Multilingual Plane](https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane).I repeat the truth I hold dearest: it is remarkable anything works at all.## Parting linksMonica Dinculescu has[a great post on how emoji work under the hood](https://meowni.ca/posts/emoji-emoji-emoji/)if you want to go deeper. I highly recommend it!And I'll end with one more plug for my[interactive surrogate pair explorer](/invalid-surrogate-pairs/) where you cantype any emoji and see this breakdown yourself, in case you missed the link upabove! I think it's a nice way to visually see and interactive with the conceptsdiscussed here.[![Screenshot from my surrogate pair explorer](https://georgemandis.s3-us-west-1.amazonaws.com/surrogate-pair-explorer-1-2.png)](/invalid-surrogate-pairs/)[![Screenshot from my surrogate pair explorer](https://georgemandis.s3-us-west-1.amazonaws.com/surrogate-pair-explorer-2-2.png)](/invalid-surrogate-pairs/)[![Screenshot from my surrogate pair explorer](https://georgemandis.s3-us-west-1.amazonaws.com/surrogate-pair-explorer-3-2.png)](/invalid-surrogate-pairs/)[![Screenshot from my surrogate pair explorer](https://georgemandis.s3-us-west-1.amazonaws.com/surrogate-pair-explorer-4-2.png)](/invalid-surrogate-pairs/)", "date_published": "2026-05-14T00:00:00.000Z" }, { "id": "https://george.mand.is/2026/05/migrating-from-netlify-to-cloudflare-pages/", "url": "https://george.mand.is/2026/05/migrating-from-netlify-to-cloudflare-pages.txt", "title": "Migrating from Netlify to Cloudflare Pages", "content_html": "I've hosted this site on Netlify for years. It's been fine. Netlify is good atwhat it does, and for a static [Eleventy](https://www.11ty.dev/) blog with 400+posts, \"fine\" is really all you need.But I kept wanting one thing Netlify doesn't give you: server logs.Not analytics. I have [Plausible](https://plausible.io/) for that. I mean rawaccess logs. Who's pulling my RSS feed? Which bots are crawling the plain-textversions of posts I make available? Is anyone actually using the JSON feed? Whatcurious search engines are visiting me in the ead of night? When a post hits thefront page of Hacker News, what does that traffic actually look like at therequest level? These kinds of questions and mysteries can only be surfaced whenyou start to dig into the raw access logs and start poking around.Netlify doesn't offer this. Well, they _sort of_ do, but not without giving themmore money. I was already giving Cloudflare money, and I'm generally fond oftheir offerings, so I was more interested in finding a solution there.## Why CloudflareI already use Cloudflare heavily for other projects. Workers, R2, Pages, the AIstuff. My DNS was already proxied through them. So the question wasn't really\"where should I move?\" but \"why haven't I moved yet?\"The answer was inertia. Netlify's deploy-on-push workflow is genuinely nice, andI didn't want to break something that's worked for me for nearly a solid decade.But once I actually sat down to do it, the migration was straightforward. TheEleventy build is identical. The `_redirects` file format is nearly the same(with some caveats I'll get to). And Cloudflare Pages gives you the samegit-push-to-deploy experience. The truth is I had to change almost nothing inthe repo itself.## The actual migrationThe short version:1. **Create a Cloudflare Pages project** connected to the GitHub repo. Set the build command (`npm run build`), output directory (`_site`), done. This part is genuinely easy once you find the right UI flow. Cloudflare's dashboard is... confusing. Workers and Pages are merged under one section, and the \"Create\" flow defaults to creating a Worker, not a Pages project. There's a small \"Looking to deploy Pages?\" link at the bottom of the page. I missed it twice.![The page Cloudflare shows you when making a new worker. I am embarrassed so say I missed this link about Page Workers a couple times.](https://georgemandis.s3-us-west-1.amazonaws.com/logexplorer/cloudlflare-page-worker.png)2. **Move the `/random` endpoint.** I had a Netlify Function that fetches a special JSON array of all post URLs that Eleventy generates, picks one at random, and 302 redirects. Super hacky and one of my favorite things about my own site. This became a [Cloudflare Pages Function](https://developers.cloudflare.com/pages/functions/) at `functions/random.js`. Same logic, different export format (`onRequest` instead of `exports.handler`).3. **Clean up `_redirects`.** Cloudflare Pages supports the same `_redirects` format as Netlify for simple path-to-path redirects. But it does _not_ support two things I was using: cross-domain redirects (for `georgemandis.com` → `george.mand.is`, etc.) and `200` status proxying to external URLs. The cross-domain redirects moved to Cloudflare Redirect Rules in the dashboard. I had to look up the syntax for dynamic redirects and found [Simon Willison's write-up on this](https://til.simonwillison.net/cloudflare/redirect-rules) more helpful than Cloudflare's own docs. Cloudflare has some amazing tools, but I wish they were more clearly and consistently documented sometimes. The proxy rules were for Plausible, which needed a different solution.4. **Drop the dead weight.** I had two other Netlify Functions that I wasn't using anymore: a Micropub endpoint for publishing from iA Writer and a home-rolled Stripe payment handler for sponsorships (I'm just leaning in to [Github sponsors](https://github.com/sponsors/georgemandis) now for that).## The Plausible proxy problemHere's where it got a little more interesting. I proxy Plausible Analyticsthrough my own domain so it works even when ad-blockers block `plausible.io`. OnNetlify, this was three lines in `_redirects`:```/js/script.js https://plausible.io/js/script.js 200/js/script.tagged-events.js https://plausible.io/js/script.tagged-events.js 200/api/event https://plausible.io/api/event 200```The `200` status tells Netlify to reverse-proxy the request rather thanredirect. Cloudflare Pages doesn't support this. It didn't fit cleanly intoPages Functions either, because file-based routing means `/js/script.js` wouldneed a file at `functions/js/script.js.js`. It technically _could_ have worked,but would have been very strange.The solution: a standalone Cloudflare Worker with route matching. The Workerintercepts requests to `/js/script*` and `/api/event` on my domain, proxies themto Plausible, and lets everything else pass through to Pages.In the Cloudflare ecosystem, when in doubt, just throw another worker on thepile...But I actually kind of liked this pattern. I ended up making this its ownproject and set it up as a GitHub template:- [plausible-cf-worker](https://github.com/georgemandis/plausible-cf-worker).![Screenshot of the plausible-cf-worker GitHub repo page](https://georgemandis.s3-us-west-1.amazonaws.com/logexplorer/plausible-cf-worker-screenshot.png)The code is about 70 lines, and it does a few things beyond a naive proxy:- **Edge caches the script** so repeated page loads don't hit plausible.io at all- **Strips cookies** from event POST requests for privacy- **Forwards the client IP** via `X-Forwarded-For` so Plausible counts unique visitors correctly instead of seeing the Worker's IP- **Returns 404 for anything else** so it's not an open proxy to plausible.ioThe nice thing about a standalone Worker with route-based matching is that itworks independently of your hosting. It sits at the Cloudflare edge andintercepts matching requests before they reach your origin, whether that originis Pages, Netlify, or anything else. It started working on my live siteimmediately, even before I finished the Pages migration, because the DNS wasalready proxied through Cloudflare.This also means I can reuse it for any other site I proxy through Cloudflare.All I have to do is just add more routes to `wrangler.toml`:```tomlroutes = [ { pattern = \"george.mand.is/js/script*\", zone_name = \"mand.is\" }, { pattern = \"george.mand.is/api/event\", zone_name = \"mand.is\" }, { pattern = \"some-other-site.com/js/script*\", zone_name = \"some-other-site.com\" }, { pattern = \"some-other-site.com/api/event\", zone_name = \"some-other-site.com\" },]```One `npx wrangler deploy` and it's live.If you use Plausible and Cloudflare, feel free to copy the `plausible-cf-worker`template and tell me if it was useful.## The logging questionRight, the whole reason I initially did this.My first attempt was a Pages middleware that logged every request to an R2bucket. Structured JSON, organized by date and hour, the whole thing. It worked!I thought I was so clever avoiding the extra money Cloudflare wanted to chargerfor their Log Explorer offering (still less than Netlify, but why pay when youcan be clever?).And then I looked at my R2 dashboard an hour later and saw 6,900 Class Aoperations. At$4.50 per million operations, this was going to cost more per month than just buying Cloudflare's Log Explorer add-on ($1per GB ingested), which at my traffic levels would cost approximately nothing.So I deleted the middleware and bought Log Explorer. Sometimes the boring answeris the right one.And truthfully, Log Explorer offers much more intersting logs than what I washome-rolling myself with this approach and a great interface for exploring themwith. I can use their query builder to build filters along 125 differentparameters. I have a nice little saved query specifically for trackign pings onmy RSS feeds (the original question) as well as checking-in on who mightactually be looking at the plain-text versions of my blog posts.![Screenshot of Log Explorer with a custom filter for seeing who has viewed plain-text versions of my blog posts](https://georgemandis.s3-us-west-1.amazonaws.com/logexplorer/cloudflare-log-explorer.png)## Was it worth it?The site works the same as before—same build, same deploy flow, same content—buthere are the things I gained:- Access logs (finally!)- The Plausible proxy as a reusable standalone thing (fun)- Everything under one roof (DNS, CDN, hosting, workers, storage)- The option to do more interesting things later (D1 for search? Workers for dynamic features? Durable Objects for real-time shenanigans? GUess we'll see!)The things I learned:- Cloudflare's dashboard UX for distinguishing Workers from Pages is kind of confusing- `_redirects` compatibility between Netlify and Cloudflare Pages is close but not identical- Per-request R2 writes are expensive; think about [Class A operations](https://developers.cloudflare.com/r2/pricing/#class-a-operations) before you build a logging pipeline- A standalone Worker with route matching is a pretty clean pattern for proxying third-party services through your domain (i.e. [plausible-cf-worker](https://github.com/georgemandis/plausible-cf-worker))If you're on Netlify and considering a move, the migration is less scary than itseems. The Eleventy build doesn't change at all. Most of the work is shufflingNetlify-specific things (functions, proxy redirects) into their Cloudflareequivalents.", "date_published": "2026-05-13T00:00:00.000Z" }, { "id": "https://george.mand.is/2026/05/meet-patui-ms-paint-for-the-terminal-with-vim-controls/", "url": "https://george.mand.is/2026/05/meet-patui-ms-paint-for-the-terminal-with-vim-controls.txt", "title": "Meet PaTUI: MS Paint for the Terminal, with Vim Controls", "content_html": "Every once in a while, a revolutionary product comes along that changeseverything. One is very fortunate if they get to work on even _one_ of these intheir career.Today I'm proud to introduce three revolutionary products of this caliber. Thefirst is a state-of-the-art image editor channeling the elegance of MS Paint.The second is an unparalleled user-interface experience continuing thelong-standing, intuitive traditions of Vim. And the third is software leveragingthe blazingly performant, brilliantly designed programming language that isJavaScript.So that's three things: an image editor in the class of MS Paint. Anunimaginably intuitive user-interface paradigm rivaling Vim. A software productfinally realizing the raw horsepower and sensible typing decisions ofJavaScript._MS Paint. Vim. JavaScript._Are you getting it?![The PaTUI screen with an image of the author's face loaded](https://georgemandis.s3-us-west-1.amazonaws.com/patui/patui-screenshot-me.png)These are not three separate products. This is _one_ product, and we're callingit **PaTUI**.![The PaTUI screen with the word PaTUI drawn in a pixel-art-meets-Comic-Sans style logo](https://georgemandis.s3-us-west-1.amazonaws.com/patui/patui-splash.png)## Welcome to PaTUIHomage to[one of the most iconic product launches of this century](https://www.youtube.com/watch?v=5J-47F8Hrdw)aside, what is PaTUI?PaTUI is a terminal-based image editor. Load an image (PNG or JPEG, locally orvia URL), and it renders as colored block characters in your terminal. Thenpaint on it in ~~visual~~ \"paint\" mode, erase, fill, type text or apply retrofilters. When your masterpiece is complete you can export your work as a JPEG,PNG or ANSI art. In all exports, WYSIWYG.No, we don't dare impose the dogma of higher-fidelities on your artistic vision.What are we, Photoshop? _Please_.This isn't ~~an Arby's~~ Photoshop. This is _PaTUI_.## \"Zoom, Enhance\"Each pixel in the image is represented by block characters wrapped with ANSIescape codes to render colors. We're using \"True\" color with RGB escape codes(e.g. `echo -e \"\\e[48;2;255;0;255m \\e[0m\"`) so the vibrance of your originalimage always shines through.And, because the original image really is kept around in memory, you can use the\"zoom and enhance\" feature to zero in on as much or little detail as you need.![The PaTUI screen with the zoom feature engaged on the loaded image](https://georgemandis.s3-us-west-1.amazonaws.com/patui/patui-zoom.png)## What pixels want: Vim-based controlsIt has Vim-style modal controls (`i` for paint mode, `hjkl` to move, `dd` todelete a \"row\" of pixels, `yy` to yank, `u` to undo), a 16-color palette you canselect with `!@#$%^&*()`, an extended CSS-compatible color palette you canaccess with `:set color <cornflowerblue|salmon|rebeccapurple,etc>`, and commandslike `:w mona-lisa.png` and `:wq`.Now you can draw shapes and touch up pixels with the most intuitive controlsever invented in computing: arbitrary keys with a generous amount of `Shift`thrown in.You can always use `:help` for a more exhaustive list.![The PaTUI help screen shown when you type `:help`](https://georgemandis.s3-us-west-1.amazonaws.com/patui/patui-help-screen.png)**Vim motions on pixels.** `5j` moves down 5 rows. `dd` clears a row. `yy` yanksit, `p` pastes. `W` jumps to the next color boundary. `dG` deletes from cursorto bottom. If you know Vim, you already know how to navigate. If you don't knowVim, well, you're going to learn!**Retro palette filters.** `:palette gameboy` limits your image to the originalGame Boy's four shades of green. `:palette cga` gives you the CGA palette.`:dither` applies Floyd-Steinberg error diffusion dithering. Combine them:`:palette gameboy` then `:dither` and suddenly your photo looks like it belongson a 1989 handheld.**Find-and-replace for colors.** `:%s/blue/red/g` replaces all blue pixels withred. `:%s/~blue/red/g` does a fuzzy match -- anything in the blue family. TheVim regex muscle memory just... works here.**Text rasterization.** Press `t` to enter text mode and type charactersdirectly onto the image in the current foreground color. Font size scales withbrush size. It's exactly as janky and charming as it sounds.**Export to ANSI art.** `:w painting.ans` exports your work as ANSI escapecodes. `:wc` copies the ANSI art to your clipboard. Paste it into a terminal andit renders in color. Paste it into Slack and confuse your coworkers.## Wh...Why? Why the terminal? Why any of this?There's something satisfying about creative tools that work in environmentsdesigned for text. The terminal gives you a grid of cells, each of which candisplay a colored block character. That's your canvas. Each cell is a pixel. Theconstraints are the point.It's also just funny. The idea of bringing Vim motions to pixel art, of typing`:wq` to save a painting, of having a tool sidebar in a terminal -- it's absurdin a way that makes me happy to work on it.## How it worksPaTUI is a [Bun](https://bun.sh) app built with[Ink](https://github.com/vadimdemedes/ink), which is React for terminal UIs.Image loading and manipulation use [sharp](https://sharp.pixelplumbing.com/).State management is [zustand](https://github.com/pmndrs/zustand).The rendering pipeline: load an image with sharp, downscale it to fit theterminal viewport (accounting for the 2:1 aspect ratio of terminal characters),map each pixel to a 256-color ANSI escape code, and render it as a grid of `▀`(upper half block) characters. Each character encodes two vertical pixels usingforeground and background colors.Edits modify the source image buffer. Undo/redo is a stack of image snapshots.Filters (grayscale, palette limiting, dithering) are applied at render time andincluded in exports.## How do you pronounce PaTUI?It sounds like \"patooey,\" because that's what your images will look like.## Try It```bash# Homebrew (macOS / Linux)brew install georgemandis/tap/patui# Or from sourcebun install && bun src/index.tsx mona.png```Is it practical? Absolutely not. Is it fun? Easily the most fun 5 minutes you'llprocrastinate with today.Built during my time at the [Recurse Center](https://www.recurse.com/) on alark. View the source and open a PR if you think you can help improve it.- [github.com/georgemandis/patui](https://github.com/georgemandis/patui)", "date_published": "2026-05-06T00:00:00.000Z" }, { "id": "https://george.mand.is/2026/04/soft-launching-little-irons/", "url": "https://george.mand.is/2026/04/soft-launching-little-irons.txt", "title": "Soft-launching Little Irons", "content_html": "I built a tool to help people keep track of their job applications. In an instance of domain-name-driven application development, the project is called Little Irons ([littleirons.com](https://littleirons.com))—a tool for helping you keep track of \"all your little irons in the fire.\"![The logo for Little Irons—a job-search tool for tracking all your 'little irons' in the fire.](https://littleirons.com/icons-logo-nobg.png?__frsh_c=11e1bff9746d5f6719ccb0dfa3b55f4c7767244f)It's free, it's early, and I would love feedback: [https://littleirons.com](https://littleirons.com)The inspiration was an ancient Google Sheet I'd used forever as part of my own job + opportunity explorations. I wanted to see if I could run with this as the base and add nicer features on top inspired by software I enjoy (like Linear) and some thoughtful affordances and \"AI flourishes\" sprinkled on top, including:- A browser extension to save job descriptions from any site for opportunities you are exploring.- An AI-powered parser that can extract all the important info (title, company, pay, location, job description, etc.) from the posting itself—all you have to do is provide the URL.- Company research with citations, salary data relative to role + industry, and personalized interview preparation based on the role, company, and your experience (if you've uploaded a resume or CV).- An ICS calendar feed you can subscribe to in any software (Google, Apple, Microsoft) that shows upcoming steps for any opportunities you've created events for.- An email assistant you can forward emails to—whether it's opportunities you're interested in or followups for jobs you've applied to. It will smartly know whether to add it as a new job you're \"exploring,\" update the status of an opportunity in motion—even a tragic rejection—or proactively \"schedule\" a new event in the sequence, like a final call with the hiring manager or an on-site.The heart of the tool is a kanban view of all the job applications you have in motion, with the columns aligning to an overall \"status\" for that particular opportunity. The furthest left column labeled \"Exploring\" is a place to put any jobs you're interested in applying to. The \"Applied\" column to the right is, hopefully, self-explanatory. Any columns further to the right are for when you move forward with the process.![Little Irons kanban board showing job applications organized by pipeline stage](https://s3.us-west-2.amazonaws.com/george.mand.is/little-irons/kanban-board.png)In my experience there is often enough a distinction between the initial screening and the rest of the interview process that it's worth ratifying these as proper top-level statuses. The interview process itself though can take many forms and even multiple phases, from one-on-one conversations with stakeholders to panel-style interviews. This is where the individual \"events\" associated with an opportunity come into play.Click any job to open its detail panel without leaving the board. You get the full picture: status, salary, location, relevant skill tags, and collapsible sections for upcoming events, contacts at the company, the full job description, your notes, and attached documents. The \"Next Actions\" section surfaces what you need to do next so nothing falls through the cracks.![Little Irons job detail side panel showing a Systems Administrator position at City Tech Services](https://s3.us-west-2.amazonaws.com/george.mand.is/little-irons/job-detail-panel.png)Adding or editing a job happens through a single modal—fill in the basics and expand the Contacts and Events sections to track who you're talking to and what's scheduled. The Job Description tab stores the full posting text for easy reference later.![Little Irons edit job modal with fields for title, URL, salary, status, events, and contacts](https://s3.us-west-2.amazonaws.com/george.mand.is/little-irons/edit-job-modal.png)Beyond the board, there are a few other ways to look at your search. The Calendar gives you a monthly overview of your activity—color-coded dots mark application dates, follow-ups, and scheduled events like interviews. Click any date to see what happened that day. You can also generate an iCalendar feed to sync everything into Google Calendar, Outlook, or Apple Calendar.![Little Irons calendar view for April 2026 showing application dates and follow-ups](https://s3.us-west-2.amazonaws.com/george.mand.is/little-irons/calendar-view.png)The Stats page turns your job search into numbers. Top-line metrics show total jobs tracked, applications sent, response rate, and offer rate. A pipeline funnel breaks down how many jobs sit at each stage, and an outcomes section tallies your offers, rejections, withdrawals, and ghostings. The activity chart tracks your momentum over the last 30 days—helpful for keeping yourself accountable during a long search.![Little Irons stats dashboard showing pipeline metrics, response rates, outcomes, and activity chart](https://s3.us-west-2.amazonaws.com/george.mand.is/little-irons/stats-dashboard.png)The Timeline presents your job search as a chronological story—every status change appears as an entry on an alternating feed with color-coded dots. It's a useful way to look back and see how active you've been or spot patterns in your pipeline.![Little Irons timeline view showing a chronological feed of job search events](https://s3.us-west-2.amazonaws.com/george.mand.is/little-irons/timeline-view.png)And for people who prefer spreadsheets, the List view shows everything in a sortable table. Click any column header to sort, filter by status, or select multiple jobs for bulk actions.![Little Irons list view showing a sortable table of all tracked jobs with a status filter dropdown](https://s3.us-west-2.amazonaws.com/george.mand.is/little-irons/list-view.png)The UI is admittedly a bit rough around the edges—something I plan on improving as I get more users to test with. Press Cmd/Ctrl+K to open the Linear-inspired command palette to quickly search jobs, jump between views, change statuses, or create new entries without reaching for the mouse.![Little Irons command palette showing quick actions for searching jobs, navigating views, and filtering](https://s3.us-west-2.amazonaws.com/george.mand.is/little-irons/command-palette.png)To be clear: this is not intended to be an automated bulk-application tool. It's a simple, focused place to try and organize a real job search.The site is completely free—all features, including the AI ones—and all you need to join is a GitHub account. If you find it useful and want to support ongoing development, I have a [GitHub Sponsors page](https://github.com/sponsors/georgemandis).Give it a shot and let me know what you think!", "date_published": "2026-04-15T00:00:00.000Z" }, { "id": "https://george.mand.is/2025/09/more-dynamic-cronjobs/", "url": "https://george.mand.is/2025/09/more-dynamic-cronjobs.txt", "title": "More dynamic cronjobs", "content_html": "I remember learning about cronjobs in the early 2000s. I could tell the computer to go _do_ something, on a recurring basis, forever, even when I wasn't there. They felt like magic!We didn't have [Crontab.guru](https://crontab.guru) or AI to ask for figuring out some of the more complex specifications. Just the [man pages](https://man.openbsd.org/crontab.5) and good old-fashioned trial and error—mostly error in my case.But while you could do fun, complex specifications of recurring intervals, you couldn't quite specify something quite as dynamic as \"run this script every Tuesday at 7am _unless it's the last Tuesday_ of the month...\"Or at least, you couldn't strictly through the crontab specification syntax. But I had a recent, mildly embarrassing epiphany that it's not hard at all to add arbitrary checks to your crontab to account for more complex and dynamic scenarios.Want to run a script every Tuesday of the month at 7am _except_ for the last Tuesday? That's easy—set up your crontab to run every Tuesday at 7am, but add a little check to make sure the _next_ week is still part of the same month:```shell0 7 * * Tue [ \"$(date -v+7d '+%m')\" = \"$(date '+%m')\" ] && /path/to/your_command```If it's not part of the same month, that means we're on the _last_ Tuesday for the month and the script won't run.**Note:** *The `-v` flag is for the macOS/BSD flavors of `date`. On Linux you'd want to use `-d +7 days` instead.*This really has nothing to do with cronjobs at all and everything to do with the [POSIX \"test\" command](https://www.unix.com/man_page/posix/1p/test/) which is the thing we're using with those square brackets. I'm used to seeing and utilizing them in shell scripts, but for whatever reason I never thought to reach for that tool here in the crontab.You could just as easily rewrite it like this, skipping the bracket shorthand, which is probably easier to read:```shell0 7 * * Tue test \"$(date -v+7d '+%m')\" = \"$(date '+%m')\" && /path/to/your_command```It never crossed my mind until recently to add slightly more complex checks at the crontab level.### Other clever cronjob things you can do:#### Holiday-only cronjobsMaybe fetch a list of all the US Holidays for a given year and store them in a handy `HOLIDAYS.txt` file somewhere:```shellcurl -s https://date.nager.at/api/v3/PublicHolidays/2025/US | jq -r '.[].date' > HOLIDAYS.txt```Now you can update your cronjob to run every Tuesday at 7am _except_ on Holidays:```shell0 7 * * Tue ! grep -qx \"$(date +%F)\" HOLIDAYS.txt && /path/to/your_command```Or inversely, maybe run a holiday-only script that checks once a day```shell@daily grep -qx \"$(date +%F)\" HOLIDAYS.txt && /path/to/your_special_holiday_command```#### Only run on sunny daysThe [National Weather Service](https://weather.gov) makes all kinds of fun data available (if you can find it...). How about a script that runs every hour, but only when the weather is clear?```shell@hourly curl -s \"https://api.weather.gov/gridpoints/TOP/32,81/forecast/hourly\" | jq -r '.properties.periods[0].shortForecast' | grep -qi clear && /path/to/your_command```Or maybe when the weather is cloudy?```shell@hourly curl -s \"https://api.weather.gov/gridpoints/TOP/32,81/forecast/hourly\" | jq -r '.properties.periods[0].shortForecast' | grep -qi cloudy && /path/to/your_command```#### Only run when there's something newsworthyOr maybe we get in line with every-other-startup I'm aware of and throw AI at the problem, only running our script when the LLM gods have decided there is something newsworthy:```shell@hourly curl -s \"https://news.google.com/rss?hl=en-US&gl=US&ceid=US:en\" | llm --system \"Reply strictly 'yes' or 'no'. Does anything in the news today suggest it is a good reason to run a script that I only want to send when the world is on fire and crazy and terrible things are happening?\" | tr -d '[:space:]' | tr '[:upper:]' '[:lower:]' | grep -qx yes && /path/to/oh_no```", "date_published": "2025-09-21T00:00:00.000Z" }, { "id": "https://george.mand.is/2025/07/building-a-go-link-toy-with-deno/", "url": "https://george.mand.is/2025/07/building-a-go-link-toy-with-deno.txt", "title": "Building a Go Link Toy with Deno", "content_html": "![Screenshot of my Go Links Web Manager tool](https://georgemandis.s3-us-west-1.amazonaws.com/go-links-window.png)For some reason I've been thinking about [go links](https://meta.wikimedia.org/wiki/Go_links), which are really just a fancy word for link shorteners. They're useful at larger companies for providing static, easy-to-remember URLs that can take employees directly to pages for benefits explanations, requesting time-off, internal documentation and all sorts of things.Some companies go the extra mile and make these links available on aesthetically pleasing URLs that only work when you're plugged into the corporate network. Something like:- <a href=\"http://go/payroll\">http://go/payroll</a>- <a href=\"http://go/holidays\">http://go/holidays</a>- <a href=\"http://go/metrics\">http://go/metrics</a>- <a href=\"http://go/evilmoonbaseplans\">http://go/evilmoonbaseplans</a>If you work at a company utilizing go links, there's a chance these links might even work for you! If that last link works, you might want to consider finding a new job.## Projects > SolutionsI wanted a simple personal go link system I could run on my machine, mostly for fun. I decided I wanted to make my own go link implementation just for me, local only to my network and perhaps even just my particular machine.You might be thinking _\"Why not just bookmark the links in your browser?\"_ to which I would say _\"Hey, I'm looking for a Sunday afternoon project, not a solution!\"_Other requirements:- **`go/whatever`** URLs, because they feel fancy- A cute little web-interface I can navigate to for managing the URLs, shortcodes- A CLI-driven component to the tool so I could manage and even access the links from the shell by invoking `golinks [shortcode]`- Some basic stat tracking every time I use one of these, just becauseIf you want to cut to the chase and play with the toy I built, check out the repo:- [github.com/georgemandis/golinks](https://github.com/georgemandis/golinks)I also used this as an opportunity to continue exploring [Deno](https://deno.com). I've been intrigued with it ever since it came on the scene, and I've found it particularly nice for building little one-off CLI toys.Another fun aspect is you can run the following and install my `golinks` tool globally:```bashdeno install --global --allow-read --allow-write --allow-env --allow-net --allow-run jsr:@georgemandis/golinks```I [published the package](https://jsr.io/@georgemandis/golinks) on JSR just to get a feel for what's going on over there. Might save that for a different blog post.## Fancy URLsHalf the fun is having those fancy TLD-less URLs. I elaborate a little bit in the README for the repo, but it boils down to adding additional aliases for the loopback address (i.e. `localhost`) on your machine.To do that in Unix and MacOS environments all you have to do is add a row to your `/etc/hosts` file:```echo \"127.0.0.1 go\" | sudo tee -a /etc/hosts```Note, `tee` command is just a nice way to write our input to a file and stdout at the same time. It’s not a command I use often, so I thought I’d explain it.## Running the ServerFor me, I'm happy to just run it in the background and forget about it:```golinks --server &```I always have my [terminal](https://ghostty.org) open, so this very lazy solution is Good Enough™ for me. The `&` runs it as a background process and if I want to stop it I can `ps -e | grep \"golink\"` it, find the PID and `kill -9` it.But a few more fun options are available if you want to be less lazy:- **Set it up as a proper service at launch**. I actually went down this route, implemented it and decided I _didn't_ want to commit. Maybe it's irrational, but I like keeping my at-launch processes as clean and minimal as possible.- **Run it on a second computer on my network.** This is actually my favorite approach, conceptually, and combined with [Tailscale](https://tailscale.com) it's surprisingly powerful. I have my \"under the bed\" computer that handles some home automation and other projects. I can run `golinks` on this machine and map the Tailscale IP address to `http://go` in my `/etc/hosts` file. Neat!## Future ConsiderationsLike any good project, this one has plenty of rabbit holes to fall into. Here are some other ways I could continue to invest in the project:- I'm really bothered by the \"insecure\" nature of these links. Getting SSL certificates working for local URLs was a pain last I checked, and _many_ years ago I had a script to automate this. I'm pretty sure no longer works.- When in doubt, add AI. I actually added the \"description\" field with this somewhat in mind. It wouldn't take very much to open up the list of shortened links to an LLM and start querying over it.I don't foresee myself investing more time in this unless I really find it useful or people start contributing to/forking/using it, but it was a fun Sunday afternoon project.", "date_published": "2025-07-13T00:00:00.000Z" }, { "id": "https://george.mand.is/2025/06/openai-charges-by-the-minute-so-make-the-minutes-shorter/", "url": "https://george.mand.is/2025/06/openai-charges-by-the-minute-so-make-the-minutes-shorter.txt", "title": "OpenAI Charges by the Minute, So Make the Minutes Shorter", "content_html": "Want to make OpenAI transcriptions faster and cheaper? Just speed up your audio.I mean that very literally. Run your audio through [ffmpeg](https://gist.github.com/georgemandis/4fd62bf5027b7a058f913d5dc32c2040) at 2x or 3x before transcribing it. You’ll spend fewer tokens and less time waiting with almost no drop in transcription quality.That’s it!Here’s a script combining of all my favorite little toys and tricks to get the job. You’ll need [yt-dlp](https://github.com/yt-dlp/yt-dlp), [ffmpeg](https://ffmpeg.org) and [llm](https://github.com/simonw/llm) installed.```bash# Extract the audio from the videoyt-dlp -f 'bestaudio[ext=m4a]' --extract-audio --audio-format m4a -o 'video-audio.m4a' \"https://www.youtube.com/watch?v=LCEmiRjPEtQ\" -k;# Create a low-bitrate MP3 version at 3x speedffmpeg -i \"video-audio.m4a\" -filter:a \"atempo=3.0\" -ac 1 -b:a 64k video-audio-3x.mp3;# Send it along to OpenAI for a transcriptioncurl --request POST \\ --url https://api.openai.com/v1/audio/transcriptions \\ --header \"Authorization: Bearer $OPENAI_API_KEY\" \\ --header 'Content-Type: multipart/form-data' \\ --form file=@video-audio-3x.mp3 \\ --form model=gpt-4o-transcribe > video-transcript.txt;# Get a nice little summarycat video-transcript.txt | llm --system \"Summarize the main points of this talk.\"```I just saved you time by jumping straight to the point, but read-on if you want more of a story about how I accidentally discovered this while trying to summarize a 40-minute talk from Andrej Karpathy.Also read-on if you’re wondering why I didn’t just use the built-in auto-transcription that YouTube provides, though the short answer there is easy: I’m sort of a doofus and thought—incorrectly—it wasn’t available. So I did things the hard way.### I Just Wanted the TL;DW(atch)A former colleague of mine sent me [this talk](https://www.youtube.com/watch?v=LCEmiRjPEtQ) from Andrej Karpathy about how AI is changing software. I wasn’t familiar with Andrej, but saw he’d worked at Tesla. That coupled with the talk being part of a Y Combinator series and 40 minutes made me think “Ugh. Do I… really want to watch this? Another 'AI is changing everything' talk from the usual suspects, to the usual crowds?”If ever there were a use-case for dumping something into an LLM to get the gist of it and walk away, this felt like it. I respected the person who sent it to me though and wanted to do the noble thing: use AI to summarize the thing for me, blindly trust it and engage with the person pretending I had watched it.My first instinct was to pipe the transcript into an LLM and get the gist of it. [This script](https://gist.github.com/simonw/9932c6f10e241cfa6b19a4e08b283ca9) is the one I would previously reach for to pull the auto-generated transcripts from YouTube:```bashyt-dlp --all-subs --skip-download \\ --sub-format ttml/vtt/best \\ [url]```For some reason though, no subtitles were downloaded. I kept running into an error!Later, after some head-scratching and rereading [the documentation](https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#subtitle-options), I realized my version (2025.04.03) was outdated.**Long story short**: Updating to the latest version (2025.06.09) fixed it, but for some reason I did not try this _before_ going down a totally different rabbit hole. I guess I got this little write-up and exploration out of it though.If you care more about summarizing transcripts and less about the vagaries of audio-transcriptions and tokens, this is the correct answer and your off-ramp.### My Transcription WorkflowI already had an old, home-brewed script that would extract the audio from any video URL, pipe it through [whisper](https://github.com/openai/whisper) locally and dump the transcription in a text file.That worked, but I was on dwindling battery power in a coffee shop. Not ideal for longer, local inference, mighty as my M3 MacBook Air still feels to me. I figured I would try offloading it to [OpenAI’s API](https://platform.openai.com/docs/guides/speech-to-text) instead. Surely that would be faster?### Testing OpenAI’s Transcription ToolsOkay, using the `whisper-1` model it’s _still_ pretty slow, but it gets the job done. Had I opted for the model I knew and moved on, the story might end here.However, out of curiosity, I went straight for the newer `gpt-4o-transcribe` model first. It’s built to handle multimodal inputs and promises faster responses.I quickly hit another roadblock: there’s a 25-minute audio limit and my audio was nearly 40 minutes long.### Let's Try Something ObviousAt first I thought about trimming the audio to fit somehow, but there wasn’t an obvious 14 minutes to cut. Trimming the beginning and end would give me a minute or so at most.An interesting, weird idea I thought about for a second but never tried was cutting a chunk or two out of the middle. Maybe I would somehow still have enough info for a relevant summary?Then it crossed my mind—**what if I just sped up the audio before sending it over?** People listen to podcasts at accelerated 1-2x speeds all the time.So I wrote a [quick script](https://gist.github.com/georgemandis/4fd62bf5027b7a058f913d5dc32c2040):```bashffmpeg -i video-audio.m4a -filter:a \"atempo=2.0\" -ac 1 -b:a 64k video-audio-2x.mp3```Ta-da! Now I had something closer to a 20 minute file to send to OpenAI.I uploaded it and… it worked like a charm! [Behold the summary](https://gist.github.com/georgemandis/b2a68b345262b94782fa6b08e41fbcf2) bestowed upon me that gave me enough confidence to reply to my colleague as though I had watched it.But there was something... interesting here. Did I just stumble across a sort of obvious, straightforward hack? Is everyone in the audio-transcription business already doing this and am I just haphazardly bumbling into their secrets?I had to dig deeper.### Why This Works: Our Brains Forgive, and So Does AIThere’s an interesting parallel here in my mind with optimizing images. Traditionally you have lossy and lossless file formats. A lossy file-format kind of gives away the game in its description—the further you crunch and compact the bytes the more fidelity you’re going to lose. It works because the human brain just isn’t likely to pick-up on the artifacts and imperfectionBut even with a “lossless” file format there are tricks you can lean into that rely on the limits of human perception. One of the primary ways you can do that with a PNG or GIF is reducing the number of unique colors in the palette. You’d be surprised by how often a palette of 64 colors or fewer might actually be enough and perceived as significantly more.There’s also a parallel in my head between this and the brain’s ability to still comprehend text with spelling mistakes, dropped words and other errors, i.e. [transposed letter effects](https://en.wikipedia.org/wiki/Transposed_letter_effect). Our brains have a knack for filling in the gaps, and when you go looking through the world with magnifying glass you'll start to notice lots of them.Speeding up the audio starts to drop the more subtle sounds and occasionally shorter words from the audio, but it doesn’t seem to hurt my ability to _comprehend_ what I’m hearing—even if I do have to focus. These audio transcription models seem to be pretty good at this as well.### Wait—how far can I push this? Does It Actually Save Money?Turns out yes. OpenAI [charges for transcription](https://platform.openai.com/docs/pricing) based on audio tokens, which scale with the duration of the input. Faster audio = fewer seconds = fewer tokens.Here are some rounded numbers based on the 40-minute audio file breaking down the audio input and text output token costs:| Speed | Duration (seconds) | Audio Input Tokens | Input Token Cost | Output Token Cost || ------------- | ------------------ | ------------------ | ---------------- | ----------------- || 1x (original) | 2,372 | NA (too long) | NA | NA || 2x | 1,186 | 11,856 | $0.07 | $0.02 || 3x | 791 | 7,904 | $0.04 | $0.02 |That’s a solid 33% price reduction on input tokens at 3x! However the bulk of your costs for these transcription models are still going to be the output tokens. Those are priced at $10 per 1M tokens whereas audio input tokens are priced at $6 per 1M token as of the time of this writing.Also interesting to note—my output tokens for the 2x and 3x versions were exactly the same: 2,048. This kind of makes sense, I think? To the extent the output tokens are a reflection of that model's ability to understand and summarize the input, my takeaway is a “summarized” (i.e. reduced-token) version of the same audio yields the same amount of comprehensibility.This is also probably a reflection of the 4,096 token ceiling on transcriptions generally when using the `gpt-4o-transcription` model. I suspect half the context window is reserved for the output tokens and this is basically reflecting our request using it up in its entirety. I suspect we might get diminishing results with longer transcriptions.But back to money.So the back-of-the-envelope calculator for a single transcription looks something like this:```text6 * (audio_input_tokens / 1_000_000) + 10 * (text_output_tokens / 1_000_000);```That does _not_ quite seem to jibe with the estimated cost of $0.006 per minute stated on the pricing page, at least for the 2x speed. That version (19-20 minutes) seemed to cost about $0.09 whereas the 3x version (13 minutes) cost about $0.07 (pretty accurate actually), if I’m adding up the tokens correctly.```text# Pricing for 2x speed6 * (11_856 / 1_000_000) + 10 * (2_048 / 1_000_000) = 0.09# Pricing for 3x speed6 * (7_904 / 1_000_000) + 10 * (2_048 / 1_000_000) = 0.07```It would seem that estimate isn’t just based on the length of the audio but also some assumptions around how many tokens per minute are going to be generated from a normal speaking cadence.That’s… kind of fascinating! I wonder how [John Moschitta’s](https://en.wikipedia.org/wiki/John_Moschitta_Jr.) feels about this.Comparing these costs to `whisper-1` is easy because the pricing table more confidently advertises the cost—not “estimated” cost—as a flat $0.006 per minute. I’m assuming that’s minute of audio processed, not minute of inference.The `gpt-4o-transcription` model actually compares pretty favorably.| Speed | Duration | Cost || ----- | ------------ | ----- || 1x | 2372 | $0.24 || 2x | 1186 seconds | $0.12 || 3x | 791 seconds | $0.08 |### Does This Save Money?In short, yes! It’s not particularly rigorous, but it seems like we reduced the cost of transcribing our 40-minute audio file by 23% from $0.09 to $0.07 simply by speeding up the audio.If we could compare to a 1x version of the audio file trimmed to the 25-minute limit, I bet we could paint an even more impressive picture of cost reduction. We kind of can with the `whisper-1` chart. You could make the case this technique reduced costs by 67%!### Is It Accurate?I don’t know—I didn’t watch it, lol. That was the whole point. And if that answer makes you uncomfortable, buckle-up for this future we're hurtling toward. Boy, howdy.More helpfully, I didn’t compare word-for-word, but spot checks on the 2x and 3x versions looked solid. 4x speed was too fast—the transcription started getting hilariously weird. So, 2x and 3x seem to be the sweet spot between efficiency and fidelity, though it will obviously depend on how fast the people are speaking in the first place.### Why Not 4x?When I pushed it to 4x the results became [comically unusable](https://gist.github.com/georgemandis/1ec4ef084789f92ee06ac6283338a194).![Output of a 4x transcription mostly repeating \"And how do we talk about that?\" over and over again](https://georgemandis.s3-us-west-1.amazonaws.com/4x-speed-transcription-min.png)That sure didn't stop my call to summarize from [trying](https://gist.github.com/georgemandis/1ec4ef084789f92ee06ac6283338a194#file-summarization-md) though.Hey, not the worst talk I've been to!### In SummaryAlways, in short, to save time and money, consider doubling or tripling the speed of the audio you want to transcribe. The trade-off is, as always, fidelity, but it’s not an insignificant savings.Simple, fast, and surprisingly effective.### TL;DR- OpenAI charges for transcriptions based on audio duration (`whisper-1`) or tokens (`gpt-4o-transcribe`).- You can **speed up audio** with `ffmpeg` before uploading to save time and money.- This reduces audio tokens (or duration), lowering your bill.- **2x or 3x speed** works well.- **4x speed**? Probably too much—but fun to try.If you find problems with my math, have questions, found a more rigorous study qualitatively comparing different output speeds please [get in touch](https://george.mand.is/contact)! Or if you thought this was so cool you want to [hire me](https://george.mand.is/hire) for something fun...<style>table:not(.hljs-ln) { border:1px rgba(0,0,0, 0.5) solid; min-width: 100%; background:rgba(255,255,255,0.9); color:#000; font-family: system-ui; font-size:0.7rem;}table:not(.hljs-ln) thead tr { background: rgba(0,0,0, 0.05);}</style>", "date_published": "2025-06-24T00:00:00.000Z" } ] }