Keith Schacht's Weblog: AI

Remember Clippy: Screen-aware voice AI in the browser

2026-02-12T21:31:05+00:00

A friend and I built a browser prototype that answers questions about whatever’s on your screen using getDisplayMedia, client-side wake-word detection, and server-side multimodal inference.

Try it here: clippy.keithschacht.com
Best in Chrome. Desktop only. No sign up.

Hard parts:

Getting the model to point to specific UI elements
Keeping it coherent across multi-step workflows (“Help me create a sword in Tinkercad”)
Preventing the infinite mirror effect and confusion between window vs full-screen sharing
Keeping voice → screenshot → inference → voice latency low enough to feel conversational

We packaged it as “Clippy” for fun, but the real experiment is letting a model tool-call fresh screenshots to help it gather more context.

One practical use case is remote tech support — I'm sending this to my mom next time she calls instead of screen sharing.

Comment on HN discussion
Email me: krschacht at gmail
Subscribe — updates on this + other AI experiments

Tags: AI

Task Master: Voice-first todo list that updates live as you talk

2025-12-27T15:34:46+00:00

I built a demo of a voice AI task manager. You speak naturally and it updates your visible task list in real time.

Try it here: taskmaster.keithschacht.com
Web-based. Desktop or mobile. No sign up.

I find it helpful to talk aloud to figure out my priorities. I’ve wired up voice AI to many daily routines and this task manager is one of the most useful. In the morning, I sit down at my computer with a cup of coffee, pull up Task Master, and start rambling:

“Mark that first task as done. Actually, undo that. Add a task to proofread it one more time. Move that to the top. Snooze the next two until tomorrow …”

I built this to explore AI UI. My key observations:

I grew up with sci-fi characters talking to computers and wanted to test whether that interaction actually works in practice.
Voice is great for input but poor for output beyond short responses; visual feedback has much higher bandwidth.
Speaking is 2–3× faster than typing, and LLMs work great when you can talk in a loose, stream-of-consciousness way.
We’re in the command-line era of LLM interfaces.

It’s built on LiveKit with a Rails web UI. It listens continuously, maps speech to tool calls, and operates with the full task-list state so it can make sense of ambiguous references (e.g. “the third item,” “the thing with my kids”).

This is intentionally rough and incomplete, it’s a demo not a production app. Tasks are saved server-side, and it’s tied to your anonymous browser session. My personal version of this app has dates, task descriptions, and the ability to snooze items. I focused this demo on the core interactions with the goal of making it feel polished and smooth so people could try it. I’m interested in feedback on the interaction model rather than feedback on this as a product.

Code is here: github.com/keithschacht/taskmaster

I’m especially curious about:

For you, does speaking feel meaningfully faster or more fluid than clicking and typing?
When you make a mistake or want to edit, does correcting it feel natural?
Want to collaborate collaborating on building personal AI tools?

Comment on HN discussion
Email me: krschacht at gmail
Subscribe — updates on this + other AI experiments

Tags: AI

Quoting Dario’s interview with Lex Friedman

2025-02-09T22:34:27+00:00

Dario from Anthropic is the most articulate and thoughtful person I’ve found on the subject of AI. In this interview he made a fantastic point I had not heard:

Let’s say it was 1995 and Moore’s Law was making computers faster and everyone was saying, “some day we’ll have super computers and we’ll be able to sequence the genome and do all these great things.” But there is no discrete point where we pass a threshold and then have super computers.

Super computers is a term we use but it’s a vague term we use to describe computers which are a lot faster than what we have today. I feel the same way about AGI. There is a smooth exponential. And if, by AGI, you mean AI is getting better and better and will do more and more of what humans do until it’s smarter than humans and it will continue to get smarter from there, then I believe in AGI. But if AGI is some discrete thing, which is how many people talk about it, then it’s just a meaningless buzzword.

But he then goes on to reference his essay in which he gives one of the best descriptions of an almost discrete point I’ve read! He doesn’t call this AGI but instead uses a great descriptor:a country of geniuses in a datacenter:

In terms of pure intelligence4, it is smarter than a Nobel Prize winner across most relevant fields – biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.

In addition to just being a “smart thing you talk to”, it has all the “interfaces” available to a human working virtually, including text, audio, video, mouse and keyboard control, and internet access. It can engage in any actions, communications, or remote operations enabled by this interface, including taking actions on the internet, taking or giving directions to humans, ordering materials, directing experiments, watching videos, making videos, and so on. It does all of these tasks with, again, a skill exceeding that of the most capable humans in the world.

It does not just passively answer questions; instead, it can be given tasks that take hours, days, or weeks to complete, and then goes off and does those tasks autonomously, in the way a smart employee would, asking for clarification as necessary.

It does not have a physical embodiment (other than living on a computer screen), but it can control existing physical tools, robots, or laboratory equipment through a computer; in theory it could even design robots or equipment for itself to use. The resources used to train the model can be repurposed to run millions of instances of it (this matches projected cluster sizes by ~2027), and the model can absorb information and generate actions at roughly 10x-100x human speed5. It may however be limited by the response time of the physical world or of software it interacts with.

Each of these million copies can act independently on unrelated tasks, or if needed can all work together in the same way humans would collaborate, perhaps with different subpopulations fine-tuned to be especially good at particular tasks.

— Dario’s interview with Lex Friedman

Tags: AI

Study: Waymo robocars are safer than human drivers

2025-01-07T18:59:03+00:00

Study: Waymo robocars are safer than human drivers

An insurance company reported that Waymo has logged 25 mmillion fully autonomous miles and showed a 88% reduction in property damage claims and a 92% reduction in bodily injury claims. In other words, if humans had driven these same miles there would have been 8x as many property damage claims and 13x as many bodily injury claims.

Via Daniel Miessler

Tags: AI

Vampire game based around AI voice: SUCK UP!

2024-11-26T15:47:33+00:00

Vampire game based around AI voice: SUCK UP!

This is one of the best examples I’ve seen of games creatively incorporating LLMs. You are a vampire character, you walk up to houses in a neighborhood and as the player you actually talk aloud to your computer and the NPCs talk back. You have to convince these NPCs to let you into their house.

Tags: AI

Jina AI tool for simplifying webpages

2024-11-03T20:49:03+00:00

Jina AI tool for simplifying webpages

This is a really handy tool for simplify a webpage before passing it into an LLM. It turns HTML into semi-structured markdown.

Tags: AI

Using LLM to process video

2024-11-03T20:11:46+00:00

Using LLM to process video

I have done a lot of experimenting with passing screenshots into an LLM to give it additional context, but I really want to try passing video directly into the LLM. I believe Gemini is the only one that supports it. This is a summary of Simon Willison’s recent experiment with Gemini for video.

Tags: AI

Automating app development with LLM

2024-11-03T19:37:19+00:00

Automating app development with LLM

I keep automating more of my day-to-day programming using various tools, and this guy did a clever wrap around LLM. He writes a unit test in python, passes it to LLM, and it writes the code necessary to get the test to pass. This technique could prove useful for what I want to do with rails development.

Tags: AI

Andressen and Horowitz on AI Robots

2024-11-03T18:59:49+00:00

Andressen and Horowitz on AI Robots

In this discussion between Andreessen and Horowitz, they pointed out that we are on the verge of embodied AI being a generally useful too. My key takeaway was this:

General purpose robots could do everything from be assistants in your home, build houses, fight wars, etc. This is being made possible by all the software advances in AI in addition to the hardware advances in actuators, batteries, and vision systems.

The U.S. currently leads on the software side of AI, but China has a significant lead on the hardware side. The U.S. has basically made manufacturing illegal in the U.S. through environmental regulation, minimum wage requirements, and unions so U.S. companies have outsourced more advanced hardware to China. Now more than ever, with our strained relationship with China, we need to build these capabilities in the U.S. and at the same time U.S. regulators have their eyes on regulating the software AI capabilities.

Tags: AI