A few months ago, Apple released FastVLM, a Visual Language Model (VLM) that offered near-instant processing of high-resolution images. Now, you can try it out if you have a Mac with Apple Silicon.
See also: All the news coming to Apple TV+ in September
When we first covered FastVLM, we explained that it leveraged MLX, Apple's own open machine learning framework designed specifically for Apple Silicon, to deliver up to 85x faster video captioning, while being more than 3x smaller than similar models.
Since then, Apple has further worked on the project, which can now be found on Hugging Face, not just on GitHub. On Hugging Face, you can load the lighter version, FastVLM-0.5B, directly from your browser and try it out for yourself.
Depending on your hardware, it may take a while to load. On a MacBook Pro M2 Pro 16GB, it took a few minutes. But once it loaded, the model began to accurately describe my appearance, the room behind me, various expressions, and objects in its field of view. You can customize the prompt that the model will consider as it updates the subtitle live, or you can choose from a few suggestions.
If you want to go further, you can try using a virtual camera app to feed video into the tool and watch it describe many scenes instantly with detail, at a point where it becomes difficult to understand what is happening. This highlights how fast and accurate the model can be.
What's particularly interesting about this experiment is that it runs locally in the browser, meaning no data ever leaves the device and it can even work offline. This would be a great use case for mobile devices and assistive technology, where lightweight and low latency will be paramount to unlocking better use cases.
See also: Apple vulnerability: PoC Exploit released for zero-day bug
It is worth noting that the demonstration runs on the lightest model with 0.5 billion parameters, while the FastVLM family also includes larger and more powerful variants with 1.5 billion and 7 billion parameters. With larger models, the performance and speed could be improved even further, although running it directly in the browser would probably not be feasible.
Apple's FastVLM (Fast Vision Language Model) is an innovative visual language model (VLM), designed for extremely fast and efficient image processing, with low latency and high accuracy , ideal for real-world applications on Apple devices such as iPhone, iPad and Mac Apple Machine Learning Research+1 .
The core of FastVLM is based on a new vision encoder called FastViTHD—a hybrid architecture that combines convolutional stages and transformer-based blocks. This allows it to process high-resolution images quickly and efficiently, drastically reducing the number of visual tokens and improving Time-to-First-Token (TTFT) speed without losing accuracy. Apple Machine Learning ResearcharXiv.
According to experiments, FastVLM‑0.5B is 85 times faster than LLaVA‑OneVision‑0.5B on TTFT, while its encoder is 3.4 times smaller, with similar performance in the Apple Machine Learning ResearcharXiv benchmarks. The larger versions (FastVLM‑7B, with LLM based on Qwen2‑7B) are 7.9 times faster than methods such as Cambrian‑1‑8B, while maintaining high accuracy fastvΙm.siteHugging FaceAibase News.
FastVLM manages to precisely balance input image resolution, latency , and accuracy, thanks to strategic encoder selection and its simplified architecture—without the need for complex techniques like token pruning or dynamic tiling Apple Machine Learning ResearcharXiv.
See also: Apple zero-day: New vulnerability found – Update now
Apple has released code and pre‑built checkpoints (0.5B, 1.5B, 7B) openly on GitHub, as well as a demo app for iOS/macOS via the MLX environment GitHubAibase News. This shows the company's strategy to promote on‑device processing, ensuring speed, user privacy and offline operation without cloud connection
