Local AI on NPUs: Building a zero-cloud photo deduplicator with OpenCLIP

$ python export_ov.py --batch 1

exporting image encoder to OpenVINO IR (static batch=1) ...
IR: ~/photo-dedup/dedup/models/clip-vitb32-b1.xml
input : [1,3,224,224] 
output: [1,512] 

The one-time export bakes the PyTorch encoder into a static-batch OpenVINO IR—one build per batch size, loaded by the NPU runs that follow.

$ sudo ./.venv/bin/python embed.py ./scan --backend npu

backend: NPU (compiled clip-vitb32-b1.xml, hint=LATENCY, batch=1)
preprocessing: /usr/sbin/magick (parallel, 4 workers)
prepared 373 crop(s) in 0.0s
[1/373] photo0000.png     26.1 ms -> photo0000.vitb32.npy
[2/373] photo0002.png     25.6 ms -> photo0002.vitb32.npy
[3/373] photo0020.png     24.9 ms -> photo0020.vitb32.npy
…
embedded 373 in 5.7s (65.3 img/s) [backend=npu, batch=1], 0 skipped (cached), avg power 27.3 W

A naive, synchronous NPU loop improves throughput to ~25 milliseconds per image, but leaves the accelerator idle between dispatch calls. Power consumption significantly decreases to 27W

To unlock the silicon’s true potential, I restructured the script to use batched inference and applied OpenVINO’s THROUGHPUT performance hint. By feeding the NPU batches of 16 images simultaneously, I saturated its execution units and eliminated the dispatch bottleneck. Per-photo inference time dropped from ~36 ms on the CPU to about 13 ms on the NPU, and end-to-end throughput climbed from roughly 28 photos per second (a serial, single-image pass) to over 92—a 3x win. But raw speed wasn’t the real victory—efficiency was. The NPU silently chewed through thousands of images in the background, drawing a mere 22W, allowing processing of a massive media archive on battery power without triggering the fans.