projects — bennytay

Creating a persistent memory layer for robotic deployments, aggregating context data (records, history, regulations) with physical data (MCAP files + ROS + URDF + logs etc). Like supermemory.ai but for robots. With more deployment data, the memory is automatically updated + can be used as a data flywheel.
Memory is aimed to be used for downstream applications (LLM agents, VLA policies, simulators, eval/training pipelines and deployment/compliance tooling).
Memory system = bi-temporal claim graph + episodic/spatial memory with immutable evidence and revision/invalidation semantics.

* still in progress *

Allows you to ask open-ended semantic questions while the drone is flying (“ChatGPT for drone video”), very applicable to industry + makes monitoring tools + surveillance process more accessible to non-technical operators.
Qwen3-VL-2B as pretrained base (4bit mode).
Frozen vision tower initially.
Image to language merger/projector + LoRA adapter is trained in Qwen’s attention and MLP layers.
Completion-only supervised fine tuning on structured aerial tasks.

extension of the aerial tracking system below · shared aero_vlm repository

Drones are great for surveillance / detection deployments but it’s hard to run advanced models on them because of limited onboard compute and battery concerns. I sought to build the lowest cost and most efficient onboard system possible.
Fine-tuning of YOLO26n on VisDrone for compute-constrained on-board detection.
Targeted MaixCAM2/AX630C inference; switchable ByteTrack and BoT-SORT tracking (depending on drone movement, if it’s just hovering vs moving).
This was the basis for my drone VLM project, which extended it on the same codebase.

base system for the VLM extension above · shared aero_vlm repository

Over 200 downloads in just over a month of publication.
World action models depend on training data that record agent-environment causality. Therefore, we mapped the current landscape by data through the lens of causal completeness, distilling it into three key tiers.
This is followed by an analysis of the saturation of the data collection economy, and discusses the problem of 'scaling in the dark' -> there is a lack of proper benchmarking / taxonomy that measures how well a dataset actually improves model performance.

“Mercor / Airtasker for physical data” -> phone-first egocentric robotics data marketplace to scale robot model training, and people earn money off of recording themselves.
Raw footage is fused with LiDAR & IMU sensor measurements and subjected to pre-processing with hand pose estimation, instance segmentation, object detection, transcripts, self-narration to capture human reasoning, and 3D Gaussian splats, natural language search through videos.
Winner @ hack48 hackathon.

We scan your whole photo library to understand context about you and your relationship with your friends - algorithm automatically recommends events to do together.
Perception model + inference engine + on-device language model (all of this is completely on-device for privacy).
Server-side = public event data + crash reports.

First-of-its-kind ‘make a bet on anything’ social prediction market where you gamble with virtual tokens and have group leaderboards.
Added an AI evidence arbiter for bet-resolution support.
Winner @ UNSW DevSoc × Atlassian hackathon.

Returns risk reports on business targeted towards Tranche 2 requirements (extremely complicated + time-consuming task for SMEs).
Distilled this entire process into an easy-to-use automated + AI-assisted app that only takes 2-3 hours to complete (vs weeks of work + cost of hiring a professional to do it for you).
Winner @ build club x Folklore ventures x Lovable hackathon.

projects: