Weights are not published yet. This repository holds the full recipe: corpus builder, training script, ONNX export and a Colab notebook that runs all of it. Weights and measured results follow when the training run finishes. There are no benchmark numbers on this page because none have been measured.
What it is
The base model is Qwen3-0.6B. It can discuss machine learning, but everything it knows about the field came from general web text. Vashtra continues training it on the ML literature itself, then fine-tunes it on machine learning questions that practitioners asked and answered.
At 0.6B it fits on a phone, an edge device, or a browser tab. The demo runs entirely client side on WebGPU, so nothing typed into it leaves the page.
How it is trained
| Stage | Data | What it teaches |
|---|---|---|
| A. Continued pretraining | Full-text arXiv ML papers, plus 117k cs.LG abstracts | Vocabulary, concepts, and how an argument in a paper is built |
| B. Supervised fine-tuning | ML, stats and CS Q&A, arXiv tasks, smoltalk2 mixture | Answering questions and following instructions |
Both stages are full-parameter. At this size a full fine-tune beats LoRA for domain adaptation and still fits a free Colab T4, using about 10GB of 16GB.
Why full text and not just abstracts
Abstracts teach a model what papers claim. Full text teaches it how the claim was argued, which is what you actually want when someone asks it to explain a method. The corpus mixes both: abstracts for breadth across the field, full papers for depth.
Why general chat data stays in the mixture
Fine-tuning a 0.6B model purely on domain data reliably destroys its ability to hold a normal conversation. About 22k general examples stay in the fine-tuning mixture to prevent that. It costs a little domain sharpness and keeps the model usable as an assistant.
One detail that is easy to get wrong
Qwen3's chat template inserts an empty <think></think>
block into the final assistant turn. Tokenising message by message drops it, and
you end up training on a format the model never sees at inference. The encoder
here is checked against apply_chat_template and round-trips to it
exactly. The failure is silent, so it is worth testing for.
Running it yourself
Open the notebook in Colab, set the runtime to a T4 GPU, and run all. Everything pushes to your own Hugging Face repositories.
python build_corpus.py # build and push the corpus
python train_vashtra.py # stage A, stage B, push the model
python export_onnx.py # ONNX and quantisation for the browser demo
The corpus is around 90M tokens, which is more than one free Colab session will
get through. Stage A takes a token budget and pushes checkpoints to the Hub, so
--resume continues a run that was cut short rather than starting
over.
Limits
It is a 0.6B model. It explains concepts reasonably and gets specifics wrong, particularly exact numbers, named papers and recent work. Do not cite it, and check anything that matters.