AstraGPT-7B
Abstract
A 7B decoder-only transformer with a custom BPE tokenizer, grouped-query attention at 28 query / 4 KV heads, RoPE at θ=1M and a SwiGLU feed-forward. The model was reasoning-tuned with LoRA on 193,841 instruction-reasoning-answer triples, giving native <think> traces. Every architectural component was implemented from first principles in PyTorch.
Read on Hugging Face 
On this page
Abstract
A 7B decoder-only transformer with a custom BPE tokenizer, grouped-query attention at 28 query / 4 KV heads, RoPE at θ=1M and a SwiGLU feed-forward. The model was reasoning-tuned with LoRA on 193,841 instruction-reasoning-answer triples, giving native <think> traces. Every architectural component was implemented from first principles in PyTorch.
Overview
AstraGPT-7B is a decoder-only language model built and trained by Tantra AI Labs, including a custom BPE tokenizer trained from scratch (64k vocabulary, later mapped to a 152,064 runtime vocabulary) and a supervised fine-tuning pipeline written in PyTorch.
The model was reasoning-tuned via LoRA on a private rig with 2× NVIDIA RTX 4090 GPUs (48GB VRAM), on a self-generated dataset of 193,841 instruction-reasoning-answer triples.
Architecture
| Component | Spec |
|---|---|
| Parameters | 7B (fp16, 15.2 GB) |
| Layers | 28 transformer blocks, pre-norm RMSNorm |
| Hidden size | 3,584 |
| Attention | Grouped-Query, 28 query / 4 KV heads |
| Positional encoding | RoPE, θ = 1,000,000 |
| Feed-forward | SwiGLU, intermediate 18,944 |
| Context window | 32,768 tokens |
| Vocabulary | 152,064 (custom BPE lineage) |
Training recipe
| Parameter | Value |
|---|---|
| Method | LoRA (PEFT) via Unsloth |
| Rank / alpha | 16 / 32 |
| Target modules | q, k, v, o, gate, up, down projections |
| Learning rate | 2e-4, cosine, 5% warmup |
| Batch | effective 16 (gradient accumulation 8) |
| Precision | BF16 mixed |
| Optimizer | AdamW 8-bit |
| Hardware | 2× RTX 4090 (48GB), Ubuntu 22.04, CUDA 12.x |
| Duration | ~3 to 4 hours |
| Post-training | LoRA adapter merged into base weights |
Reasoning behaviour
The model emits chain-of-thought inside <think> tags. Reasoning is triggered explicitly by appending "<think>\n" after the chat template; without it the model answers directly.
Example: for 15 × 47 the trace decomposes the product into 15 × 40 + 15 × 7 = 600 + 105, then commits to 705.
Limitations
- Hallucination: can produce confident but incorrect answers; verify claims.
- Mathematics: complex multi-step problems can fail; 7B is a small model.
- Language: English-primary.
- Reasoning trigger: <think> works most reliably with an explicit prefix.
Cite this
@misc{astragpt7b2026,
author = {Aditya Wakharkar},
title = {AstraGPT-7B: A 7B LLM with Custom Tokenizer and Chain-of-Thought Reasoning},
year = {2026},
organization = {Tantra AI Labs},
url = {https://huggingface.co/tantra-ai-labs/AstraGPT-7B}
}The canonical version of this work lives on Hugging Face. If a number here and the card disagree, the card wins.