Glossary
Big words, small meanings! Rocky explain. On every review, words with a dotted line show this help when you hover or tap them.
70B / 9B (parameters)
B means billion parameters, the numbers a model learned. 70B = 70 billion. More parameters usually means smarter but much bigger and slower.
AI agent
An AI that takes actions in steps (runs commands, edits files, browses) instead of only chatting.
baseline
The normal way of doing the same job, used as the yardstick. Rocky compares every claim against a baseline on the same machine.
bf16
A 16-bit number format for model weights ("brain float"). Full quality, but big: about 2 bytes per parameter, so a 72B model is about 145 GB.
claim test
Rocky measures the headline claim ("10x faster!") on real tasks against a baseline.
commit
One saved version of a repo's code. Rocky pins the exact commit so anyone can rerun the same code.
context window
How much text a model can look at in one go: your prompt, files, and the chat so far.
Docker / container
A sealed, throwaway box for running software. Rocky tests in fresh containers with no network so nothing leaks in or out.
file cache
Spare RAM the operating system uses to keep recently read files. It can make repeat reads much faster, which can flatter a test.
GPU
Graphics card. AI models run on them because they do huge amounts of math in parallel.
greedy decoding
Always pick the single most likely next token, with no randomness. Same input gives the same output, so runs can be compared word for word.
Hugging Face
The main website where open AI models and datasets are shared and downloaded.
layer streaming
Loading a model one slice (layer) at a time from disk instead of all at once. Saves memory, costs a lot of time.
license (Apache-2.0, MIT)
The rules for using the code. Apache-2.0 and MIT both allow free use, including commercial use.
llama.cpp
A fast open-source engine for running quantized models on ordinary hardware. ollama is built on it.
LLM
Large language model: the kind of AI that reads and writes text, like the models behind chat assistants.
MCP
Model Context Protocol: a standard plug that lets AI assistants use outside tools and data (files, browsers, databases).
MCP server
A small program that offers tools to an AI assistant over MCP, for example "search my docs" or "open a browser".
NVMe SSD
A fast kind of storage drive. Layer streaming reads from it constantly, so drive speed sets the pace.
ollama
A popular free app for running AI models on your own computer, usually in 4-bit form.
OOM
Out of memory: the program asked for more memory than the machine has, so it crashed.
prefill
The time a model spends reading your prompt before it writes the first word. Long prompts make prefill slower.
Q4 / 4-bit
A quantized model that stores each weight in about 4 bits. Roughly a quarter the size of bf16 and much faster, slightly lower quality.
quantization
Shrinking a model by storing its numbers with fewer bits (for example 4-bit instead of 16-bit). Much smaller and faster, with a small quality cost.
RAG
Retrieval-augmented generation: look up relevant documents first, then let the model answer using them.
RSS (process memory)
Resident set size: all the RAM a running program is holding right now, including libraries and cached data, not just the model.
seconds per token
The flip side of tok/s: how long you wait for each new word-piece. 64 seconds per token means about a minute per word.
shards / layer files
A big model split into many smaller files on disk so pieces can be loaded one at a time.
shared (unified) memory
Some machines let the CPU and GPU use one big memory pool. That is not the same as a graphics card that physically has only 4 GB.
smoke test
Rocky's quickest test: fresh sandbox, does it install and does the README example run?
tok/s (tokens per second)
How fast a model writes. Reading speed is about 5 to 8 tok/s, so anything below 1 tok/s feels painfully slow.
token
The small chunks a model reads and writes. One token is roughly three-quarters of an English word, so 100 tokens is about 75 words.
VRAM / GPU memory
The memory on a graphics card. A "4 GB card" has 4 GB of it. Models normally have to fit here to run fast.