OpenScholar Install Guide: Academic Search AI on Your GPU

Two ways to run academic search AI locally with OpenScholar: spin up the full local stack with vLLM and a GPU, or run inference-only with the --api flag against OpenAI. The second path is faster to set up and works on a laptop. The first is cheaper at scale and keeps queries private. For most researchers reaching this page – people wanting to test the pipeline before committing infrastructure – the API path wins. We’ll cover both, but lead with the one that gets you a working answer in 20 minutes.

OpenScholar is the open-source retrieval-augmented model from Ai2 and the University of Washington, published in Nature in February 2026. The corpus: 45 million open-access papers, roughly 236 million passage embeddings. Why deploy it yourself instead of the demo at open-scholar.allen.ai? Custom corpora, programmatic batch queries, no rate limits – and the ability to point the retriever at your own document set.

System requirements

No Docker image, no pip package. You’re building from source. Here’s the honest split – exact hardware numbers aren’t pinned in the README, so treat these as working estimates rather than official specs; check the repo for the latest guidance before provisioning:

The part most tutorials skip entirely: running the full peS2o retriever locally against the 45M-paper index is a RAM problem, not a GPU problem. The GitHub README states directly that due to the index scale – 200+ million embeddings across 45 million papers – the peS2o retriever requires substantial CPU memory. Plan accordingly. Most users skip local retrieval entirely and rely on the Semantic Scholar API instead.

Installation

Clone the repo, create a clean conda environment, install dependencies. Python is pinned to 3.10.0 in the README – worth following that exactly given the vLLM dependency chain.

git clone https://github.com/AkariAsai/OpenScholar.git
cd OpenScholar

conda create -n os_env python=3.10.0
conda activate os_env

pip install -r requirements.txt
python -m spacy download en_core_web_sm

That sequence comes straight from the GitHub README. The spaCy step – easy to miss, not bundled into requirements.txt – will throw a load error on first run if you skip it. Ask me how I know.

First-time configuration

Three keys, one of which is easy to forget. The retriever hits Semantic Scholar for paper metadata. Optionally, you.com handles web search augmentation – but the README buries this: “sign up for you.com web API and set the key.” Without it, web retrieval just doesn’t happen; Semantic Scholar still works. For API mode, your OpenAI key goes in a plain text file (e.g. ~/.openai_key) passed as --api_key_fp.

1. Request a Semantic Scholar API key. Approval takes a few business days.
2. Optional: you.com API key for web retrieval.
3. For API mode: OpenAI (or Together AI / Anyscale) key saved to a file.

The fast path: Start with API mode using your OpenAI key. According to the research abstract, the OS-GPT4o pipeline improves GPT-4o’s correctness by 12% – same pipeline, zero GPU setup. Switch to local OS-8B once you’ve confirmed the tool fits your use case.

Verify the install

Create test.jsonl with one query, then run inference. Local OS-8B path:

python run.py 
 --input_file test.jsonl 
 --model_name OpenScholar/Llama-3.1_OpenScholar-8B 
 --use_contexts 
 --output_file out.jsonl 
 --top_n 10 --llama3 --zero_shot

Model name, top_n, flags – all from the README example. For API mode, swap the local model:

python run.py 
 --input_file test.jsonl 
 --model_name "gpt-4o" 
 --api "openai" 
 --api_key_fp ~/.openai_key 
 --use_contexts 
 --output_file out.jsonl 
 --top_n 10 --llama3 --zero_shot

Success looks like: out.jsonl with an answer containing bracketed citation IDs that map back to retrieved passages. Turns out the --api flag does more than swap the LLM – when set, run.py initializes an OpenAI client instead of loading a local vLLM model, so a clean API-mode run also validates your retriever and reranker chain without touching GPU setup at all.

Here’s a question worth sitting with before you go further: what does “working” actually mean for your use case? Bracketed citations in the output file is a technical pass. Whether the retrieved passages are the right papers for your domain is a different question – one you won’t answer by running the test query.

Common errors

• vLLM CUDA mismatch – vLLM in requirements.txt is pinned to a specific CUDA version. If nvidia-smi shows CUDA 11.x, pip install succeeds but vLLM crashes on first model load. Fix: upgrade the driver or install a matching vLLM wheel manually.
• OOM on OS-8B load – vLLM pre-allocates KV cache aggressively. If the model fails to load, check your GPU’s available VRAM against the vLLM documentation for your pinned version.
• Semantic Scholar 429s – without an API key you’re on the public throttle. Symptom: empty ctxs arrays in your output. There’s no documented workaround; wait for key approval.
• spaCy model not found – the en_core_web_sm download isn’t in requirements.txt. Run it manually after pip install.
• Reranker path errors – if you pass --ranking_ce, you also need --reranker OpenScholar/OpenScholar_Reranker. Forgetting the second flag throws an unhelpful KeyError rather than a clear missing-argument message.

Upgrade and uninstall

No semantic versions. No release tags – as of writing. Upgrades mean git pull then pip install -r requirements.txt inside the same conda env. Check the requirements diff before running it; vLLM bumps can silently break CUDA compatibility.

To remove cleanly: conda env remove -n os_env, delete the OpenScholar directory, and clear your Hugging Face model cache (~/.cache/huggingface/) if you want the model weights back. The peS2o index – if you downloaded it from Google Drive – lives wherever you put it and isn’t auto-managed.

Is this the right tool?

Honest take: for one-off questions about a literature corpus, the hosted demo at openscholar.allen.ai answers in seconds and costs nothing. Local deployment only pays off for batch processing, private corpora, or retriever fine-tuning. The team is also working on a follow-up model – DR Tulu (arXiv 2511.19399) – that builds on OpenScholar’s approach and adds multi-step search, so another install cycle is plausible within months.

Pick your path and commit to one. For API mode: grab your OpenAI key, install the env, run the test command. For local OS-8B: provision hardware that meets vLLM’s requirements first, then come back. Don’t try both simultaneously.

FAQ

Can I run OpenScholar without a GPU?

Yes. Pass --api openai (or Together AI / Anyscale). Retrieval, reranking, and the self-feedback loop still run locally. Only the language model call goes remote.

How much does it cost to run OS-GPT4o per query?

The pipeline sends top_n=10 retrieved passages plus the prompt to GPT-4o – potentially twice if self-feedback is enabled. That’s a lot of tokens per query. OpenAI’s pricing changes, so calculate against the current GPT-4o rate with your expected passage lengths before running any batch jobs. A small pilot run of 10-20 queries first is the sensible move.

Why does the official benchmark have a knowledge cutoff?

The ScholarQABench reference answers were written by domain experts between July and September 2024 (per the ScholarQABench GitHub README). Papers published after that window can still be retrieved – the system has no retrieval cutoff – but the human reference answers won’t reflect them. Which means your model can produce a more current answer than the benchmark expects and still score lower. Worth knowing before you draw conclusions from variant comparisons.