# RepoBench-R benchmark — Hanzo AI

> Cross-file code retrieval benchmark: typed links over imports and symbols beat dense retrieval across RepoBench-R settings and splits.

[Benchmarks](https://hanzo.ai/benchmarks) / RepoBench-R

measured

# RepoBench-R

The retrieval half of RepoBench: a completion point needs one snippet from another file, and the imports, definitions and identifiers say which. Regular expressions over the source, no model reading anything.

dataset

[RepoBench-R](https://huggingface.co/datasets/tianyang/repobench-r)

licence

CC BY-NC-ND 4.0

introduced by

[RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://arxiv.org/abs/2306.03091) — Liu, Xu, McAuley, ICLR 2024

harness

[hanzoai/benchmarks · brain](https://github.com/hanzoai/benchmarks/tree/main/brain)

`node results.mjs`regenerates every table on this page from the raw runs

## What is measured

A completion point needs one snippet from another file, and the file’s imports, definitions and identifiers say which. RepoBench-R gives about twelve candidates per item with one of them correct, in two settings: `cross-file-first`, the first cross-file line of a file, and `cross-file-random`, any of them. Python. Retrieval is scored on its own, apart from generation, which is the reason this benchmark is here — it isolates the part a memory is responsible for.

Typed links are regular expressions over `import`, `def`, `class` and identifiers. No model reads anything and nothing reads the answer. On cross-file-first they alone reach 31.4 R@1 against dense retrieval’s 18.8, which is the whole finding: on code, the structure is in the source text and an embedding is the roundabout way to it.

## Results

### cross-file-first · dev split, 200 items

dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item

configuration

R@1

R@3

R@5

MRR

nDCG@5

easy R@1

hard R@1

dense only

24.0

56.5

75.0

44.9

50.5

28.0

20.0

typed links only (no model)

34.5

63.0

77.5

52.2

56.9

41.0

28.0

dense + typed links

35.5

66.0

80.0

53.4

58.6

44.0

27.0

### cross-file-first · test split, 500 items

dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item

configuration

R@1

R@3

R@5

MRR

nDCG@5

easy R@1

hard R@1

dense only

18.8

51.4

72.8

40.8

46.1

24.8

12.8

typed links only (no model)

31.4

61.0

75.4

50.5

54.5

37.2

25.6

dense + typed links

30.2

63.0

78.8

50.5

55.6

38.4

22.0

### cross-file-random · dev split, 200 items

dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item

configuration

R@1

R@3

R@5

MRR

nDCG@5

easy R@1

hard R@1

dense only

22.5

49.0

69.5

41.8

46.1

27.0

18.0

typed links only (no model)

34.0

58.0

76.5

51.2

55.3

38.0

30.0

dense + typed links

26.5

59.0

79.5

47.1

53.3

33.0

20.0

### cross-file-random · test split, 500 items

dense generator all-MiniLM-L6-v2 · no configuration frozen on dev · one gold snippet among about twelve candidates per item

configuration

R@1

R@3

R@5

MRR

nDCG@5

easy R@1

hard R@1

dense only

24.6

55.6

74.6

45.0

50.2

29.6

19.6

typed links only (no model)

29.6

63.2

79.0

49.9

55.3

35.2

24.0

dense + typed links

31.2

64.6

77.8

51.0

55.8

35.6

26.8

Hanzo AI Cloud

### Build on Hanzo Cloud

[Start on Hanzo Cloud](https://cloud.hanzo.ai)[View Models & Pricing](https://hanzo.ai/pricing)

[PreviousCronQuestions](https://hanzo.ai/benchmarks/cronquestions)

[All Benchmarks Board](https://hanzo.ai/benchmarks)

[NextLoCoMo · subject scope](https://hanzo.ai/benchmarks/locomo-subject-scope)

Other benchmarks[LoCoMo](https://hanzo.ai/benchmarks/locomo) · [MemoryAgentBench](https://hanzo.ai/benchmarks/memoryagentbench) · [LongMemEval](https://hanzo.ai/benchmarks/longmemeval) · [CronQuestions](https://hanzo.ai/benchmarks/cronquestions) · [LoCoMo · subject scope](https://hanzo.ai/benchmarks/locomo-subject-scope) · [LoCoMo-Conv](https://hanzo.ai/benchmarks/locomo-conv) · [Fleet residency](https://hanzo.ai/benchmarks/fleet) · [Live agent footprint](https://hanzo.ai/benchmarks/goroutine) · [Sandbox cold start](https://hanzo.ai/benchmarks/sandbox) · [Engine prefill on CUDA vs llama.cpp](https://hanzo.ai/benchmarks/inference) · [GPQA-Diamond](https://hanzo.ai/benchmarks/gpqa)

raw runs and scoring: [hanzoai/benchmarks · brain](https://github.com/hanzoai/benchmarks/tree/main/brain) · regenerated by `node results.mjs`