Issues

• wrong evaluation in mmbench?

  open-compass/VLMEvalKit#1710 · Filter by author garychan22 opened on Sep 30, 2026

• ScreenSpot: click points in 0-1000 coordinates (Qwen-VL style) are scored as misses

  open-compass/VLMEvalKit#1706 · Filter by author yepapa-nest opened on Sep 24, 2026

• anls_compute measures distance on normalized strings but divides by raw length, so whitespace-padded predictions score near 1.0 on DocVQA/InfoVQA/WildDoc

  open-compass/VLMEvalKit#1705 · Filter by author shaurya416 opened on Sep 23, 2026

• [Bug] Missing images and outdated questions/answers in ZeroBench

  open-compass/VLMEvalKit#1703 · Filter by author Pulyong opened on Sep 22, 2026

• extract_characters_regex: reversed containment test scores empty/missing predictions as 'A'

  open-compass/VLMEvalKit#1701 · Filter by author rrrxxx0510 opened on Sep 22, 2026

• cal_scores: compute_mcq_score scores unparseable GT+prediction as correct (False == False yields 1.0)

  open-compass/VLMEvalKit#1697 · Filter by author rrrxxx0510 opened on Sep 21, 2026
  1

• olmOCRBench: evaluate() returns None, so the score never reaches status.json

  open-compass/VLMEvalKit#1690 · Filter by author jaeminSon opened on Sep 16, 2026
  4

• Structured extra_records output is discarded for the entire dataset if any single sample fails

  open-compass/VLMEvalKit#1665 · Filter by author williamwuyantao opened on Sep 1, 2026
  1

• OpenAI-compat: GPT4V api_base is the full /v1/chat/completions URL, not /v1

  open-compass/VLMEvalKit#1661 · Filter by author cursor opened on Aug 28, 2026
  2

• Interop idea: a VLMEvalKit-openeval-adapter for portable eval results

  open-compass/VLMEvalKit#1652 · Filter by author adhabnr-ux opened on Aug 23, 2026

• Add Pluto-AI-Labs/Apollo-VL-Edge-3B to Leaderboard

  open-compass/VLMEvalKit#1632 · Filter by author SiddharthFX opened on Aug 14, 2026

• Qwen2.5-VL-7B模型，对评测集LogicVista，从推理结果中提取答案错误，导致误判

  open-compass/VLMEvalKit#1621 · Filter by author suddenlee opened on Aug 6, 2026

• MMMU Open-ended问题无法正确评测

  open-compass/VLMEvalKit#1617 · Filter by author identzz-ai opened on Aug 4, 2026

• OpenVLM: one-decimal Overall makes the board un-checkable, and hides that 8 boards' #1 leads are under 1.2 sigma

  open-compass/VLMEvalKit#1615 · Filter by author ipezygj opened on Aug 1, 2026
  1

• 有些基准没有放数据url

  open-compass/VLMEvalKit#1614 · Filter by author yangyangyang127 opened on Jul 31, 2026

• LiveMMBench空链接

  open-compass/VLMEvalKit#1600 · Filter by author yangyangyang127 opened on Jul 16, 2026

• 请问flyai-vl模型的评测什么时候可以支持？

  open-compass/VLMEvalKit#1598 · Filter by author zhangxuewei-xirui opened on Jul 7, 2026

• [Bug] Thyme: sandbox image fed in assistant turn + KV-cache reused across re-encoded turns corrupts multi-round vision context

  open-compass/VLMEvalKit#1590 · Filter by author Irisicy4 opened on Jun 28, 2026
  1

• Adding [BenchCAD]: would you accept execution-based image→CadQuery-code tasks (heavier deps)?

  open-compass/VLMEvalKit#1582 · Filter by author HaozheZhang6 opened on Jun 25, 2026

• [Bug] CharXiv_descriptive_val DATASET_MD5 appears to be incorrect after #1555

  open-compass/VLMEvalKit#1580 · Filter by author kail8 opened on Jun 23, 2026

• Qwen 3.5支持

  open-compass/VLMEvalKit#1562 · Filter by author identzz-ai opened on Jun 1, 2026

• SEEDBench_IMG & MMStar: option text "None" parsed as NaN by pandas

  open-compass/VLMEvalKit#1559 · Filter by author identzz-ai opened on May 28, 2026

• Ovis2.6-80B-A3B 评测模型预计什么时候可以支持？

  open-compass/VLMEvalKit#1553 · Filter by author 1359347500cwc opened on May 25, 2026

• 是否能发布pypi源

  open-compass/VLMEvalKit#1537 · Filter by author zq1134241486-beep opened on May 8, 2026

• model：llava-1.5-7b，data：mmebench

  open-compass/VLMEvalKit#1529 · Filter by author waowaocyj opened on Apr 27, 2026