Introducing the SEAL Leaderboards! We rank LLMs using private datasets that can’t be gamed. Vetted experts handle the ratings, and we share our methods in detail openly!
Check out our leaderboards at scale.com/leaderboard!
Which evals should we build next?
We added Llama 3.1 405B onto the SEAL Leaderboards and it does not disappoint! Here's how it stacks up:
- 
#1 in Instruction Following
- 
#2 in GSM1k
- 
#4 in Coding
SEAL evals are private, expert evals that refresh periodically: scale.com/leaderboard
SEAL and Red Team at present a position paper outlining what we’ve learned from red teaming LLMs so far—what matters, what’s missing, and how model safety fits into broader system safety and monitoring.
scale.com/research/red_t
scale.com/blog/rethink-r
(1/6) Bringing together diverse mindsets – from in-the-trenches red teamers to ML & policy researchers, we write a position paper arguing crucial research priorities for red teaming frontier models, followed by a roadmap towards system-level safety, AI monitoring, and
Red team-generated prompts
Focused on universal harm scenarios
Transparent evaluation methods
SEAL evals are private, expert evals that refresh periodically: scale.com/leaderboard
Introducing Humanity's Last Exam by & : 3,000 open-source reasoning questions where even the leading model (OpenAI o1) scores single digit accuracy.
The hardest AI benchmark yet - and it's open source 
scl.ai/hle-paper
Today, we’re introducing VISTA—a new rubric-based visual task assessment benchmark that pushes beyond simple Q&A.
The leading models achieve under 40% on this eval, compared to a human baseline of ~55.4%. This highlights that
Exciting update: Claude 3.5 Sonnet is now #1 in Coding on the SEAL leaderboard (scale.com/leaderboard)! 
Calling all experts and PhDs! Scale and CAIS are launching "Humanity's Last Exam" to develop the toughest open-source LLM benchmark.
We need your challenging questions to push AI models to their limits! Selected questions earn co-authorship and a share of $500k in prizes.
Leaderboard:
1/N) Excited to share our recent work at @scale_AI, "Jailbreaking to Jailbreak (J2)".
We present a novel LLM-as-red-teamer approach in which a human jailbreaks a refusal-trained LLM to make it willing to jailbreak itself or other LLMs. We refer to this process as
Math - we released the GSM1k last month. Today, we augmented it with human ratings to account for chatty yet correct responses.
Explore the GSM1k leaderboard as part of SEAL Leaderboards. We were glad to see LLMs have mostly nailed grade school math!
Big update: 4 new SEAL multilingual leaderboards are LIVE — Arabic, Chinese, Japanese, and Korean!
Arabic: Gemini 1.5 Pro (gemini-exp-1121) leads the pack
Chinese: Gemini 1.5 Pro (gemini-1.5-pro-exp-0827) holds the crown
Japanese & Korean: o1-preview dominates
See
AI agents are crossing into the real world.
But when they act independently—who’s watching?
At Scale, we’re building Agent Oversight: a platform to monitor, intervene, and align autonomous AI.
We’re hiring engineers (SF/NYC) to tackle one of the most urgent problems in AI.
