Commit Graph

Select branches

Hide Pull Requests

StellaAthena-patch-1

StellaAthena-patch-2

acc_mutual

adamlin120/main

add-chat-templating

add-paloma-benchmark

add-siqa

add_humaneval

afrimmlu-edit

ai2d

aime24

alt_worlds

api

apilint

autobatchtest

b281b09

b_limit

baber

based-tasks

bchat

big-merge-from-gakada

big-refactor

bjudge

brier_score

cach

calibration

chat_

chat_template

cloze

comma

commonsense_qa

convert_gen

cost

custom-aggregation

dash_topk

dm_math

feature/update-kotask-prompt

fix-fewshot-bbh-parsing

fix-len0-continuations

fix-task-table

gemini_

gen

genericize-args

group-config

hailey-multimodal-prototyping

inverse-scaling-tasks

japanese-tasks

ko

lama_primed_negated

lintang-multimodal-prototyping

lintangsutawika-patch-1

lintangsutawika-patch-2

lintangsutawika-patch-3

lintangsutawika-patch-4

lintangsutawika-patch-5

llama

logprob-completions

ls

main

masakhane-io/main

master

math

mathvista

mc-answer-prompt-experiment

mela

metric-loading

metrics

metricsv2

mm-llm

mmlu

mmlu-pro

mmlu-pro-changes

mmlu-test

mmlu_llama

multilingual

multimodal-evals

multimodal-support

multiprompt

newdocs

newmetrics_

newpy

next0.5

nextdocs

no-bootstrap

nous

num_fewshot_fix

parentheses

patch-group-metric-aggregation

polyglot

prefill

presets

pythia

qwen_math

readme

readme-overwrite

reason

refactorTask_!

refactor_05

repeats

revert-2083-patch-1

revert-780-fix-metrics

rm_multiple_target

roadmap

rwkv

scipy-optional

scrolls

siq

smolrefact

standardize_mcqa

standardize_metrics

steer-merge

t5v2-alt-plus

t5v2-evals

taskmanager_

term-freq-triviaqa

testing

testtasks

thinktest

tinyBenchmarks

toxicity

toxicity-test

truncation

update-pile-tasks

vllm_

weight_by_size

#1

#100

#1001

#1002

#1004

#1005

#1007

#1008

#1009

#101

#1011

#1014

#1018

#1020

#1024

#1025

#1029

#103

#1030

#1031

#1032

#1033

#1035

#1036

#1037

#1038

#1039

#104

#1040

#1041

#1045

#1046

#1047

#1048

#1049

#105

#1051

#1052

#1059

#106

#1060

#1061

#1062

#1063

#1064

#1065

#1066

#1069

#107

#1074

#1076

#1076

#1077

#108

#1080

#1080

#1082

#1087

#1089

#109

#1093

#1094

#1096

#1097

#1099

#110

#1104

#1106

#1108

#1109

#111

#1110

#1113

#1114

#1116

#1118

#112

#1120

#1122

#1123

#1123

#1124

#1126

#1127

#1128

#1130

#1133

#1136

#1138

#1139

#1141

#1142

#1144

#1145

#1146

#1147

#1148

#1149

#1150

#1154

#1156

#1166

#1167

#1167

#1171

#1172

#1173

#1174

#1178

#1180

#1181

#1183

#1184

#1185

#1186

#1188

#1189

#119

#1190

#1191

#1193

#1194

#1195

#1197

#1198

#1200

#1203

#1206

#1208

#1212

#1215

#1216

#1218

#1219

#1221

#1221

#1222

#1227

#1228

#1229

#123

#1230

#1238

#124

#1241

#1243

#1246

#1252

#1255

#1257

#1258

#1259

#1261

#1263

#1265

#1266

#1267

#1268

#1270

#1271

#1273

#1275

#1276

#1277

#1279

#128

#1280

#1281

#1284

#1285

#1287

#1289

#129

#1290

#1298

#130

#1305

#1306

#1312

#1313

#1314

#1315

#1316

#1318

#1320

#1321

#1322

#1322

#1324

#1325

#1326

#1327

#1328

#1329

#133

#1330

#1331

#1332

#1334

#1335

#1336

#1337

#1339

#1341

#1345

#1348

#1349

#135

#1354

#1355

#1356

#1357

#1358

#1359

#136

#1363

#1364

#1367

#1369

#1371

#1372

#1373

#1374

#1378

#1379

#138

#1380

#1384

#1385

#1387

#139

#1390

#1394

#1398

#140

#1400

#1402

#1404

#1405

#141

#1412

#1414

#1418

#1419

#142

#1420

#1425

#1427

#1429

#143

#1430

#1434

#1438

#144

#1440

#1441

#1443

#1445

#1446

#1447

#1448

#145

#1450

#1453

#1456

#1458

#146

#1460

#1462

#1464

#1465

#1466

#1467

#1469

#147

#1474

#1475

#1477

#1480

#1481

#1481

#1482

#1484

#1487

#1488

#1489

#1491

#1495

#1498

#1499

#150

#1500

#1502

#1505

#1506

#1507

#1508

#1509

#151

#1511

#1516

#1517

#152

#1520

#1526

#1527

#1528

#153

#1530

#1532

#1533

#1534

#1535

#1536

#154

#1540

#1541

#1542

#1545

#1546

#1548

#1549

#155

#1551

#1553

#1557

#1557

#156

#1563

#1566

#1567

#1569

#1571

#1571

#1573

#1578

#1579

#1584

#1587

#1588

#1589

#159

#1592

#1593

#1594

#1597

#1598

#1599

#1601

#1604

#1605

#1607

#1608

#161

#1611

#1612

#1614

#1615

#1616

#1618

#1619

#162

#1620

#163

#1630

#1633

#1635

#1636

#1640

#1641

#1647

#1649

#1649

#1651

#1654

#1655

#1655

#1656

#1657

#1657

#166

#1661

#1668

#1669

#167

#1670

#1674

#168

#1680

#1685

#1686

#169

#1694

#1699

#170

#1705

#1705

#1706

#1706

#1708

#1708

#1709

#1710

#1712

#172

#1721

#1724

#1726

#1727

#1728

#1730

#1731

#1732

#1736

#1737

#1740

#1741

#1745

#1749

#1750

#1753

#1753

#1756

#1758

#176

#1760

#1763

#1764

#1766

#1769

#177

#1773

#1775

#1776

#1777

#1779

#1780

#1784

#1785

#1786

#1787

#1789

#179

#1790

#1791

#1792

#1793

#1794

#1794

#1796

#1798

#180

#1800

#1803

#1806

#1807

#181

#1810

#1815

#1815

#182

#1825

#1828

#183

#1832

#1834

#1838

#1843

#1844

#1847

#1848

#1849

#1851

#1852

#1853

#1853

#1855

#1856

#1857

#1858

#1859

#1860

#1860

#1861

#1862

#1862

#1863

#1865

#1867

#1868

#1871

#1872

#1873

#1874

#1876

#1878

#1878

#1880

#1886

#1888

#1890

#1893

#1895

#1897

#1898

#1900

#1901

#1902

#1902

#1905

#1909

#1912

#1913

#1914

#1916

#1917

#1919

#1921

#1921

#1922

#1922

#1924

#1926

#1927

#1928

#1929

#1930

#1931

#1933

#1939

#194

#1940

#1942

#1943

#1943

#1946

#1946

#1948

#195

#1950

#1954

#1955

#1956

#1961

#1964

#1968

#1969

#1970

#1973

#1974

#1975

#1975

#1977

#1979

#198

#1981

#1982

#1983

#1986

#1987

#1988

#1989

#1990

#1991

#1992

#1993

#1994

#1995

#1996

#1997

#1998

#1999

#1999

#2

#200

#2003

#2004

#2004

#2006

#2008

#2009

#201

#2010

#2011

#2013

#2013

#2015

#2017

#2019

#2020

#2020

#2022

#2022

#2023

#2024

#2029

#203

#2030

#2031

#2031

#2032

#2034

#2036

#2037

#2038

#2039

#204

#2041

#2042

#2044

#2045

#2046

#2047

#2048

#2049

#2050

#2050

#2051

#2052

#2053

#2054

#2055

#2056

#2057

#2058

#206

#2060

#2067

#2073

#2074

#2075

#2079

#2080

#2082

#2083

#2084

#2085

#2087

#209

#2090

#2092

#2093

#2097

#2098

#2099

#2099

#210

#2100

#2103

#2104

#2105

#2106

#2106

#2110

#2111

#2112

#2113

#2113

#2114

#2116

#2118

#2119

#2123

#2124

#2125

#2127

#2128

#2129

#213

#2132

#2132

#2139

#214

#2149

#215

#2153

#2154

#2155

#2156

#2157

#216

#2162

#2168

#2169

#2172

#2174

#2176

#2176

#2181

#2184

#2186

#2187

#219

#2191

#2198

#2203

#2206

#2207

#2207

#2208

#2208

#2209

#2209

#2215

#2216

#2217

#2218

#2219

#222

#2223

#2227

#2228

#2229

#223

#2230

#2232

#2235

#2236

#2238

#2239

#2241

#2243

#2244

#2245

#2246

#2247

#2249

#225

#2250

#2251

#2251

#2258

#2259

#226

#2262

#2264

#2265

#2267

#2269

#227

#2270

#2270

#2271

#2274

#2275

#2277

#228

#2280

#2282

#2283

#2288

#229

#2297

#2298

#2299

#230

#2300

#2304

#2307

#2309

#2311

#2312

#2312

#2314

#2315

#232

#2320

#2321

#2325

#2327

#233

#2332

#2333

#2334

#2336

#2337

#2339

#234

#2340

#2341

#2349

#2350

#2351

#2353

#2357

#2359

#236

#2363

#2364

#2366

#2368

#2369

#2369

#237

#2372

#2374

#2375

#2376

#2378

#2378

#2380

#2383

#2384

#2384

#2385

#2387

#2388

#2389

#239

#2390

#2391

#2391

#2393

#2396

#2397

#240

#2401

#2403

#2404

#2407

#2408

#2409

#2413

#2417

#2419

#2419

#242

#2420

#2421

#2426

#2427

#243

#2433

#2434

#2435

#2437

#2438

#2439

#244

#2440

#2443

#2446

#2449

#245

#2450

#2450

#2452

#2453

#2456

#2457

#2459

#246

#2461

#2462

#2464

#2465

#2473

#2474

#2475

#2476

#2481

#2482

#2482

#2484

#2485

#2486

#2486

#2488

#2489

#2492

#2493

#2495

#2496

#2497

#2499

#250

#2500

#2502

#2502

#2503

#2504

#2507

#2508

#2509

#2510

#2512

#2512

#2514

#2515

#2515

#2517

#2518

#252

#2520

#2521

#2524

#2531

#2534

#2535

#2536

#254

#2540

#2541

#2542

#2542

#2543

#2544

#2545

#2546

#2549

#2554

#2556

#2556

#2558

#2559

#256

#2560

#2565

#2566

#2567

#2568

#2570

#2570

#2574

#2575

#2576

#258

#258

#2580

#2582

#2582

#2585

#2587

#259

#2593

#2593

#2595

#2596

#2599

#260

#2600

#2601

#2601

#2602

#2603

#2604

#2606

#2606

#2609

#261

#2611

#2613

#2615

#2620

#2621

#2621

#2622

#2624

#2627

#2628

#2629

#263

#2630

#2632

#2634

#2636

#2637

#264

#2641

#2642

#2643

#2645

#265

#2650

#2651

#2652

#2653

#2654

#2655

#2655

#2658

#2659

#266

#2660

#2662

#2663

#2667

#2668

#267

#2670

#2670

#2671

#2673

#2675

#2678

#2679

#268

#2680

#2681

#2683

#2684

#2686

#2687

#2688

#2688

#2689

#269

#2691

#2692

#2694

#2697

#2699

#2699

#270

#2700

#2701

#2703

#2705

#2708

#2709

#271

#2710

#2711

#2712

#2712

#2714

#2716

#2718

#2718

#2719

#272

#2721

#2724

#2727

#2728

#2729

#273

#2732

#2734

#2735

#2736

#2737

#2738

#2739

#2739

#274

#2742

#2746

#2749

#275

#2750

#2753

#2754

#2755

#2757

#2759

#276

#2760

#2761

#2762

#2765

#2768

#2769

#277

#2771

#2772

#2772

#2773

#2777

#2778

#278

#2781

#2784

#2785

#2786

#2787

#2788

#279

#2790

#2792

#2795

#2795

#2796

#2797

#2799

#280

#2802

#2804

#2805

#2805

#2806

#2806

#2807

#2808

#2809

#281

#2810

#2811

#2812

#2812

#2813

#2814

#2816

#2817

#2818

#2819

#282

#2820

#2822

#2824

#2825

#2826

#2827

#2828

#2829

#283

#2831

#2832

#2832

#2835

#2835

#2837

#2837

#2840

#2841

#2842

#2845

#2846

#2846

#2849

#2850

#2854

#2855

#2856

#2856

#2857

#2859

#2859

#286

#2860

#2862

#2865

#2866

#2867

#2869

#2869

#2870

#2873

#2876

#2877

#2878

#2879

#288

#2882

#2888

#2888

#2889

#289

#2892

#2893

#2893

#2895

#2899

#290

#2902

#2906

#2906

#2908

#2908

#2909

#2909

#291

#2910

#2911

#2912

#2912

#2913

#2915

#2916

#2917

#2918

#2919

#292

#2921

#2923

#2923

#2924

#2924

#2925

#2928

#2929

#2937

#2939

#294

#2940

#2942

#2943

#2945

#2946

#2947

#2951

#2951

#2955

#2956

#2957

#2957

#2958

#2959

#296

#2962

#2967

#2969

#297

#2970

#2972

#2973

#2977

#2979

#2980

#2981

#2983

#2985

#2987

#299

#2991

#2992

#2993

#2995

#2997

#2998

#2999

#3

#300

#3000

#3002

#3006

#3007

#3008

#3009

#301

#3011

#3013

#3016

#3016

#3018

#3018

#3019

#302

#3020

#3020

#3023

#3024

#3025

#3029

#3029

#303

#3030

#3031

#3031

#3032

#3032

#3034

#3035

#3036

#3036

#3037

#3037

#3039

#3040

#3041

#3043

#3043

#3047

#3049

#3049

#305

#3052

#3054

#3057

#3058

#3058

#3059

#3059

#306

#3060

#3061

#3062

#3063

#3064

#3064

#3066

#3066

#3068

#3069

#307

#3070

#3071

#3071

#3072

#3073

#3074

#3076

#3078

#3079

#308

#3084

#3084

#3085

#3085

#3087

#3087

#3088

#3088

#309

#3090

#3091

#3092

#3094

#3097

#3098

#3099

#310

#3101

#3102

#3105

#3105

#3106

#3107

#3108

#3109

#3109

#3112

#3113

#3114

#3117

#3124

#3127

#3128

#313

#3131

#3132

#3133

#3135

#3136

#3137

#3138

#314

#3140

#3141

#3141

#3144

#3145

#3146

#3149

#3149

#315

#3151

#3153

#3155

#3156

#3157

#3157

#3158

#3159

#316

#3160

#3160

#3162

#3163

#3164

#3167

#3168

#3169

#3169

#317

#3172

#3173

#3174

#3176

#318

#3180

#3181

#3182

#3184

#3185

#3186

#3189

#319

#3191

#3192

#3192

#3194

#3197

#3198

#3198

#3199

#3201

#3202

#3204

#3205

#3205

#3206

#3208

#3209

#321

#3211

#3213

#3215

#3215

#3217

#3218

#3219

#322

#3220

#3221

#3222

#3223

#3225

#3227

#3229

#3230

#3232

#3232

#3234

#3235

#3235

#3238

#3239

#324

#3240

#3240

#3241

#3242

#3246

#3247

#3248

#3249

#325

#3250

#3250

#3254

#3254

#3255

#3256

#3256

#3257

#3259

#326

#3261

#3262

#3262

#3263

#3264

#3265

#3267

#3268

#327

#3271

#3272

#3273

#3277

#3279

#328

#3281

#3283

#3284

#3287

#3288

#329

#3290

#3292

#3294

#3296

#3299

#33

#330

#3300

#3300

#3301

#3303

#3304

#3304

#3305

#3309

#3309

#331

#3311

#3313

#3313

#3315

#3316

#3317

#332

#3321

#3327

#333

#3331

#3334

#3337

#3338

#334

#3340

#3343

#3344

#3347

#3348

#3349

#3350

#3351

#3351

#3353

#3354

#3354

#3356

#3359

#336

#3361

#3363

#3363

#3364

#3367

#3368

#3372

#3373

#3374

#3377

#3378

#3379

#338

#3380

#3381

#3381

#3384

#3384

#3386

#3386

#3387

#3387

#3388

#3392

#3393

#3394

#3395

#3398

#3406

#3407

#3407

#3408

#341

#3410

#3411

#3411

#3412

#3413

#3415

#3418

#3420

#3422

#3425

#3426

#3428

#3429

#343

#3432

#3432

#3433

#3436

#3437

#344

#3440

#3442

#3443

#3444

#3446

#3449

#3450

#3453

#3454

#3459

#3459

#3461

#3462

#3463

#3464

#3464

#3465

#3467

#3468

#3469

#3470

#3471

#3472

#3473

#3474

#3476

#3478

#3478

#3479

#3480

#3480

#3482

#3485

#3485

#3487

#3488

#3489

#349

#3492

#3494

#3494

#3496

#35

#3500

#3503

#3504

#3506

#3506

#3507

#3507

#3508

#3509

#3510

#3510

#3514

#3515

#3517

#3518

#352

#3520

#3521

#3522

#3523

#3525

#3526

#3527

#3529

#3530

#3531

#3532

#3534

#3534

#3535

#3536

#3539

#3540

#3541

#3541

#3542

#3542

#3543

#3544

#3545

#3546

#3547

#3547

#3549

#355

#3550

#3551

#3552

#3553

#3553

#3554

#3554

#3555

#3555

#3556

#3556

#3559

#3561

#3563

#3564

#3565

#3567

#3568

#3568

#357

#3570

#3571

#3572

#3573

#3574

#3575

#3577

#3578

#3579

#358

#3580

#3581

#3582

#3584

#3585

#3586

#3588

#3589

#359

#3591

#3592

#3593

#3593

#3594

#3595

#3596

#3597

#3597

#3598

#36

#3600

#3600

#3601

#3602

#3603

#3603

#3604

#3604

#3605

#3606

#3607

#3608

#3609

#3609

#362

#3620

#3620

#3622

#3622

#3623

#3623

#3625

#3625

#3626

#3628

#363

#3630

#3631

#3633

#3634

#3635

#3638

#364

#3640

#3641

#3641

#3642

#3642

#3644

#3644

#3647

#3647

#3648

#3649

#3649

#365

#3650

#3651

#3651

#3653

#3653

#3654

#3655

#3655

#3656

#3657

#3658

#3659

#366

#3660

#3662

#3663

#3663

#3664

#3665

#3667

#3667

#3668

#3668

#3669

#3670

#3671

#3671

#3672

#3672

#3673

#3675

#3677

#3678

#3678

#3679

#3679

#3680

#3680

#3681

#3681

#3682

#3682

#3684

#3684

#3685

#3685

#3686

#3686

#3687

#3689

#3690

#3691

#3692

#3693

#3694

#3695

#3695

#3696

#3699

#37

#370

#3700

#3701

#3702

#3703

#3704

#3705

#3705

#3706

#3706

#3707

#3707

#3708

#3709

#3709

#371

#3710

#3710

#3711

#3713

#3713

#3714

#3715

#3715

#3716

#3718

#3719

#3720

#3720

#3721

#3722

#3722

#3723

#3724

#3725

#3726

#3726

#3727

#3728

#3729

#373

#3731

#3732

#3733

#3734

#3735

#3736

#3736

#3737

#3738

#3739

#3740

#3740

#3741

#3741

#3742

#3743

#3744

#3744

#3745

#3746

#3746

#3747

#3748

#3750

#3750

#3751

#3751

#3752

#3752

#3753

#3753

#3754

#3754

#3755

#3755

#3756

#3756

#3757

#3757

#3758

#3759

#3759

#376

#3760

#3761

#3761

#3762

#3762

#3763

#3764

#3766

#3766

#3767

#3767

#3768

#3769

#3769

#377

#3770

#3770

#3772

#3772

#3773

#3773

#3775

#3775

#3777

#3777

#3778

#3778

#3779

#3780

#3781

#3781

#3783

#3783

#3784

#3785

#3785

#3786

#3786

#3787

#3787

#3788

#3789

#3789

#379

#3790

#3792

#3793

#3793

#3794

#3794

#3795

#3796

#3796

#3797

#3797

#3798

#3798

#3799

#3799

#38

#380

#3800

#3800

#3801

#3802

#3803

#3804

#3804

#3806

#3807

#3807

#3808

#3808

#3809

#3809

#381

#3810

#3811

#3811

#3812

#3813

#3814

#3815

#3815

#3816

#3817

#3818

#3818

#3819

#3819

#3821

#3821

#3822

#3823

#3823

#3824

#3825

#3825

#3826

#3827

#3828

#3828

#3829

#3829

#3830

#3830

#3832

#3834

#3834

#3835

#3835

#3836

#3837

#3838

#3838

#3839

#3839

#384

#3840

#3840

#3841

#3842

#3843

#3843

#3844

#3844

#3845

#3845

#3847

#3848

#3850

#3850

#3851

#3851

#3852

#3853

#3855

#3855

#3856

#3856

#3857

#3858

#3858

#3860

#3861

#3861

#3862

#3862

#3863

#3863

#3864

#3864

#3865

#3865

#3867

#3868

#3868

#3869

#3869

#3870

#3871

#3871

#3872

#3872

#3873

#3873

#3874

#3874

#3875

#3875

#3876

#3876

#3878

#3878

#3879

#3879

#388

#3880

#3880

#3882

#3883

#3883

#3884

#3885

#3886

#3886

#3887

#3888

#3888

#3889

#389

#3890

#3890

#3891

#3891

#3892

#3892

#3893

#3893

#3894

#3894

#3896

#3896

#3897

#3898

#3899

#39

#390

#3901

#3902

#3902

#3903

#3903

#3904

#3906

#3906

#3907

#3907

#3908

#3908

#3909

#3909

#391

#3910

#3910

#3913

#3913

#3914

#3914

#3916

#3917

#3917

#3918

#3918

#3919

#3919

#3920

#3920

#3921

#3922

#3922

#3923

#3925

#3925

#3927

#3927

#3928

#3928

#3929

#3929

#393

#3931

#3932

#3933

#3933

#3934

#3936

#3937

#3938

#3938

#3939

#394

#3940

#3941

#3941

#3942

#3943

#3944

#3945

#3945

#3946

#3948

#3948

#3949

#3949

#395

#3950

#3951

#3951

#3952

#3952

#3953

#3953

#3954

#3956

#3956

#3959

#396

#3960

#3963

#3963

#3965

#3968

#3968

#3969

#3969

#397

#3970

#3971

#3972

#3973

#3973

#3974

#3975

#3976

#3977

#3978

#3979

#3980

#3981

#3981

#3982

#3983

#3983

#3984

#3987

#3987

#3988

#3988

#3990

#3990

#3991

#3992

#3993

#3994

#3994

#3995

#3997

#3997

#3998

#3999

#3999

#4

#4000

#4000

#4001

#4001

#4003

#4004

#4004

#4006

#4006

#4008

#4008

#4009

#4009

#4010

#4012

#4013

#4014

#4015

#4015

#4016

#4018

#4018

#402

#4020

#4021

#4021

#4023

#4024

#4025

#4026

#4027

#4027

#4029

#4029

#403

#4032

#4033

#4033

#4034

#4035

#4037

#4039

#4040

#4041

#4042

#4042

#4043

#4044

#4045

#4047

#4048

#4049

#4049

#405

#4050

#4051

#4051

#4053

#4054

#4055

#4055

#4056

#4057

#4057

#4059

#406

#4060

#4061

#4064

#4064

#4065

#4065

#4066

#4067

#4067

#4068

#4069

#4069

#407

#4071

#4072

#4073

#4073

#4074

#4074

#4075

#4075

#4076

#4077

#4077

#4078

#4078

#4079

#4080

#4081

#4082

#4082

#4085

#4085

#4087

#4087

#4088

#4088

#4089

#4089

#409

#4090

#4090

#4091

#4091

#4092

#4092

#4094

#4094

#4096

#4096

#4098

#4098

#4099

#4099

#41

#410

#4100

#4100

#4101

#4101

#4102

#4102

#4103

#4103

#4104

#4104

#4105

#4105

#4106

#4106

#4107

#4107

#4108

#4108

#4109

#4109

#411

#4110

#4110

#4111

#4111

#4112

#4112

#4113

#4113

#4114

#4114

#4115

#4115

#4116

#4116

#4117

#4117

#4118

#4118

#4119

#4119

#4120

#4120

#4121

#4121

#4122

#4122

#4123

#4123

#4124

#4125

#4126

#4127

#4127

#4128

#4128

#4129

#4129

#4130

#4130

#4131

#4132

#4132

#4134

#4134

#4136

#4137

#4138

#4138

#4139

#4139

#414

#4140

#4141

#4144

#4146

#4146

#4148

#4148

#4149

#4149

#415

#415

#4150

#4150

#4151

#4151

#4154

#4154

#4156

#4157

#4157

#416

#4160

#4160

#4164

#4165

#4165

#4166

#4166

#4167

#4167

#4168

#4170

#4170

#4172

#4172

#4173

#4173

#4174

#4174

#4176

#4176

#4177

#4177

#4179

#4179

#418

#4180

#4180

#4181

#4181

#4182

#4182

#4184

#4184

#4186

#4186

#4188

#4188

#4189

#4190

#4190

#4192

#4192

#4194

#4194

#4195

#4195

#4196

#4196

#4197

#4197

#4198

#4198

#4199

#4199

#42

#4200

#4200

#4202

#4202

#4204

#4204

#4205

#4205

#4206

#4206

#4208

#4208

#4209

#4209

#4210

#4211

#4211

#4214

#4214

#4215

#4215

#4216

#4216

#4217

#4219

#4219

#422

#4221

#4221

#4222

#4222

#4224

#4224

#4226

#4226

#4227

#4227

#4228

#4228

#4229

#4229

#4231

#4231

#4232

#4232

#4235

#4235

#4237

#4237

#4239

#4239

#4240

#4240

#4241

#4241

#4242

#4242

#4244

#4244

#4245

#4245

#4246

#4246

#4247

#4247

#4248

#4248

#4249

#4249

#425

#4250

#4250

#4252

#4252

#4253

#4253

#4254

#4254

#4257

#4257

#4258

#4258

#4259

#4259

#426

#4260

#4260

#4261

#4261

#4262

#4262

#4263

#4264

#4264

#4265

#4266

#4267

#4267

#4269

#4269

#427

#4271

#4271

#4273

#4273

#4275

#4275

#4279

#4279

#4280

#4280

#4281

#4281

#4284

#4284

#4285

#4285

#4287

#4287

#4289

#4289

#429

#4290

#4290

#4291

#4291

#4292

#4292

#4293

#4293

#4294

#4294

#4295

#4295

#4296

#4296

#4297

#4297

#4298

#4298

#4299

#4299

#43

#430

#4300

#4300

#4301

#4301

#4302

#4302

#4303

#4303

#4306

#4306

#4307

#4307

#4308

#4308

#4309

#4309

#4310

#4310

#4311

#4311

#4313

#4313

#4314

#4314

#4315

#4315

#4316

#4316

#4317

#4317

#4318

#4318

#432

#4320

#4320

#4323

#4323

#4324

#4324

#4325

#4325

#4326

#4326

#4327

#4327

#4328

#4328

#4329

#4329

#4330

#4330

#4331

#4331

#4332

#4332

#4333

#4333

#4336

#4336

#4337

#4337

#4338

#4338

#4339

#4339

#4340

#4340

#4341

#4341

#4342

#4343

#4343

#4344

#4344

#4345

#4346

#4346

#4347

#4347

#4348

#4348

#4349

#4349

#435

#4351

#4351

#4352

#4352

#4355

#4355

#4356

#4356

#4357

#4357

#4358

#4358

#4359

#4359

#4360

#4360

#4362

#4362

#438

#441

#444

#444

#446

#447

#448

#45

#46

#465

#467

#47

#472

#473

#474

#477

#479

#48

#480

#481

#482

#483

#484

#485

#486

#488

#49

#490

#492

#493

#497

#50

#501

#503

#51

#515

#516

#517

#519

#52

#520

#522

#523

#525

#526

#528

#529

#53

#531

#532

#533

#534

#536

#537

#54

#542

#544

#545

#546

#55

#550

#551

#552

#553

#555

#556

#557

#558

#559

#56

#560

#562

#563

#564

#565

#566

#567

#568

#57

#572

#574

#575

#576

#577

#578

#579

#580

#581

#582

#584

#585

#587

#59

#590

#593

#596

#598

#599

#600

#601

#606

#608

#609

#61

#610

#612

#613

#615

#616

#617

#618

#619

#620

#621

#623

#624

#625

#627

#629

#63

#630

#631

#637

#638

#639

#64

#640

#642

#643

#644

#645

#646

#647

#648

#65

#650

#651

#652

#653

#654

#655

#657

#658

#659

#66

#660

#661

#664

#666

#669

#67

#671

#673

#674

#675

#676

#678

#679

#68

#680

#682

#684

#685

#686

#687

#688

#689

#69

#690

#692

#693

#695

#699

#70

#700

#701

#702

#703

#706

#707

#71

#710

#711

#712

#718

#719

#72

#720

#722

#723

#724

#725

#728

#729

#73

#730

#732

#733

#734

#735

#736

#738

#739

#74

#740

#741

#742

#743

#744

#745

#746

#747

#749

#75

#750

#751

#752

#753

#754

#755

#756

#757

#758

#759

#761

#763

#764

#765

#766

#768

#769

#770

#771

#772

#773

#774

#775

#776

#777

#779

#780

#781

#782

#783

#784

#785

#786

#788

#789

#79

#791

#792

#793

#794

#795

#797

#798

#799

#80

#802

#803

#807

#809

#81

#810

#812

#813

#815

#816

#817

#819

#82

#820

#821

#822

#824

#826

#828

#829

#830

#831

#833

#834

#838

#839

#84

#840

#842

#843

#844

#845

#847

#848

#85

#850

#851

#852

#853

#854

#856

#857

#858

#859

#86

#860

#861

#862

#863

#864

#865

#868

#87

#870

#871

#872

#874

#876

#877

#878

#88

#880

#882

#883

#885

#888

#889

#89

#890

#893

#895

#897

#899

#90

#900

#902

#904

#905

#906

#907

#908

#909

#91

#910

#912

#913

#914

#915

#916

#917

#918

#920

#922

#923

#925

#925

#926

#927

#929

#931

#934

#935

#936

#936

#937

#943

#944

#948

#949

#952

#953

#954

#955

#956

#957

#958

#965

#966

#967

#968

#969

#97

#971

#977

#979

#98

#981

#985

#986

#987

#99

#990

#994

#996

#997

#999

v0.0.1

v0.2.0

v0.3.0

v0.4.0

v0.4.1

v0.4.10

v0.4.11

v0.4.12

v0.4.13

v0.4.2

v0.4.3

v0.4.4

v0.4.5

v0.4.6

v0.4.7

v0.4.8

v0.4.9

v0.4.9.1

v0.4.9.2

v0.5.0.dev1

Mono Color

• d6de816439 fix(basqueglue): load the f1 metric with evaluate (#4076) main Ruilin Gao 2026-09-14 06:51:05 -04:00
• e54687eac0 fix(bbh): guard find_match against an all-empty capture tuple (#4125) 3 people 2026-09-14 03:48:32 -07:00
• 761c18cfd2 feat(tasks): add zero-shot CoT variants for MedMCQA and MedQA (#4081) Yigit TuncerandBaber Abbasi 2026-09-14 13:47:47 +03:00
• 75d1f2b152 Restore the FLAN RTE prompt-2 task (#4156) DivanshuandBaber Abbasi 2026-09-14 15:51:04 +05:30
• 2e1fd07142 Fix SampleResult docs: arguments shape and score types (#4137) omid511andomid511 2026-09-14 13:50:38 +03:30
• 53089ea8a4 fix(cli): preserve signed integer argument types (#4136) Divanshu 2026-09-14 15:10:04 +05:30
• 2abbf682a3 fix(cli): allow braces in key-value arguments (#4144) Divanshu 2026-09-14 15:04:42 +05:30
• ad8737ae7f gguf: support modern llama.cpp logprobs format (fixes #1437) (#4012) 4 people 2026-09-10 21:29:22 +05:30
• b954108c9b Feat/plugin filter metric registration (#4014) 3 people 2026-09-01 09:51:28 -04:00
• 69c005f265 feat(registry): runtime plugin registration for model backends (#4013) Thiago CrepaldiandClaude 2026-09-01 05:33:29 -04:00
• 889c656bdd chore(version): 0.4.14.dev0 (#4072) Baber Abbasi 2026-09-01 01:46:12 +05:00
• ddd6722043 chore(version): 0.4.13 (#4071) v0.4.13 Baber Abbasi 2026-09-01 01:38:54 +05:00
• 2fca61d4f1 fix: make --use_cache keys hashable for multimodal image/byte requests (#4040) feiandBaber Abbasi 2026-09-01 01:12:16 +08:00
• 3d019dc2fa Fix/indicparam citation (#4066) Vivek Patel | Vivek codes ❶andVivek Patel 2026-08-31 18:49:35 +05:30
• 8c1291c306 fix(evaluator): report a sample count that does not depend on metric order (#4068) Arthi ArumugamandBaber Abbasi 2026-08-31 18:43:09 +05:30
• 6bfdbcbdf2 feat(physics_gre): add InflectionAI Physics GRE multiple-choice task (#3853) bonoandbongho.lee 2026-08-31 21:30:09 +09:00
• c1b3b3a33e Add IndicXNLI Gujarati task (indicxnli_gu) (#4056) Bhaumik PatelandBaber Abbasi 2026-08-29 19:48:05 +05:30
• a966806e4d Add Uncheatable Eval (#3442) Ziqing HuangandBaber Abbasi 2026-08-28 12:42:49 -07:00
• d1193e39a2 fix(scripts): repair requests_caching.py entrypoint (bad kwargs + helper defined after use) (#4059) Tai An 2026-08-28 12:35:12 -07:00
• 9fd734a220 chore(pre-commit) (#4060) Baber Abbasi 2026-08-28 23:18:52 +05:00
• 0fdda000ec fix(ruler): resolve the tokenizer name before the cached lookup (#4048) Gordeeva Nataliia 2026-08-28 21:16:37 +03:00
• b101fbf1ce fix(minerva_math): stop sqrt shorthand normalization from corrupting indexed roots (#4037) 3 people 2026-08-29 01:51:59 +08:00
• 95469434ff fix(minerva_math): thousands-separator comma strip fuses bare digit tuples ("0,1" -> "01") (#4039) feiandBaber Abbasi 2026-08-29 00:37:27 +08:00
• 17a4506c2a fix(minerva_math): score an answer identical to the gold as correct (#4034) Gordeeva Nataliia 2026-08-28 19:03:47 +03:00
• ebe1704a45 fix(minerva_math): correct few-shot prompt LaTeX (#4045) Ben YuanandBaber Abbasi 2026-08-28 23:25:50 +08:00
• 0f8479cc83 Fix: Prevent infinite loop when max_seq_lengths < 4096 in prepare_niah.py (#3372) Binayak DeyandBaber Abbasi 2026-08-27 23:42:08 +05:30
• 02a1125e53 feat(cli): allow key=value to be passed to --metadata; linting (#4054) Baber Abbasi 2026-08-27 23:06:20 +05:00
• bc4354e865 add GreekMMLU (official native-sourced benchmark) task configuration (#3581) mersin konomiandBaber Abbasi 2026-08-27 16:29:57 +03:00
• 55cf7888ec feat(tydiqa): add TyDiQA Gold Passage tasks (9 languages) (#4044) bono 2026-08-27 22:02:35 +09:00
• 7e2c8d00db feat(tasks): add LongProc benchmark (6 task types, 16 configs) (#3544) Xi Ye 2026-08-27 06:42:50 -06:00
• 06e3cb5f31 feat: add chrf++ aggregation and metric (word_order=2) (#3780) Krishna VenkateshandBaber Abbasi 2026-08-27 18:08:50 +05:30
• 0cce2dfff1 Pass dataset_kwargs for Unitxt tasks (#3230) Matthew PrahlandBaber Abbasi 2026-08-27 08:28:10 -04:00
• b93b6b2238 Fix the Unitxt init method to set the task name (#3225) Matthew Prahl 2026-08-27 08:20:44 -04:00
• d8c66720e6 fix(cache): create missing cache parent directories (#4047) Divanshu 2026-08-27 17:32:55 +05:30
• 8b955c4839 Add mlqa tag to run all variants. (#2977) 3 people 2026-08-27 08:29:15 -03:00
• dd417662e5 MMLU Task Name (#3660) Joshua Zhang 2026-08-26 12:38:13 -04:00
• 2927029716 Fix MultiChoiceRegexFilter prefix-shadowing of choice text (#3884) Shardul DeshpandeandBaber Abbasi 2026-08-26 22:02:01 +05:30
• e1ace28331 Fix group stderr to match weight_by_size=False (unweighted) aggregation (#3882) Shardul Deshpande 2026-08-26 20:30:56 +05:30
• ec73eb76fe Fix ContextSampler crash when the few-shot pool contains duplicate eval_doc rows (#3790) Kyle 2026-08-26 09:38:28 -05:00
• 3cc2a76f91 test(registry): add tests for higher_is_better directions (#4050) Baber Abbasi 2026-08-26 19:09:10 +05:00
• ccde35c889 Fix TER metric direction and correct chrF docstring (#3993) Leshem Choshen 2026-08-26 09:15:20 -04:00
• a8931d6fb1 Serialize all numpy scalar types in JSON output, not just int64/int32 (#3885) Shardul Deshpande 2026-08-26 18:05:04 +05:30
• 496d65eafc Align task name in README with yaml (#4025) James A. Michaelov 2026-08-26 09:30:21 -03:00
• eb31833f56 feat(models): add raw onnxruntime backend for Model Builder ONNX exports (#3984) amd-sourjya 2026-08-26 05:22:33 -07:00
• 64f3d0924f fix: local directory with task name no longer shadows registered task (#3670) Naomi Loughlin 2026-08-24 14:14:40 -04:00
• 6937d5f334 fix(med_prescriptions): require both keys before combining complaints and diagnosis (#4041) Tai AnandBaber Abbasi 2026-08-24 11:12:21 -07:00
• ffb2f7b0df fix: fall back to tokenizer.eos_token when decode returns empty string (#3657) Ganesh R 2026-08-24 19:22:45 +05:30
• 59b416c740 fix(irokobench): repair afrimmlu/afrimgsm/afrixnli task registration and update READMEs (#3841) Philip Sinitsin 2026-08-24 14:43:19 +01:00
• a95910d156 Add cieaCOVA task into catalan bench (#4043) Baber Abbasiandsgs97ua 2026-08-24 18:35:50 +05:00
• d439eaa034 Add make_table regression coverage (#3788) Aryan PuttaandBaber Abbasi 2026-08-24 08:38:03 -04:00
• 1f96192998 Add Terretaqa task into Catalanbench (#3784) Ivan Martinez Murillo 2026-08-24 14:29:17 +02:00
• 3113a98c26 fix: make delete_cache a no-op when the cache directory is absent (#4035) fei 2026-08-24 20:22:53 +08:00
• 4e7e0d47f3 fix(filters): format_span only normalizes labels, not entity text (#3887) Kyle DickinsonandBaber Abbasi 2026-08-21 10:56:54 -04:00
• 9067fdb83e fix: normalize scalar seed from a config file the way the CLI does (#4003) Madan kumarandBaber Abbasi 2026-08-21 19:35:25 +05:30
• 404ac7c415 fix(config): parse string dictionary arguments (#4020) Zhewen Tan 2026-08-21 21:23:14 +08:00
• fd75bb98fb Enable function resolving for custom tasks (#3992) Zhaojiang SunandBaber Abbasi 2026-08-21 21:19:52 +08:00
• b0195a94f4 feat(models): add cross-platform onnxruntime-genai backend + refactor winml (#3960) 3 people 2026-08-21 08:20:42 -04:00
• 1323ffe16f fix(api): support think_end_token for chat completions (#3959) buduoqiu 2026-08-21 00:02:09 +08:00
• e0ec2b4bfa Honor configured timeout for synchronous API requests (#3995) Siavash 2026-08-20 08:43:43 -07:00
• 3e093bc90a fix(huggingface): detect max_length from nested text_config (Gemma3 multimodal) (#3916) Dan MercedeandBaber Abbasi 2026-08-20 08:34:10 -07:00
• e321c2221c megatron_lm.py: run linter (#4024) Baber Abbasi 2026-08-20 19:01:56 +05:00
• 8a8a069f4f fix(hf): pass max_cpu_memory through to accelerate's max_memory (#4016) Tai An 2026-08-20 07:00:06 -07:00
• cc9b6c2a2a chore: update pre-commit hooks and workflow packages (#4023) Baber Abbasi 2026-08-20 18:43:24 +05:00
• 4850314509 fix megatron v0.18 compatibility (#3991) oberpierre 2026-08-20 14:49:14 +02:00
• 8a07e1110d feat(legalbench): add Contract NLI suite (14 NDA entailment tasks) (#3954) bono 2026-08-14 18:42:09 +09:00
• 73da0818e8 Putnam Axiom (#3998) 5 people 2026-08-14 13:46:16 +05:00
• 72d2638514 fix: resolve fewshot gen_prefix against the fewshot doc (#3979) Aditya Singh 2026-08-14 00:16:03 -07:00
• e90e5b656b fix: handle auto:N batch_size format in API models (#3970) Muhammad Abdullah Rasheed 2026-08-11 09:24:39 +01:00
• 3264352d1a fix: prevent ValueError when batch_size="auto" is passed to neuronx model (#3971) Muhammad Abdullah Rasheed 2026-08-11 09:23:46 +01:00
• 6e61725220 fix(samplers): exclude eval docs from first-n samples (#3978) Siavash 2026-08-11 01:15:27 -07:00
• f1a0cb5f1d Fix fewshot_config.split precedence in TaskConfig (#3937) Andrew ChenandClaude Opus 4.8 2026-08-11 15:49:44 +08:00
• f9472c4d55 fix(jsonschema_bench): add per-sample validation timeout to prevent eval hangs (#3923) Han-Cheol Cho 2026-08-11 16:30:12 +09:00
• ad3ee81919 fix(afrixnli): use Jinja braces in prompt_1 doc_to_text (#3944) Solaris-star 2026-08-10 19:31:31 +08:00
• a97cfc217e fix(longbench): skip blank leading line in code_sim_score extraction (#3921) Cameron ShinnandBaber Abbasi 2026-08-10 03:40:26 -07:00
• 877a609346 Fix broken afrobench group task references (afrisenti/mafand prompt_2) (#3847) Daoyuan Li 2026-08-10 03:08:14 -07:00
• fe9a1c7936 fix(tmmluplus): load from ikala/tmmluplus so the task works on datasets>=4 (#3950) Gowtham Sai G 2026-08-10 15:26:29 +05:30
• 7575f6e692 fix(tasks): use script-less parquet mirrors for mathqa, siqa, and moral_stories (#3943) Shubz 2026-08-10 10:54:20 +01:00
• 671b1550ff fix(scrolls): load SCROLLS without the removed dataset script (#3975) Ayaan Gazali 2026-08-09 09:15:57 -07:00
• d245db26a9 fix(mmlusr): restore dataset loading for all 171 MMLU-SR tasks (#3972) Ayaan Gazali 2026-08-09 09:06:12 -07:00
• a5cdfdd4ae fix(arithmetic): load arithmetic tasks without the removed dataset script (#3976) Ayaan Gazali 2026-08-09 08:49:57 -07:00
• 434d473452 fix(tasks): use namespaced dataset paths for webqs, medmcqa, and wmt16 (#3942) Ayaan Gazali 2026-08-09 08:30:28 -07:00
• 68bab35e13 fix(tasks): use namespaced dataset path for wsc273 (#3946) Ayaan GazaliandClaude Fable 5 2026-08-09 08:28:16 -07:00
• 09a9193378 fix(tests): pass dataset_kwargs in test_download (#3980) Ayaan Gazali 2026-08-09 08:27:25 -07:00
• 0423d98e20 fix(spanish_bench): load wnli_es without the removed dataset script (#3982) Ayaan Gazali 2026-08-09 01:09:50 -07:00
• 62b0963449 fix(prost): load PROST without the removed dataset script (#3977) Ayaan Gazali 2026-08-09 01:06:31 -07:00
• f4d4b3de3e feat(portuguese_bench): add ASSIN2 RTE and STS tasks (#3812) bono 2026-07-14 05:18:15 +09:00
• 0236a71c39 Add IndicParam: MCQ benchmark for 12 low-resource Indic languages (#3826) Himanshu Joshi 2026-07-14 01:41:30 +05:30
• 4b6c8f1653 fix(tasks): use namespaced dataset paths for xnli/xcopa/paws-x/xquad (#3870) Taksh KothariandCursor 2026-07-14 01:39:28 +05:30
• 97a5e2c710 fix(kormedmcqa): extract answer choice per paper Appendix B (#3842) Philip Sinitsin 2026-06-24 20:51:08 +01:00
• 8d3d131af4 Fix space in North Macedonian task identifiers in INCLUDE suite (#3848) Daoyuan Li 2026-06-24 12:23:44 -07:00
• 0334aec529 feat(legalbench): add HELM-lite LegalBench subset (5 tasks) (#3860) bonoandbongho.lee 2026-06-25 04:19:32 +09:00
• 8a5dec73a5 fix sglang args (#3817) wm901115nwpu 2026-06-23 00:13:18 +08:00
• 8200000520 Warn when vLLM ignores device argument (#3803) SophusandParadoxiar 2026-06-22 23:42:29 +08:00
• 62ebbf57fa fix: keep anthropic stop sequences nonempty (#3822) Yufeng He 2026-06-22 23:09:44 +08:00
• 1e814746f0 fix(ruler): replace defunct hotpot dataset host with pinned HF mirror (#3806) Tai An 2026-06-22 08:02:41 -07:00
• a212af8e18 Fix "trasnlation" typo in DarijaBench translation task names (#3824) Daoyuan Li 2026-06-22 08:01:02 -07:00
• 1b16d59c18 fix: strip whitespace from input/target for FDA, SWDE, and SQuAD_completion tasks (#3795) Ephraiem SarabamounandBaber Abbasi 2026-06-22 07:36:47 -07:00
• 1dd9310873 task name was set to set to humaneval_single_line_infilling_light instead of humaneval_random_span_infilling_light causing key error (#3768) Jaydeep Borkar 2026-06-02 11:06:38 -04:00
• f8594275c0 update global piqa (#3816) 3 people 2026-06-02 19:59:48 +05:00
• e7e6bc67fc feat: 0.5.0.dev2 next0.5 Baber Abbasi 2026-05-11 19:24:39 +05:00