Methodology

Right now: we aggregate, we don't score

• This draft shows only figures published by others (SEA-HELM, Revolab, pendakwah.tech, ILMU). Each table names the owner, the date and a link.
• We never merge scores from different tests into one number. Different tests measure different things.
• Vendor-run or vendor-claimed results are flagged. Results where a maker ranks its own model are flagged as such.
• Prices are shown in RM only when taken from an official RM price page (currently ILMU). Other vendors' prices are coming soon, converted from official USD pages with the exchange rate and date shown.

Coming: our own weekly tests

• Chat: public Malay test sets (e.g. MalayMMLU subsample) plus a small private set of Manglish / code-switched prompts, scored by exact match and double-blind human review. Same prompts and settings for every model.
• Speech-to-text: word error rate on public Malaysian speech data, normalised the same way for every model; plus noisy / code-switched clips.
• Text-to-speech: blind pairwise listening test by native Malaysian raters (formal BM and casual Manglish sentences), plus intelligibility via round-trip transcription.
• Details, cost and schedule are in our internal plan; we will publish test sets' descriptions and settings before the first scored week.

Independence

Sponsors and paying vendors never change rankings. Paid "get benchmarked" results are published as-is with a "paid listing" label, and the same test is available to everyone.

Corrections

Spot a wrong number or a missing model? A correction contact will be added at launch (placeholder).