v0.1.1 — Beta · Apache-2.0 · on PyPI as argus-llm

Heuristic quality scores for your LLM outputs

ARGUS-AI is a Beta (0.1.x) Python library that computes G-ARVIS scores for LLM responses across six dimensions. All scorers are rule-based heuristics (word overlap, regex, keyword lists), not trained models, and the repo has no accuracy benchmarks against labelled data. Use the scores as trend signals to monitor over time, not as verdicts on any single response.

View on GitHub See an example
pip install argus-llm
quickstart.py
agentic_eval.py
monitoring.py
# pip install argus-llm (import name: argus_ai)
import argus_ai
argus = argus_ai.init(profile="enterprise")
result = argus.evaluate(
prompt="What causes climate change?",
response="Greenhouse gas emissions from fossil fuels are the primary driver.",
context="Climate change is driven by human activities releasing greenhouse gases.",
model_name="claude-sonnet-4",
latency_ms=1200.0,
input_tokens=45, output_tokens=30, cost_usd=0.002,
)
Output on argus-llm 0.1.1:
Composite: 0.755
Passing: True
Safety: 1.000
Scores depend on your inputs and may change between releases. Groundedness here is low (0.167) because the response shares few words with the context: the heuristic measures word overlap, not meaning.
from argus_ai.types import AgenticEvalRequest
workflow = AgenticEvalRequest(
prompt="Research competitors and generate report",
response="Report generated with 5 competitor analyses.",
steps_planned=8,
steps_completed=7,
steps_failed=2,
steps_recovered=1,
retries=3,
total_cost_usd=0.45,
)
result, agentic = argus.evaluate_agentic(workflow)
Agentic Metrics
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Agent Stability Factor 0.632
Error Recovery Rate 0.540
Cost Per Completed Step 0.357
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Output on argus-llm 0.1.1.
from argus_ai.monitoring.thresholds import ThresholdConfig
from argus_ai.monitoring.alerts import AlertRule, AlertSeverity
argus = argus_ai.init(
profile="healthcare",
thresholds=ThresholdConfig(safety_min=0.90),
on_alert=lambda msg, res: pagerduty.trigger(msg),
)
Illustrative output (hand-written example, not a captured run):
Request 1 Composite: 0.872 Safety: 0.950 ✓ PASS
Request 2 Composite: 0.714 Safety: 0.720 ✗ FAIL
[CRITICAL] safety score 0.720 below threshold 0.90
PII detected: emails=1, phones=1
Request 3 Composite: 0.698 Safety: 0.850 ✗ FAIL
[HIGH] accuracy score 0.580 below threshold 0.65
speculation_penalty: 0.15
Monitor Statistics
safety: 2/3 breaches (67%)
accuracy: 1/3 breaches (33%)
composite: 2/3 breaches (67%)
[SUSTAINED] safety degradation: 67% of last 3 evaluations below threshold

Six dimensions of LLM quality

Each response is scored 0 to 1 by a rule-based heuristic, and a weighted composite combines the six. Treat the numbers as trend signals to watch over time, not as verdicts on a single response.

G
Groundedness
Word overlap between the response and the context you supply, with small adjustments for attribution and speculation phrases. Neutral 0.5 when no context is given. A signal of drift from the provided context, not a hallucination detector.
A
Accuracy
Token-level F1 against a ground-truth answer when you supply one, plus simple internal-consistency and numeric checks. Flags obvious mismatches with a reference; it does not verify facts.
R
Reliability
Truncation markers, JSON and markdown format checks, response length relative to the prompt, and latency tiers. A structural-quality signal.
V
Variance
Wording signals within a single response: hedging versus definitive phrases and vocabulary diversity. Multi-run variance analysis is not part of this package.
I
Inference Cost
Token, cost, and latency ratios computed from the numbers you pass in. Useful for budget tracking.
S
Safety
Regex and keyword scan for PII-like patterns (email, SSN, card, phone), a short toxicity word list, and system-prompt or jailbreak marker phrases. It will miss things and produce false positives, and does not replace dedicated safety or security tooling.

Agentic evaluation metrics

Three formula-based metrics, each in [0, 1], for monitoring autonomous workflows from step counts, retries, and cost that you supply.

ASF
Agent Stability Factor
Completion ratio × (1 − failure rate) × a retry-penalty consistency factor. A signal of whether a workflow reliably finishes what it starts.
ERR
Error Recovery Rate
Recovered steps divided by failed steps, with a small bonus for fewer retries per recovery. Shows how many failed steps were later recovered.
CPCS
Cost Per Completed Step
Cost per completed step scored against a budget per step (default $0.10). Higher is cheaper relative to that budget.

5 weight profiles

Enterprise, healthcare, finance, consumer, agentic. The profiles only change how the six scores are weighted into the composite; they do not provide or certify regulatory compliance.

Threshold monitoring

Sliding-window breach detection with configurable alert rules, severity levels, and callback hooks.

Prometheus & OTEL

Export G-ARVIS scores as Prometheus metrics, or emit them through the OpenTelemetry API to whatever exporter you configure. Not tested against specific vendor backends.

Provider wrappers

InstrumentedAnthropic and InstrumentedOpenAI clients that attach a G-ARVIS score to each response.

Open test suite

Tests for the scoring, monitoring, alerting, and SDK layers live in the public repository. The repo has no accuracy benchmarks against labelled data.

Fast, local evaluation

Pure-Python heuristics with no network calls. Measured on one machine (Xeon E3-1505M, Python 3.11, default logging): median about 0.6 ms, p99 about 2.6 ms per evaluation. Your numbers will vary.

What is in the package, and what is not

Available now — Apache 2.0

argus-llm (import argus_ai)

  • G-ARVIS composite scorer
  • 6 heuristic dimension scorers
  • ASF, ERR, CPCS agentic metrics
  • Small SDK with init / evaluate
  • Threshold monitoring & alerting
  • Prometheus & OpenTelemetry export
  • Anthropic & OpenAI wrappers
  • @argus_evaluate decorator
Planned — not available

ARGUS Platform (commercial)

  • Autonomous Correction Loop
  • Prompt Optimizer
  • LLM-as-Judge evaluation
  • Multi-run variance analysis
  • Dashboard UI

None of this is included in the package or available to buy today. There are no dates or pricing.

Install argus-llm. Score your LLM calls.

Open source (Apache-2.0), Beta, rule-based heuristics. pip install argus-llm, then import argus_ai.

View on GitHub Contact Us

Questions or feedback: contact form or email anil@ambharii.com