Pewarta Proxy

📋 Server Request & Proxy Logs

Loading active proxy settings...
Developed by Khairul Adha (adhexa.id)
Mengambil log server...

⚙️ Pengaturan Proxy Target

Skip to content

Add fixless audit support - #109

Open
GabeBlanchard wants to merge 3 commits into
masterfrom
gabe/fixless-audits
Open

GabeBlanchard wants to merge 3 commits into
masterfrom
gabe/fixless-audits

Conversation

@GabeBlanchard

@GabeBlanchard GabeBlanchard commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Pull Request Summary

Description

Adds first-class support for fixless audits: structured feedback on specific parts of a completed task's response, without modifying the response. Until now customers had to build the nested JSON for PUT /v1/audits by hand.

New public API:

  • ScaleClient.create_fixless_audit(task_id, target_response_url, accepted, feedback_items=None, comments=None, metadata=None) → PUT /v1/audits
  • ScaleClient.get_audit(audit_id) → GET /v1/audit/:id
  • ScaleClient.get_task_audits(task_id) → GET /v1/audits/task/:taskId (newest first)
  • Task.create_fixless_audit(target_response_url, accepted, **kwargs) and Task.audits()
  • scaleapi.audits: FeedbackItem (.error / .comment / .confirmation / .flag), Scope (.annotation / .scene / .point / .polygon / .lss), and the ErrorCategory, Severity, FeedbackItemType and AuditResult enums. frames takes a frame index or an inclusive (start, end) range.
audit = client.create_fixless_audit(
    task.id,
    task.response["annotations"]["url"],
    accepted=False,
    feedback_items=[
        FeedbackItem.error(Scope.annotation("cuboid_12", frames=(0, 9)), ErrorCategory.Geometry, Severity.Severe),
        FeedbackItem.error(Scope.point([412.0, 230.5], frames=4), ErrorCategory.Missing),
    ],
)

Design notes:

  • Validation stays on the API. The helpers only shape the payload, so client rules can't drift from the server's. Invalid audits raise ScaleInvalidRequest (400).
  • target_response_url is required. Customers already have it from the completed tasks they pulled down (task.response["annotations"]["url"]).
  • Completed tasks only. No stage_instance_id parameter.
  • FeedbackItem.error defaults to Severity.Standard, the baseline grading weight. A code comment explains this.
  • Bug fix in Api._api_request: it only treated 200 as success, and PUT /v1/audits responds 201 Created. A successful audit would have raised ScaleException(code=201) even though it was created. It now accepts 200 or 201; other status codes behave as before. PUT is not in the retry list, so no duplicate audits can come from retries.

Assumptions and dependencies:

  • Fixless audits must be enabled for the task's project, and the caller needs a team Admin or Manager API key (enforced by the API).
  • The API currently returns 409 both when fixless audits aren't enabled and when target_response_url isn't the task's latest response. The client maps 409 to ScaleDuplicateResource, which is misleading but unchanged here. Both cases are documented in the docstring.
  • A separate monorepo change (not yet merged) makes the API reject audits on tasks that aren't completed, with a 422. This SDK does not depend on it.

How did you test your code?

  • Created new unit tests in tests/ for the newly implemented methods: tests/test_audits.py, 6 tests. They run offline with mocks (payload and scope serialization, the PUT request body, and 201 handling), so they don't need SCALE_TEST_API_KEY. The live test project can't run fixless audits, so there is no live end-to-end test.
  • Updated existing unit tests in tests/ to cover changes made to existing methods

Ran pytest tests/test_audits.py on Python 3.8 and the repo's pre-commit hooks (isort, black, flake8, pylint) on all changed files. README.rst renders cleanly with readme_renderer, the same check twine check --strict uses.

Checklist

  • I have commented my code in details, particularly in hard-to-understand areas
  • I have updated Readme.rst document with examples for newly implemented public methods ("Create a Fixless Audit")
  • I have reviewed Deployment and Publishing Guide for Python SDK document
  • I incremented the SDK version in _version.py (2.18.4 → 2.19.0, a new feature)
  • In order to release a new version, a "Release Summary" needs to be prepared and published after the merge

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new actionable issues were found.

Summary

Adds helpers to create and fetch fixless audits without changing task responses.

  • Clients can create audits with feedback tied to parts of a task response.
  • Clients can fetch an audit or list a task's audits.
  • The request path accepts HTTP 201 responses as successful results.
  • DocumentTranscription task cases no longer run in the client test suite.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Task["Task.create_fixless_audit"] --> Client["ScaleClient.create_fixless_audit"]
    Feedback["FeedbackItem and Scope"] --> Payload["Build audit payload"]
    Client --> Payload
    Payload --> API["PUT /v1/audits"]
    API --> Created["201 response"]
    Created --> Audit["Audit object"]
Loading

Reviews (3) · Last reviewed commit: "Remove DocumentTranscription live tests" · Reviewed by Greptile

Adds ScaleClient.create_fixless_audit (PUT /v1/audits), get_audit and
get_task_audits, plus Task.create_fixless_audit / Task.audits. Feedback
items are built with FeedbackItem and Scope helpers instead of
hand-written JSON; validation stays on the API.

Also treats 201 Created as success in the HTTP layer: PUT /v1/audits
responds 201, which previously raised even though the audit was created.

Bumps version to 2.19.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread tests/test_audits.py Resolved
GabeBlanchard and others added 2 commits October 28, 2026 11:00
Covers get_audit, get_task_audits (including an empty list), returned
Audit objects, and Task.create_fixless_audit / Task.audits forwarding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The API no longer serves POST /v1/task/documenttranscription (the task
type was removed server-side in July 2026), so these tests 404 on every
CI run. TaskType.DocumentTranscription is left in place; removing it
would be a breaking change for a separate PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant