feat(api): retain error spans and sample remaining traces at 50% by AchoArnold · Pull Request #1034 · NdoleStudio/httpsms

AchoArnold · 2026-10-29T17:34:37Z

Summary

Use the existing logger to mark warning/error spans with httpsms.must_sample=true; do not copy log messages into spans.
Logger.WithSpan now receives the actual span rather than only its IDs. This removes the separate logger wrapper and all caller-offset machinery.
Filter completed spans before the existing batch exporter: retain errors, recorded exceptions, explicit flags, database spans over 200ms, and truncated telemetry; export 50% of the rest using a deterministic hash of the full trace ID, including UUID-shaped IDs.
Standalone logs and metric measurements are not sampled. Metric exemplars use only the stable sampled trace-ID subset so they do not point to spans discarded by this policy. Log messages, correlation fields, decorations, callers, and destinations are preserved.

Minimal scope

Keep the original GORM logger and dedicated-database logging setup; only update existing span bindings.
No second GORM logger, configuration layer, parameter filters, stack-walking, or log-severity text parsing.
The extra test-double edits only update the WithSpan signature; no service, handler, listener, or validator behavior changes.
Keep the two heartbeat database context-propagation fixes so database spans remain connected to their requests.
Keep focused tests for sampling decisions, retention flags, slow database boundaries, truncation, and unchanged structured log output.

Tradeoffs

All spans are still recorded locally; this reduces exported volume, not instrumentation overhead.
Retained spans outside the normal 50% sample can have missing normal parent or sibling spans.
Slow-database retention uses the span's measured duration and database attributes, independently of GORM's log timing/configuration.
The shared processor is used by the Axiom and Google trace providers.

Validation

go test ./pkg/telemetry ./pkg/services ./pkg/repositories ./pkg/handlers ./pkg/listeners ./pkg/validators ./pkg/middlewares ./pkg/di
go build ./...
No new lint issues across the same packages using golangci-lint run --new-from-rev=origin/main with Go 1.26.0.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d

AchoArnold · 2026-10-29T17:34:44Z

@greptileai review

codacy-production · 2026-10-29T17:35:49Z

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 113 complexity · 0 duplication

Metric Results

Complexity 113

Duplication 0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer}
_{TIP This summary will be updated as you push new changes.}

greptile-apps · 2026-10-29T17:37:26Z

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no new blocking issues were found.

Summary

The PR retains warning/error spans and applies a trace-ID-based 50% sample to remaining spans.

Trace exporters send flagged spans and half of the rest.
Warnings and errors now leave clues on their active spans.
GORM query errors and slow queries now link to active spans.

No new actionable issues were found. No previous review threads were supplied. Tests were inspected but not run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Span ends] --> B{Trace ID in 50% sample?}
  B -->|Yes| E[Queue for export]
  B -->|No| C{Error, exception, warning, or dropped telemetry?}
  C -->|Yes| E
  C -->|No| D[Do not export]

_{Reviews (2) · Last reviewed commit: "fix(api): address trace sampling review ..." · Reviewed by Greptile}

Copilot

🟡 Changes recommended

Raw error text can expose customer content and API keys in traces, while span retention, production database warning capture, and log caller attribution remain unreliable.

4 open issues

What changed in this PR

This PR reduces exported API trace volume by retaining warning/error spans and deterministically sampling 50% of remaining spans after completion. All spans remain locally recorded.

Changes:

Adds a shared sampling processor for Axiom and Google trace providers.
Links application and GORM warning/error logs to active spans.
Adds tests for sampling, error retention, logger decoration, and event limits.

File	Description
`api/pkg/telemetry/span_logger.go`	Adds span-linked warning, error, and fatal logging.
`api/pkg/telemetry/sampling_span_processor.go`	Filters completed spans using severity and trace-ID sampling.
`api/pkg/telemetry/sampling_span_processor_test.go`	Tests sampling decisions and retention signals.
`api/pkg/telemetry/otel_tracer.go`	Wraps context loggers with span-aware logging.
`api/pkg/telemetry/gorm_logger.go`	Routes GORM logs through span-aware loggers.
`api/pkg/di/container.go`	Installs the shared processor for both trace providers.

_{Review effort: Balanced}

Retain spans with truncated telemetry and capture dedicated database slow-query warnings. Preserve original callers for span-linked logs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d

AchoArnold · 2026-10-29T18:32:39Z

@greptileai review

Copilot

🟡 Changes recommended

Unredacted log messages can expose credentials and message bodies in trace exports, while warning spans may still be dropped when event attributes are truncated.

4 open issues

1 previously missed issue

In code that hasn't changed since last review

Limit request bodies copied into trace events

api/pkg/telemetry/span_logger.go:48

[P2] This copies unrestricted error text into queued spans; HTTPRequestLogger and MessageHandler.PostSend include entire request bodies in their warnings. Large rejected requests therefore increase trace-buffer memory use and export payload size in addition to the existing log traffic. Omit the message from the span event, keeping the severity marker for retention and the full text in the standalone log.

3 resolved since last review

_{Review effort: Balanced}

Keep standalone log output intact and simplify telemetry conditions and test fixtures. Propagate heartbeat update contexts and defer SQL formatting until a log is emitted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d

Copilot

🟡 Changes recommended

SQL parameters can leak despite redaction, uppercase warning and error events can be dropped, and delegated GORM logs lose the original query call site.

3 open issues

4 resolved since last review

_{Review effort: Balanced}

Keep original query callers in standard GORM log output and honor opt-in parameter filtering. Retain warning and error events regardless of severity capitalization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d

Copilot

🔵 Needs a closer look

Fatal logging’s interaction with span completion and telemetry flushing warrants human assessment.

No open issues 🎉

3 resolved since last review

_{Review effort: Balanced}

Remove the extra GORM logger, configuration layer, parameter filters, and caller repair. Retain database spans over 200ms in the export processor without changing logging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d

Bind actual spans through Logger.WithSpan instead of adding a logger wrapper. Remove caller offsets and unused severity parsing while preserving log output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d

Copilot

🟡 Changes recommended

The processor can discard spans with truncated link data and leave metric exemplars pointing to discarded spans.

2 open issues

_{Review effort: Balanced}

Share the trace-ID decision with exemplar filtering without dropping metric measurements. Retain spans with dropped links or link attributes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d

Copilot

🔵 Needs a closer look

The interaction between UUIDv4 trace IDs and shared sampling warrants human assessment because it can exclude all normal spans and their exemplars.

No open issues 🎉

2 resolved since last review

_{Review effort: Balanced}

Avoid UUID version and variant bits biasing the shared 50% decision. Keep exported spans and metric exemplars aligned for UUID-shaped IDs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d

AchoArnold · 2026-10-30T00:22:41Z

Addressed the UUID sampling advisory in 0467d46. The previous ratio sampler rejected all normal spans in a deterministic set of valid UUIDv4-shaped trace IDs because its decision used bits affected by the UUID variant. The shared span/exemplar decision now hashes the full 16-byte trace ID and selects half the hash space. This removes that fixed-bit bias without adding configuration or changing metric measurements. Added a UUID-shaped population regression and updated both export/exemplar fixtures to exercise opposite outcomes.

Copilot

🔵 Needs a closer look

Slow-query retention needs production-like validation because the tests synthesize span attributes rather than exercise the GORM tracing plugin’s actual attributes and timing.

No open issues 🎉

_{Review effort: Balanced}

feat(api): retain error spans and sample remaining traces at 50%

6c1881c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 60fb2b83-6e9d-4656-9f60-6183c61b716d

AchoArnold requested a balanced review from Copilot October 29, 2026 18:17

Copilot started reviewing on behalf of AchoArnold October 29, 2026 18:17

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

Comment thread api/pkg/telemetry/span_logger.go Resolved

Comment thread api/pkg/telemetry/gorm_logger.go Resolved

Comment thread api/pkg/telemetry/sampling_span_processor.go Resolved

Comment thread api/pkg/telemetry/span_logger.go Resolved

AchoArnold requested a balanced review from Copilot October 29, 2026 18:31

Copilot started reviewing on behalf of AchoArnold October 29, 2026 18:32

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

Comment thread api/pkg/di/container.go Resolved

Comment thread api/pkg/di/container.go Resolved

Comment thread api/pkg/telemetry/gorm_logger.go Resolved

AchoArnold requested a balanced review from Copilot October 29, 2026 18:46

Copilot started reviewing on behalf of AchoArnold October 29, 2026 18:46

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

Comment thread api/pkg/telemetry/gorm_logger.go Resolved

Comment thread api/pkg/telemetry/gorm_logger.go Resolved

Comment thread api/pkg/telemetry/sampling_span_processor.go Resolved

AchoArnold requested a balanced review from Copilot October 29, 2026 18:59

Copilot started reviewing on behalf of AchoArnold October 29, 2026 19:00

View session

Copilot AI reviewed Oct 29, 2026

View reviewed changes

AchoArnold and others added 2 commits October 30, 2026 01:34

AchoArnold requested a balanced review from Copilot October 30, 2026 00:01

Copilot started reviewing on behalf of AchoArnold October 30, 2026 00:01

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

Comment thread api/pkg/telemetry/sampling_span_processor.go Resolved

Comment thread api/pkg/telemetry/sampling_span_processor.go Resolved

AchoArnold requested a balanced review from Copilot October 30, 2026 00:11

Copilot started reviewing on behalf of AchoArnold October 30, 2026 00:11

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

AchoArnold requested a balanced review from Copilot October 30, 2026 00:22

Copilot started reviewing on behalf of AchoArnold October 30, 2026 00:22

View session

Copilot AI reviewed Oct 30, 2026

View reviewed changes

📋 Server Request & Proxy Logs

⚙️ Pengaturan Proxy Target

Conversation

AchoArnold commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Minimal scope

Tradeoffs

Validation

Uh oh!

AchoArnold commented Oct 29, 2026

Uh oh!

codacy-production Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Up to standards ✅

Uh oh!

greptile-apps Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AchoArnold commented Oct 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

AchoArnold commented Oct 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AchoArnold commented Oct 29, 2026 •

edited

Loading

codacy-production Bot commented Oct 29, 2026 •

edited

Loading

greptile-apps Bot commented Oct 29, 2026 •

edited

Loading