tPjI8UdZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z ddl mZdd lmZdd lmZdd lmZdd lmZdd lmZmZddlmZmZmZddlmZm Z m!Z!ddl"m#Z#dZ$e e%d<dZ&e e%d<dZ'e e%d<dZ(e e%d<e)e)e*e*fdfZ+e e%d<e)e+e+e*dze*dzfZ,e e%d<e iZ-e ee*e*fe%d<eZ.e e%d<dee*e*fdzde*dzfd Z/d!eddfd"Z0d#ee*e*fde*fd$Z1d#ee*e*fde*fd%Z2ed&d&'Gd(d)Z3Gd*d+Z4y),aPer-request multi-tenant tracer routing. When a request carries team/key vendor credentials in ``standard_callback_dynamic_params``, or the key/team config resolved at auth names a destination project or a service name, its spans must export through a ``TracerProvider`` whose OTLP headers carry those credentials / that project, or whose Resource carries that ``service.name``. ``TenantTracerCache`` builds and caches one provider per distinct (credentials, project, service name) tuple, and otherwise hands back the logger's default tracer. This lets a single logger fan requests out to many tenants without needing a logger per tenant. N) OrderedDict)Mapping) dataclass)MappingProxyType)Final TypeAliasquote)TracerProvider)Tracer)verbose_logger)OTEL_SERVICE_NAME_METADATA_KEYS) ExporterSpecOpenTelemetryV2Config)build_tracer_providerexporter_transport get_tracer)dynamic_otlp_endpointdynamic_otlp_headersproject_routing_headers)StandardCallbackDynamicParams)console in_memoryinmemorymemory_NON_OTLP_KINDS) otlp_grpcgrpc _GRPC_KINDS_MAX_CACHED_PROVIDERS@_MAX_RETIRED_PROVIDERS. _HeaderItems _RouteKey _NO_HEADERS_SERVICE_NAME_KEYS auth_metadatareturnc@sytfdtDdS)zThe per-request ``service.name`` override for this key/team, if any. ``None`` keeps the env-configured default (``OTEL_SERVICE_NAME``). Nc3nK|],}j|xsdjxs).yw)N)getstrip).0keyr(strippeds o/home/oem/CyberBrain26-unified/.venv/lib/python3.12/site-packages/litellm/integrations/otel/plumbing/routing.py z&tenant_service_name..[s6hcARARSVAWA][]@d@d@f4fH4fhs*55)nextr')r(r1s`@r2tenant_service_namer5Ss$  h/h  providerc~ |jy#t$r }tjd|Yd}~yd}~wwxYw)adFlush + stop an evicted provider's processors (reclaims their threads). ``TracerProvider.shutdown`` force-flushes each ``SpanProcessor`` before stopping it, so any spans already handed to a ``BatchSpanProcessor`` are exported rather than dropped. Best-effort: a shutdown failure must not break the request that triggered the eviction. z1OTel V2: error shutting down evicted provider: %sN)shutdown Exceptionr debug)r7es r2_shutdown_providerr=`s;U UPRSTTUs <7<headerscNdjd|jDS)N,c30K|]\}}|d|yw)=Nr/r0values r2r3z'_plain_header_string..osGese1UG$Gsjoinitemsr>s r2_plain_header_stringrJns 88Gw}}G GGr6cNdjd|jDS)zPercent-encode values so one containing the ``k=v,k=v`` separators (e.g. a project name with a comma) survives; ``parse_env_headers`` decodes it back. r@c3FK|]\}}|dt|dyw)rBr,)safeNr rDs r2r3z)_encoded_header_string..vs'Wese1U5r234Ws!rFrIs r2_encoded_header_stringrNrs 88Ww}}W WWr6T)frozenslotsc:eZdZUdZeed<eed<dZedzed<y) TenantRoutea+The tracer to create a span on, plus whether it must root its own trace. ``detached`` is True when the routed span exports to a DIFFERENT backend than the request's root span, which always exports through the default tracer. A detached span roots a fresh trace with a link back to the request trace for correlation, so the destination account is not left holding a child whose parent it never received. It is driven by whether routing headers were actually applied to an owned exporter, not merely requested: a credential or project route whose callback owns no exporter those headers can reach exports through the default backend unchanged, so it stays parented like an unrouted span. Credential routing (a team/key's own vendor account) is one detaching case: the root, auth, and db spans stay on the operator's default backend while the LLM-call span exports to the tenant's account, so parenting it into the request trace makes the tenant account show a fragmented span with a missing parent. Project routing (Phoenix) is the other: Phoenix assigns a whole trace to one project by whichever of its spans arrives first, so a project-routed span parented into the request trace gets dragged into the project of the default-exported request spans and the header does nothing. Both mirror the v1 loggers, which exported each request under its own backend-local root. Service-name routing does NOT detach: it relabels ``service.name`` on the SAME operator backend, where the parent is present. tracerdetachedNr7) __name__ __module__ __qualname____doc__r __annotations__boolr7r rCr6r2rRrRys$2 NN '+Hnt#*r6rRc eZdZdZdededzdeddfdZdedzddfd Z dd e d e dzd e eefdzde fd Z dede eefde eefdedzdedzdef dZdedzfdZd e dzde eeffdZd e eefdzde eeffdZ dde eefde eefdedzdedzdef dZ ddede eefde eefdedzdef dZy)TenantTracerCachezQTenant-scoped ``TracerProvider`` cache keyed by routing headers and service name.config callback_nameN tracer_namer)c4||_|_||_tj|_t |_i|_t |_ tfd|jD}d|v|_ d|vxsd|v|_ d|_d|_y)Nc3hK|])}|jk(st|j+ywN)ownerrkind)r/specr^s r2r3z-TenantTracerCache.__init__..s,( .2$**XeJe tyy )( s22httprF)_config_callback_name _tracer_name threadingLock_lockr _providers_open_span_counts_retiredtuple exporters_project_routable_credential_routable_warned_project_unroutable_warned_credential_unroutable)selfr]r^r_owned_transportss ` r2__init__zTenantTracerCache.__init__s  +'&NN, M =?;F= #(( 6<6F6F( # "(+;!;$*.>$>$\&L\B\!*/'-2*r6r7cn|y|j5|jj|ddz }|dkDr||j|< dddy|jj|d||jv}|jj|ddddr t |yy#1swYxYw)a]Drop one open-span count; shut a retired provider down once drained. ``None`` (the default route) is a no-op so callers can release a ``TenantRoute.provider`` unconditionally. The shutdown itself runs outside the lock: it force-flushes over the network and must not stall every concurrently routing request. Nr)rlrnr-popror=)rvr7 remainingdraineds r2releasezTenantTracerCache.releases    ZZ .#5599(AFJI1}3<&&x0  . .  " " & &x 6%6G MM  h - .  x (  . .s5B+AB++B4defaultdynamic_paramsr(c|j|}|j|}t|}|s|s| t|dSt |j |}t t|jt t|j||f}|j5|j|||||} |jj| ddz|j| <|j} ddd t| tt |j t#|xs t#|| S#1swYNxYw)uReturn the tracer (and trace-detachment flag) for this request. Use ``default`` unless the request's dynamic credentials, its key/team project, or its key/team service name require a scoped tracer, in which case build (or reuse) one. The cache is a bounded LRU: the least-recently-used provider is flushed and shut down on overflow so its exporter threads don't accumulate. A routed provider is returned already held — its open-span count is incremented in the same critical section as the cache update — so a concurrent overflow eviction can't shut it down between selection and the caller's span start. The caller must ``release`` it exactly once. NF)rSrTrrz)rSrTr7)_credential_headers_project_headersr5rRrrhrpsortedrHrl_cached_provider_lockedrnr-_evicted_on_overflow_lockedr=rrirZ) rvrrr(credential_headersproject_headers service_nameendpoint cache_keyr7evicteds r2 route_forzTenantTracerCache.route_forsO&%)$<$<^$L!%!6!6}!E1-@ !/l>Rg> >00C0C^T &+1134 5 &..01 2    ZZ @"::-,H04/E/E/I/I(TU/VYZ/ZD " "8 ,!==?G  @   w 'h(9(9:/*Fd3E.F   @ @s )AEErrrrrc|jj|}||jj||St|j ||||}||j|<|Srb)rmr- move_to_endr_routed_config)rvrrrrrcachedbuilts r2rz)TenantTracerCache._cached_provider_lockedsi++I6   OO ' ' 2M,    2OX| \ &+ " r6ct|jtkry|jjd\}}|jj |ddk(r|Sd|j |<t|j tkry|j jd\}}|jj|d|S)aPop the LRU provider past the cap; return it if the caller must shut it down. A provider with open spans is retired to drain instead: stopping its processors while a span opened at ``pre_call`` is still live would silently drop that span at end instead of exporting it. Retirees are themselves capped, so the stalest one is shut down (and its open-span count dropped, making its eventual ``release`` a no-op) once too many pile up rather than letting them accumulate a thread each. NF)lastr) lenrmr!popitemrnr-ror#r{)rv_r overfloweds r2rz-TenantTracerCache._evicted_on_overflow_lockeds t #8 8__,,%,8 7  ! ! % %gq 1Q 6N!% g t}} !7 7 --5-9 A "":t4r6ct|j|xst}|r |jr|S|js'd|_t j d|jtS)aThe per-request dynamic OTLP credentials, if this cache can apply them. A callback owning only a console/in_memory exporter has nowhere to stamp them, so the span would export to the operator's default backend unchanged; routing there and detaching would orphan it on the very backend that holds its parent. Warn once and keep the default tracer. TzOTel V2: %s request carries dynamic credentials, but the callback owns no OTLP exporter to stamp them onto; spans export to the default backend.)rrhr&rsrur warning)rvr requesteds r2rz%TenantTracerCache._credential_headers.sc00C0C^TcXc D55 1115D .  " "Y##  r6ct|j|}|r |jr|S|js'd|_t j d|jt S)zThe per-request project-routing headers, if this cache can apply them. A gRPC-only exporter can't (the project header route is HTTP-only), so the request warns once and stays on the env-configured default project. TzOTel V2: %s key/team config names a per-request project, but its exporter is not OTLP/HTTP and the project header is HTTP-only; spans stay in the default project.)rrhrrrtr rr&)rvr(rs r2rz"TenantTracerCache._project_headersBs^ 343F3F V D22 ...2D +  " "###   r6c |jjDcgc]}|j||||}}|d|in||d}|jj|Scc}w)uClone the config, rewriting headers on the callback's own exporter. Both header sets apply only to the exporter ``self._callback_name`` contributed (``spec.owner``). A request that carries one tenant's Arize key must never rewrite the headers of a co-configured Langfuse or self-hosted collector exporter, which would leak that key to a different backend. Dynamic credentials REPLACE the exporter's headers — they are the tenant's complete credential set. Project headers APPEND instead: the preset's static headers carry the backend auth (Phoenix's ``Authorization``), which must survive routing to a project. rq)rqrupdate)rgrq_routed_exporter model_copy)rvrrrrrerqrs r2rz TenantTracerCache._routed_configUsy, ..   ! !$(:OX V  )5(<[) $PYkwBx ||&&f&55 sArec|jj}|j|jk7s|tvr|S|r t |n |j }|r+|tvr#djd|t|fDn|}d|fd|ffD cic]2\}} |dk(r||j k7s|dk(r|||jk7r|| 4} }} | s|S|j| Scc} }w)Nr@c3&K|] }|s| ywrbrC)r/parts r2r3z5TenantTracerCache._routed_exporter..~s^dY]T^sr>rr) rdlowerrcrhrrJr>rrGrNrr) rvrerrrrdbaseroutedfieldrErs r2rz"TenantTracerCache._routed_exporterrsiioo' ::,, ,0GKBT*+=>Z^ZfZf4{#: HH^t-CO-T&U^ ^ #,V!4z86L M u"v'=#(<T]]AZ 5L  "tEtf'EE  s7C!rb)NN)rUrVrWrXrstrrxr r~r rrrRrr%rrrrrrrrCr6r2r\r\s[ 3% 3Tz 3 3  3D) 5)$)237 - - 6<- sCx(4/ -  - ^$CH-!c*  *  Dj  $^d-B,2ORV2Vcfhkck[l(gc3h.?$.F7SVX[S[K\. $#' 6#CH-6!c*6* 6 Dj 6  6D $ FF$CH-F!c* F * F  Fr6r\)5rXrj collectionsrcollections.abcr dataclassesrtypesrtypingrr urllib.parser opentelemetry.sdk.tracer opentelemetry.tracer litellm._loggingr litellm.constantsr&litellm.integrations.otel.model.configrr,litellm.integrations.otel.plumbing.providersrrr!litellm.integrations.otel.presetsrrrlitellm.types.utilsrrrYrr!r#rprr$r%r&r'r5r=rJrNrRr\rCr6r2rs ##!"#2&+=V  >HG + U* #u"!#"c3h 45 i5\<tS4ZOP 9P(8(< U738$ %<<E; wsCx'84'? C$J  U UD UH'#s("3HHXGCH$5X#X $d# + +$ +FkFkFr6