Commit 46be034
authored
feat(llm-obs): added report evaluator integration from pydantic (#17032)
## Description
Add integration for pydantic ai report evaluators that return scalars (i.e. return a `ScalarResult` object) to be submitted as summary evaluators in experiments sdks
<!-- Provide an overview of the change and motivation for the change -->
## Testing
Pydantic summary report evaluators are working in experiments. I used this [test script](https://github.com/DataDog/llm-obs/pull/245/changes#diff-ab2adacd550f0dd442d7bcdbf2a73ed6bb5064ad6f5ad5ddbe7a19ec45875df8) which generated [this experiment](https://dd.datad0g.com/llm/experiments/0a70a7d8-215b-44a5-9b12-cbd045d99d6b?project=jenn-test-pydantic-evaluators).
Added unit tests as well confirming that both sync and async pydantic summary evaluators are working and that unsupported types (i.e. non ScalarReport types) fail as expected.
<!-- Describe your testing strategy or note what tests are included -->
## Risks
<!-- Note any risks associated with this change, or "None" if no risks -->
None
## Additional Notes
<!-- Any other information that would be helpful for reviewers -->
Co-authored-by: jennifer.mickel <jennifer.mickel@datadoghq.com>1 parent ee88ac2 commit 46be034
17 files changed
Lines changed: 808 additions & 974 deletions
File tree
- .riot/requirements
- ddtrace/llmobs
- releasenotes/notes
- tests/llmobs
This file was deleted.
Lines changed: 37 additions & 54 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
| 5 | + | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
14 | 13 | | |
15 | 14 | | |
16 | | - | |
17 | | - | |
| 15 | + | |
| 16 | + | |
18 | 17 | | |
19 | 18 | | |
20 | 19 | | |
21 | 20 | | |
22 | | - | |
23 | 21 | | |
24 | 22 | | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
| 23 | + | |
29 | 24 | | |
30 | 25 | | |
31 | 26 | | |
32 | | - | |
33 | 27 | | |
34 | | - | |
35 | 28 | | |
36 | | - | |
| 29 | + | |
37 | 30 | | |
38 | | - | |
39 | | - | |
40 | | - | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
41 | 34 | | |
42 | 35 | | |
43 | 36 | | |
44 | | - | |
45 | | - | |
| 37 | + | |
| 38 | + | |
46 | 39 | | |
47 | | - | |
48 | | - | |
49 | 40 | | |
50 | | - | |
51 | | - | |
52 | | - | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
53 | 44 | | |
54 | | - | |
55 | 45 | | |
56 | 46 | | |
57 | | - | |
58 | 47 | | |
59 | 48 | | |
60 | 49 | | |
| |||
64 | 53 | | |
65 | 54 | | |
66 | 55 | | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
74 | 64 | | |
75 | 65 | | |
76 | | - | |
77 | 66 | | |
78 | 67 | | |
79 | 68 | | |
80 | | - | |
81 | | - | |
82 | 69 | | |
83 | | - | |
84 | | - | |
| 70 | + | |
| 71 | + | |
85 | 72 | | |
86 | 73 | | |
87 | 74 | | |
88 | 75 | | |
89 | | - | |
90 | | - | |
91 | | - | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
92 | 80 | | |
93 | 81 | | |
94 | 82 | | |
95 | 83 | | |
96 | 84 | | |
97 | 85 | | |
98 | | - | |
99 | 86 | | |
100 | 87 | | |
101 | 88 | | |
102 | 89 | | |
103 | | - | |
| 90 | + | |
104 | 91 | | |
105 | | - | |
106 | | - | |
| 92 | + | |
| 93 | + | |
107 | 94 | | |
108 | 95 | | |
109 | | - | |
110 | | - | |
| 96 | + | |
111 | 97 | | |
112 | 98 | | |
113 | 99 | | |
| |||
118 | 104 | | |
119 | 105 | | |
120 | 106 | | |
121 | | - | |
122 | | - | |
123 | | - | |
| 107 | + | |
124 | 108 | | |
125 | 109 | | |
126 | 110 | | |
127 | | - | |
| 111 | + | |
128 | 112 | | |
129 | 113 | | |
130 | 114 | | |
131 | 115 | | |
132 | | - | |
133 | 116 | | |
134 | | - | |
135 | | - | |
| 117 | + | |
136 | 118 | | |
137 | 119 | | |
138 | 120 | | |
139 | | - | |
140 | 121 | | |
141 | 122 | | |
| 123 | + | |
142 | 124 | | |
143 | 125 | | |
144 | 126 | | |
145 | 127 | | |
146 | 128 | | |
147 | 129 | | |
148 | 130 | | |
| 131 | + | |
149 | 132 | | |
150 | 133 | | |
151 | 134 | | |
0 commit comments