Repository navigation
[bot] Merge master/94382166 into rel/dev - #1949
Merged
Merged
Conversation
The dashboard simulated reply always ended with "Date range: ...". A case without a date_range key read as None and was answered with "all time", although such a case does not look at the date filter. That reply can contradict the cases it is sent for. When a case expects existing charts that carry their own date filter (this month, last 12 months), "all time" leads the agent to author all-time copies instead of reusing them, and the case fails on the missing chart ids although the agent did what the reply said. The reply now says the date range only when the key is present, the same rule the scoring already follows. A null date_range still asks for all time, so the cases that state one get the same reply as before. jira: QA-29724 risk: low
An edit case stopped at the first turn that asked back instead of
patching, by design: the simulated reply is built for creation and
names charts and a date range, which answers nothing an edit asks.
An edit whose wording left room for a question, such as where in a
section new charts should go, therefore failed without a patch even
when the agent's question was reasonable and offered a default.
An edit case can now carry `clarification_reply`, the text its
author wants the simulated user to send. When the agent asks back,
that text is sent once. Nothing is generated or guessed, so the
reply stays fixed across runs and says only what the fixture says.
Whether a turn asked is decided by classify_reply, as the document
skill does, so a turn that asked nothing, such as one reporting a
rejected patch, is not nudged into a retry. A turn that only asks to
go ahead ("Shall I apply this?") is told "Yes, go ahead.", as the
conversation runner does; that does not spend the case's reply, and
the turn cap still bounds it. A real question that ends in "Please
confirm" is still a question, read with asks_in_text, and gets the
case's reply.
The run still stops when the case has no reply, or when the agent
asks a second time. It now records the question it stopped on as a
note (not scored) in the assertion message, so triage can tell a
fixture that needs a reply from an agent that did the wrong thing.
A structured clarifyingQuestions part is quoted in place of the
prose beside it, and a run whose turn cap ends it before the reply
could be sent says that too. A turn that neither asked nor patched
is quoted in a note of its own.
These stops were reported as budget_exhausted, although turns were
left. They now report a new LoopExit, stopped_early, so a count of
exhausted budgets no longer includes runs the harness ended itself.
A refusal case, answered in one turn on purpose, reports it too.
A run that does hit the cap keeps budget_exhausted.
The key must be a non-empty string and is rejected on a creation
case, which keeps its reply built from the charts and date range.
jira: QA-29515
risk: low
…eply fix(gooddata-eval): make the dashboard simulated user follow the case
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1949 +/- ##
===========================================
+ Coverage 86.52% 86.55% +0.03%
===========================================
Files 349 349
Lines 26957 27011 +54
===========================================
+ Hits 23324 23379 +55
+ Misses 3633 3632 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 9438216 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/43349502886).