Biocontainment Observations TAC to CAMP export - #1019
RameshRapa wants to merge 3 commits into
Conversation
… the Biocontainmnet Observations.
…fb_tac_camp_bioObs
…tenCO, FoodEnrichmentCO and DehydrationCO so that the ETL merge populates them in TAC_src.BiocontainmentObservations
| RETURN; | ||
| END; | ||
|
|
||
| INSERT INTO dbo.editCommentBuffer (tableName, tid, editComment) |
There was a problem hiding this comment.
This write to editCommentBuffer broke every write back on Moe for the demo branch. CAMP's update trigger needs the editCommentBuffer row to exist before the update so the insert has to move above it, plus a cleanup for the rows that lose the rowversion check.
| END; | ||
|
|
||
| INSERT INTO dbo.editCommentBuffer (tableName, tid, editComment) | ||
| SELECT 'BiocontainmentObservation', targetTid, 'TAC ETL sync' |
There was a problem hiding this comment.
CAMP's audit trail can't tell who made the edit in TAC. We should put the TAC user's email or username in there instead of a fixed string, since we are able to retrieve that from the labkey db. editComment would be the only place in CAMP's audit that can show who made the change.
| l.NasalDischarge, l.FeedEaten, l.FeedEaten_CO, l.FoodEnrichment, | ||
| l.FoodEnrichment_CO, l.Stool, l.FluidIntake, l.Dehydration, | ||
| l.Dehydration_CO, l.Comment | ||
| FROM TAC_src.BiocontainmentObservations l |
There was a problem hiding this comment.
Right now on approval, or a CAMP row coming back through the import, gets updated in CAMP with the same values and writes audit rows for an edit nobody made. Can we only fan out rows whose values actually differ from CAMP?
| FluidIntake = COALESCE(m.FluidIntake, b.FluidIntake), | ||
| Dehydration = COALESCE(m.Dehydration, b.Dehydration), | ||
| Dehydration_CO = COALESCE(m.Dehydration_CO, b.Dehydration_CO), | ||
| Comment = COALESCE(m.Comment, b.Comment) |
There was a problem hiding this comment.
Comment is the one nullable column, so COALESCE means clearing it in TAC never reaches CAMP, and the import pulls the old one back in TAC. This one should just be a straight assignment instead
| -- Log (non-fatal) any not-yet-fanned-out landing rows that didn't match | ||
| -- a CAMP row. | ||
| IF EXISTS (SELECT 1 FROM TAC_src.BiocontainmentObservations l | ||
| WHERE (l.FannedOutDate IS NULL OR l.FannedOutDate < l.Modified) |
There was a problem hiding this comment.
FannedOutDate in 4 places is set from SQL Server's clock but compared against Labkey's modified, so any skew between two servers can make a later edit look already sent. If we store the modified value we pushed instead of GETDATE() both sides come from the same clock
| WHERE (l.FannedOutDate IS NULL OR l.FannedOutDate < l.Modified) | ||
| AND NOT EXISTS (SELECT 1 FROM @matched m WHERE m.landingObjectId = l.objectid) | ||
| ); | ||
| RAISERROR('TAC_src.usp_FanOutBiocontainmentObservations: %d row(s) had no matching ObjectId in dbo.BiocontainmentObservation and were skipped', 10, 1, @unmatchedCount) WITH NOWAIT; |
There was a problem hiding this comment.
Severity 10 RAISERRORs don't come back through SqlExecutor, so the skipped and conflict counts never show up in the ETL log. Can the proc return counts as a result set and have the task log them?
| Comment = COALESCE(m.Comment, b.Comment) | ||
| OUTPUT inserted.tid, m.landingTid INTO @updated (targetTid, landingTid) | ||
| FROM dbo.BiocontainmentObservation b | ||
| INNER JOIN @matched m ON m.targetTid = b.tid AND b.timestamp = m.capturedTimestamp; |
There was a problem hiding this comment.
Since the fan out updates every row in one statement, one value CAMP's check constraints reject rolls back the whole batch, and the same row blocks every write back every 10 minutes after that. We should skip rows that would fail constraints and report them in the counts instead of blocking the entire record set
| </alternateKeys> | ||
| <columnTransforms> | ||
| <column source="createdBy" target="createdBy" transformClass="org.labkey.snprc_ehr.columnTransforms.UserTransform"/> | ||
| <column source="modifiedBy" target="modifiedBy" transformClass="org.labkey.snprc_ehr.columnTransforms.UserTransform"/> |
There was a problem hiding this comment.
createdBy and modifiedBy should already be Labkey user IDs. The UserTransforms here create a deactivated @noreply-txbiomed.org style user for every editor. I think we should drop the columnTransforms block here
2 participants
This PR has code changes for Biocontainment Observations TAC to CAMP export data implementation.