Repository navigation
DO NOT MERGE - Testing integration of #415, #417 and #418 on staging - #430
Closed
yuechao-qin wants to merge 6 commits into
Closed
yuechao-qin wants to merge 6 commits into
yuechao-qin wants to merge 6 commits into
Conversation
…than in the shared serializer _serialize_kubernetes_object_to_compact_dict no longer removes command lines; a _serialize_kubernetes_object_to_compact_dict_and_censor_it wrapper does, and the three call sites that persist a launched object use it.
…give up quietly Reading a container log goes through the API server's proxy to the node's kubelet, and a read that fails has almost always lost the log for good. The caller retries `upload_log` five times, so each lost log cost its single loop thread five requests and five waits for a log that would never arrive. Both Kubernetes launchers now read within their own budget and return quietly when it runs out. The budget wraps the read only: a failed write to storage still raises and still earns the caller's retries.
A container log is read through the API server's proxy to the node's kubelet. When the cluster-autoscaler reclaims the node, that proxy fails with a 500 whose body reports a failed dial of port 10250. The log died with the node, so retrying cannot recover it; the read is a lost cause from the first attempt. `_get_log_by_pod_key` now returns None for that error, beside the existing branches for a deleted Pod (404) and a container that has not started (400), and logs a warning carrying the full response body. The match is deliberately narrow: a 500 whose body names the container-log endpoint and a failed dial. Any other 500 still raises, so a reworded or unrelated error surfaces instead of being swallowed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this branch exists
Three open pull requests all touch the Kubernetes launcher and all want the same staging environment to prove themselves. This branch carries the three together so one staging deploy exercises all of them.
f2470fc,b709e6b,9c63943eabb056f15728d,ab15742Why the base is not
masterThe base is
3001032, which is the commitoasis-backend'sstagingbranch currently points itsbackend/submodule at. It is not onmaster: it is #396's work before that was squashed intodb041c2, so staging is also missingea250bc(#411) and1766484(#425).Basing the integration branch here means the only difference staging sees is the three pull requests themselves, and nothing else rides along.
Conflicts resolved
Two, both the same shape: #415 and #418 each append a new module-level function at the same point in
kubernetes_launchers.py. Both sides were kept, and #418's later rename of_api_exception_bodyto_extract_api_exception_bodywas applied on top.Tests
Full suite on this branch: 598 passed.
blackreports no changes.Do not merge
This branch exists to be deployed to staging and then thrown away. It can never be fast-forwarded onto
master, because its base is not onmaster. Each pull request merges on its own.