fix: TestPatchResourceTimeout flaky test by googs1025 · Pull Request #132715 · kubernetes/kubernetes · GitHub | Latest TMZ Celebrity News & Gossip | Watch TMZ Live
Skip to content

fix: TestPatchResourceTimeout flaky test #132715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

fix: TestPatchResourceTimeout flaky test #132715

wants to merge 1 commit into from
+13 −7

Conversation

googs1025
Copy link
Member

What type of PR is this?

/kind failing-test
/kind flake

What this PR does / why we need it:

Which issue(s) this PR is related to:

fix: #132712

Special notes for your reviewer:

Does this PR introduce a user-facing change?

None

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

None

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. kind/flake Categorizes issue or PR as related to a flaky test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 4, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 4, 2025
@googs1025
Copy link
Member Author

after this change:

➜  kubernetes git:(fix/flaky_patch) ✗ stress -p 1 ./handlers.test                                                                       
5s: 134 runs so far, 0 failures, 1 active
10s: 328 runs so far, 0 failures, 1 active
15s: 519 runs so far, 0 failures, 1 active
20s: 699 runs so far, 0 failures, 1 active
25s: 878 runs so far, 0 failures, 1 active
30s: 1059 runs so far, 0 failures, 1 active
35s: 1226 runs so far, 0 failures, 1 active
40s: 1396 runs so far, 0 failures, 1 active
45s: 1560 runs so far, 0 failures, 1 active
50s: 1729 runs so far, 0 failures, 1 active
55s: 1901 runs so far, 0 failures, 1 active
1m0s: 2074 runs so far, 0 failures, 1 active
1m5s: 2191 runs so far, 0 failures, 1 active
1m10s: 2324 runs so far, 0 failures, 1 active
1m15s: 2487 runs so far, 0 failures, 1 active
1m20s: 2660 runs so far, 0 failures, 1 active
1m25s: 2834 runs so far, 0 failures, 1 active
1m30s: 3011 runs so far, 0 failures, 1 active
1m35s: 3186 runs so far, 0 failures, 1 active
1m40s: 3361 runs so far, 0 failures, 1 active
1m45s: 3524 runs so far, 0 failures, 1 active
1m50s: 3694 runs so far, 0 failures, 1 active
1m55s: 3863 runs so far, 0 failures, 1 active
2m0s: 4028 runs so far, 0 failures, 1 active
2m5s: 4189 runs so far, 0 failures, 1 active

@googs1025
Copy link
Member Author

before this change:

➜  kubernetes git:(fix/flaky_patch) ✗ stress -p 1 ./handlers.test                                                                       
5s: 129 runs so far, 0 failures, 1 active
10s: 323 runs so far, 0 failures, 1 active

/var/folders/j3/b8896xf92g7d60x2ghdpcnd40000gn/T/go-stress-20250704T113628-2677718024
E0704 11:36:42.373343   51767 request.go:120] "Encoding failed of request object" err="encoding failed: v1.DeleteOptions is not suitable for converting to \"meta.k8s.io/v1\" in scheme \"k8s.io/apiserver/pkg/endpoints/handlers/delete_test.go:91\"" auditID="" gvr="/v1, Resource=pods" obj="&DeleteOptions{GracePeriodSeconds:*30,Preconditions:nil,OrphanDependents:nil,PropagationPolicy:*Background,DryRun:[],IgnoreStoreReadErrorWithClusterBreakingPotential:nil,}"
E0704 11:36:42.373615   51767 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: encoding is not allowed for this codec: <nil>" logger="UnhandledError"
E0704 11:36:42.373638   51767 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"encoding is not allowed for this codec: <nil>\"}: encoding is not allowed for this codec: <nil>" logger="UnhandledError"
E0704 11:36:42.374887   51767 delete.go:414] "failed to authorize" err="error while checking permission for \"unsafe-delete-ignore-read-errors\", unexpected error"
E0704 11:36:42.377425   51767 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 24.083µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
--- FAIL: TestPatchResourceTimeout (0.00s)
    rest_test.go:466: Starting test TestPatchResourceTimeout
    rest_test.go:514: Working with patchType application/json-patch+json
    rest_test.go:514: Working with patchType application/merge-patch+json
    rest_test.go:514: Working with patchType application/strategic-merge-patch+json
    rest_test.go:612: TestPatchResourceTimeout: expected error Timeout: request did not complete within requested timeout - context deadline exceeded, but got context deadline exceeded
FAIL


ERROR: exit status 1

15s: 517 runs so far, 1 failures (0.19%), 1 active
20s: 705 runs so far, 1 failures (0.14%), 1 active
25s: 883 runs so far, 1 failures (0.11%), 1 active
30s: 1057 runs so far, 1 failures (0.09%), 1 active
35s: 1221 runs so far, 1 failures (0.08%), 1 active

@googs1025
Copy link
Member Author

cc @liggitt @jpbetz @ylink-lfs

@googs1025
Copy link
Member Author

/assign @liggitt @jpbetz

I reproduced it locally, made some changes, and ran a stress test. This instability test basically does not occur again.

@ylink-lfs
Copy link
Contributor

I think it's a bypass method. Is the root cause hard to investigate in a short time?

@googs1025
Copy link
Member Author

I'm not sure why sometimes ctx doesn't seem to encapsulate the error every time, but instead directly returns the "context deadline exceeded" message. This seems to be a very rare case (as can be seen from flaky), so I think we can just use strings.Contains to check for errors.

@googs1025
Copy link
Member Author

In addition, this case is a test of a timeout case, I don't think it affects the main process. It is not a fix to the root (I think the root cause of ctx returning different errors is relatively complex and not introduced by the original PR), but it is reasonable and effective in the short term. 🤔

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: googs1025
Once this PR has been reviewed and has the lgtm label, please ask for approval from jpbetz. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@googs1025 googs1025 force-pushed the fix/flaky_patch branch 3 times, most recently from d930b82 to 65e3593 Compare July 4, 2025 04:19
@googs1025
Copy link
Member Author

new change test:

➜  kubernetes git:(fix/flaky_patch) ✗ stress -p 1 ./handlers.test                                                                       
5s: 114 runs so far, 0 failures, 1 active
10s: 297 runs so far, 0 failures, 1 active
15s: 480 runs so far, 0 failures, 1 active
20s: 667 runs so far, 0 failures, 1 active
25s: 847 runs so far, 0 failures, 1 active
30s: 1003 runs so far, 0 failures, 1 active
35s: 1143 runs so far, 0 failures, 1 active
40s: 1311 runs so far, 0 failures, 1 active
45s: 1480 runs so far, 0 failures, 1 active
50s: 1647 runs so far, 0 failures, 1 active
55s: 1788 runs so far, 0 failures, 1 active
1m0s: 1946 runs so far, 0 failures, 1 active
1m5s: 2105 runs so far, 0 failures, 1 active
1m10s: 2266 runs so far, 0 failures, 1 active
1m15s: 2414 runs so far, 0 failures, 1 active
1m20s: 2581 runs so far, 0 failures, 1 active
1m25s: 2748 runs so far, 0 failures, 1 active
1m30s: 2919 runs so far, 0 failures, 1 active
1m35s: 3087 runs so far, 0 failures, 1 active
1m40s: 3250 runs so far, 0 failures, 1 active
1m45s: 3406 runs so far, 0 failures, 1 active
1m50s: 3540 runs so far, 0 failures, 1 active
1m55s: 3694 runs so far, 0 failures, 1 active
2m0s: 3835 runs so far, 0 failures, 1 active
2m5s: 4002 runs so far, 0 failures, 1 active
2m10s: 4171 runs so far, 0 failures, 1 active
2m15s: 4343 runs so far, 0 failures, 1 active
2m20s: 4486 runs so far, 0 failures, 1 active
2m25s: 4655 runs so far, 0 failures, 1 active
2m30s: 4816 runs so far, 0 failures, 1 active
2m35s: 4984 runs so far, 0 failures, 1 active
2m40s: 5151 runs so far, 0 failures, 1 active
2m45s: 5325 runs so far, 0 failures, 1 active
2m50s: 5495 runs so far, 0 failures, 1 active
2m55s: 5616 runs so far, 0 failures, 1 active
3m0s: 5778 runs so far, 0 failures, 1 active
3m5s: 5946 runs so far, 0 failures, 1 active
3m10s: 6111 runs so far, 0 failures, 1 active
3m15s: 6284 runs so far, 0 failures, 1 active
3m20s: 6456 runs so far, 0 failures, 1 active
3m25s: 6631 runs so far, 0 failures, 1 active
3m30s: 6795 runs so far, 0 failures, 1 active
3m35s: 6955 runs so far, 0 failures, 1 active
3m40s: 7107 runs so far, 0 failures, 1 active
3m45s: 7272 runs so far, 0 failures, 1 active
3m50s: 7441 runs so far, 0 failures, 1 active
3m55s: 7610 runs so far, 0 failures, 1 active
4m0s: 7759 runs so far, 0 failures, 1 active
4m5s: 7923 runs so far, 0 failures, 1 active
4m10s: 8069 runs so far, 0 failures, 1 active
4m15s: 8217 runs so far, 0 failures, 1 active
4m20s: 8383 runs so far, 0 failures, 1 active
4m25s: 8551 runs so far, 0 failures, 1 active
4m30s: 8723 runs so far, 0 failures, 1 active
4m35s: 8889 runs so far, 0 failures, 1 active
4m40s: 9050 runs so far, 0 failures, 1 active
4m45s: 9221 runs so far, 0 failures, 1 active
4m50s: 9386 runs so far, 0 failures, 1 active
4m55s: 9559 runs so far, 0 failures, 1 active
5m0s: 9724 runs so far, 0 failures, 1 active
5m5s: 9889 runs so far, 0 failures, 1 active
5m10s: 10055 runs so far, 0 failures, 1 active

@googs1025
fix: TestPatchResourceTimeout flaky test
3689e1f 
Signed-off-by: googs1025 <googs1025@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@sttts sttts Awaiting requested review from sttts

Assignees

@jpbetz jpbetz

@liggitt liggitt

Labels
area/apiserver cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. kind/flake Categorizes issue or PR as related to a flaky test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Flaking Test] TestPatchResourceTimeout get inconsistent err return
5 participants
@googs1025 @k8s-ci-robot @ylink-lfs @jpbetz @liggitt

TMZ Celebrity News – Breaking Stories, Videos & Gossip

Looking for the latest TMZ celebrity news? You've come to the right place. From shocking Hollywood scandals to exclusive videos, TMZ delivers it all in real time.

Whether it’s a red carpet slip-up, a viral paparazzi moment, or a legal drama involving your favorite stars, TMZ news is always first to break the story. Stay in the loop with daily updates, insider tips, and jaw-dropping photos.

🎥 Watch TMZ Live

TMZ Live brings you daily celebrity news and interviews straight from the TMZ newsroom. Don’t miss a beat—watch now and see what’s trending in Hollywood.