test: adds pytest-xdist to speed up processing of CI/CD checks by chalmerlowe · Pull Request #2153 · googleapis/python-bigquery · GitHub | Latest TMZ Celebrity News & Gossip | Watch TMZ Live
Skip to content

test: adds pytest-xdist to speed up processing of CI/CD checks #2153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Apr 11, 2025
Merged

test: adds pytest-xdist to speed up processing of CI/CD checks #2153

merged 8 commits into from
Apr 11, 2025

Conversation

chalmerlowe
Copy link
Collaborator

DO NOT MERGE.
WIP experiments with pytest-xdist.

@chalmerlowe chalmerlowe added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Apr 2, 2025
@product-auto-label product-auto-label bot added size: xs Pull request size is extra small. api: bigquery Issues related to the googleapis/python-bigquery API. labels Apr 2, 2025
@product-auto-label product-auto-label bot added size: s Pull request size is small. and removed size: xs Pull request size is extra small. labels Apr 3, 2025
chalmerlowe
chalmerlowe commented Apr 11, 2025
View reviewed changes
@chalmerlowe chalmerlowe removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Apr 11, 2025
@chalmerlowe chalmerlowe changed the title experimentation using pytest-xdist test: adds pytest-xdist to speed up processing of CI/CD checks Apr 11, 2025
chalmerlowe
chalmerlowe commented Apr 11, 2025
View reviewed changes
@chalmerlowe
Copy link
Collaborator Author

@Linchin

Using pytest-xdist for tests that run under pytest shortens the time for a number of our CI/CD checks. In this image, darker green means more time savings. For some checks we cut total time by five minutes.

Our two worst offenders do not benefit, because:

  • PRESUBMIT: has nox sessions internally and though each is less than 2 minutes long, each goes through a virtualenv setup, package install, etc and does so sequentially thus by the time we finish ~10 sessions, we have aggregated 13 minutes). This should be broken into individual CI/CD checks (out of scope).
  • LINTING-TYPING: does not use pytest, instead these tests run using mypy, pytype, etc. pytype is the long pole in tent at 10 minutes. Despite this, these should be broken into individual CI/CD checks to allow more of them to run in parallel (out of scope). It may be possible to provide a wrapper to run pytype against multiple files in parallel using multiprocessing or subprocess modules (out of scope).
Screenshot 2025-04-11 at 6 37 57 AM

@chalmerlowe chalmerlowe marked this pull request as ready for review April 11, 2025 11:47
@chalmerlowe chalmerlowe requested review from a team as code owners April 11, 2025 11:47
@chalmerlowe chalmerlowe requested review from tswast and nicain April 11, 2025 11:47
tswast
tswast approved these changes Apr 11, 2025
View reviewed changes
noxfile.py
@@ -129,6 +130,7 @@ def default(session, install_extras=True):
# Run py.test against the unit tests.
session.run(
"py.test",
"-n=auto",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the default is the number of CPUs on the instance, right? I recall in bigframes we set this a bit higher than that and it seemed to help.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you are correct, that the default is the number of CPUs on the instance.

Linchin
Linchin approved these changes Apr 11, 2025
View reviewed changes
Copy link
Contributor

@Linchin Linchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the tests a lot faster!

@chalmerlowe chalmerlowe merged commit 1cabacb into main Apr 11, 2025
17 checks passed
@chalmerlowe chalmerlowe deleted the test-xdist-experimentation branch April 11, 2025 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tswast tswast tswast approved these changes

@Linchin Linchin Linchin approved these changes

@nicain nicain Awaiting requested review from nicain nicain is a code owner automatically assigned from googleapis/python-samples-owners

Assignees

@leahecole leahecole

Labels
api: bigquery Issues related to the googleapis/python-bigquery API. size: s Pull request size is small.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@chalmerlowe @tswast @Linchin @leahecole

TMZ Celebrity News – Breaking Stories, Videos & Gossip

Looking for the latest TMZ celebrity news? You've come to the right place. From shocking Hollywood scandals to exclusive videos, TMZ delivers it all in real time.

Whether it’s a red carpet slip-up, a viral paparazzi moment, or a legal drama involving your favorite stars, TMZ news is always first to break the story. Stay in the loop with daily updates, insider tips, and jaw-dropping photos.

🎥 Watch TMZ Live

TMZ Live brings you daily celebrity news and interviews straight from the TMZ newsroom. Don’t miss a beat—watch now and see what’s trending in Hollywood.