feat(dataproc): Add Spark Job to Cluster sample #13459
Conversation
|
Here is the summary of changes. You are about to add 1 region tag.
This comment is generated by snippet-bot.
|
product-auto-label
bot
added
samples
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @aman-ebay, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request adds a new Python sample to the repository, showcasing the submission of a Spark job to a Dataproc cluster with specific driver node group configurations. The sample provides a practical example of interacting with the Dataproc API to manage Spark workloads and retrieve their results.
Highlights
- New Sample Addition: This pull request introduces a brand new Python sample script,
submit_spark_job_to_driver_node_group_cluster.py, to thedataproc/snippetsdirectory. - Spark Job Submission: The new sample demonstrates how to programmatically submit a Spark job (specifically, the SparkPi example) to a Google Cloud Dataproc cluster using the
google-cloud-dataprocclient library. - Driver Node Group Configuration: A key feature of this sample is the inclusion of
driver_scheduling_config, allowing users to specify resource allocations (like memory and vcores) for the Spark job's driver on the Dataproc cluster's driver node group. - Job Output Retrieval: The script also includes logic to retrieve and print the output of the submitted Spark job, demonstrating how to parse the
driver_output_resource_uriand download the output from Google Cloud Storage.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
glasnt
added
kokoro:force-run
kokoro-team
removed
the
kokoro:force-run
glasnt
changed the title
glasnt
added
the
kokoro:force-run
| def test_workflows(capsys): | ||
|
|
||
| # create temporary cluster for test | ||
| create_cluster.create_cluster(PROJECT_ID, REGION, CLUSTER_NAME) |
There was a problem hiding this comment.
I assume"submit_spark_job_to_driver_node_group_cluster.submit_job" will fail unless the cluster that is created is a driver node group cluster (see https://cloud.google.com/dataproc/docs/guides/node-groups/dataproc-driver-node-groups#create_a_driver_node_group_cluster):
gcloud dataproc clusters create CLUSTER_NAME
--region=REGION
--driver-pool-size=SIZE
--driver-pool-id=NODE_GROUP_ID
There was a problem hiding this comment.
I'll adjust the test accordingly 👍
There was a problem hiding this comment.
It doesn't look like there's any example of how to create a driver node group cluster in the python docs. Since I'll have to add it for the test, I'll set it up so that it could also be used in your content. I'll give you more details on the bug internally 👍
There was a problem hiding this comment.
Thanks. Please surround with region tags if the docs should pull in a selected code snippet.
* Create submit_spark_job_to_driver_node_group_cluster.py * add a system test, copied from instantiate_line_workflow, create cluster tests * use create cluster sample to create cluster * black, isort * ensure job check is specified in output * create node cluster manually --------- Co-authored-by: aman-ebay <amancuso@google.com>
Description
Fixes b/424371877
Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.
Checklist
nox -s py-3.9(see Test Environment Setup)nox -s lint(see Test Environment Setup)