chore: add fast crc32c concat implementation by BenWhitehead · Pull Request #3148 · googleapis/java-storage · GitHub | Latest TMZ Celebrity News & Gossip | Watch TMZ Live
Skip to content

chore: add fast crc32c concat implementation #3148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 24, 2025

Conversation

BenWhitehead
Copy link
Collaborator

This new implementation provides a constant time computation regardless of the length of B, rather than the previous implementation which took linear time relative to the length of B.

Benchmark summary

CPU: AMD Ryzen Threadripper PRO 3945WX 12-Cores
OS: Debian Testing

# JMH version: 1.37
# VM version: JDK 11.0.17, OpenJDK 64-Bit Server VM, 11.0.17+8
# VM invoker: /home/benwhitehead/opt/java/jdk-11.0.17+8/bin/java
# VM options: -Xms1g -Xmx1g -XX:+AlwaysPreTouch
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 4 iterations, 30 s each
# Measurement: 8 iterations, 30 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Sampling time
Benchmark                                  (objectSize)    Mode      Cnt     Score    Error  Units
Crc32cBenchmark.concatCrc32c                     131072  sample  9243752     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                     262144  sample  9015106     0.078 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                     524288  sample  9010257     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                    1048576  sample  9165519     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                    2097152  sample  8993038     0.075 ±  0.001  us/op
Crc32cBenchmark.crc32cCombineGoogle              131072  sample  2070639   115.806 ±  0.018  us/op
Crc32cBenchmark.crc32cCombineGoogle              262144  sample  1029209   233.019 ±  0.045  us/op
Crc32cBenchmark.crc32cCombineGoogle              524288  sample   516517   464.360 ±  0.106  us/op
Crc32cBenchmark.crc32cCombineGoogle             1048576  sample   262118   915.067 ±  0.354  us/op
Crc32cBenchmark.crc32cCombineGoogle             2097152  sample   131385  1825.685 ±  0.834  us/op

This new implementation provides a constant time computation regardless of the length of B, rather than the previous implementation which took linear time relative to the length of B.

#### Benchmark summary

CPU: AMD Ryzen Threadripper PRO 3945WX 12-Cores
OS: Debian Testing
```
# JMH version: 1.37
# VM version: JDK 11.0.17, OpenJDK 64-Bit Server VM, 11.0.17+8
# VM invoker: /home/benwhitehead/opt/java/jdk-11.0.17+8/bin/java
# VM options: -Xms1g -Xmx1g -XX:+AlwaysPreTouch
# Blackhole mode: full + dont-inline hint (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 4 iterations, 30 s each
# Measurement: 8 iterations, 30 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Sampling time
```

```
Benchmark                                  (objectSize)    Mode      Cnt     Score    Error  Units
Crc32cBenchmark.concatCrc32c                     131072  sample  9243752     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                     262144  sample  9015106     0.078 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                     524288  sample  9010257     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                    1048576  sample  9165519     0.074 ±  0.001  us/op
Crc32cBenchmark.concatCrc32c                    2097152  sample  8993038     0.075 ±  0.001  us/op
Crc32cBenchmark.crc32cCombineGoogle              131072  sample  2070639   115.806 ±  0.018  us/op
Crc32cBenchmark.crc32cCombineGoogle              262144  sample  1029209   233.019 ±  0.045  us/op
Crc32cBenchmark.crc32cCombineGoogle              524288  sample   516517   464.360 ±  0.106  us/op
Crc32cBenchmark.crc32cCombineGoogle             1048576  sample   262118   915.067 ±  0.354  us/op
Crc32cBenchmark.crc32cCombineGoogle             2097152  sample   131385  1825.685 ±  0.834  us/op
```
@BenWhitehead BenWhitehead requested a review from a team as a code owner June 13, 2025 21:23
@product-auto-label product-auto-label bot added size: l Pull request size is large. api: storage Issues related to the googleapis/java-storage API. labels Jun 13, 2025
@BenWhitehead BenWhitehead requested a review from BrandonY June 13, 2025 21:27
@BenWhitehead BenWhitehead merged commit b956eb9 into main Jun 24, 2025
24 checks passed
@BenWhitehead BenWhitehead deleted the checksumming/01/fast-crc32c-concat branch June 24, 2025 19:35
@BenWhitehead BenWhitehead added this to the e2e-crc32c-media milestone Jun 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/java-storage API. size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants

TMZ Celebrity News – Breaking Stories, Videos & Gossip

Looking for the latest TMZ celebrity news? You've come to the right place. From shocking Hollywood scandals to exclusive videos, TMZ delivers it all in real time.

Whether it’s a red carpet slip-up, a viral paparazzi moment, or a legal drama involving your favorite stars, TMZ news is always first to break the story. Stay in the loop with daily updates, insider tips, and jaw-dropping photos.

🎥 Watch TMZ Live

TMZ Live brings you daily celebrity news and interviews straight from the TMZ newsroom. Don’t miss a beat—watch now and see what’s trending in Hollywood.