Skip to content

Commit 2c7ec20

Browse files
committed
docs: correct dynamic batching visualization and timings
Updated diagram PNGs to accurately reflect dynamic batching behavior. Inference triggers immediately upon reaching 'max_batch_size', rather then waiting for the timeout Ajusted launching time in the README to match the corrected visual flow.
1 parent 4fe2b90 commit 2c7ec20

3 files changed

Lines changed: 1 addition & 1 deletion

File tree

Conceptual_Guide/Part_2-improving_resource_utilization/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ Let's take the previous example and discuss the effect of adding multiple models
7979

8080
For a "no dynamic batching" case, as there are two models to execute, the queries are distributed equally. Users can also add [priorities](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#priority) to prioritize or de-prioritize any specific instance group.
8181

82-
When considering the case of multiple instances with dynamic batches enabled, the following happens. Owing to the availability of another instance, query `B` which arrives with some delay can be executed using the second instance. With some delay allocated, instance 1 gets filled and launched by time `T = X/2` and since queries `D` and `E` stack up to fill up to the maximum batch size, the second model can start inference without any delay.
82+
When considering the case of multiple instances with dynamic batches enabled, the following happens. Owing to the availability of another instance, query `B` which arrives with some delay can be executed using the second instance. With some delay allocated, instance 1 gets filled and launched by time `T = X/3` and since queries `D` and `E` stack up to fill up to the maximum batch size, the second model can start inference without any delay.
8383

8484
The key takeaway from the above examples is that the Triton Inference Server provides flexibility with respect to policies related to creating more efficient batching, thus enabling better resource utilization, resulting in reduced latency and increased throughput.
8585

85.6 KB
Loading
83.2 KB
Loading

0 commit comments

Comments
 (0)