* docs: add AWS (EC2/SageMaker) deployment + benchmarking guide
- Add an AWS deployment tutorial for EC2 + SageMaker
- Fix SageMaker example indentation and link to the new guide
- Add the new guide to the docs toctree
* docs: address AWS deployment review feedback
* Apply suggestion from code review
---------
Co-authored-by: Fahad Alghanim <[email protected]>
Co-authored-by: Alvaro Bartolome <[email protected]>
* fix(num_devices): fix num_shard/num devices auto compute when NVIDIA_VISIBLE_DEVICES == "all"
the computed num_shards was always 1 in this case, no matter what
* fix(num_devices): make TGI shard auto compute compliant with nvidia-container-toolkit in cdi mode
* feat: support max_image_fetch_size to limit
* fix: update model path for test
* fix: adjust model repo id for test again
* fix: apply clippy lints
* fix: clippy fix
* fix: avoid torch build isolation in docker
* fix: bump repo id in flash llama tests
* fix: temporarily avoid problematic repos in tests
Custom masks are padded to the shape `[batch_size, max_len, max_len]`.
However, flashinfer expects an unpadded mask of the shape
`[sum(q_len[i] * k_len[i] for i in range(batch_size)]`.
This change unpads the custom mask (currently only used by Gemma 3)
to this shape (assuming q_len == k_len, since we only use the custom
mask during prefill).
* chore: prepare version 3.3.5
* black
* neuron: black
* Update hf-xet in uv lockfile
* Attempt to fix API doc check failure
Add `error_type` where missing.
* Pin redocly version
* Sync redocly with Nix for now
---------
Co-authored-by: Daniël de Kok <[email protected]>
* Disable Cachix pushes
This is not safe until we have sandboxed builds. For TGI alone
this might not be a huge issue, but with Cachix caching disabled
in hf-nix, TGI CI would build all the packages and push it to
our cache.
* fix: bump docs
---------
Co-authored-by: drbh <[email protected]>
* chore(neuron): update to optimum-neuron 0.3.0
Dependencies were changed accordingly, because Neuron SDK was updated to
v2.24.
* test: sample is not deterministic
Also modify the temperature in decode test to avoid granite early
stopping.
* test(neuron): adjust expectations after graph changes
* test(neuron): use greedy for stop sequences
---------
Co-authored-by: David Corvoysier <[email protected]>
* chore(neuron): use optimum-neuron 0.2.1
* test(neuron): adjust expectations
Since the latest optimum-neuron uses a new modeling for granite and
qwen, the greedy outputs are slighly different.
* test(neuron): add phi3 and qwen3 tests
* chore(neuron): use optimum-neuron 0.2.2
* chore(neuron): bump version to 0.2.0
* refactor(neuron): use named parameters in inputs helpers
This allows to hide the differences between the two backends in terms of
input parameters.
* refactor(neuron): remove obsolete code paths
* fix(neuron): use neuron_config whenever possible
* fix(neuron): use new cache import path
* fix(neuron): neuron config is not stored in config anymore
* fix(nxd): adapt model retrieval to new APIs
* fix(generator): emulate greedy in sampling parameters
When on-device sampling is enabled, we need to emulate the greedy
behaviour using top-k=1, top-p=1, temperature=1.
* test(neuron): update models and expectations
* feat(neuron): support on-device sampling
* fix(neuron): adapt entrypoint
* tests(neuron): remove obsolete models
* fix(neuron): adjust test expectations for llama on nxd
