RHEL AI: ilab model serve command failing with error "AssertionError: please set tensor_parallel_size to less than max local gpu count"

Issue

• ilab model serve command failing with error.

$ ilab model serve
INFO 2024-10-04 08:00:38,834 instructlab.model.serve:145: Using model '/var/home/cloud-user/.cache/instructlab/models/granite-7b-redhat-lab' with -1 gpu-layers and 4096 max context size.
INFO 2024-10-04 08:00:38,834 instructlab.model.serve:149: Serving model '/var/home/cloud-user/.cache/instructlab/models/granite-7b-redhat-lab' with vllm
INFO 2024-10-04 08:00:38,835 instructlab.model.backends.vllm:230: vLLM starting up on pid 34 at http://127.0.0.1:8000/v1
INFO 10-04 08:00:50 api_server.py:212] vLLM API server version 0.5.2.2
.
.
INFO 10-04 08:00:51 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/app-root/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 282, in <module>
    run_server(args)
.
.
 File "/opt/app-root/lib64/python3.11/site-packages/vllm/executor/multiproc_gpu_executor.py", line 50, in _init_executor
    assert world_size <= cuda_device_count_stateless(), (
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: please set tensor_parallel_size to less than max local gpu count