SYCLDeviceManager Class Reference

SYCL device manager. More...

Collaboration diagram for SYCLDeviceManager:

Public Member Functions
	SYCLDeviceManager (int &default_device_num, int &num_devices, int local_rank, int local_size)

Static Public Member Functions
static sycl::queue &	getDefaultDeviceDefaultQueue ()
	access the the DeviceManager owned default queue. More...

Private Attributes
int	sycl_default_device_num
std::vector< syclDeviceInfo >	visible_devices

Static Private Attributes
static std::unique_ptr< sycl::queue >	default_device_queue
	the global singleton which can be used to access the default queue of the default device. More...

Detailed Description

SYCL device manager.

Definition at line 50 of file SYCLDeviceManager.h.

Constructor & Destructor Documentation

SYCLDeviceManager()

SYCLDeviceManager	(	int &	default_device_num,
		int &	num_devices,
		int	local_rank,
		int	local_size
	)

Definition at line 42 of file SYCLDeviceManager.cpp.

References qmcplusplus::app_log(), qmcplusplus::app_warning(), SYCLDeviceManager::default_device_queue, qmcplusplus::determineDefaultDeviceNum(), SYCLDeviceManager::sycl_default_device_num, and SYCLDeviceManager::visible_devices.

    : sycl_default_device_num(-1)
{
#if defined(ENABLE_OFFLOAD)
  const size_t omp_num_devices = omp_get_num_devices();
  visible_devices.reserve(omp_num_devices);
  for (int id = 0; id < omp_num_devices; id++)
  {
    omp_interop_t interop;
#pragma omp interop device(id) init(prefer_type("level_zero"), targetsync : interop)

    int err = -1;
    const std::string omp_backend_name(omp_get_interop_str(interop, omp_ipr_fr_name, &err));
    if (err != omp_irc_success)
      throw std::runtime_error("omp_get_interop_str(omp_ipr_fr_name) failed!");

    if (omp_backend_name.find("level_zero") != 0)
      throw std::runtime_error("Interop between OpenMP and SYCL is only supported when both implementations are built "
                               "on top of Level Zero API.");

    auto hPlatform = omp_get_interop_ptr(interop, omp_ipr_platform, &err);
    if (err != omp_irc_success)
      throw std::runtime_error("omp_get_interop_ptr(omp_ipr_platform) failed!");
    auto hContext = omp_get_interop_ptr(interop, omp_ipr_device_context, &err);
    if (err != omp_irc_success)
      throw std::runtime_error("omp_get_interop_ptr(omp_ipr_device_context) failed!");
    auto hDevice = omp_get_interop_ptr(interop, omp_ipr_device, &err);
    if (err != omp_irc_success)
      throw std::runtime_error("omp_get_interop_ptr(omp_ipr_device) failed!");

    const sycl::platform sycl_platform =
        sycl::ext::oneapi::level_zero::make_platform(reinterpret_cast<pi_native_handle>(hPlatform));

    const sycl::device sycl_device =
        sycl::ext::oneapi::level_zero::make_device(sycl_platform, reinterpret_cast<pi_native_handle>(hDevice));

    visible_devices
        .emplace_back(sycl::ext::oneapi::level_zero::make_context({sycl_device},
                                                                  reinterpret_cast<pi_native_handle>(hContext),
                                                                  true /* keep the ownership, no transfer */),
                      sycl_device, interop);
  }

#else
  // Potentially multiple GPU platform.
  std::vector<sycl::platform> platforms = sycl::platform::get_platforms();
  if (platforms.empty())
    throw std::runtime_error("Cannot find SYCL platforms!");

  // find out devices from the first platform with GPUs.
  std::vector<sycl::device> devices;
  app_log() << "Visible SYCL platforms are :" << std::endl;
  for (auto& platform : platforms)
  {
    std::vector<sycl::device> gpu_devices = platform.get_devices(sycl::info::device_type::gpu);
    const auto gpu_count                  = gpu_devices.size();
    bool selected                         = false;
    if (devices.empty() && gpu_count > 0)
    {
      selected = true;
      devices  = std::move(gpu_devices);
    }
    app_log() << (selected ? " ** " : "    ") << platform.get_info<sycl::info::platform::name>() << " with "
              << gpu_count << " GPUs." << std::endl;
  }
  app_log() << std::endl;

  visible_devices.reserve(devices.size());
  for (int id = 0; id < devices.size(); id++)
    visible_devices.emplace_back(sycl::context(devices[id]), devices[id]);
#endif

  const size_t sycl_device_count = visible_devices.size();
  if (num_devices == 0)
    num_devices = sycl_device_count;
  else if (num_devices != sycl_device_count)
    throw std::runtime_error("Inconsistent number of SYCL devices with the previous record!");
  if (sycl_device_count > local_size)
    app_warning() << "More SYCL devices than the number of MPI ranks. "
                  << "Some devices will be left idle.\n"
                  << "There is potential performance issue with the GPU affinity.\n";
  if (num_devices > 0)
  {
    sycl_default_device_num = determineDefaultDeviceNum(sycl_device_count, local_rank, local_size);
    if (default_device_num < 0)
      default_device_num = sycl_default_device_num;
    else if (default_device_num != sycl_default_device_num)
      throw std::runtime_error("Inconsistent assigned SYCL devices with the previous record!");
    default_device_queue = std::make_unique<sycl::queue>(visible_devices[sycl_default_device_num].get_context(),
                                                         visible_devices[sycl_default_device_num].get_device(),
                                                         sycl::property::queue::in_order());
    if (!visible_devices[sycl_default_device_num].get_device().has(sycl::aspect::ext_intel_free_memory))
      app_warning()
          << "Free memory queries always return 0 due to inactive 'oneAPI' System Resource Management (sysman). "
          << "Set environment variable ZES_ENABLE_SYSMAN to 1 to activate the query feature." << std::endl;
  }
}

Member Function Documentation

getDefaultDeviceDefaultQueue()

sycl::queue & getDefaultDeviceDefaultQueue ( )

static

access the the DeviceManager owned default queue.

Restrict the use of it to performance non-critical operations. Note: CUDA has a default queue but all the SYCL queues are explicit.

Definition at line 140 of file SYCLDeviceManager.cpp.

References SYCLDeviceManager::default_device_queue.

Referenced by qmcplusplus::getSYCLDefaultDeviceDefaultQueue().

{
  if (!default_device_queue)
    throw std::runtime_error("SYCLDeviceManager::getDefaultDeviceQueue() the global instance not initialized.");
  return *default_device_queue;
}

Member Data Documentation

default_device_queue

std::unique_ptr< sycl::queue > default_device_queue

staticprivate

the global singleton which can be used to access the default queue of the default device.

Definition at line 56 of file SYCLDeviceManager.h.

Referenced by SYCLDeviceManager::getDefaultDeviceDefaultQueue(), and SYCLDeviceManager::SYCLDeviceManager().

sycl_default_device_num

int sycl_default_device_num

private

Definition at line 52 of file SYCLDeviceManager.h.

Referenced by SYCLDeviceManager::SYCLDeviceManager().

visible_devices

std::vector<syclDeviceInfo> visible_devices

private

Definition at line 53 of file SYCLDeviceManager.h.

Referenced by SYCLDeviceManager::SYCLDeviceManager().

---

The documentation for this class was generated from the following files:

• /home/pk7/projects/qmc/for_cron_doxygen/qmcpack/src/Platforms/SYCL/SYCLDeviceManager.h
• /home/pk7/projects/qmc/for_cron_doxygen/qmcpack/src/Platforms/SYCL/SYCLDeviceManager.cpp