offload/docs/declare_target_indirect.md

# Overview

The indirect clause enables **indirect device invocation** for a procedure:
> 19 An indirect call to the device version of a procedure on a device other than the host<br>
> 20 device, through a function pointer (C/C++), a pointer to a member function (C++) or<br>
> 21 a procedure pointer (Fortran) that refers to the host version of the procedure.

# Compiler support
### Offload entry metadata (C++ FE)

For each function declared as **declare target indirect** C++ FE generates the following offload metadata:

```c++
// Entry 0 -> Kind of this type of metadata (2)
// Entry 1 -> Mangled name of the function.
// Entry 2 -> Order the entry was created.
```

The offloading metadata uses new `OffloadEntriesInfoManagerTy::OffloadingEntryInfoKinds::OffloadingEntryInfoDeviceIndirectFunc`  metadata kind.

### Offload entries table

The offload entries table that is created for the host and for each of the device images currently have entries for **declare target** global variables, **omp target** outlined functions and constructor/destructor thunks for **declare target** global variables.


Compiler will also produce an entry for each procedure listed in **indirect** clause of **declare target** construct:
```C++
struct llvm::offloading::EntryTy {
  void *addr;       // Pointer to the function
  char *name;       // Name of the function
  size_t size;      // 0 for function
  int32_t flags;    // OpenMPOffloadingDeclareTargetFlags::OMP_DECLARE_TARGET_FPTR
  int32_t reserved; // Reserved
};
```

### Run-time dispatch in device code

When an indirect function call is generated by a FE in **device code** it translates the original function pointer (which may be an address of a host function) into the device function pointer using a translation API, and uses the resulting function pointer for the call.

Original call code:

```
  %0 = load void ()*, void ()** %fptr.addr
  call void %0()
```

Becomes this:

```
  %0 = load void ()*, void ()** %fptr.addr
  %1 = bitcast void ()* %0 to i8*
  %call = call i8* @__kmpc_target_translate_fptr(i8* %1)
  %fptr_device = bitcast i8* %call to void ()*
  call void %fptr_device()
```

Device RTLs must provide the translation API:

```c++
// Translate \p FnPtr identifying a host function into a function pointer
// identifying its device counterpart.
// If \p FnPtr matches an address of any host function
// declared as 'declare target indirect', then the API
// must return an address of the same function compiled
// for the device. If \p FnPtr does not match an address
// of any host function, then the API returns \p FnPtr
// unchanged.
EXTERN void *__kmpc_target_translate_fptr(void *FnPtr);
```

# Runtime handling of function pointers

`OpenMPOffloadingDeclareTargetFlags::OMP_DECLARE_TARGET_FPTR` is a new flag to distinguish offload entries for function pointers from other function entries.  Unlike other function entries (with `size` equal to 0) `omptarget::InitLibrary()` will establish mapping for function pointer entries in `Device.HostDataToTargetMap`.

For each `OMP_DECLARE_TARGET_FPTR` entry in the offload entries table `libomptarget` creates an entry of the following type:

```c++
struct __omp_offloading_fptr_map_ty {
  int64_t host_ptr; // key
  int64_t tgt_ptr;  // value
};
```

Where `host_ptr` is `llvm::offloading::EntryTy::addr` in a **host** offload entry, and `tgt_ptr` is `llvm::offloading::EntryTy::addr` in the corresponding **device** offload entry (which may be found using the populated `Device.HostDataToTargetMap`).

When all `__omp_offloading_function_ptr_map_ty` entries are collected in a single host array, `libomptarget` sorts the table by `host_ptr` values and passes it to the device plugin for registration, if plugin supports optional `__tgt_rtl_set_function_ptr_map` API.

Plugins may provide the following API, if they want to support **declare target indirect** functionality:

```c++
// Register in a target implementation defined way a table
// of __omp_offloading_function_ptr_map_ty entries providing
// mapping between host and device addresses of 'declare target indirect'
// functions. \p table_size is the number of elements in \p table_host_ptr
// array.
EXTERN void __tgt_rtl_set_function_ptr_map(
    int32_t device_id, uint64_t table_size, __omp_offloading_fptr_map_ty *table_host_ptr);
```

# Sample implementation

This section describes one of potential implementations.

A FE may define the following global symbols for each translation module containing **declare target indirect**, when compiling this module for a device:

```c++
// Mapping between host and device functions declared as
// 'declare target indirect'.
__attribute__((weak)) struct __omp_offloading_fptr_map_ty {
  int64_t host_ptr; // key
  int64_t tgt_ptr;  // value
} *__omp_offloading_fptr_map_p = 0;

// Number of elements in __omp_offloading_fptr_map_p table.
__attribute__((weak)) uint64_t __omp_offloading_fptr_map_size = 0;
```

`__tgt_rtl_set_function_ptr_map(int32_t device_id, uint64_t table_size, __omp_offloading_fptr_map_ty *table_host_ptr)` allocates device memory of size `sizeof(__omp_offloading_fptr_map_ty) * table_size`, and transfers the contents of `table_host_ptr` array into this device memory.  An address of the allocated device memory area is then assigned to `__omp_offloading_fptr_map_p` global variables on the device.  For example, in **CUDA**, a device address of `__omp_offloading_fptr_map_p` may be taken by calling `cuModuleGetGlobal`, and then a pointer-sized data transfer will initialize `__omp_offloading_fptr_map_p` to point to the device copy of `table_host_ptr` array.  `__omp_offloading_fptr_map_size` is assigned to `table_size` the same way.

An alternative implementation of `__tgt_rtl_set_function_ptr_map` may invoke a device kernel that will do the assignments.

`__kmpc_target_translate_fptr(void *FnPtr)` API uses binary search to match `FnPtr` against `host_ptr` inside the device table pointed to by `__omp_offloading_fptr_map_p`.  If the matching key is found, it returns the corresponding `tgt_ptr`, otherwise, it returns `FnPtr`.

# TODO: Optimization for non-unified_shared_memory

If a program does not use **required unified_shared_memory**, and all function pointers are mapped (not a requirement by OpenMP spec), then an implementation may avoid the runtime dispatch code for indirect function calls (i.e. `__kmpc_target_translate_fptr` is not needed) and also `__tgt_rtl_set_function_ptr_map` is not needed.  `libomptarget` will just map the function pointers as regular data pointers via `Device.HostDataToTargetMap`.