UML, like all Linux ports, has to provide to the generic kernel all of the facilities that it needs in order to run. A basic platform facility is a distinction between an unprivileged user mode and a privileged kernel mode. Hardware platforms provide a built-in mechanism for switching between these two modes and enforcing the lack of privileges in user mode. However, Linux provides no such mechanism to its processes, so UML constructs it using the ptrace system call tracing mechanism.
UML has a special thread whose main job is to ptrace almost all of the other threads. When a process is in user space, its system calls are being intercepted by the tracing thread. When it's in the kernel, it's not under system call tracing. This is the distinction between user mode and kernel mode in UML.
The transition from user mode to kernel mode is done by the tracing thread. When a process executes a system call or receives a signal, the tracing thread forces the process to run in the kernel if necessary and continues it without system call tracing.
The transition back is also performed by the tracing thread, but it's requested by the process. When it's finished executing in the kernel, because it finished either a system call or trap, it sends a signal (SIGUSR1) to itself. This is intercepted by the tracing thread, which restores the process state if necessary and continues the process with tracing on.