Notes:
One possibility is to convert system calls into signals. The signal handler would run on the UML kernel stack, and would execute the system call. When entering the kernel, a mode switch turning off system call tracing is required. This could be done fairly naturally by the default behavior of a signal being blocked while its handler is running.
Another possibility is to add the notion of an unprivileged context. A process would have a privileged context, and possibly an unprivileged context. A process running in its unprivileged context would not be able to enter the kernel. Instead, any attempt to do so would cause it to switch to its privileged context, which would examine the unprivileged state and decide whether to allow it to proceed. In this scheme, the UML kernel would be the privileged context, and its userspace would run in unprivileged context.