37 lines
1.9 KiB
Python
37 lines
1.9 KiB
Python
|
"""The `rpc` package handles most aspects of the actual execution of dbt's RPC
|
|||
|
|
server (except for the server itself and the client tasks, which are defined in
|
|||
|
|
the `task.remote` package).
|
|||
|
|
|
|||
|
|
The general idea from a thread/process management perspective (ignoring the
|
|||
|
|
--single-threaded flag!) is as follows:
|
|||
|
|
|
|||
|
|
- The RPC server runs a web server, in particular `werkzeug`, which manages a
|
|||
|
|
thread pool.
|
|||
|
|
- When a request comes in, werkzeug spins off a thread to manage the
|
|||
|
|
request/response portion. dbt itself has basically no control over this
|
|||
|
|
operation - from our viewpoint request/response cycles are fully
|
|||
|
|
synchronous.
|
|||
|
|
- synchronous requests are defined as methods in the `TaskManager` and handled
|
|||
|
|
in the responding thread directly.
|
|||
|
|
- Asynchronous requests (defined in `tasks.remote`) are kicked off wrapped in
|
|||
|
|
`RequestTaskHandler`s, which manage a new process and a new thread.
|
|||
|
|
- The process runs the actual dbt request, logging via a message queue
|
|||
|
|
- eventually just before process exit, the process places an "error" or
|
|||
|
|
"result" on the queue
|
|||
|
|
- The thread monitors the queue, taking logs off the queue and adding them
|
|||
|
|
to the `RequestTaskHandler`'s `logs` attribute.
|
|||
|
|
- The thread also monitors the `is_alive` state of the process, in case
|
|||
|
|
it is killed "unexpectedly" (including via `kill`)
|
|||
|
|
- When the thread sees an error or result come over the queue, it join()s
|
|||
|
|
the process.
|
|||
|
|
- When the thread sees that the process has disappeared without placing
|
|||
|
|
anything on the queue, it checks the queue one last time, and then acts
|
|||
|
|
as if the queue received an 'Unexpected termination' error
|
|||
|
|
- `kill` commands pointed at an asynchronous task kill the process and allow
|
|||
|
|
the thread to handle cleanup and management
|
|||
|
|
- When the RPC server receives a shutdown instruction, it:
|
|||
|
|
- stops responding to requests
|
|||
|
|
- `kills` all processes (triggering the end of all processes, right!?)
|
|||
|
|
- exits (all remaining threads should die here!)
|
|||
|
|
"""
|