Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog.

Asyncio

• Developing with asyncio https://docs.python.org/3/library/asyncio-dev.html

• =PYTHONASYNCIODEBUG= = 1

• =-X dev=

• =asyncio.run(debug=True)=

• =loop.set_debug()=

• Enable =ResourceWarning=

• logging.getLogger("asyncio").setLevel(...)

• AOSA Book: Web crawler with Asyncio Coroutines [2020-05-09 Sat]

http://www.aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html

• TODO Understand how non-blocking sockets actually work

• TODO Dig into the different event loops
  Understand how the different event loops sockets actually work at the kernel / C level, and visualize it using ebpf.

• TODO Check out the selectors module in python

• TODO Read through the new asyncio modules, read about =Event=

• Notes

• "Stack Ripping": Loss of context because of the broken event loop.

• Coroutine takes ~3k of memory, compared to ~50k per thread

• "Note a nice feature of async programming with callbacks: we need no mutex around changes to shared data"

• =PyEval_EvalFrameEx= takes a frame object and evaluates bytecode in the context of that frame.

• =inspect= module also looks pretty good for looking at where I am

• Python stack frames are allocated in the heap!

• How generators work:

  • gen_fn.code.co_flags & generator_bit = 1
  • when python "compiles" the gen_fn to bytecode
  • creates generators when the flag is set
  • Python generator = stack_frame + code
    • =gen.gi_code.co_name=
    • The stack frame isn't on any actual stack
    • Frame has a last instruction pointer: =f_lasti=

• Coroutines

  • Run with a driver that waits for futures to resolve
  • As part of the callback for the resolution, sends out the next step
  • Using "send" from generators
  • The event loop is a distinct concept

• This would be much harder to read if I hadn't extensively used coroutines in Advent of Code.

• Python's =yield from=

  • Makes a generator act as if it were another generator
  • The return value of the inner generator is the value sent to the outer generator

• Handles exceptions correctly as well!

• To callers, advancing a generator is the same as advancing an iterator

• "The real framework addresses zero-copy I/O, fair scheduling, exception handling, etc"

• "Real" AsyncIO

  • =from asyncio import Queue=
    • =yield from q.get()=
    • =q.task_done()=
    • =q.put_nowait()=
  • =loop = asyncio.get_event_loop()=
  • =loop.run_until_complete(...)=
  • =ClientSession= does connection pooling, keep alives
  • =asyncio.Task= <- actually drives the future forward
  • Explicitly call =worker.cancel()= to clean up pending tasks

• Generator cancel:

  • Throw an exception into the generator from the outside
  • Task explicitly sends a =CancelledError= to the coroutine

• Coroutines are only interruptible at yield points

• The crawler deals with redirects by stepping through them to avoid redownloading the same pages again and again.

• A =Task= is a =Future= that /resolves itself/

• =yield from= --> =await=

• Asyncio & Music

  • TODO Check out UVLoop Understand how nodejs's event loop works.

  • TODO Write about /how/ to digest a technical document/book

  • TODO Write about /why/ to digest a technical document/book

— Kunal