Repository navigation
Ability to send a message from Python back to Elixir #20
Description
Activity
Python can easily call C and this library ships with C code, so one option would be to define a C function alongside our C code and then call it from Python. However, in order to do that, you would probably need to store the environment somewhere (if we don't already do that).
And yes, we would need to expose this function somewhere. Perhaps this lib could ship with a .py file in priv, I am not sure of the best approach.
Right, after looking at the implementation for a moment, there is a python prelude already in the NIF where the function could be exposed
https://github.com/livebook-dev/pythonx/blob/main/c_src/pythonx.cpp#L388
Following the example of
pythonx_handle_io_write, we could expose a function that is quite similar that does theenif_sendbut off of user input https://github.com/livebook-dev/pythonx/blob/main/c_src/pythonx.cpp#L1399I think I could throw a prototype together here
Yeah, I think this could be done very similar to IO write.
Perhaps this lib could ship with a .py file in priv, I am not sure of the best approach.
We could patch the default globals to include an extra function that we would define, or rather a
pythonxpython module with such function, imported by default. We would do this as part of initialization, similarly to how we define custom stdout/stdin objects. The only issue is that this makes the Python code specific to Pythonx, but maybe it's fine for specific use cases.Note that with IO you can already send messages wherever you want by specifying custom
:stdout_device, although it only works with text. I assume you want to send a Python object as the message. One thing to note though is that once you get that Python object, any operations on that object will requires the GIL. If the long-running Python code doesn't hold GIL for much, then it should be fine, but if it does, it can result in bottlenecks.Note that with IO you can already send messages wherever you want by specifying custom :stdout_device, although it only works with text.
That wasn't clear to me but is obvious in retrospect
I assume you want to send a Python object as the message
Correct, although I think there are two modes that I might care about
- References to the Python objects that I can manipulate directly
- Copies of the Python objects encoded to Elixir terms that I never will manipulate
My motivations here are more for eventing that I think is better suited to 2
The issue with 1 is precisely that you would trigger the
gilso you would have to be very careful about it. Perhaps one option is to allow only binaries to be sent...For 2 we also need GIL, we don't really want to decode within the NIF itself, because that has limitations. In other words instead of 2. we would actually do 1. and you call
Pythonx.decode/1, so it's the same.What about 3, only send binaries which we already encode as binaries before sending them?
@josevalim isn't that equivalent to setting the
:stdout_device, or do you mean something else?Yes, it would be equivalent.
Well, one slight difference, you would be to send it to different PIDs if we allowed PIDs to be encoded to Python.
Right. I may try to find some time to prototype something here, will report back
But to be clear, if IO works for you, then go for it. That’s how you would communicate through a port anyway.
Hi everyone, we potentially have a nice usecase for this library at CyanView, I'm working on PoC at the moment.
@davydog187 did you have a chance to make prototype for exchanging events between python and elixir?Do you have an example of what you are trying to do? It is worth saying that you can emulate this by calling the Python code, having it stop once you have the intermedia result, yielding back to Elixir, and then you call it back passing the previous state/objects. This is probably even preferable to letting Python run in a dirty scheduler for really long periods of time.
At the moment I work on integration with streamdeck and since we are running purely Elixir on our devices we would love to continue doing that, Python is being used to listen for "button pressed" events from the streamdeck and I would love to pass those events back to Elixir code, so it's a long running process. Worse case scenario I will pass those events as api calls back to my Elixir app.
- I have not had a chance to implement anything here yet, although our use case sounds similar to yours. (Btw we also work with live streaming video at TV Labs, we should talk!) Our use case is similar, we have SDKs that make a connection to a device, and we want to receive events and pass them back to Elixir. We currently do this over a service interface, but we've found that in failure modes, its very difficult to reconcile the state between two stateful systems. We'd prefer to have Elixir monitor the connection. We already take this to the extreme elsewhere, embedding the GStreamer framework as a NIF in our application, so running Python on a dirty scheduler doesn't scare me
I see! I'm honestly not sure how I feel about having a whole Python application running forever within a dirty scheduler. Unless you are really doing something high performance that requires memory sharing, I think using something like
erlport(or similar) for I/O communication or an API would be a better fit. But maybe it would all be fine too, I just never tried it!It's possible that erlport would be a better solution, but experimenting with Pythonx is simply too tempting
I have a working POC for this, and we can figure out the details later -- like if the Python objects should be encoded as Erlang terms or passed as opaque handles, or just make it an option for the user to choose.
Reacted by Dave Lucia, Daniil Popov and Alexander GubarevIs there any progress here? I would love to be able to use this feature.
See #38 :)
We merged #38. You can try it by installing Pythonx from GitHub
{:pythonx, github: "livebook-dev/pythonx"}. Feedback is welcome :)Reacted by GenericJam
We have a few interesting use cases for Pythonx at TV Labs, and one of the features that would make it particularly useful, is if long-running Python programs could be executed, with the ability to send messages back to the owning process (or any process).
There is precedent for a feature like this, since NIFs themselves can send message through the enif_send interface.
@cocoa-xu is on vacation this week or I would have asked her, but I imagine this is really only possible if we are able to pull the BEAM header files into Python, or find a way to expose a function in Python that can lift the message up to the BEAM layer.
Thoughts?