py_shm (erlang_python v5.0.0)

View Source

Shared memory regions between Erlang and Python contexts.

A region is a fixed-size file mapped MAP_SHARED through iommap. Erlang reads it with no copy (binary/3) and writes with one copy (write/3); Python maps the same file and sees it as a buffer (erlang.SharedMemory), in every context mode. The handle is a plain term, {$py_shm', Id, Path, Size}', so it travels inside any call argument or result.

   {ok, Shm} = py_shm:new(64 * 1024 * 1024),
   ok = py_shm:write(Shm, 0, Data),
   {ok, Sum} = py_context:call(Ctx, myapp, sum_floats, [Shm]),
   Out = py_shm:binary(Shm, 0, 1024),
   ok = py_shm:close(Shm).

iommap is an optional dependency: add {iommap, "1.1.3"} to your deps. Without it new/1,2 returns {error, iommap_not_available}.

The module also backs shared py_buffers (py_buffer:new(#{shared => true})): a region used as a ring, with the write position and the closed flag in a header page and flow control through the _py_buffer_wait and _py_buffer_consumed callbacks the Python side calls.

Owns: the region table (ETS), the backing files, and the ring state of shared buffers. Talks to: iommap (through apply/3, optional dependency), py_buffer (shared variant), py_callback (registers _py_buffer_wait, _py_buffer_consumed, _py_buffer_state). Never: maps memory into Python itself; that is _erlang_impl/_shm.py in each interpreter.

Summary

Functions

Whether iommap is available: shared memory needs it.

A binary over the region, no copy. It stays valid after close/1 (the mapping is kept as long as the binary is referenced). Its bytes are the region's: they change when Python writes, so treat it as a snapshot only once the callee is done with the handle, or use read/3 for a copy.

Create a shared streaming buffer (ring of RingSize bytes).

Append Data; blocks up to Timeout ms while the ring is full.

Close the region: the file is removed and the iommap handle closed. Python mappings stay valid until the wrapper is closed or collected.

Create a region of Size bytes owned by the calling process.

Create a region. Options: owner (pid whose exit closes the region, default the caller); writable (default true): with false Python maps the region read-only, so a buggy or hostile callee cannot change it.

Private directory for region files: /dev/shm when it exists (memory backed), else a 0700 directory under TMPDIR.

Copy Len bytes out of the region.

Read-only view of a handle for the Python side; Erlang keeps writing.

Copy Data into the region at Offset.

Types

buffer/0

-type buffer() :: {'$py_buffer', pos_integer(), binary(), pos_integer()}.

shm/0

-type shm() :: {'$py_shm' | '$py_shm_ro', pos_integer(), binary(), non_neg_integer()}.

Functions

available()

-spec available() -> boolean().

Whether iommap is available: shared memory needs it.

binary(_, Offset, Len)

-spec binary(shm(), non_neg_integer(), non_neg_integer()) -> binary().

A binary over the region, no copy. It stays valid after close/1 (the mapping is kept as long as the binary is referenced). Its bytes are the region's: they change when Python writes, so treat it as a snapshot only once the callee is done with the handle, or use read/3 for a copy.

buffer_close(_)

-spec buffer_close(buffer()) -> ok.

buffer_info(_)

-spec buffer_info(buffer()) -> {ok, map()} | {error, term()}.

buffer_new(Opts)

-spec buffer_new(map()) -> {ok, buffer()} | {error, term()}.

Create a shared streaming buffer (ring of RingSize bytes).

buffer_write(Buf, Data)

-spec buffer_write(buffer(), binary()) -> ok | {error, term()}.

buffer_write(_, Data, Timeout)

-spec buffer_write(buffer(), binary(), timeout()) -> ok | {error, term()}.

Append Data; blocks up to Timeout ms while the ring is full.

close(Buf)

-spec close(shm() | buffer()) -> ok.

Close the region: the file is removed and the iommap handle closed. Python mappings stay valid until the wrapper is closed or collected.

handle_call(Req, From, State)

handle_cast(Msg, State)

handle_info(Info, State)

info(_)

-spec info(shm()) -> {ok, map()} | {error, term()}.

init(_)

new(Size)

-spec new(pos_integer()) -> {ok, shm()} | {error, term()}.

Create a region of Size bytes owned by the calling process.

new(Size, Opts)

-spec new(pos_integer(), map()) -> {ok, shm()} | {error, term()}.

Create a region. Options: owner (pid whose exit closes the region, default the caller); writable (default true): with false Python maps the region read-only, so a buggy or hostile callee cannot change it.

private_dir()

-spec private_dir() -> string().

Private directory for region files: /dev/shm when it exists (memory backed), else a 0700 directory under TMPDIR.

read(_, Offset, Len)

-spec read(shm(), non_neg_integer(), non_neg_integer()) -> {ok, binary()} | {error, term()}.

Copy Len bytes out of the region.

read_only(_)

-spec read_only(shm()) -> shm().

Read-only view of a handle for the Python side; Erlang keeps writing.

size(_)

-spec size(shm()) -> non_neg_integer().

start_link()

terminate(Reason, State)

write(_, Offset, Data)

-spec write(shm(), non_neg_integer(), binary()) -> ok | {error, term()}.

Copy Data into the region at Offset.