MediaPipe框架- 框架概念

Framework concepts

框架概念

Before using MediaPipe Framework, familiarize yourself with the following concepts.

在使用MediaPipe Framework之前，请熟悉以下概念。

The basics

基础

Packet

数据包

The basic data flow unit. A packet consists of a numeric timestamp and a shared pointer to an immutable payload. The payload can be of any C++ type, and the payload's type is also referred to as the type of the packet. Packets are value classes and can be copied cheaply. Each copy shares ownership of the payload, with reference-counting semantics. Each copy has its own timestamp. See also Packet.

​基本数据流单元。数据包由一个数字时间戳和一个指向不可变有效负载的共享指针组成。有效载荷可以是任何C++类型，有效载荷的类型也被称为数据包的类型。数据包是价值类，可以廉价复制。每个副本共享有效负载的所有权，具有引用计数语义。每个副本都有自己的时间戳。另请参阅数据包。

Graph

图

MediaPipe processing takes place inside a graph, which defines packet flow paths between nodes. A graph can have any number of inputs and outputs, and data flow can branch and merge. Generally data flows forward, but backward loops are possible. See Graphs for details.

​MediaPipe处理发生在一个图中，该图定义了节点之间的数据包流路径。一个图可以有任意数量的输入和输出，数据流可以分支和合并。通常数据向前流动，但也可以向后循环。有关详细信息，请参见图。

Nodes

节点

Nodes produce and/or consume packets, and they are where the bulk of the graph’s work takes place. They are also known as “calculators”, for historical reasons. Each node’s interface defines a number of input and output ports, identified by a tag and/or an index. See Calculators for details.

​节点产生或消耗数据包，它们是图的大部分工作发生的地方。由于历史原因，它们也被称为“计算器”。每个节点的接口定义了许多输入和输出端口，这些端口由标记或索引标识。有关详细信息，请参阅计算器。

Streams

流

A stream is a connection between two nodes that carries a sequence of packets, whose timestamps must be monotonically increasing.

流是两个节点之间的连接，承载一系列数据包，这些数据包的时间戳必须单调增加。

Side packets

侧面数据包

A side packet connection between nodes carries a single packet (with unspecified timestamp). It can be used to provide some data that will remain constant, whereas a stream represents a flow of data that changes over time.

节点之间的侧分组连接承载单个分组（具有未指定的时间戳）。它可以用于提供一些将保持不变的数据，而流表示随时间变化的数据流。

Packet Ports

数据包端口

A port has an associated type; packets transiting through the port must be of that type. An output stream port can be connected to any number of input stream ports of the same type; each consumer receives a separate copy of the output packets, and has its own queue, so it can consume them at its own pace. Similarly, a side packet output port can be connected to as many side packet input ports as desired.

端口具有关联的类型；通过端口传输的数据包必须是那种类型。输出流端口可以连接到任意数量的相同类型的输入流端口；每个使用者都接收一个单独的输出数据包副本，并有自己的队列，因此可以按照自己的速度使用它们。类似地，侧分组输出端口可以根据需要连接到任意多的侧分组输入端口。

A port can be required, meaning that a connection must be made for the graph to be valid, or optional, meaning it may remain unconnected.

端口可能是必需的，这意味着必须建立连接才能使图有效，或者是可选的，这也意味着它可能保持未连接。

Note: even if a stream connection is required, the stream may not carry a packet for all timestamps.

注意：即使需要流连接，流也可能不携带所有时间戳的数据包。

Input and output

输入和输出

Data flow can originate from source nodes, which have no input streams and produce packets spontaneously (e.g. by reading from a file); or from graph input streams, which let an application feed packets into a graph.

数据流可以起源于没有输入流的源节点，并自发地产生数据包（例如，通过从文件中读取）；或者从允许应用程序将数据包喂到到图中的图输入流。

Similarly, there are sink nodes that receive data and write it to various destinations (e.g. a file, a memory buffer, etc.), and an application can also receive output from the graph using callbacks.

类似地，也有接收数据并将其写入各种目的地（例如，文件、内存缓冲区等）的汇聚节点，应用程序也可以使用回调从图中接收输出。

Runtime behavior

运行时行为

Graph lifetime

图生存期

Once a graph has been initialized, it can be started to begin processing data, and can process a stream of packets until each stream is closed or the graph is canceled. Then the graph can be destroyed or started again.

一旦一个图被初始化，它就可以开始处理数据，并且可以处理一个数据包流，直到每个数据包流被关闭或该图被取消。然后可以销毁或重新启动图形。

Node lifetime

节点生存期

There are three main lifetime methods the framework will call on a node:

框架将在节点上调用三种主要的生存期方法：

• Open: called once, before the other methods. When it is called, all input side packets required by the node will be available.
• Open：在其他方法之前调用一次。当它被调用时，节点所需的所有输入端数据包都将可用。
• Process: called multiple times, when a new set of inputs is available, according to the node’s input policy.
• 处理：根据节点的输入策略，当一组新的输入可用时，调用多次。
• Close: called once, at the end.
• 关闭：在最后调用一次。

In addition, each calculator can define constructor and destructor, which are useful for creating and deallocating resources that are independent of the processed data.

此外，每个计算器都可以定义构造函数和析构函数，这对于创建和释放独立于已处理数据的资源非常有用。

Input policies

输入策略

The default input policy is deterministic collation of packets by timestamp. A node receives all inputs for the same timestamp at the same time, in an invocation of its Process method; and successive input sets are received in their timestamp order. This can require delaying the processing of some packets until a packet with the same timestamp is received on all input streams, or until it can be guaranteed that a packet with that timestamp will not be arriving on the streams that have not received it.

默认的输入策略是按时间戳对数据包进行确定性排序。节点在调用其Process方法时同时接收相同时间戳的所有输入；并且以它们的时间戳顺序接收连续的输入集合。这可能需要延迟对一些分组的处理，直到在所有输入流上接收到具有相同时间戳的分组，或者直到可以保证具有该时间戳的数据包不会到达尚未接收到它的流。

Other policies are also available, implemented using a separate kind of component known as an InputStreamHandler.

其他策略也可用，使用一种称为InputStreamHandler的独立组件来实现。

See Synchronization for more details.

​有关更多详细信息，请参阅同步。

Real-time streams

实时流

MediaPipe calculator graphs are often used to process streams of video or audio frames for interactive applications. Normally, each Calculator runs as soon as all of its input packets for a given timestamp become available. Calculators used in real-time graphs need to define output timestamp bounds based on input timestamp bounds in order to allow downstream calculators to be scheduled promptly. See Real-time Streams for details.

​MediaPipe计算器图通常用于处理交互式应用程序的视频或音频帧流。通常，每个计算器在给定时间戳的所有输入数据包可用后立即运行。实时图中使用的计算器需要基于输入时间戳边界来定义输出时间戳边界，以便能够及时调度下游计算器。有关详细信息，请参阅实时流。