Lambda Layer

2000년 도입 · 논문 2편에서 사용

Lambda layers are a building block for modeling long-range dependencies in data. They consist of long-range interactions between a query and a structured set of context elements at a reduced memory cost. Lambda layers transform each available context into a linear function, termed a lambda, which is then directly applied to the corresponding query. Whereas self-attention defines a similarity kernel between the query and the context elements, a lambda layer instead summarizes contextual information into a fixed-size linear function (i.e. a matrix), thus bypassing the need for memory-intensive attention maps.

출처: LambdaNetworks: Modeling Long-Range Interactions Without Attention

소개 논문: LambdaNetworks: Modeling Long-Range Interactions Without Attention