Pipeline Recipe

A pipeline is defined by a Recipe, which is essentially a YAML composed of sections on, variable, output and multiple components. Here is a sample recipe demonstrating a pipeline that accepts a prompt as input, forwards the prompt to OpenAI, and retrieves the response text.

#Define a Pipeline Recipe

The pipeline recipe comprises four essential fields: on, variable, output, and component.

#On

The on section indicates how the pipeline is executed. Pipeline supports various execution methods.

In the pipeline recipe, you can define the execution method in YAML format.

• Run-on-Trigger: All pipelines support this method by default, requiring no additional setup.
• Run-on-Event: Configure event handling in the on section. For example, to set up the pipeline to receive events from Slack:
  
  

  on:

  slack-0:

  type: slack

  event: EVENT_NEW_MESSAGE

  config:

  channel-names:

  - channel-0

  
  

For more information, see the Run-on-Trigger, Run-on-Event, and

#Variable

The variable section defines variables for the pipeline, allowing components to reference this data.

• title: The variable title displayed on the Console.
• description: A description of the variable, also displayed on the Console.
• format: Specifies the format of the variable. Refer to Instill Type for more details.
• required: Indicates whether the variable is required. Default is false.
• default: Specifies the default value of the variable.

Users can input data into variables when running the pipeline. For further information, refer to Run-on-Trigger.

#Output

The output section specifies the output of the pipeline, serving as the response data for the Run-on-Trigger method:

• title: The title of the output displayed on the Console.
• description: A description of the output displayed on the Console.
• value: The value of the output, which can be referenced from any component data.

#Component

The component section consists of multiple components, which can be generic, AI, data, application, or operator components.

• Each component must have a unique ID (e.g., openai-0), adhering to the RFC1034 rule, which permits alphabets, numbers, and hyphens.
• For AI, data, application, or operator components, set the following fields:
  • type: Indicates the component type.
  • task: Specifies the component's task, listed in the component definition's tasks field.
  • input: Defines the input data schema, described in the spec.componentSpecification field.
  • setup: Configures the component, e.g., specifying an api-key for connection. This is also detailed in the spec.componentSpecification field.
  • condition: (optional) Sets a condition to determine whether the component executes.
• For an iterator, set the following fields:
  • input: The array input for iteration.
  • output-elements: Specifies the output of the iterator.
  • component: An array of components executed within the iterator.

#Data Flow

#Reference Data

In pipeline components, use a reference syntax to establish data flow in the input field of each component:

A reference utilizes special syntax enclosed in single curly brackets. For example:

1. Reference data from the pipeline variable: ${variable.KEY} (you can include any fields defined in recipe.variable).
2. Reference data from a component: ${comp-id.input.KEY} or ${comp-id.output.KEY} (available fields are described in the componentSpecification of each component).

This method functions as a variable reference, transferring values from an upstream component to the input while preserving the original data type.

If you prefer not to use data references, you can also establish constant values in the recipe.

#Reference Secrets

In addition to referencing pipeline input or component settings, you can reference secrets from the secret management system. These secret field values remain hidden in the recipe, ensuring that keys are protected when the pipeline is shared or published. For example: "api-key": "${secrets.my-openai-key}". For further information, see Secret Management.

In the above example:

• prompt: ${variable.prompt} indicates that the prompt in the OpenAI component is referenced from the prompt variable.
• api-key: "${secret.my-openai-key}" signifies that the API key in the OpenAI component is sourced from the secrets management system.

#Control Flow

Control flow in pipeline components is facilitated through the condition field in each component, which determines whether the component will be executed.

#Example Configuration

This configuration defines the execution condition. Pipeline interprets this to decide component execution. We support two condition types: "condition on value" and "condition on status".

#Condition on Value

The following syntax is supported for conditions:

• Logic Operators: &&, ||
• Comparison Operators: <, >, <=, >=, ==, !=
• Not Operator: !
• Parentheses: ()

#Examples

• Condition on string value:

  
  

  condition: ${variable.a-condition-str} == TARGET_CONDITION_STR

  
  

• Condition on number value:

  
  

  condition: ${variable.a-condition-num} > 1

  
  

• Condition on boolean value:

  
  

  condition: ${variable.a-condition-bool}

  
  

• Complex condition:

  
  

  condition: (${variable.a-condition-bool} && ${comp-a.output.x} == 1) || ${comp-b.output.y} < 1

  
  

• Always false condition:

  
  

  condition: false

  
  

#Condition on Status

In addition to conditions based on values, we support conditions based on status. We provide a boolean value called status.completed, allowing you to specify whether a component's execution is complete.

#Examples

• Execute the component after comp-a is completed:

  
  

  condition: ${comp-a.status.completed}

  
  

• Combine status with value condition:

  
  

  condition: ${variable.a-condition-bool} || ${comp-a.status.completed}