💧 Pipeline

#About

Pipeline is a powerful tool designed to build end-to-end unstructured data pipelines. It leverages the capabilities of 3rd-party data, AI, and applications functionalities and seamlessly connect with Model and Instill Artifact via Instill Component components.

#Create Pipeline

#Pipeline Recipe

A pipeline is defined by a Recipe, which is essentially a YAML composed of sections on, variable, output and multiple components. Here is a sample recipe demonstrating a pipeline that accepts a prompt as input, forwards the prompt to OpenAI, and retrieves the response text.

#Instill Type

Pipeline defines several Instill Types as data type identifiers. These types simplify the creation of pipelines by eliminating the complexity of converting unstructured data types. The supported Instill Types include primitive data types such as boolean, string, integer, number, and json, as well as unstructured data types like document, image, audio, and video, along with array data types.

These Instill Types enable users to efficiently build pipelines that manage unstructured data in ETL workflows.

#Create Pipeline

Pipeline offers multiple methods for creating a pipeline, including HTTP and gRPC APIs, the Python/TypeScript SDK, and the web Console. For more details, please refer to the Create Pipeline page.

#Pipeline Management

Pipeline also provides a set of methods for pipeline management, including listing, updating, and deleting pipelines. For more information, please refer to the Pipeline Management page.

#Secret Management

In a pipeline, it may be necessary to hide certain credential data from the recipe. Pipeline provides a method to set up secret values and reference them in a recipe, rather than directly exposing sensitive information. For more information, please refer to the Secret Management page.

#Integrations

Integrations allow users to set up components that connect with external services. For more information, please refer to the Integrations page.

#Release Pipeline

Version Control provides enhanced flexibility and management by enabling tracking of modifications in your pipelines and applications over time. This functionality empowers users to leverage various versions of the same pipeline for purposes such as backup, recovery, A/B testing, among others. Instill Pipeline offers a pipeline release feature that allows users to manage versions and control deployments.

#Run Pipeline

Pipeline provides multiple methods to run a pipeline, including:

• Run-on-Trigger: Offers HTTP, gRPC, and HTTP with multipart/form-data APIs for executing a pipeline.
• Run-on-Event: Provides webhook endpoints to receive events from vendors.

#Share Pipeline

Pipeline offers the capability to disseminate your creations either via a share link or by setting them to public visibility. This allows other users to view your pipeline. For more information, please refer to the Share Pipeline page.

#View Runs

Pipeline enables users to list and search historical pipeline runs, facilitating monitoring and analysis of pipeline execution. For more information, please refer to the View Runs page.