Set Up Local Development Environment

Instill Core is currently under rapid development. The project is built with the microservice architecture. To develop each service independently, we assign profiles to each service in the docker-compose-latest.yml file in Core. This allows us to selectively enable services for various purposes, e.g., debugging, development.

INFO

Services are associated with profiles through the profiles attribute, which takes an array of profile names. A service will be started only if one of its profile names is activated. A service without profiles will always be started.

Core's profiles are defined as exclusion profiles, i.e., instead of launching a service and its dependencies, they'll launch all the services except the specified one. This optimizes the development experience, where a service is built locally in order to apply some changes in the source code while its dependencies tend to keep unchanged.

The following profiles are available on Core:

all - start all services
exclude-api-gateway - start all the services but api-gateway
exclude-artifact - start all the services but artifact-backend
exclude-console - start all the services but console
exclude-mgmt - start all the services but mgmt-backend
exclude-model - start all the services but model-backend
exclude-pipeline - start all the services but pipeline-backend

Use one of the profile names to develop the corresponding service:

# In the `instill-core` repository folder
make latest PROFILE=all # or other profile name

The following guideline shows a specific example of how to develop the pipeline-backend service. For more details, you can read the service's contribution guidelines.

#Start dependent services for `pipeline-backend`

On the local machine, clone the instill-core repository in your workspace, move to the repository folder, and launch all the dependent services:

# Clone the core repository
git clone https://github.com/instill-ai/instill-core.git
cd instill-core
# Use profile `exclude-pipeline` to launch all dependent services
# for `pipeline-backend`
make latest PROFILE=exclude-pipeline

#Run dev `pipeline-backend`

Clone the pipeline-backend repository in your workspace and move to the repository folder:

git clone https://github.com/instill-ai/pipeline-backend.git
cd pipeline-backend

#Build & run the dev image

make build-dev
make dev

Now, you have the Go project set up in the container, in which you can compile and run the binaries together with the integration test in each container shell.

#Start the `pipeline-backend` server

docker exec -it pipeline-backend bash
go run ./cmd/migration
go run ./cmd/init
go run ./cmd/main

#Start the Temporal worker

docker exec -it pipeline-backend bash
go run ./cmd/worker

#Run the integration tests

During local development, you can run the integration test to make sure your latest pipeline-backend works as intended:

docker exec -it pipeline-backend bash
make integration-test API_GATEWAY_URL=api-gateway:8080 DB_HOST=pg-sql

#Remove the dev container

make rm

#Shut down all dependent Core services

# In the `instill-core` repository folder
make down