INTRODUCING

Open source library to build your ML pipelines

Keep track of your functions, pipelines and models and deploy everywhere.

How it works

Track your functions

Add the pipeline decorator on top of your functions and define your pipeline.

Run your pipeline

Debug, run and schedule your pipelines.

Deploy your pipelines locally

Automatically get docker files with all dependencies, an endpoint and documentation to run your ML pipeline anywhere.

Join BETA waitlist

Thank you for joining the waitlist! If you are selected, we will be in touch shortly.
Oops! Something went wrong while submitting the form.
Please refresh the page or try again later.

Transform your code into production-ready pipelines

Organise your code in re-usable functions and pipelines.

An environment for each pipeline

Each pipeline is enclosed with all libraries and dependencies required to run your code.

A data snapshot per run

Create a snapshot of all data and code on each run so you can inspect and reproduce your code.

Deploy to your infrastructure

Automatically generate an endpoint and docker files to run your pipeline in your own infrastructure.

Out-of-the-box pipelines

Built-in pipelines and functions to facilitate data processing, model fine-tuning and much more.

.py

Copy

1

2

3

4

5

6

7

8

9

10

11

12

13

14

from pipeline import function

@function
def xgboost_predict(data: dict) -> list:
    """
    Run predictions with my xgboost model
    """

    # Use custom pipeline function to load cached xgboost model    
    xb_model = pipeline.XGBModel.load_remote("mystic://project/xgboost_model")
    y_pred = xb_model.predict(data["x_data"])    
    return y_pred

.py

Copy

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

from pipeline import Pipeline
from pipeline.objects import File
from pipeline.models.hugging_face import TransformersModel

# Finetune GPT-J for your application
with Pipeline(pipeline_name="GPT-J_finetune") as pipeline:    
    my_data = File(type="path", is_input=True)    
    hf_model = TransformersModel("EleutherAI/gpt-j-6B")    
    finetuned_model = hf_model.train(my_data)    
    pipeline.output(finetuned_model)

gptj_pipeline = Pipeline.get_pipeline("GPT-J_finetune")
finetuned_model = gptj_pipeline.run("my_data/data.csv")

.py

Copy

1

2

3

4

5

6

7

8

from pipeline import get_pipeline
from pipeline.docker import create_api

gpt_j_pipeline = get_pipeline("GPT-J_inference_fp16")
docker_path = create_api(pipelines=[gpt_j_pipeline])

CometML

Feast

GCloud

HF

Horovod

JupyterNB

Kafka

Keras

Neptune

Pandas

PL

Pytorch

Ray

ScikitLearn

Spark

S3

TensorBoard

TensorFlow

Weights&Bias

XGBoost

And many more...

Compatible with all your tools

No limits to how you can integrate Pipeline with your data and ML libraries.

Your code, your style

Every developer and organisation has different ways of connecting data with models. Pipeline is an abstraction layer that wraps your code so that it can run anywhere.

Integrates with everything

From your favourite data buckets, batch prediction libraries and streaming services. From feature stores to visualisation libraries, whatever your workflow is, we support it.

Other features

Out-of-the box pipelines

We provide functions and pipelines that will help with your data-science needs, from data-storage access to training ML models.

Schedule your runs

Define when and how to run your pipeline and set custom triggers.

Execution and time analysis

Inspect your pipeline and functions to optimise your code for speed and find-out where your bottleneck is.

Online predictions

Connect to your streaming infrastructure in your functions and execute the pipeline as soon as new data comes in.

Batch processing

Create your data batch in your pipeline and run when ready.

Deploy to your infrastructure

Automatically generate all files required to deploy and run your pipeline within your own infrastructure.

Environment granularity

Each function and pipeline can be wrapped with custom dependencies so you can easily port your code to other machines.

Compatible with all your tools

No limits to how you can integrate Pipeline with your data and ML libraries.

And much more...

Ready to deploy?

Serverless cloud for your Pipelines

Check out our cloud product and get scalable infrastructure and monitoring tools for your pipelines.