💻Local Datasets and Runs

This walkthrough will teach you how to start Runs locally on you machine and use local datasets on the Coretex platform.

Local Dataset

Using your Local Datasets on the Coretex platform is very simple. First, install the Coretex library using pip:

pip install coretex

In your python code you will have to import the Dataset class of type that you want to use (i.e. ObjectDetectionDataset) and its local class which usually has the same name with the prefix "Local" (i.e. LocalObjectDetectionDataset).

from coretex import ObjectDetectionDataset, LocalObjectDetectionDataset

In this example, we use CustomDataset type which is used for undefined Project Types (Other). To learn more about Project Types check out this link: Types.

Only one modification in your task code is necessary in order to use the local Dataset.

In main.py file in main() function at its very beginning you will need to specify your local Dataset path. Like this:

from pathlib import Path

from coretex import CustomDataset, LocalCustomDataset, ExecutingExperiment
from coretex.project import initializeProject


def main(experiment: ExecutingExperiment[CustomDataset]) -> None:
    experiment.parameters["dataset"] = LocalCustomDataset(Path("path/to/dataset"))
    
    # pass your dataset to train() function    
    model = train(dataset = experiment.dataset)
    
    # validate the trained model
    validate(model = model)


if __name__ == "__main__":
    initializeProject(main)

This line has to be at the very beginning of your main()function in your task.

You can access your dataset samples using:

experiment.dataset.samples  # list of all samples in the dataset

# use simple list iteration to access each sample in the dataset

for sample in dataset.samples:
    sample.unzip()  # unzips sample
    sampleData = sample.load()  # access sample's data like name, path, etc.

Local Experiment

Starting local Runs on your machine using Coretex is straightforward.

The first step is downloading the Coretex library through PyPi:

pip install coretex

The next step is making the necessary changes, required for running the Task on the Coretex platform.

Refer to this tutorial to find out more about those changes: Migrate your tasks to Coretex.

After you've done all necessary modifications, open your terminal and write the following:

python main.py --username [email protected] --password ******* --projectId 1023

This will start the Coretex Run on your machine locally. Message output of successful start will look something like this:

>> [MLService] Login successful

This message is a sign of successful authentication. Keep in mind that you have to specify an id of your Coretex dataset in the experiment.config file or to set up a local Dataset in your task.