Uploading Custom Python Pickle/Model

What will you learn

1. How to create a pickle object from python class.

2. How to upload pickle object to NetiLab python server.

After completing this tutorial you will be able to upload different python pickles and/or ML models to NetiLab python server.

Prerequisites

• Access to NetiLab application with enough permissions to create and run processes.

• Python server configured for user’s workspace.

Introduction

Pickle or pickling is a process where a Python object (class for example) is converted into binary format with all of its variables an parameters stored. On the other hand unpickling is a reverse operation. In contrast to the raw .py file, pickle can contain also other data like variable values and entire trained ML model. After unpickling operation, you can execute every of the class functions.

In NetiLab Python server pickle file is stored and it then become available for using it as a separate node in any process. In that way, you can write custom logic and functionality with python and add it into NetFlow.

Steps

Step 1

First step is creating a pickle file. We can do so with python dill library. Below example creates a pickle file from python class

import dill as pickle

class DummyModel:
    # Libraries used inside the class needs to be imported in this way,
    # otherwise there can be some errors with imports after unpickling operation.
    np = __import__("numpy")
    pd = __import__("pandas") 
    rand = __import__("random")   

    def dummy_predict(self, df):
        """
        A function that simulates ML prediction.
        """
        df["Prediction"] = self.rand.random()
        return df
    
# Create an instance of a class
x = DummyModel()

# Write class data into pickle object
y = pickle.dumps(x)

# Store pickle object into file
with open('dummy_model_pickle.txt', 'w') as f:
    f.write(y.hex())

After running this code, a file named dummy_model_pickle.txt should be generated.

Step 2

Now we will upload created pickle file to NetiLab Python server. Find process Upload Pickle and click Run. Process should stop on a file upload form. Upload previously created pickel file and click continue. Process should then successfully finish.

Pickle file is then stored on Python server can then be used as a node.

When pickling a Python object all custom module imports should be included as a class or function in the same class. I.e. pickling operation does not include module imports from other files on its own.