TechOne

TechOne doesn't expose an API that MODLR can call directly, so this uses MODLR's Workaround integration method: TechOne is configured to export scheduled reports as CSV files to an SFTP location, and a Process on the MODLR side connects out to pull those files down before reading them in.

TIP

Every TechOne environment is configured a little differently. If your export setup doesn't match the approach below, MODLR support is happy to investigate further alongside you - submit a request for TechOne integration advice.

Exports are often split across more than one remote folder (for example, a main data export folder alongside a separate reporting-specific folder) - the Process should check each folder that's relevant.

Pulling the files

A Process connects to TechOne's SFTP export location using ftp.ConnectWithKeyFile, lists the files in each remote export folder, and downloads any that haven't already been pulled down (skipping files already present on the MODLR filesystem, using datasource.fileSize as a simple "already downloaded" check):

function pre() {
    // This function is called once before the process is executed.
    // Use this to setup prompts.
    script.log('process pre-execution parameters parsed.');
}

function begin() {
    // This function is called once at the start of the process
    script.log('process execution started.');

    let sftpClient = ftp.ConnectWithKeyFile(
        "sftp",
        "sftp.example.com",
        22,
        "modlr_user",
        "/path/to/key_file",
        security.getSecret("TECHONE_SFTP_PASSPHRASE")
    );

    // Check each remote folder that TechOne exports into.
    const remoteFolders = [
        "/export/main",
        "/export/reporting"
    ];

    remoteFolders.forEach(remoteFolder => {
        let files = JSON.parse(sftpClient.Directory(remoteFolder));

        files.forEach(item => {
            if (!item.isDirectory) {
                if (datasource.fileSize("/incoming/techone/" + item.name) == 0) {
                    console.log(remoteFolder + "/" + item.name);
                    sftpClient.Download(remoteFolder + "/" + item.name, "/incoming/techone/" + item.name);
                }
            }
        });
    });
}

function data(record) {
    // This function is called once for each line of data on the second cycle
    // Use this to build dimensions and push data into cubes
}

function end() {
    // This function is called once at the end of the process
    script.log('process execution finished.');
}

This is the same technique documented generically in Flat File Load: Option 1.

Reading the files

Once downloaded, the CSV exports are read with datasource.readFile - see Flat File Load: Reading the file in a Process for the pattern.