Returning Data from a Python (or Node.js, Perl, PHP, QShell, PASE, etc.) Script
Dumb Joke of the Month
My first choice for a career was to play hide & seek! After all, great players are hard to find.
COMMON POWERUP 2024
As I write this blog post, I am preparing to attend COMMON’s POWERUp 2024 in Fort Worth, TX (May 20-24, 2024). If you’ll be there, please stop by and say “hi”! If you’d like to know more about the event, you can find details here: https://www.common.org/powerup2024/
Returning Data from a Python (or Node.js, Perl, PHP, QShell, PASE, etc.) Script
Q: I’ve written a Python script and am calling it from CL.
The CL program passes input to the Python script in the 1st parameter, the
script does some work, and then tries to return its output through the 2nd
parameter. I’ve debugged it and the output looks good in the Python program,
but the CL program isn’t receiving it. What’s wrong?
A. On IBMi, Python (and most other code written in an open source language) runs in the PASE environment, which works like Unix. In Unix, parameters are input-only, so outputting through the parameter isn’t going to work. Thankfully, there are other ways to send output, and I’ll describe one of them in this blog post.
The PASE Environment
That is the purpose of the PASE environment: It provides a runtime that’s more-or-less equivalent to the runtime environment in AIX. This is very useful because most open source software is available for Unix or Unix-like systems (such as Linux or MacOS) and therefore can easily be made to run on AIX. With PASE, it can also be run on IBM i.
Indeed, that is how Java, PHP, Perl, Node.js and Python (amongst others) work on IBM i. They run in the PASE environment.
Understanding the Problem
In AIX and other Unix systems, parameters passed to a program or script are input-only. That means you can read data passed from your CL program, but you cannot return data back through the parameters.
Unix software also offers a concept of “standard input, output and error streams”. These are streams of text data that are automatically available in Unix-like software. Standard input (“stdin”) is attached to the user’s keyboard by default but can be redirected to come from a program or file. Likewise, standard output (“stdout”) is written to the user’s screen by default but can be redirected to a program or file. Standard error (“stderr”) is similar to standard output, but by convention is only used for error messages.
What this means is that although you cannot return output through the parameter list, you can write your Python script (as well as software in any of the other languages in the PASE environment) so that it returns its output via standard output. Then your CL program can read the standard output to get the results.
Example Python Script
The person who submitted this question provided an example Python script that reads the files in a directory. It should be noted that there are many ways to read files in a directory, and some of them are simpler than calling a Python script. However, this is still a useful exercise because there are many, many other things that Python can do, to say nothing of the other languages in the PASE environment, so knowing how to return data is invaluable.
Here is the original program that I received, except that I modified it to send its output to stdout instead of trying to write it to a parameter:
import os
import sys
# Retrieve path to read from the
parameters
root_path = sys.argv[1]
def get_next_level_items(root_path):
items = []
for item in os.listdir(root_path):
full_path = os.path.join(root_path, item)
if os.path.isdir(full_path):
items.append((item, ‘directory‘))
elif os.path.isfile(full_path):
items.append((item, ‘file‘))
items.sort(key=lambda x: (x[1], x[0]))
return [item[0] for item in items]
# list the files in that path and
print to stdout
next_level_items = get_next_level_items(root_path)
print(next_level_items)
The sys.argv[1] variable is the first parameter passed to the program. The print() function is what writes the data to standard output.
Processing Stdout with the IBM-Supplied QSH CommandUnfortunately, the tools that IBM provides with IBM i for running commands in PASE or QShell are not very good at handling standard input or output. IBM does provide a means of writing the output to a file, however.
Therefore, we can write the output to a file and then read that file in our program, the code that follows is one way to do that:
/* +
* TO COMPILE: +
* CRTPF QTEMP/QSHOUTPUT RCDLEN(5000) +
* CRTBNDCL CALLPY SRCFILE(QCLSRC) +
* DLTF QTEMP/QSHOUTPUT +
*/
pgm
DCL VAR(&JOBNO) TYPE(*CHAR) LEN(6)
DCL VAR(&TEMPFILE) TYPE(*CHAR) LEN(50)
DCL VAR(&OUTPUT) TYPE(*CHAR) LEN(75)
DCL VAR(&DIR) TYPE(*CHAR) LEN(256) VALUE(‘/QIBM’)
DCL VAR(&CMD) TYPE(*CHAR) LEN(500)
DCL VAR(&EOF) TYPE(*LGL) VALUE(‘0’)
DCLF FILE(QSHOUTPUT)
/* Change the value of &DIR to whatever folder you want +
to list */
RTVJOBA NBR(&JOBNO)
CHGVAR VAR(&TEMPFILE) VALUE(‘/tmp/tempfile’ +
*CAT &JOBNO +
*TCAT ‘.txt’)
CHGVAR VAR(&OUTPUT) VALUE(‘FILE=’ *CAT &TEMPFILE)
ADDENVVAR ENVVAR(QIBM_QSH_CMD_ESCAPE_MSG) +
VALUE(Y) +
REPLACE(*YES)
ADDENVVAR ENVVAR(QIBM_QSH_CMD_OUTPUT) +
VALUE(&OUTPUT) +
REPLACE(*YES)
CHGVAR VAR(&CMD) VALUE(‘/QOpenSys/pkgs/bin/python3’ +
*BCAT ‘/path/to/listfiles.py’ +
*BCAT &DIR)
QSH CMD(&CMD)
MONMSG QSH0000 EXEC(DO)
/* FIXME: An error occurred! Normally you’d want to +
notify the user or write it to a log or something here +
The error should be in the temp file so for a quick and +
dirty solution, I’ll display the file. */
DSPF STMF(&TEMPFILE)
RETURN
ENDDO
DLTF QTEMP/QSHOUTPUT
MONMSG CPF0000
CRTPF QTEMP/QSHOUTPUT RCDLEN(5000)
CPYFRMSTMF FROMSTMF(&TEMPFILE) +
TOMBR(‘/qsys.lib/qtemp.lib/qshoutput.file/q+
shoutput.mbr’) MBROPT(*ADD)
DOWHILE COND(&EOF *EQ ‘0’)
RCVF
MONMSG MSGID(CPF0864) EXEC(CHGVAR &EOF VALUE(‘1’))
IF (&EOF *EQ ‘0’) DO
/* FIXME: The &QSHOUTPUT variable now has the output from the +
Python script. Replace the following code with whatever you wish +
To do with that output. */
SNDUSRMSG MSG(&QSHOUTPUT)
ENDDO
ENDDO
RMVLNK OBJLNK(&TEMPFILE)
DLTF QTEMP/QSHOUTPUT
endpgm
You’ll need to replace /path/to/listfiles.py with the proper name of the python script on your system.
To be honest, I think this program is really clumsy. There are many different approaches that could’ve been done better, but they would require more background explanations and more set up on each system that tries to use this program, and I don’t want to make this blog post any longer than I have to.
In particular, you should be using the PATH variable to control where it finds the python3 interpreter rather than hard-coding the path (just as you don’t hard-code library names in your CL programs, you shouldn’t hard code IFS directory locations.)
A More Elegant Solution
I also find it clumsy that it writes the output to a temporary file, then copies that to another temporary file, and then finally reads the file in the CL program. It would be a lot more elegant to read the data directly from the Python script’s stdout, and more performant, too!
For that reason, I created an open source project called UNIXCMD that provides a way to treat a QShell or PASE command as if it is a file. This way, you can write data to the program (which it could read via it’s stdin) or read data that the program has written to it’s stdout as if you were working with a file. However, under the covers, it is not a file, the data is simply passed directly from the Unix program to your program using RPG’s open access.
To use this example, you will need to download and install the UNIXCMD utility from my website (the link follows). Don’t worry, this is open source software, and is completely free of charge. https://www.scottklement.com/unixcmd/
Here is an RPG example using UNIXCMD:
**free
dcl-f UNIX disk(5000)
usage(*input:*output)
handler(‘UNIXCMDOA’: cmd)
usropn;
dcl-s cmd char(5000);
// FIXME: Change this to the folder you want to list
dcl-c dir ‘/QIBM’;
dcl-ds rec qualified;
data char(5000);
end-ds;
cmd = ‘/QOpenSys/pkgs/bin/python3’
+ ‘ /path/to/listfiles.py ‘
+ dir;
// OPEN starts the command running
open UNIX;
read UNIX rec;
dow not %EOF(UNIX);
// FIXME: Change this to use the data in the record
// to do something useful
snd-msg rec.data;
read UNIX rec;
enddo;
// FIXME: ‘close’ will give an error if something goes
// wrong, we should be monitoring for that
close UNIX;
*inlr = *on;
Notice the HANDLER keyword on the DCL-F. This means that when you do operations to the UNIX file, you aren’t really working with a file. Instead, it will call the UNIXCMDOA program (which is called an “Open Access Handler”.) The UNIXCMDOA program will start up the QShell or PASE program and connect it to a “pipe” which is a communications device for communicating data between programs in a Unix environment.
When you perform an OPEN operation, it starts the command running. A READ operation reads data from the pipe (which is connected to the program’s stdout and stderr) and the CLOSE operation tells the running program that we’re done. Since errors are typically reported when the program ends (i.e. during the CLOSE operation) if something goes wrong, the CLOSE is probably where you want to monitor for errors.
This solution is much more efficient and eliminates the temporary files. (Like the CL solution, however, you should remove the hard-coded directories and use the PATH variable for best results.) The disadvantage, of course, is that you have to install the UNIXCMD software.
UNIXCMD In CL Programs
Since the CL programming language does not have an equivalent to RPG’s open access, you won’t be able to use standard file access operations in CL. Instead, UNIXCMD provides a set of commands for opening, reading, writing and closing a pipe that you can use in your CL program.
PGM
DCL VAR(&CMD) TYPE(*CHAR) LEN(5000)
DCL VAR(&REC) TYPE(*CHAR) LEN(5000)
DCL VAR(&DIR) TYPE(*CHAR) LEN(256)
DCL VAR(&EOF) TYPE(*LGL) VALUE(‘0’)
/* FIXME: Change this to the dir you want to list */
CHGVAR VAR(&DIR) VALUE(‘/QIBM’)
CHGVAR VAR(&CMD) VALUE(‘/QOpenSys/pkgs/bin/python3’ +
*BCAT ‘/path/to/listfiles.py’ +
*BCAT &DIR)
OPNPIPE CMD(&CMD)
RCVPIPE RCD(&REC) EOF(&EOF)
DOWHILE COND(&EOF *EQ ‘0’)
/* FIXME: Change this to use &REC rather than printing +
it to the screen */
SNDUSRMSG MSG(&REC)
RCVPIPE RCD(&REC) EOF(&EOF)
ENDDO
CLOPIPE
ENDPGM
The same caveats apply to this as the RPG example.
It would be nice if IBM provided tools like this with the operating system. After all, they have been working hard to provide us with access to these open source environments. Shouldn’t they provide better ways to integrate them?
Until they do, give UNIXCMD a try!