[Solved] Please see my problem, believe me it is easy to solve

by


I’m not sure I completely understand what you’re trying to do, but I’ll give it a shot since you seem to have asked this question many times already (which usually isn’t a good idea). I believe that there’s a lack of clarity in your question – it would help a lot if you could clarify what your end goal is (i.e. how do you want this to behave?)

I think you mentioned two separate problems here. The first is that you expect a new line of ‘******’ to be placed before each separate piece of data returned from your script. This is something that can’t be relied on – check out the answer to this question for more detail: Order of process.stdout.on( ‘data’, … ) and process.stderr.on( ‘data’, … ). The data will be passed to your stdout handler in chunks, not line-by-line, and any amount of data can be provided at a time depending how much is currently in the pipe.

The part I’m most confused about is your phrasing of “to get intermediate ouput of the python script in my nodejs script”. There’s not necessarily any “immediate” data – you can’t rely on data coming in at any particular time with your process’s stdout handler, its going to hand you data at a pace determined by the Python script itself and the process its running in. With that said, it sounds like your main problem here is the timeout happening on your POST. You aren’t ever ending your request – that’s why you’re getting a timeout. I’m going to assume that you want to wait for the first chunk of data – regardless of how many lines it contains – before sending a response back. In that case, you’ll need to add res.send, like this:

    // form submit request
app.post('/formsubmit', function(req, res){

    csvData = req.files.csvfile.data.toString('utf8');
    filteredArray = cleanArray(csvData.split(/\r?\n/))
    csvData = get_array_string(filteredArray)
    csvData = csvData.trim()
    
    var keywords = req.body.keywords
    keywords = keywords.trim()

    // Send request to python script
    var spawn = require('child_process').spawn;
    var process = spawn('python', ["./webextraction.py", csvData, keywords, req.body.full_search])

    var outarr = []

    // process.stdout.on('data', (data) => {
    //   console.log(`stdout: ${data}`);
    // });
    
    // Keep track of whether we've already ended the request
    let responseSent = false;

    process.stdout.on('data', async function(data){

        console.log("\n ************* ")
        console.log(data.toString().trim())
        outarr.push(data.toString().trim())
        console.log("\n ************* ")
        
        // If the request hasn't already been ended, send back the current output from the script
        // and end the request
        if (!responseSent) {
            responseSent = true;
            res.send(outarr);
        }
    });

});

12

solved Please see my problem, believe me it is easy to solve