How to wait for api action to complete

Here is the output from a run of my solution.

root@hhgp:~# date;bin/vol1;date
Thu 19 Aug 18:45:34 BST 2021
,..mke2fs 1.44.5 (15-Dec-2018)
Discarding device blocks: done                            
Creating filesystem with 5242880 4k blocks and 1310720 inodes
Filesystem UUID: ec4a7b8c-bbfa-4c27-be80-f81766698807
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done   

Thu 19 Aug 18:45:53 BST 2021
root@hhgp:~# 

The commas and full stops before mke2fs indicate the 5 second waits before the volume becomes active and the volume is attached to the linode respectively.

The stages are

1. Create the volume
2. Wait for volume to become active
3. Wait for volume to be attached
4. Prepare the volume for use (format and mount)

It seems that stage 2 could be omitted as stage 3 happens later, but I prefer to leave it in.

The error handling is pretty basic and could be improved. Anybody is free to use, generalise and/or improve this as they see fit.

Here's the bash script

#!/bin/bash
set +x

# Allocate and format a volume via Linode api

# Requires httpie and jq

# Run as root

token=$(</etc/linode/api/volume_token)
linode_id=$(</etc/linode/api/linode_id)
vol="data"

rm /root/apiresp 2>/dev/null
rm /root/volresp 2>/dev/null
rm /root/lsresp 2>/dev/null

# Allocate a new volume
http --pretty format POST https://api.linode.com/v4/volumes \
    Authorization:"Bearer $token" \
    label="$vol" \
    size:=20 \
    linode_id:="$linode_id" \
    > apiresp
id=$(jq .id apiresp)
if [ "$id" = "null" ]; then
    echo "Volume create failed"
    cat apiresp
    exit 1
fi

fspath=$(jq .filesystem_path apiresp)
fspath="${fspath%\"}"
fspath="${fspath#\"}"

# Wait for status to become "active" (It's probably "creating" at this stage)
status=$(jq .status apiresp)
until [ $status = "\"active\"" ]; do
    echo -n ','
    sleep 5

    # View details of new volume
    http --pretty format GET "https://api.linode.com/v4/volumes/$id" \
        Authorization:"Bearer $token" \
        > volresp
    status=$(jq .status volresp)
    if [ "$status" = "null" ]; then
        echo "Volume view failed"
        cat volresp
        exit 2
    fi
done

# Wait for the volume to become attached
touch lsresp
ls -l "$fspath" 1> lsresp 2>/dev/null
lsresp=$(<lsresp)
while [ "$lsresp" = "" ]; do
    echo -n '.'
    sleep 5
    ls -l "$fspath" 1> lsresp 2>/dev/null
    lsresp=$(<lsresp)
done

# Make volume ready to use
mkfs.ext4 "$fspath" || exit 3
mkdir -p "/mnt/$vol" || exit 4
mount "$fspath" "/mnt/$vol" || exit 5
exit 0

Hopefully, there is a simple way to wait for completion of an operation in the api, but I couldn't find one. (I'm running this on the linode).

If you're doing this in a program (not a shell script), spawn a thread to make and wait for the API call. When the api call completes, have the thread signal the main thread with the results of the API call before terminating by joining the main thread.

-- sw

Thanks @stevewi, but I am using a shell script.

The response to the create gives "status": "creating" rather than "active" as per the example in the Api Documentation.

By complete, I mean the action complete not just the api call. In my case it is complete when the volume has been created and attacheds to the linode.

So… you can use PHP or Perl or Python or Ruby to achieve the same thing. All support some notion of threads.

If you used Perl, you can use WWW::Curl, JSON and Http::Resonse to do all the heavy lifting. Ruby & Python have similar facilities.

I personally would pick Ruby…expressive, simple & lets you focus on your job rather than mechanics. Also, Ruby doesn’t suffer from the “any sequence of random punctuation is a valid Perl program” syndrome so when you look at your program six months later to fix a bug or add a new feature, you can reconstruct your design vision.

Just a suggestion…

If you’re stuck on a shell script, use jobs (background processes), signals (trap) and some sort of async I/O mechanism (shared memory?) for signaling.

Shells don’t support threads. They’re really behind the curve on this…

You may find shell scripts just too limiting. I’ve run into that conundrum many many times.

— sw

@stevewi, I don't understand your obsession with asynchronous processing. It sounds like overkill to me. Polling is good enough as there is nothing to do until the volume has been created and attached.

I don't need advice on coding or language selection, after all I've been doing this for 50 years.

All I'm trying to find is the best way to tell when the volume is available for formating.

@stevewi, I don't understand your obsession with asynchronous processing. It sounds like overkill to me. Polling is good enough as there is nothing to do until the volume has been created and attached.

Polling is so 1970s…when there were no such things as multi-core CPUs. Polling holds the CPU/core while accomplishing nothing. Asynchronous processing allows the OS to prioritize other things ahead of the waiting thread/process.

I don't need advice on coding or language selection, after all I've been doing this for 50 years.

Fair enough. Riddle me this, Batman…how was I supposed to know that?

-- sw

I give up. Not once have you addressed the question I asked.

Just use busy-waiting. You have apparently decided that was the best way anyway…

-- sw

Don't be ridiculous. Haven't you heard of sleep?.

Why complicate things with threads just to suspend the task for a while. The OS does the work for you.

@dinosore
From what I gather from this discussion, you want to execute a task in the Linode API, but wait until that specific task is complete before moving on to the next one, in this case, creating a block storage volume. If I have correctly determined what you'd like to do, you want to make the HTTPS call to the API, but have that call return only when the volume is set up and prepared for further actions, attaching to a Linode, etc. If I'm right, to my knowledge, there's no way that this can be done. I think the API will return success or failure immediately when you send the API call to create a block storage volume, or anything else. This doesn't indicate whether it's actually ready for further actions, and I don't think there's a way to wait until a block storage volume, or anything else created via the API, is ready for further action within the API call itself.

I don't know what the Linode staff would have to say about this, or if there is a way to do such a thing. In your case, my solution would be to attempt the next step in your sequence of actions every five seconds or so in a loop, no longer than twelve iterations. At the twelth, it'll be approximately one minute later, and you should probably assume failure, regardless of whether or not the volume was successfully created. You could also catch the failure error and determine why it failed, if it's because the volume isn't yet ready, or for some other reason.

I don't really like this solution myself, I wish there was a way you could make an HTTPS call to the API that would wait until the creation process is complete, but this would introduce more problems on Linode's side, keeping multiple connections alive and open creates more load on their servers.

Good luck in your task, I hope I've been helpful.

Blake

Don't be ridiculous. Haven't you heard of sleep?.

What exactly about

SLEEP=<sleep interval>
...
do_something
while [ $(done) -ne 0 ] 
do
   sleep $SLEEP 
done
...

would you not consider to be busy-waiting?

Why complicate things with threads just to suspend the task for a while.

I thought I stipulated that threads were not an option in a shell script:

Shells don’t support threads. They’re really behind the curve on this…

You write:

The OS does the work for you.

What exactly about threads would you consider not to be part of the OS?

Whatev… There are ways to do what you want in a shell script that involve neither threads nor massive amounts of busy-waiting. You just have to think outside the box a bit (kill/trap are your friends). We each opt for what we know… ¡Adiós, amigo!

-- sw