Back

also, for #curl / #bash aficionados: how do I actually DO this, like in a script

https://docs.joinmastodon.org/api/guidelines/#pagination

(I'm trying to get a list of all the people I follow, via the Mastodon API, but of course it paginates. I can get the "next" link with some gnarly-looking `curl -D` and `grep -Po` stuff but do I just, like, do a for loop to repeatedly do it until there isn't any response?)

@nev this particular task is a surprisingly annoying pain in the ass.

If you're doing this with curl, you have to..:

1. use `--dump-header [filename]` to preserve the response headers in a temp file;

2. then convert those headers from DOS to unix line endings;

3. then pick out the `Link: ...` header;

4. then parse its `rel=...` components into key/value pairs;

5. _then_ you can use the `next` k/v pair to fetch your next page of results.

It's fiddly. The GitHub REST API does this too.

@gnomon so i ended up doing

curl -s -D - -o /dev/null "$url" | grep -Po '(?<=link\: \<).+(?=\>\; rel\=\"next\")'

which I'm sure is terrible and non-portable but it works for me lmao

@nev NICE

@nev does that structure mean you're downloading each page twice, once for the content and once to throw away the content but extract the link header?

(Which isn't necessarily a problem, it's fine)

@gnomon yeah, I wanted to see the output in standard out as it proceeded. But I suppose I could combine them into one? (A cursory search seems to show that I can use tee & process substitution?)

@nev you could, yep!

In fact you wouldn't even have to mess with tee, process substitution alone would work - something like:

result=''
verbose=1
url_next='https://whatever_apicall_to_get_page1'
until [[ ! $url_next == http* ]]; do
result=$(curl --dump-header '.headers' --url "$url_next")
if [[ $verbose == 1]]; then echo "$result"; fi
# url_next=$(grep ... '.headers.') # step to get the next link, or nothing
# step to do whatever with "$result" here
done

@gnomon just making sure i have this right: this puts the header info into a file called ".headers" for separate grepping? and all "verbose" does is print $result to stdout?

@gnomon thank you so much for your help with this! this is what i eventually went with, though of course it's not carved in stone and there's likely more efficient ways to do this: https://codeberg.org/flippingrocks/pages/raw/branch/main/media/downloads/mastodon-fws.sh

@nev you're welcome, I'm delighted that was helpful!!

@nev ooh I have a trick to show you about managing temp files, I'll pastebin up a thing for you as soon as I get off this bus

@nev well this is coming a bit late, sorry about that. I wrote you a little example about using a `trap ... EXIT` handler in a bash script to automatically remove temp directories and files on program exit, even if that exit is unclean or an error.

https://paste.sr.ht/~gnomon/e3f2bd6723565d81438cffe5b993e1485d0d0652