Follow

proof-of-concept for importing packagesite.yaml into FreshPorts. The steps are:

1 - From each line of 32500-line yaml file, extract 3 fields creating a csv file
2 - load cvs file into db

Step 2 takes seconds.

I need help / advice with step 1 which takes 3 minutes.

Code and examples are found here:

git.langille.org/FreshPorts/pa

Thank you

Combined suggestions from Fosstodon & from bsd.network:

$ time jq -rc '[1, .origin, .name, .version] |
@tsv
' < ~/tmp/FreeBSD\:12\:amd64/latest/packagesite.yaml > packagesite.csv

real0m1.351s
user0m1.295s
sys0m0.055s

$ time ./import-via-copy-packagesite.py

real0m1.731s
user0m0.131s
sys0m0.008s

The data get in there fast enough.

Next step, go from that raw data into normalized form. That should be easier & faster now that it's in a [] database [on ].

Thank you.

Show thread
Sign in to participate in the conversation
Mastodon

Server run by the main developers of the project 🐘 It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!