Boring technical question which neither the documentation nor ChatGPT can answer.
I want to automate screenshots using #Python's #Selenium Webdriver with #Chrome.
This works - but the non-logged in experience of Twitter forces dark-mode.
I want a light-mode screenshot.
I can't find a way to set SEC-CH-PREFERS-COLOR-SCHEME in Selenium. Nor can I find a light-mode toggle for non-logged in Twitter users.
Any clues gang?
(I want to automate replacing my Twitter embeds with a screenshot.)
Aha! There's a way to cheat!
The old Twitter embed API lets you manually specify colour scheme - and is easy to screenshot.
Hopefully I can replace all the Twitter embeds on my website with images (and, yes, I will add alt text).
Thanks @samir for giving me the inspiration.
Any #Selenium experts in the house?
All my screenshots are slightly fuzzy. Why?!?!
I'm using the Chromium webdriver. Grabbing the area with `element.screenshot_as_png`
Is it because I have fractional scaling? Wayland?
I have set `chrome_options.add_argument("--ozone-platform=wayland")` but to no avail.
This'll be a two-drink minimum problem, I reckon.
OK, if you have lots of Twitter embeds in your website, I've written an easy(ish) way to replace them with static images & alt text.
https://github.com/edent/Tweet2Img
Run:
`python tweet2img.py 1234`
And you'll get a ready-to-paste piece of HTML with a data-encoded image.
`<a href="https:/ /twitter.com/edent/status/123"><img src="data:image/webp;base64,Ukl..." width="550" height="439" alt="text from tweet."/></a>`
It also saves the WebP and alt text if you need them.
Someone complained that an image with decent alt text wasn't accessible.
So now I'm practising Spite-Driven-Development to create a semantic HTML embed.
Doesn't look *totally* accurate - but not bad for a dozen lines of CSS.
Spite-Driven-Development continues.
I now have a scrap of semantic HTML which adequately replicates an embedded Tweet.
TODO:
* Add images
* Add entities (hashtags, mentions, links)
* Add parent of thread
* Add quoted tweet
* Add ... whatever else
Should be able to hijack the embed API to get most metadata (no number or replies / retweets though).
Parents and Quotes are just <blockquotes> within <blockquotes>. So that was easy!
I think that's good enough for the basics.
This code will (eventually) convert all your embedded Tweets into proper semantic HTML without any tracking back to Twitter.
Rough code at:
https://github.com/edent/Tweet2Embed
OK gang! This is ready for you to test.
python tweet2html.py --css 1234
That will take a Twitter ID and return HTML and CSS for you to embed in your website with no calling back to Twitter.
Features: Avatars inlined as Base64 WebP
Hyperlinks don't use t.co
Hashtags and @ mentions linked
Semantic time
and
counts
Try it out at https://github.com/edent/Tweet2Embed
Feedback and pull requests very welcome!
Now also shows parents to replies, and full content of quotes.
And polls!
HTML validates!
Play with the code at https://github.com/edent/Tweet2Embed
I'll be doing this for Mastodon later - because I really don't like iFrame embeds!
OK! I *think* I've finished.
You can now "rescue" any embedded Tweet and recreate it as simple HTML - no tracking.
Includes:
Avatars inlined as WebP
Hyperlinks don't use t.co
Hashtags & @ mentions linked
Includes reply threads & quote Tweets
Semantic time
Schema.org metadata
Cards
Polls
,
&
counts
One command. No API key needed.
Code at https://github.com/edent/Tweet2Embed
Feedback *very* much welcome!
@Edent Lots of missing python packages for that.
For example, apt doesn't have python3-webdriver (or what I need for "from webdriver_manager.firefox import GeckoDriverManager").
Maybe publish a list of what use would need?
@derickr how do I go about doing that?
I think I just pip installed everything I needed. Is there a way to find out what? I've never properly published a Python package before.
@Edent I don't really know either. I think there usually is some kind of file listing them that you can install?
@derickr I've added a requirements.txt - hopefully that works!
@Edent That worked, after using virtual env, and installing xclip:
python3 -m venv /tmp/t2i
/tmp/t2i/bin/pip3 install -r requirements.txt
sudo apt install xclip
/tmp/t2i/bin/python3 tweet2img.py 1821809135920550280
@derickr woo!
I used the Wayland version - good to know it works with the x version.
@Edent That’s pretty cool! I could see this being super useful for static sites where you can just automate any tweet in Markdown on its own line
@Edent I mean, they use a million lines of JS to provide an embed, but there’s nothing stopping you using their same CSS so it’s indistinguishable
@Edent yikes. I applaud the effort to attempt to emulate their look with decent semantic HTML. Keep it up :)
@thin_line @sil
Try it and let me know
quick test with anthropic's claude 3.5 sonnet
https://gist.github.com/hollisticated-horse/3f98702594da7979765a908e5c9d8097
I think it's pretty close.
@Edent suggestion (if possible) expand the short links?
@ben see previous posts - on the todo list.
The expanded URls are in the entities that are returned by the API - but I'm not sure the best way of converting them back to HTML.
@Edent any chance it could handle tweet threads in the future?
I have some /epic/ threads I want to archive, 100+ tweets chained together, most of which are themselves quote-tweets
This one for example (only properly visible if you are signed in to Twitter) https://twitter.com/simonw/status/1077737871602110466
@simon Ooooh. Possibly. I used to do epic things with their threading API. Not sure if it's still possible - but I'll put it on the roadmap.
Next step is quotes and replies.
If I can get that working, then it *should* be possible.
@Edent my SpiderVerse thread would be an epic stress-test - many of the quote-tweets have embedded video, so could be hundreds of MBs if you were to fetch and base64-embed all of them!
@simon if you don't already know it, https://www.solipsys.co.uk/DiscDAG.html by @ColinTheMathmo
@simon I used to fetch and render the *content* of Twitter threads, though not the linked or embedded media. Then the nodes were clickable so you could view the original.
It was never very good, but it served my purposes:
https://www.solipsys.co.uk/Chitter/SuggestedReading.svg
I have bigger examples.
When the threads become humungous, viewing a static tree is sub-optimal.
CC: @Edent
Calling @Chartodon spine ...
Your chart is ready, and can be found here:
https://www.solipsys.co.uk/Chartodon/112939515109242805.svg
Things may have changed since I started compiling that, and some things may have been inaccessible.
In particular, the very nature of the fediverse means some toots may never have made it to my instance, in which case I can't see them, and can't include them.
The chart will eventually be deleted, so if you'd like to keep it, make sure you download a copy.
@eaton you may or may not already be familiar with @darius's <https://tinysubversions.com/twitter-archive/>.
@darius yep, that's what @eaton was talking about.
Folks should really be exporting their archives on the regular. Have you considered setting something up for people e.g. that explains that Twitter lets you do this—and give them a place for them to dump it? For those that don't want to go whole-hog on the Mastodon/Fediverse thing, but are amenable to doing just a tiny bit to be proactive about having an escape plan.
@eaton @darius @colby @Edent I recently posted the first video on my blog that was hosted by me (an mp4 file in a video element) as opposed to hosted on YouTube or similar: https://simonwillison.net/2024/Jul/29/sam-2/
It's in my S3 bucket along with all my other images, but I have to admit I do worry that some day decades into the future I may fail to pay my AWS bill and risk my S3 bucket blinking out of existence
@simon @eaton @darius @colby @Edent this actually happened to me. I wasn't using my aws account, but there was some stuff there costing me pennies per month. The credit card I used for the account expired and I didn't see the alert emails. When I finally tried to go back to the account, it was gone.
@Edent this is incredibly cool. It's it per tweet right now or can you embed a feed?
@snim2 per Tweet.
But, if that Tweet is a reply - it will get the parent as well.
There's no "conversation ID" in the free API, so it can't get full threads.
There's also no easy way to iterate over an entire feed.
@Edent this is very cool! If you don’t mind I might use it as inspiration to rebuild and replace my old, no longer working webapp https://tweet.cluster.fun/
@Marcus as per the terms of the MIT licence, you may do so only after sending me one million bitcons.
@Edent for you, I’ll give you two million bitcons
Feel free to use it. A link back would be appreciated.
@Edent assuming I ever actually get the motivation to tackle it, you betcha!