hogumathi-app

Hogumathi

The purpose of this app is three fold:

• Import the Twitter Archive into a usable format
• Provide a User Interface to the Archive and Twitter API
• Generalize the user experience across services and integrate the data

Since the Twitter API only provides 7-30 days of historical data, we download a copy of the archive and use it to augment live data from the API and enhance the user experience with the historical data.

Since there's a great migration from Twitter to Mastodon and I've previously built a Mastodon hosting company, I've added a Mastodon backend that mirrors the Twitter API access functions.

The app is designed to be run locally on a locked down system such as Windows S Mode and in the Cloud in a managed environment like Glitch.

Status

This is the merger of three projects which are on their way toward meeting in the middle.

Initially the Archive tech was developed for personal reporting, and then my quest into Python brought me to web development and I needed a known domain to build for so I could focus on learning the web tech. Prior experiments lead me to build a Mastodon client and soft launch a hosting company, so it's an avenue for me to cash in.

The intent of this release is to give a deployable Twitter and Mastodon client away to open source, and then bring the archive functionality into the UI. At the time of release the Archive functionality will be obscure but present to who looks for it. The same goes for personal reporting functionality which is even less integrated.

User Interface

The user interface is built to be as simple to customize as possible for the power user.

It connects to the regular Twitter site using the official method that is well documented on the developer site. Mastodon is almost the same It is built to be general enough to support alternative backends as well, to port the experience on one network to another.

It minimizes the need to use Javascript for customization to the extent possible, making the experience good for programmers who have not mastered modern web browser tricks.

Archive

A Twitter accounts's Archive can be requested from settings; it takes about a day to deliver and is 500MB-2GB, depending largely on how much media was posted. An archive of 50k Tweets is about 50MB of raw tex or 20MB compressed, and can be requested once per month. The Tweets file is not in a usable format, requiring processing even to read into a JSON parser.

Once in memory it's simply a large list of Tweets and requires manual coding to do anything useful. The first thing we do is put it into an SQLite3 database for fast and easy querying using SQL, a proven tool known by non-programmer business analysts worldwide.

Once the archive is in a Database, we need an interface to work with it. Since the data is derived from the live site's data, we fit it into a user interface that works like the regular Twitter app.

Technology

• Python 3
• HTMx
• SQLite3

Using HTMx allows us to get away without much Javascript on the client side. It's a throwback to earlier days of web development pre-Angular and Reach and into the era of jQuery Ajax forms, but it codifies the operations via custom HTML attributes beginning with hx-~.

Installation

• Install Python 3.7+
• pip install -r requirements.txt
• Copy sample-env.txt to .env and configure

Twitter v2 API

One needs to go to the developer.twitter.com portal and create an app and obtain the values listed in the .env file.

• Client type "Confidential client"
• Redirect URL, local: http://localhost:{PORT}/twitter/logged-in.html
• Redirect URL, Glitch: https://{PROJECT NAME}.gitch.me/twitter/logged-in.html

Tweet Archive

The Twitter archive comes as a zip file that contains a file called tweets.js which is not a valid JSON file. One can open it with their code editor and change the first line; replace window.YTD.tweet.part0 = [ with just [ and save the file as tweets.json at the ARCHIVE_TWEETS_PATH in .env. The file then needs to be loaded into a SQLite3 DB by making a POST request to the URL /twitter-archive/tweets.

• curl -XPOST http://localhost/twitter-archive/tweets

Tweet Collections

There is no UI to edit Tweet collections, but they can be created with the following JSON structure:

{"authorized_users": ["{Numeric Twitter accounts ID}"],
 "items": [
    {
        "id": "{Numeric Tweet ID}",
        "note": "My note about the Tweet",
	}
	]}

If no "authorized_users" is present then the collection is accessible to all users. By default the base.html template file contains a link to a collection named swipe-all2 which should exist in .data/collection/swipe-all2.json.

Feeds

This is a preview release, hard coded feeds are available in the get_posts method of feeds_facade.py. It includes a sample of feeds across providers and shows that they can all be represented in the same model and merged together into a timeline-like experience.

YouTube

This is a preview release, hard coded. OAuth authentication works and should be configured on one's own YouTube account. 'YouTube Data v3' access is required, to read channel info and followers/subscribers. It's one day old, so one really needs to go through the code to understand it.

The developer portal provides a client secret file that should be placed in .data/yt_client_secret.json.

Glitch Deployment

The app detects if it's running in Glitch and will configure itself accordingly. Ensure that the Twitter API is configured to allow access to your project name.

Running

For local deployment, use Python:

• python hogumathi_app.py or python3 hogumathi_app.py

Contributing

Since I intend to retain full copyright and re-license portions of the project I am only able to accept contributions with a copyright assignment, similar to how the Apache project works.

I'm open to substantial contributions with that in mind, and more importantly I'd love to pay contributors for their work; as such, you may donate to me and I can share the wealth while maintaining my business constraints. I know this may rub some free software folks the wrong way, such as a younger version of myself. These days one needs to be serious about the business of their software, and so releasing my work to the public using a business unfriendly free software license is the balance I can offer.

Patreon and Venmo

Thanks to my friends who remain, and Patreons and Venmo donors for financial support.

• https://patreon.com/harlanji - Monthly support
• https://venmo.com/harlanji - Tips for coffee

License & Copyright

The project is licensed to the public under the AGPL, requiring disclosure of the source code however it is used.

Please contact me for a commercial or copyleft license for portions of the project.

Copyright (C) 2021-22 Harlan J. Iverson; all rights reserved.