# Hogumathi

The purpose of this app is three fold:

* Import the Twitter Archive into a usable format
* Provide a User Interface to the Archive and Twitter API
* Generalize the user experience across services and integrate the data

Since the Twitter API only provides 7-30 days of historical data, 
we download a copy of the archive and use it to augment live data
from the API and enhance the user experience with the historical data.

Since there's a great migration from Twitter to Mastodon and I've previously built a 
Mastodon hosting company, I've added a Mastodon backend that mirrors the Twitter API 
access functions.

The app is designed to be run locally on a locked down system such as Windows S Mode 
and in the Cloud in a managed environment like Glitch.

## Status

This is the merger of three projects which are on their way toward meeting in the middle.

Initially the Archive tech was developed for personal reporting, and then my quest into Python brought me
to web development and I needed a known domain to build for so I could focus on learning the web tech. Prior
experiments lead me to build a Mastodon client and soft launch a hosting company, so it's an avenue for
me to cash in.

The intent of this release is to give a deployable Twitter and Mastodon client away to open source, and then bring the 
archive functionality into the UI. At the time of release the Archive functionality will be obscure but present
to who looks for it. The same goes for personal reporting functionality which is even less integrated.

## User Interface

The user interface is built to be as simple to customize as possible for the power user.

It connects to the regular Twitter site using the official method that is well documented 
on the developer site. Mastodon is almost the same It is built to be general enough to support 
alternative backends as well, to port the experience on one network to another.

It minimizes the need to use Javascript for customization to the extent possible, 
making the experience good for programmers who have not mastered modern web browser
tricks.

## Archive

A Twitter accounts's Archive can be requested from settings; it takes about a day to deliver and is 500MB-2GB,
depending largely on how much media was posted. An archive of 50k Tweets is about 50MB of raw tex or 20MB compressed,
and can be requested once per month. The Tweets file is not in a usable format, requiring processing even to
read into a JSON parser. 

Once in memory it's simply a large list of Tweets and requires manual coding to do anything useful. The first thing we do
is put it into an SQLite3 database for fast and easy querying using  SQL, a proven tool known by non-programmer business analysts worldwide.

Once the archive is in a Database, we need an interface to work with it. Since the data is derived from the live site's data, we fit it into a user interface that works like the regular Twitter app.

## Technology

* Python 3
* HTMx
* SQLite3

Using HTMx allows us to get away without much Javascript on the client side. It's a throwback
to earlier days of web development pre-Angular and Reach and into the era of jQuery Ajax forms,
but it codifies the operations via custom HTML attributes beginning with `hx-~`.

## Installation

* Install Python 3.7+
* `pip install -r requirements.txt`
* Copy `sample-env.txt` to `.env` and configure

### Twitter v2 API

One needs to go to the `developer.twitter.com` portal and create an app and obtain the values listed
in the `.env` file. 

* Client type "Confidential client"
* Redirect URL, local: `http://localhost:{PORT}/twitter/logged-in.html`
* Redirect URL, Glitch: `https://{PROJECT NAME}.gitch.me/twitter/logged-in.html`

### Tweet Archive

The Twitter archive comes as a zip file that contains a file called `tweets.js` which is not a valid JSON file.
One can open it with their code editor and change the first line; replace `window.YTD.tweet.part0 = [` with just `[`
and save the file as `tweets.json` at the `ARCHIVE_TWEETS_PATH` in `.env`. The file then needs to be loaded into a SQLite3 DB
by making a `POST` request to the URL `/twitter-archive/tweets`.

* `curl -XPOST http://localhost/twitter-archive/tweets`

### Tweet Collections

There is no UI to edit Tweet collections, but they can be created with the following JSON structure:

```
{"authorized_users": ["{Numeric Twitter accounts ID}"],
 "items": [
    {
        "id": "{Numeric Tweet ID}",
        "note": "My note about the Tweet",
	}
	]}
```

If no "authorized_users" is present then the collection is accessible to all users. By default the `base.html` template 
file contains a link to a collection named `swipe-all2` which should exist in `.data/collection/swipe-all2.json`.


### Feeds

This is a preview release, hard coded feeds are available in the `get_posts` method of `feeds_facade.py`.
It includes a sample of feeds across providers and shows that they can all be represented in the same model
and merged together into a timeline-like experience.

### YouTube

This is a preview release, hard coded. OAuth authentication works and should be configured on one's own YouTube account.
'YouTube Data v3' access is required, to read channel info and followers/subscribers. It's one day old, so one really needs
to go through the code to understand it.

The developer portal provides a client secret file that should be placed in `.data/yt_client_secret.json`.

### Glitch Deployment

The app detects if it's running in Glitch and will configure itself accordingly.
Ensure that the Twitter API is configured to allow access to your project name.

## Running

For local deployment, use Python:

* `python -m hogumathi_app` or `python3 -m hogumathi_app`

## Contributing

Since I intend to retain full copyright and re-license portions of the project I am only able to accept contributions with a copyright assignment, similar to how the Apache project works.

I'm open to substantial contributions with that in mind, and more importantly I'd love to pay contributors for their work; as such, you may donate to me and I can share the wealth while maintaining my business constraints. I know this may rub some free software folks the wrong way, such as a younger version of myself. These days one needs to be serious about the business of their software, and so releasing my work to the public using a business unfriendly free software license is the balance I can offer.

### Patreon and Venmo

Thanks to my friends who remain, and Patreons and Venmo donors for financial support.

* https://patreon.com/harlanji - Monthly support
* https://venmo.com/harlanji - Tips for coffee

## License & Copyright

The project is licensed to the public under the AGPL, requiring disclosure of the source code however it is used.

Please contact me for a commercial or copyleft license for portions of the project.

Copyright (C) 2021-22 Harlan J. Iverson; all rights reserved.