release v0.1

LICENSE-AGPL3.txt

@@ -629,7 +629,7 @@ to attach them to the start of each source file to most effectively
 state the exclusion of warranty; and each file should have at least
 the "copyright" line and a pointer to where the full notice is found.
 
-    Twitter App: A user interface and tools for the Twitter API and Archive.
+    Hogumathi: A user interface and tools for the Twitter API and Archive.
     Copyright (C) 2021-22 Harlan J. Iverson
 
     This program is free software: you can redistribute it and/or modify

README.md

@@ -1,44 +1,55 @@
-# Twitter App
+# Hogumathi
 
-The purpose of this app is two fold:
+The purpose of this app is three fold:
 
 * Import the Twitter Archive into a usable format
 * Provide a User Interface to the Archive and Twitter API
+* Generalize the user experience across services and integrate the data
 
-Since the Twitter API only provides 7 days of historical data, 
+Since the Twitter API only provides 7-30 days of historical data, 
 we download a copy of the archive and use it to augment live data
 from the API and enhance the user experience with the historical data.
 
+Since there's a great migration from Twitter to Mastodon and I've previously built a 
+Mastodon hosting company, I've added a Mastodon backend that mirrors the Twitter API 
+access functions.
+
+The app is designed to be run locally on a locked down system such as Windows S Mode 
+and in the Cloud in a managed environment like Glitch.
+
 ## Status
 
-This is the merger of two projects which are on their way toward meeting in the middle.
+This is the merger of three projects which are on their way toward meeting in the middle.
 
 Initially the Archive tech was developed for personal reporting, and then my quest into Python brought me
-to web development and I needed a known domain to build for so I could focus on learning the web tech.
+to web development and I needed a known domain to build for so I could focus on learning the web tech. Prior
+experiments lead me to build a Mastodon client and soft launch a hosting company, so it's an avenue for
+me to cash in.
 
-The intent of this release is to give a deployable Twitter client away to open source, and then bring the 
+The intent of this release is to give a deployable Twitter and Mastodon client away to open source, and then bring the 
 archive functionality into the UI. At the time of release the Archive functionality will be obscure but present
-to who looks for it.
+to who looks for it. The same goes for personal reporting functionality which is even less integrated.
 
 ## User Interface
 
 The user interface is built to be as simple to customize as possible for the power user.
 
 It connects to the regular Twitter site using the official method that is well documented 
-on the developer site.
+on the developer site. Mastodon is almost the same It is built to be general enough to support 
+alternative backends as well, to port the experience on one network to another.
 
 It minimizes the need to use Javascript for customization to the extent possible, 
 making the experience good for programmers who have not mastered modern web browser
-nuances.
+tricks.
 
 ## Archive
 
-A Twitter's accounts Archive can be requested from settings; it takes about a day to deliver and is 500MB-2GB,
+A Twitter accounts's Archive can be requested from settings; it takes about a day to deliver and is 500MB-2GB,
 depending largely on how much media was posted. An archive of 50k Tweets is about 50MB of raw tex or 20MB compressed,
 and can be requested once per month. The Tweets file is not in a usable format, requiring processing even to
 read into a JSON parser. 
 
-One in memory it's simply a large list of Tweets and requires manual coding to do anything useful. The first thing we do
+Once in memory it's simply a large list of Tweets and requires manual coding to do anything useful. The first thing we do
 is put it into an SQLite3 database for fast and easy querying using  SQL, a proven tool known by non-programmer business analysts worldwide.
 
 Once the archive is in a Database, we need an interface to work with it. Since the data is derived from the live site's data, we fit it into a user interface that works like the regular Twitter app.
@@ -53,11 +64,64 @@ Using HTMx allows us to get away without much Javascript on the client side. It'
 to earlier days of web development pre-Angular and Reach and into the era of jQuery Ajax forms,
 but it codifies the operations via custom HTML attributes beginning with `hx-~`.
 
+## Installation
+
+* Install Python 3.7+
+* `pip install -r requirements.txt`
+* Copy `sample-env.txt` to `.env` and configure
+
+### Twitter v2 API
+
+One needs to go to the `developer.twitter.com` portal and create an app and obtain the values listed
+in the `.env` file. 
+
+* Client type "Confidential client"
+* Redirect URL, local: `http://localhost:{PORT}/twitter/logged-in.html`
+* Redirect URL, Glitch: `https://{PROJECT NAME}.gitch.me/twitter/logged-in.html`
+
+### Tweet Archive
+
+The Twitter archive comes as a zip file that contains a file called `tweets.js` which is not a valid JSON file.
+One can open it with their code editor and change the first line; replace `window.YTD.tweet.part0 = [` with just `[`
+and save the file as `tweets.json` at the `ARCHIVE_TWEETS_PATH` in `.env`. The file then needs to be loaded into a SQLite3 DB
+by making a `POST` request to the URL `/twitter-archive/tweets`.
+
+* `curl -XPOST http://localhost/twitter-archive/tweets`
+
+### Tweet Collections
+
+There is no UI to edit Tweet collections, but they can be created with the following JSON structure:
+
+```
+{"authorized_users": ["{Numeric Twitter accounts ID}"],
+ "items": [
+    {
+        "id": "{Numeric Tweet ID}",
+        "note": "My note about the Tweet",
+	}
+	]}
+```
+
+If no "authorized_users" is present then the collection is accessible to all users. By default the `base.html` template 
+file contains a link to a collection named `swipe-all2` which should exist in `.data/collection/swipe-all2.json`.
+
+### Glitch Deployment
+
+The app detects if it's running in Glitch and will configure itself accordingly.
+Ensure that the Twitter API is configured to allow access to your project name.
+
 ## Contributing
 
 Since I intend to retain full copyright and re-license portions of the project I am only able to accept contributions with a copyright assignment, similar to how the Apache project works.
 
-I'm open to substantial contributions with that in mind, and more importantly I'd love to pay contributors for their work; as such, you may donate to me and I can share the wealth while maintaining my business constraints. I know this may rub some free software folks the wrong way, such as a younger version of myself. These days one needs to be serious about the business of their software, and so releasing my work to the public using a business unfriendly open source license is the balance I can offer.
+I'm open to substantial contributions with that in mind, and more importantly I'd love to pay contributors for their work; as such, you may donate to me and I can share the wealth while maintaining my business constraints. I know this may rub some free software folks the wrong way, such as a younger version of myself. These days one needs to be serious about the business of their software, and so releasing my work to the public using a business unfriendly free software license is the balance I can offer.
+
+### Patreon and Venmo
+
+Thanks to my friends who remain, and Patreons and Venmo donors for financial support.
+
+* https://patreon.com/harlanji - Monthly support
+* https://venmo.com/harlanji - Tips for coffee
 
 ## License & Copyright
 

mastodon_facade.py

@@ -0,0 +1,551 @@
+from configparser import ConfigParser
+import base64
+from flask import json, Response, render_template, request, send_from_directory, Blueprint, url_for, session, redirect
+from flask_cors import CORS
+import sqlite3
+import os
+import json
+import json_stream
+from zipfile import ZipFile
+import itertools
+from urllib.parse import urlencode as urlencode_qs
+
+import datetime
+import dateutil
+import dateutil.parser
+import dateutil.tz
+
+import requests
+
+
+DATA_DIR='.data'
+
+twitter_app = Blueprint('mastodon_facade', 'mastodon_facade',
+    static_folder='static',
+    static_url_path='',
+    url_prefix='/')
+
+
+
+
+def mastodon_model (post_data):
+    # retweeted_by, avi_icon_url, display_name, handle, created_at, text
+    
+    
+    user = post_data['account']
+    
+    source_url = post_data['url']
+    
+    avi_icon_url = user['avatar']
+    
+    url = url_for('.get_tweet_html', tweet_id=post_data['id'])
+    
+
+        
+    t = {
+        'id': post_data['id'],
+        # hugely not a fan of allowing the server's HTML out. can we sanitize it ourselves?
+        'html': post_data['content'], 
+        'url': url,
+        'source_url': source_url,
+        'created_at': post_data['created_at'],
+        'avi_icon_url': avi_icon_url,
+        'display_name': user['display_name'],
+        'handle': user['acct'],
+        
+        'author_url': url_for('.get_profile_html', user_id = user['id'], me='mastodon:mastodon.cloud:109271381872332822'),
+        'source_author_url':  user['url'],
+        
+        'public_metrics': {
+            'reply_count': post_data['replies_count'],
+            'retweet_count': post_data['reblogs_count'],
+            'like_count': post_data['favourites_count']
+        },
+        
+        'activity': post_data
+    }
+    
+
+    return t
+    
+    
+
+def register_app (instance):
+    client_name = os.environ.get('MASTODON_CLIENT_NAME')
+    redirect_uri = 'http://localhost:5004/mastodon/logged-in.html'
+    
+    url = f'https://{instance}/api/v1/apps'
+    
+    params = {
+        'client_name': client_name,
+        'redirect_uris': redirect_uri,
+        'scopes': 'read write'
+    }
+    
+    resp = requests.post(url, params=params)
+    
+    with open(f'{DATA_DIR}/mastodon-client_{instance}.json', 'wt') as f:
+        f.write(resp.text)
+        
+    return resp
+
+
+@twitter_app.get('/api/app/<instance>/register')
+def post_api_app_instance_register (instance):
+    
+    if not instance:
+        return 'pass isntance= in request params', 400
+    
+    resp = register_app(instance)
+    
+    return resp.text, resp.status_code
+
+@twitter_app.get('/api/app/<instance>')
+def get_api_app_instance (instance):
+    
+    with open(f'{DATA_DIR}/mastodon-client_{instance}.json', 'rt') as f:
+        return Response(f.read(), mimetype='application/json')
+
+@twitter_app.get('/login.html')
+def get_login_html ():
+    instance = request.args.get('instance')
+    force_login = request.args.get('force_login')
+    
+    if not instance:
+        return 'provide instance= in query string.', 400
+    
+    if not os.path.exists(f'{DATA_DIR}/mastodon-client_{instance}.json'):
+        resp = register_app(instance)
+        
+        print(resp)
+    
+    with open(f'{DATA_DIR}/mastodon-client_{instance}.json', 'rt') as f:
+        app_info = json.loads(f.read())
+    
+    params = {
+        'client_id': app_info['client_id'],
+        'redirect_uri': app_info['redirect_uri'],
+        'scope': 'read write',
+        
+        'response_type': 'code'
+    }
+    
+    if force_login:
+        params['force_login'] = force_login
+    
+    url = f'https://{instance}/oauth/authorize?' + urlencode_qs(params)
+    
+    return redirect(url)
+
+@twitter_app.get('/logged-in.html')
+def get_logged_in_html ():
+    code = request.args.get('code')
+    instance = request.args.get('instance', 'mastodon.cloud')
+    
+    if not instance:
+        return 'provide instance= in query string.', 400
+    
+    with open(f'{DATA_DIR}/mastodon-client_{instance}.json', 'rt') as f:
+        app_info = json.loads(f.read())
+    
+    params = {
+        'client_id': app_info['client_id'],
+        'client_secret': app_info['client_secret'],
+        
+        'redirect_uri': app_info['redirect_uri'],
+        'scope': 'read write',
+        
+        'grant_type': 'authorization_code',
+        'code': code,
+        # force_login: True
+    }
+    
+    url = f'https://{instance}/oauth/token'
+    
+    resp = requests.post(url, data=params)
+    
+    auth_info = json.loads(resp.text)
+    
+    
+    #auth_info = {"access_token":"6C6-hD2_OK1vDFc_WcPQnZC5KL0jOUePgQgMQELeV0k","token_type":"Bearer","scope":"read write","created_at":1670088545}
+    
+    access_token = auth_info['access_token']
+    
+    
+    url = f'https://{instance}/api/v1/accounts/verify_credentials'
+    
+    
+    headers = {
+        'Authorization': f'Bearer {access_token}'
+    }
+    
+    resp = requests.get(url, headers=headers)
+    
+    mastodon_user = json.loads(resp.text)
+    
+    me = f'mastodon:{instance}:{mastodon_user["id"]}'
+    
+    session_info = {
+        **auth_info,
+        
+        'instance': instance,
+        
+        'id': mastodon_user['id'],
+        'acct': mastodon_user['acct'],
+        'username': mastodon_user['username'],
+        'display_name': mastodon_user['display_name'],
+        'source_url': mastodon_user['url'],
+        'created_at': mastodon_user['created_at'],
+    }
+    
+    session[me] = session_info
+    
+    return redirect(url_for('.get_latest_html', me=me))
+    
+    #return resp.text, resp.status_code
+
+
+@twitter_app.post('/data/statuses')
+def post_tweets_create ():
+    me = request.args.get('me')
+    mastodon_user = session[me]
+    
+    instance = mastodon_user['instance']
+    access_token = mastodon_user["access_token"]
+    
+    text = request.form.get('text')
+    in_reply_to_id = request.form.get('reply_to_tweet_id')
+
+    headers = {
+       'Authorization': f'Bearer {access_token}'
+    }
+    params = {
+        'text': text
+    }
+    
+    if in_reply_to_id:
+        params['in_reply_to_id'] = in_reply_to_id
+    
+    url = f'https://{instance}/api/v1/statuses/{tweet_id}/bookmark'
+    
+    resp = requests.post(url, data=params, headers=headers)
+    
+    new_status = json.loads(resp.text)
+    
+    new_tweet_id=new_status['id']
+    
+    if 'HX-Request' in request.headers:
+        return render_template('partial/compose-form.html', new_tweet_id=new_tweet_id, me=me)
+    else:
+        return resp.text, resp.status_code
+
+@twitter_app.post('/data/status/<tweet_id>/retweet')
+def post_tweet_retweet (tweet_id):
+    me = request.args.get('me')
+    mastodon_user = session[me]
+    
+    instance = mastodon_user['instance']
+    access_token = mastodon_user["access_token"]
+
+    headers = {
+       'Authorization': f'Bearer {access_token}'
+    }
+
+    url = f'https://{instance}/api/v1/statuses/{tweet_id}/reblog'
+    
+    resp = requests.post(url, headers=headers)
+    
+    # new_status = json.loads(resp.text)
+    
+    # new_tweet_id=new_status['id']
+    ## old status is in reblog: {id:} property.
+    
+    # if 'HX-Request' in request.headers:
+    #     return 'retweeted, new ID: ' + new_tweet_id
+    # else:
+    return resp.text, resp.status_code
+
+@twitter_app.get('/status/<tweet_id>.html')
+def get_tweet_html (tweet_id):
+    return 'tweet: ' + tweet_id
+
+
+@twitter_app.get('/profile/<user_id>.html')
+def get_profile_html (user_id):
+    me = request.args.get('me')
+    mastodon_user = session[me]
+    
+    instance = mastodon_user['instance']
+    access_token = mastodon_user["access_token"]
+    
+    max_id = request.args.get('max_id')
+
+    headers = {
+       'Authorization': f'Bearer {access_token}'
+    }
+    params = {}
+    
+    try:
+        # if the UID isn't numeric then we'll fallback to string.
+        url = f'https://{instance}/api/v1/accounts/{int(user_id)}'
+    except:
+        url = f'https://{instance}/api/v1/accounts/lookup'
+        params = {
+            'acct': user_id
+        }
+    
+    resp = requests.get(url, params=params, headers=headers)
+
+    account_info = json.loads(resp.text)
+    
+    
+    user_id = account_info["id"]
+    
+    url = f'https://{instance}/api/v1/accounts/{user_id}/statuses'
+    
+    params = {}
+    
+    if max_id:
+        params['max_id'] = max_id
+    
+    resp = requests.get(url, params=params, headers=headers)
+
+    tweets = json.loads(resp.text)
+    #tweets = tweet_source.get_timeline("public", timeline_params)
+    
+    max_id = tweets[-1]["id"] if len(tweets) else ''
+    tweets = list(map(mastodon_model, tweets))
+    
+    print("max_id = " + max_id)
+    
+    query = {}
+    
+    if len(tweets):
+        query = {
+            #'next_data_hx_select': '#tweets',
+            'next_data_url': url_for('.get_profile_html', user_id=user_id, me=me, max_id=max_id),
+            'next_page_url': url_for('.get_profile_html', user_id=user_id, me=me, max_id=max_id)
+        }
+    
+    user = {
+            'id': user_id
+        }
+    
+    #return Response(response_json, mimetype='application/json')
+    
+    if 'HX-Request' in request.headers:
+        return render_template('partial/tweets-timeline.html', user = user, tweets = tweets, query = query, 
+            me=me, mastodon_user=mastodon_user)
+    else:
+        return render_template('user-profile.html', user = user, tweets = tweets, query = query, 
+            me=me, mastodon_user=mastodon_user)
+
+
+@twitter_app.route('/latest.html', methods=['GET'])
+def get_timeline_home_html (variant = "reverse_chronological"):
+    # retweeted_by, avi_icon_url, display_name, handle, created_at, text
+    me = request.args.get('me')
+    mastodon_user = session[me]
+    
+    token = mastodon_user.get('access_token')
+    max_id = request.args.get('max_id')
+    
+    # comes from token or session
+    user_id = "ispoogedaily"
+    
+    if not token:
+        print("No token provided or found in environ.")
+        return Response('{"err": "No token."}', mimetype="application/json", status=400) 
+    
+    
+    tweet_source = MastodonAPSource("https://mastodon.cloud", token)
+    
+    timeline_params = {
+        'local': True
+    }
+    
+    if max_id:
+        timeline_params['max_id'] = max_id
+    
+    tweets = tweet_source.get_timeline("public", timeline_params)
+    
+    max_id = tweets[-1]["id"] if len(tweets) else ''
+    tweets = list(map(mastodon_model, tweets))
+    
+    print("max_id = " + max_id)
+    
+    query = {}
+    
+    if len(tweets):
+        query = {
+            'next_data_url': url_for('.get_timeline_home_html', max_id=max_id, me=me),
+            'next_page_url': url_for('.get_timeline_home_html', max_id=max_id, me=me)
+        }
+    
+    user = {
+            'id': user_id
+        }
+    
+    #return Response(response_json, mimetype='application/json')
+    
+    return render_template('tweet-collection.html', user = user, tweets = tweets, query = query,
+        me=me, mastodon_user=mastodon_user)
+
+
+
+@twitter_app.get('/bookmarks.html')
+def get_bookmarks_html ():
+    me = request.args.get('me')
+    mastodon_user = session[me]
+    
+    instance = mastodon_user['instance']
+    access_token = mastodon_user["access_token"]
+    
+    max_id = request.args.get('max_id')
+
+    headers = {
+       'Authorization': f'Bearer {access_token}'
+    }
+    params = {}
+    
+    url = f'https://{instance}/api/v1/bookmarks'
+    
+    params = {}
+    
+    if max_id:
+        params['max_id'] = max_id
+    
+    resp = requests.get(url, params=params, headers=headers)
+
+    tweets = json.loads(resp.text)
+    #tweets = tweet_source.get_timeline("public", timeline_params)
+    
+    max_id = tweets[-1]["id"] if len(tweets) else ''
+    tweets = list(map(mastodon_model, tweets))
+    
+    print("max_id = " + max_id)
+    
+    query = {}
+    
+    if len(tweets):
+        query = {
+            'next_data_url': url_for('.get_bookmarks_html', me=me, max_id=max_id),
+            'next_page_url': url_for('.get_bookmarks_html', me=me, max_id=max_id)
+        }
+    
+    user = {}
+    
+    #return Response(response_json, mimetype='application/json')
+    
+    return render_template('tweet-collection.html', user = user, tweets = tweets, query = query, 
+        me=me, mastodon_user=mastodon_user)
+
+
+@twitter_app.post('/data/bookmarks/<tweet_id>')
+def post_tweet_bookmark (tweet_id):
+    me = request.args.get('me')
+    mastodon_user = session[me]
+    
+    instance = mastodon_user['instance']
+    access_token = mastodon_user["access_token"]
+    
+    max_id = request.args.get('max_id')
+
+    headers = {
+       'Authorization': f'Bearer {access_token}'
+    }
+    params = {}
+    
+    url = f'https://{instance}/api/v1/statuses/{tweet_id}/bookmark'
+    
+    resp = requests.post(url, headers=headers)
+    
+    return resp.text, resp.status_code
+
+
+@twitter_app.delete('/data/bookmarks/<tweet_id>')
+def delete_tweet_bookmark (tweet_id):
+    me = request.args.get('me')
+    mastodon_user = session[me]
+    
+    instance = mastodon_user['instance']
+    access_token = mastodon_user["access_token"]
+    
+    max_id = request.args.get('max_id')
+
+    headers = {
+       'Authorization': f'Bearer {access_token}'
+    }
+    params = {}
+    
+    url = f'https://{instance}/api/v1/statuses/{tweet_id}/unbookmark'
+    
+    resp = requests.post(url, headers=headers)
+    
+    return resp.text, resp.status_code
+
+
+class MastodonAPSource:
+    def __init__ (self, endpoint, token):
+        self.endpoint = endpoint
+        self.token = token
+        super().__init__()
+
+    def get_timeline (self, path = "home", params = {}):
+
+        url = self.endpoint + "/api/v1/timelines/" + path
+        params = {
+            **params
+        }
+        
+        
+        headers = {"Authorization": "Bearer {}".format(self.token)}
+        
+        response = requests.get(url, params=params, headers=headers)
+        
+        print(response)
+        
+        response_json = json.loads(response.text)
+        
+        return response_json
+    
+    # nice parallel is validation
+    # https://github.com/moko256/twitlatte/blob/master/component_client_mastodon/src/main/java/com/github/moko256/latte/client/mastodon/MastodonApiClientImpl.kt
+    
+    def get_mentions_timeline (self):
+        # /api/v1/notifications?exclude_types=follow,favourite,reblog,poll,follow_request
+        return
+        
+    def get_favorited_timeline (self):
+        # /api/v1/notifications?exclude_types=follow,reblog,mention,poll,follow_request
+        return
+        
+    def get_conversations_timeline (self):
+        # /api/v1/conversations
+        return
+        
+    def get_bookmarks_timeline (self):
+        # /account/bookmarks
+        return
+        
+    def get_favorites_timeline (self):
+        # /account/favourites
+        return
+    
+    
+    def get_user_statuses (self, account_id):
+        # /api/v2/search?type=statuses&account_id=
+        # /api/v1/accounts/:id/statuses
+        return
+    
+    def get_statuses (self, ids):
+        # /api/v1/statuses/:id
+        return
+        
+    def get_profile (self, username):
+        # /api/v1/accounts/search
+        return self.search('@' + username, result_type='accounts')
+        
+    def search (q, result_type = None, following = False):
+        # /api/v2/search
+        return

oauth2_login.py

@@ -0,0 +1,281 @@
+import base64
+from flask import json, Response, render_template, request, send_from_directory, Blueprint, session, redirect, g, current_app, jsonify
+
+from flask_cors import CORS
+import sqlite3
+import os
+import json
+
+from datetime import datetime, timedelta, timezone
+import dateutil
+import dateutil.parser
+import dateutil.tz
+
+import requests
+
+
+import hashlib
+import re
+from requests.auth import AuthBase, HTTPBasicAuth
+from requests_oauthlib import OAuth2Session
+
+
+from flask import url_for as og_url_for
+
+
+app_access_token = os.environ.get("BEARER_TOKEN")
+
+app_consumer_key = os.environ.get("TWITTER_CONSUMER_KEY")
+app_secret_key = os.environ.get("TWITTER_CONSUMER_SECRET")
+
+
+
+TWITTER_SCOPES = ["bookmark.read", "bookmark.write", "tweet.read", "tweet.write", "dm.read", "users.read", "like.read", "offline.access", "follows.read"]
+
+oauth2_login = Blueprint('oauth2_login', 'oauth2_login',
+    static_folder='static',
+    static_url_path='',
+    url_prefix='/oauth2')
+
+
+def url_for_with_me (route, *args, **kwargs):
+    #print('url_for_with_me')
+    if route.endswith('.static'):
+        return og_url_for(route, *args, **kwargs)
+    
+    return og_url_for(route, *args, **{'me': g.me, **kwargs})
+
+@oauth2_login.before_request
+def add_me ():
+    g.me = request.args.get('me')
+    
+    #if me.startswith('twitter') and me in session:
+    g.twitter_user = session.get(g.me)
+    
+    if not g.twitter_user:
+        return
+    
+    now = datetime.now(timezone.utc).timestamp()
+    
+    if g.twitter_user['expires_at'] < now - 10:
+        print('token is expired... refreshing')
+        # FIXME we should have a lock here in case 2 requests quickly come in.
+        #       the later will fail as of now. should be rare since static resources aren't authenticated 
+        #       and we don't use APIs w/ JS.
+        refresh_token()
+
+@oauth2_login.context_processor
+def inject_me():
+    
+    return {'me': g.me, 'twitter_user': g.twitter_user, 'url_for': url_for_with_me}
+    
+
+
+@oauth2_login.route('/logout.html')
+def get_logout_html ():
+    del session[g.me]
+    return redirect('/')
+    
+    
+# def add_me(endpoint, values):
+    ##values['me'] = request.args.get('me')
+    # g.me = request.args.get('me')
+
+# twitter_app.url_value_preprocessor(add_me)
+
+
+@oauth2_login.route('/logged-in.html')
+def get_loggedin_html ():
+    client_id = os.environ.get('TWITTER_CLIENT_ID')
+    client_secret = os.environ.get('TWITTER_CLIENT_SECRET')
+    
+    code_verifier = session['twitter_code_verifier']
+    
+    code = request.args.get('code')
+    state = request.args.get('state')
+    
+    endpoint_url = g.app_url
+    redirect_uri = endpoint_url + og_url_for('.get_loggedin_html')
+    
+    authorization_response = redirect_uri + '?code={}&state={}'.format(code, state)
+    
+    # Fetch your access token
+    token_url = "https://api.twitter.com/2/oauth2/token"
+    
+    
+
+    # The following line of code will only work if you are using a type of App that is a public client
+    auth = False
+
+    # If you are using a confidential client you will need to pass in basic encoding of your client ID and client secret.
+
+    # Please remove the comment on the following line if you are using a type of App that is a confidential client
+    auth = HTTPBasicAuth(client_id, client_secret)
+    
+    scopes = TWITTER_SCOPES
+    #redirect_uri = 'https://{}/api/logged-in'.format(os.environ.get('PROJECT_DOMAIN') + '.glitch.me')
+    oauth = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
+    
+
+    token = oauth.fetch_token(
+        token_url=token_url,
+        authorization_response=authorization_response,
+        auth=auth,
+        client_id=client_id,
+        include_client_id=True,
+        code_verifier=code_verifier,
+    )
+
+    # Your access token
+    access = token["access_token"]
+    refresh = token["refresh_token"]
+    #expires_at = token["expires_at"] # expires_in
+    expires_at = (datetime.now(timezone.utc) + timedelta(seconds=token['expires_in'])).timestamp()
+    
+    
+    # Make a request to the users/me endpoint to get your user ID
+    user_me = requests.request(
+        "GET",
+        "https://api.twitter.com/2/users/me",
+        headers={"Authorization": "Bearer {}".format(access)},
+    ).json()
+    user_id = user_me["data"]["id"]
+    user_username = user_me["data"]["username"]
+    user_name = user_me["data"]["name"]
+    
+    del session['twitter_code_verifier']
+    
+    me = 'twitter:{}'.format(user_id)
+    
+    session[ me ] = {
+        'expires_at': expires_at,
+        'access_token': access,
+        'refresh_token': refresh,
+        'id': user_id,
+        'display_name': user_name,
+        'username': user_username
+    }
+    
+    g.me = me
+    g.twitter_user = session[ me ]
+    
+    #run_script('on_loggedin', {'twitter_user': g.twitter_user})
+    
+    return redirect(url_for_with_me('twitter_v2_facade.get_timeline_home_html'))
+  
+@oauth2_login.route('/login.html')
+def get_login_html ():
+    client_id = os.environ.get('TWITTER_CLIENT_ID')
+    client_secret = os.environ.get('TWITTER_CLIENT_SECRET')
+  
+    #redirect_uri = 'https://{}/api/logged-in'.format(os.environ.get('PROJECT_DOMAIN') + '.glitch.me')
+    
+    endpoint_url = g.app_url
+    redirect_uri = endpoint_url + og_url_for('.get_loggedin_html')
+    
+    # Set the scopes
+    scopes = TWITTER_SCOPES
+
+    # Create a code verifier
+    code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
+    code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier)
+
+    # Create a code challenge
+    code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()
+    code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
+    code_challenge = code_challenge.replace("=", "")
+
+    # Start an OAuth 2.0 session
+    oauth = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
+
+    # Create an authorize URL
+    auth_url = "https://twitter.com/i/oauth2/authorize"
+    authorization_url, state = oauth.authorization_url(
+        auth_url, code_challenge=code_challenge, code_challenge_method="S256"
+    )
+    
+    session['twitter_code_verifier'] = code_verifier
+    
+    
+    return redirect(authorization_url)
+
+
+
+
+
+
+def refresh_token ():
+    """
+    Refresh twitter oauth access_token with refresh token.
+    
+    Works in the session, so must be done within a logged in request context.
+    
+    """
+    client_id = os.environ.get('TWITTER_CLIENT_ID')
+    client_secret = os.environ.get('TWITTER_CLIENT_SECRET')
+
+    token = g.twitter_user['refresh_token']
+    
+    basic_auth = base64.b64encode('{}:{}'.format(client_id, client_secret).encode('utf-8')).decode('utf-8')
+    
+    headers = {
+        'Authorization': 'Basic ' + basic_auth,
+    }
+    
+    data = {
+        'refresh_token': token,
+        'grant_type': 'refresh_token'
+    }
+    
+    
+    response = requests.post('https://api.twitter.com/2/oauth2/token', data=data, headers=headers)
+    
+    result = json.loads(response.text)
+    
+    if 'access_token' in result:
+        expires_at = (datetime.now(timezone.utc) + timedelta(seconds=result['expires_in'])).timestamp()
+        g.twitter_user['expires_at'] = expires_at
+        g.twitter_user['refresh_token'] = result['refresh_token']
+        g.twitter_user['access_token'] = result['access_token']
+        
+        session[ g.me ] = g.twitter_user
+    
+    return response.text
+
+
+@oauth2_login.route('/refresh-token', methods=['GET'])
+def get_twitter_refresh_token (response_format='json'):
+    return refresh_token()
+
+
+
+@oauth2_login.route('/app/refresh-token', methods=['GET'])
+def get_twitter_app_refresh_token ():
+    
+    basic_auth = base64.b64encode('{}:{}'.format(app_consumer_key, app_secret_key).encode('utf-8')).decode('utf-8')
+    
+    headers = {
+        'Authorization': 'Basic ' + basic_auth,
+    }
+    
+    data = {
+        'grant_type': 'client_credentials'
+    }
+    
+    response = requests.post('https://api.twitter.com/oauth2/token', data=data, headers=headers)
+    
+    result = json.loads(response.text)
+    
+    if 'access_token' in result:
+         
+        app_access_token = result['access_token']
+        
+        
+    
+    return response.text
+
+@oauth2_login.route('/accounts.html', methods=['GET'])
+def get_loggedin_accounts_html ():
+    twitter_accounts = dict(filter(lambda e: e[0].startswith('twitter:'), session.items()))
+    
+    return jsonify(twitter_accounts)

requirements-dev.txt

@@ -0,0 +1,2 @@
+pytest                   ~= 7.2.0
+responses                ~= 0.22.0

requirements.txt

@@ -4,4 +4,5 @@ Flask-Session             ~= 0.4.0
 json-stream               ~= 1.3.0
 python-dateutil           ~= 2.8.2
 requests                  ~= 2.28.0
-requests-oauthlib         ~= 1.3.1
+requests-oauthlib         ~= 1.3.1
+dataclass-type-validator  ~= 0.1.2

sample-env.txt

@@ -1,10 +1,22 @@
 FLASK_SECRET=
 
+NOTES_APP_URL=
+
+
+MASTODON_CLIENT_NAME=Hogumathi (Local Dev)
+
+# Twitter App Token cache
+BEARER_TOKEN=
+
 TWITTER_CLIENT_ID=
 TWITTER_CLIENT_SECRET=
 
+TWITTER_CONSUMER_KEY=
+TWITTER_CONSUMER_SECRET=
+
 # Path to tweet.js converted into json (barely works, bolting these two things together)
 ARCHIVE_TWEETS_PATH=
 
 # For development on localhost set value to 1
-OAUTHLIB_INSECURE_TRANSPORT=
+#OAUTHLIB_INSECURE_TRANSPORT=1
+#PORT=5004

static/theme/base.css

@@ -0,0 +1,63 @@
+
+.qt-box, .card-box {
+	padding: 4px;
+	border: 1px solid black;
+}
+
+.tweet-actions-box {
+	width: 100%;
+	background-color: lightgrey;
+	padding: 2px;
+	text-align: right;
+}
+
+#tweets .tweet.marked {
+	background-color: powderblue;
+}
+
+.theme-dark #tweets .tweet.marked {
+	background-color: lightslategray;
+}
+
+.theme-dark {
+	color: #dfdfdf;
+	background-color: #030303;
+}
+    
+	
+
+.theme-dark a {
+	color: green;
+}
+
+.theme a:visited {
+	color: orange;
+}
+
+.theme-dark .qt-box, .theme-dark .card-box {
+	border-color: #dfdfdf;
+}
+
+.theme-dark .tweet-actions-box {
+	background-color: midnightblue;
+}
+
+.theme-dark .tweet-actions-box a {
+    color: #dfdfdf;
+}
+
+.theme-dark textarea, .theme-dark input {
+	background-color: #121212;
+	border-color: #dfdfdf;
+	color: #dfdfdf;
+}
+
+.theme-dark button {
+    background-color: #121212;
+    color: #dfdfdf;
+    border-color: #dfdfdf;
+}
+
+.theme-dark #tweets .tweet {
+	background: revert;
+}

static/theme/user-14520320.css

@@ -0,0 +1,11 @@
+body  { 
+	background:	floralwhite;
+}
+
+#tweets .tweet {
+	background:		white;
+	
+	border:			1px solid silver;
+	padding:		8px;
+	margin:			8px;
+}

static/tweets-ui.js

@@ -0,0 +1,164 @@
+
+function noteCardFormat (tweet, annotation) {
+	if (!tweet) { return ""; }
+	if (!annotation) { annotation = ''; } else { annotation = '\n' + annotation + '\n'; }
+	
+	var now = new Date();
+	
+	var s = `---
+
+${formatTime(now)}
+${annotation}
+${extendedFormat(tweet)}
+
+`;
+
+	return s;
+}
+
+
+function simpleFormat (tweet) {
+	if (!tweet) { return ""; }
+
+	var s;
+	with (tweet) {
+		if( !tweet['source_url'] ) { source_url = null; }
+		
+		s = 
+`${source_url || ''}
+
+${display_name}
+@${handle}
+${created_at}
+
+${text || '(no text)'}
+`;
+	}
+	return s.trim();
+}
+
+function htmlToText(html) {
+    var temp = document.createElement('div');
+    temp.innerHTML = html;
+    return temp.textContent; // Or return temp.innerText if you need to return only visible text. It's slower.
+}
+
+function extendedFormat (tweet) {
+	if (!tweet) { return ""; }
+	
+	var s;
+	
+	if (!tweet.text && tweet.html) {
+		// FIXME: HTML to text
+		tweet.text = htmlToText(tweet.html.replaceAll('</p>', '</p>\n\n'));
+	}
+	
+	with (tweet) {
+		var total_rt_count = (public_metrics.quote_count || 0) + (public_metrics.retweet_count || 0);
+		
+		s = 
+`${source_url || ''}
+
+${display_name}
+@${handle}
+${created_at}
+${!isNaN(public_metrics.reply_count) ? public_metrics.reply_count : 'unknown'} replies, ${total_rt_count} rts, ${public_metrics.like_count} likes
+${tweet['retweeted_by'] ? 'Retweeted by ' : tweet['retweeted_by']}
+
+${text || '(no text)'}
+
+${tweet['card'] ? "Card:\n" + cardFormat(card) : ''}
+
+${tweet['quoted_tweet'] ? "Quoted Tweet:\n" + simpleFormat(quoted_tweet) : ''}
+`;
+	}
+	return s.trim();
+}
+
+
+function cardFormat (card) {
+	if (!card) { return ""; }
+	
+	var s;
+	with (card) {
+		s = 
+`${source_url || ''}
+
+${display_url}
+${title}
+${content}
+`;
+	}
+	return s;
+}
+
+function jsonFormat (tweet) {
+	return JSON.stringify(tweet, null, 2);
+}
+
+
+function formatDate (date) {
+	return date.getFullYear() 
+		+ "-" + (date.getMonth() + 1).toString().padStart(2, '0')
+		+ "-" + date.getDate().toString().padStart(2, '0');
+}
+
+function formatTime (date) {
+	return ("" + date.getHours()).padStart(2, "0")
+		+ ":" + ("" + date.getMinutes()).padStart(2, "0")
+		+ ":" + ("" + date.getSeconds()).padStart(2, "0")
+}
+
+function copyToClipboard (text) {
+	navigator.clipboard.writeText(text);
+}
+
+function tweetById (tweetId) {
+	var tweets = window.dataset.items.filter(t => t.id == tweetId);
+	
+	if (!tweets.length) {
+		return null;
+	}
+	
+	return tweets[0];
+}
+
+
+function copyTweetToClipboard (tweetId) {
+	var text = noteCardFormat(tweetById(tweetId));
+	return copyToClipboard(text)
+}
+
+async function swipeTweetToNotesApp (tweetId) {
+	const { value: annotation } = await Swal.fire({
+	  input: 'textarea',
+	  inputLabel: 'Add annotation to swipe?',
+	  inputPlaceholder: 'Type your annotation here...',
+	  inputAttributes: {
+		'aria-label': 'Type your annotation here'
+	  },
+	  showCancelButton: true,
+	  cancelButtonText: 'No annotation'
+	})
+	
+	var noteId = formatDate(new Date()) + '-swipe.md'
+	var url = notesAppUrl + '/intent/prepend-text/' + noteId
+	var text = noteCardFormat(tweetById(tweetId), annotation);
+	
+	
+	
+	
+	var params = new URLSearchParams({
+		text: text,
+		should_create: 1
+	});
+	
+	return fetch(url + '?' + params).then(function (r) {
+		Toast.fire({
+		  icon: 'success',
+		  title: 'Tweet swiped.'
+		});
+	
+		return r;
+	});
+}

templates/base.html

@@ -5,62 +5,27 @@
 	<meta name="viewport" content="width=device-width, initial-scale=1">
 	<link rel="stylesheet" href="{{ url_for('.static', filename='tachyons.min.css') }}">
 	<script src="{{ url_for('.static', filename='htmx.js') }}"></script>
+	<script src="{{ url_for('.static', filename='sweetalert2.js') }}"></script>
 	
 	{% block head %}
 	<title>{{ title | default('No Title') }}</title>
 	{% endblock %}
-<style>
-
-
-.qt-box, .card-box {
-	padding: 4px;
-	border: 1px solid black;
-}
-
-.tweet-actions-box {
-	width: 100%;
-	background-color: lightgrey;
-	padding: 2px;
-	text-align: right;
-}
-
-
-.theme-dark {
-	color: #dfdfdf;
-	background-color: #030303;
-}
-    
 	
-
-.theme-dark a {
-	color: green;
-}
-
-.theme a:visited {
-	color: orange;
-}
-
-.theme-dark .qt-box, .theme-dark .card-box {
-	border-color: #dfdfdf;
-}
-
-.theme-dark .tweet-actions-box {
-	background-color: midnightblue;
-}
-
-.theme-dark textarea, .theme-dark input {
-	background-color: #121212;
-	border-color: #dfdfdf;
-	color: #dfdfdf;
-}
-
-.theme-dark button {
-    background-color: #121212;
-    color: #dfdfdf;
-    border-color: #dfdfdf;
-}
-
-</style>
+	<link rel="stylesheet" href="{{ url_for('.static', filename='theme/base.css') }}">
+	<link rel="stylesheet" href="{{ url_for('.static', filename='theme/user-14520320.css') }}">
+<script>
+const Toast = Swal.mixin({
+  toast: true,
+  position: 'top-end',
+  showConfirmButton: false,
+  timer: 3000,
+  timerProgressBar: true,
+  didOpen: (toast) => {
+    toast.addEventListener('mouseenter', Swal.stopTimer)
+    toast.addEventListener('mouseleave', Swal.resumeTimer)
+  }
+})
+</script>
 </head>
 <body>
 
@@ -71,28 +36,51 @@ Flexbox makes logical sense but we'll go with a table-based display
 
 <div class="nav" style="position: fixed; width: 25%">
 <ul>
+	{% if twitter_user %}
 	<li><a href="{{ url_for('.get_timeline_home_html') }}">Latest Tweets</a></li>
 	<li><a href="{{ url_for('.get_conversations_html') }}">DMs</a></li>
-	<li><a hx-get="/twitter/data/mentions/{{ twitter_user.id }}?me={{ me }}" hx-target="#tweets" hx-swap="outerHTML">Mentions</a></li>
+	<li><a hx-push-url="/mentions.html?me={{ me }}" hx-get="/twitter/data/mentions/{{ twitter_user.id }}?me={{ me }}" hx-target="#tweets" hx-swap="outerHTML">Mentions</a></li>
 	<!--
 	<li><a hx-get="/twitter/data/thread/1592514352690900992?me={{ me }}" hx-target="#tweets" hx-swap="outerHTML">Test Thread</a></li>
 	<li><a hx-get="/twitter/data/conversation/1592596557009801216?me={{ me }}" hx-target="#tweets" hx-swap="outerHTML">Test Conversation</a></li>
 	<li><a hx-get="/twitter/data/tweets?ids=1592637236147027970,1592474342289330176&amp;me={{ me }}" hx-target="#tweets" hx-swap="outerHTML">Test Tweets</a></li>
 	-->
 	<li><a href="{{ url_for('.get_bookmarks_html') }}">Bookmarks</a></li>
-	<li><a href="{{ url_for('.get_logout_html') }}">Logout ({{me}})</a></li>
-
+	<li><a href="{{ url_for('.get_collection_html', collection_id='swipe-all2') }}">Test Collection</a></li>
+	
+	
+	
+	<li><a href="{{ url_for('twitter_v2_facade.oauth2_login.get_logout_html') }}">Logout ({{me}})</a></li
+	{% endif %}
+	
+	{% if mastodon_user %}
+	<li><a href="{{ url_for('.get_timeline_home_html', me=me) }}">Public Timeline</a></li>
+	<li><a href="{{ url_for('.get_bookmarks_html', me=me) }}">Bookmarks</a></li>
+	{% endif %}
 	
 	<li><a href="javascript:document.body.classList.toggle('theme-dark')">Toggle Dark Mode</a></li>
 	<li><a href="javascript:document.location.reload()">Refresh</a></li>
 </ul>
 
+{% if twitter_user or mastodon_user %}
+
 {% include "partial/compose-form.html" %}
 
 
+{% endif %}
+
+{% if False %}
+<!--
 {% include "partial/media-upload-form.html" %}
+-->
+{% endif %}
 
 {% include "partial/user-picker.html" %}
+
+{% if add_account_enabled %}
+<a href="{{ url_for('get_login_html') }}">Add account</a>
+{% endif %}
+
 </div>
 
 <div style="width: 75%; left:25%; position: absolute">
@@ -100,6 +88,11 @@ Flexbox makes logical sense but we'll go with a table-based display
 {% block content %}{% endblock %}
 
 </div>
-
+<footer style="position: absolute; bottom: -300px; right: 0; width: 100px;">
+<p>
+Powered by <a href="https://glitch.com/~hogumathi">Hogumathi</a>. Give a gift to the author with <a href="https://venmo.com/harlanji">Venmo</a> or become a <a href="https://patreon.com/harlanji">patron</a> for gifts
+like early access and coupon codes for software on <a href="https://harlanji.gumroad.com">Gumroad</a>.
+</p>
+</footer>
 </body>
 </html>

templates/following.html

@@ -0,0 +1,9 @@
+{% extends "base.html" %}
+
+{% block content %}
+<ul>
+{% for follower in following %}
+	<li><a href="{{ url_for('.get_profile_html', user_id=follower) }}">{{ follower }}</a></li>
+{% endfor %}
+</ul>
+{% endblock %}

templates/login.html

@@ -0,0 +1,40 @@
+<h1>Login</h1>
+
+{% if glitch_enabled %}
+<p>We're running on <a href="https://glitch.com/~hogumathi">Glitch</a>, meaning you can run your own copy of this app or inspect the running source code.</p>
+
+<p>We highly recommend you run your own copy for maximal security and control.</p>
+{% endif %}
+
+{% if add_account_enabled %}
+
+<p>Choose a login provider. You can login multiple times with each provider.</p>
+
+<ul>
+{% if twitter_enabled %}
+	<li><a href="{{ url_for('twitter_v2_facade.oauth2_login.get_login_html') }}">Twitter v2</a></li>
+{% endif %}
+{% if mastodon_enabled %}
+	<li><form method="GET" action="{{ url_for('mastodon_facade.get_login_html') }}">
+		<button type="submit">Mastodon</button> on instance <input name="instance" value="mastodon.social" placeholder="instance hostname"></form>
+		<p><a href="mailto:biz@harlanji.com?subject=mastodon%20server">Get your own managed Mastodon server</a></p>
+		</li>
+{% endif %}
+{% if email_enabled %}
+	<li><form method="GET" action="{{ url_for('email_facade.get_login_html') }}">
+		<button type="submit">Email</button> <input name="address" placeholder="me@example.com"></form>
+		</li>
+{% endif %}
+{% if youtube_enabled %}
+	<li><a href="{{ url_for('youtube_facade.get_api_login') }}">YouTube</a></li>
+{% endif %}
+
+</ul>
+
+{% else %}
+
+<p>Adding accounts is disabled on this instance.</p>
+
+{% endif %}
+
+{% include "partial/user-picker.html" %}

templates/partial/compose-form.html

@@ -1,4 +1,4 @@
-<form class="compose" hx-post="/twitter/tweets/create?me={{ me }}" hx-swap="outerHTML">
+<form class="compose" hx-post="{{ url_for('.post_tweets_create', me=me) }}" hx-swap="outerHTML">
 <h2>Compose</h2>
 <ul>
 	<li><textarea name="text" placeholder="Only the finest..." style="width: 100%" onchange="this.onkeyup()" onkeyup="this.form.querySelector('.compose-length').innerHTML = this.value.length" rows="6" cols="30"></textarea>
@@ -7,9 +7,13 @@
 	<li><input type="text" name="quote_tweet_id" placeholder="Quote Tweet ID" style="width: 100%">
 	<li><button type="submit" style="margin-top: 0.33em">Post</button>
 </ul>
-</form>
+
 {% if new_tweet_id %}
-<div class="flash">
-Tweet posted. <a href="/tweet/{{ new_tweet_id }}.html">View</a>.
-</div>
-{% endif %}
+<script>
+Toast.fire({
+  icon: 'success',
+  title: 'Tweet was sent; <a style="text-align: right" href="{{ url_for('.get_tweet_html', tweet_id=new_tweet_id) }}">View</a>.'
+});
+</script>
+{% endif %}
+</form>

templates/partial/media-upload-form.html

@@ -1,4 +1,5 @@
 <form action={{ url_for('.post_media_upload', me=me) }} enctype="multipart/form-data" method="POST">
+<h2>Upload Media</h2>
 <ul>
 	<li><input type="file" name="file1">
 	<li><input type="file" name="file2">

templates/partial/timeline-tweet.html

@@ -10,9 +10,16 @@
 	{% endif %}
 	
 	<a href="{{ tweet.author_url }}" class="silver">@{{ tweet.handle }}</a>
-	<a href="{{ tweet.source_url }}">{{ tweet.created_at }}</a>
+	<a href="{{ tweet.url }}">{{ tweet.created_at }}</a> [<a href="{{ tweet.source_url }}" target="tweet_{{ tweet.id }}">source</a>]
+	</p>
+	<p class="w-100">
+	
+	{% if tweet.html %}
+		{{ tweet.html | safe }}
+	{% else %}
+		{{ tweet.text | replace('<', '&lt;') | replace('\n', '<br>') | safe }}
+	{% endif %}
 	</p>
-	<p class="w-100">{{ tweet.text | replace('<', '&lt;') | replace('\n', '<br>') | safe }}</p>
 
 	{% if tweet.quoted_tweet %}
 	<div class="dt qt-box">
@@ -24,6 +31,35 @@
 	</div>
 	</div>
 	{% endif %}
+	
+	{% if tweet.replied_tweet %}
+	<p style="color: silver">
+	Replying to:
+	</p>
+	<div class="reply_to w-100" style="border: 1px solid silver; padding: 6px">
+		<div class="dt reply-box">
+		<div class="dt-row">
+			{% with tweet = tweet.replied_tweet %}
+				{% include "partial/timeline-tweet.html" %}
+			{% endwith %}
+
+		</div>
+		</div>
+	</div>
+	{% elif tweet.replied_tweet_id %}
+	<p style="color: silver">
+	Replying to:
+	</p>
+	<p class="reply_to w-100" style="border: 1px solid silver; padding: 6px">
+		<a href="{{ url_for('.get_tweet_html', tweet_id=tweet.replied_tweet_id, view='replies', marked_reply=tweet.id ) }}">View in Thread</a>.
+	</p>
+	{% endif %}
+	
+	{% if tweet.note %}
+	<p class="note w-100" style="border: 1px solid black; background-color: yellow; padding: 6px">
+		{{ tweet.note.replace('\n', '<br>') | safe }}
+	</p>
+	{% endif %}
 
 	{% if tweet.photos %}
 	<p class="w-100">
@@ -41,7 +77,7 @@
 		<p>VIDEOS</p>
 		<ul>
 		{% for video in tweet.videos %}
-			<li><img class="w-100" src="{{ video.preview_image_url }}" crossorigin="" referrerpolicy="no-referrer" onclick="this.src='{{ video.image_url }}'"></li>
+			<li><img class="w-100" src="{{ video.preview_image_url }}" referrerpolicy="no-referrer" onclick="this.src='{{ video.image_url }}'"></li>
 		{% endfor %}
 		</ul>
 
@@ -59,14 +95,23 @@
 		</div>
 		{% endif %}
 		
+		
+				
+
+		{% if False and tweet.replied_tweet %}
+			<a href="{{ tweet.replied_tweet.url }}">View Parent</a>
+			<a href="{{ url_for('.get_tweet_html', tweet_id=tweet.replied_tweet.conversation_id) }}">View Conversation</a>
+		{% endif %}
+		
 		{% if tweet.public_metrics %}
 		
 
-		<p class="w-100">replies: {{ tweet.public_metrics.reply_count }},
-						 quotes: {{ tweet.public_metrics.quote_count }},
-						 rt: {{ tweet.public_metrics.retweet_count }},
-						 likes: {{ tweet.public_metrics.like_count }}
-						 </p>
+		<p class="w-100">
+		{% for k, v in tweet.public_metrics.items() %}
+			{{ k.replace('_count', 's').replace('ys', 'ies').replace('_', ' ') }}: {{ v }}, 
+		{% endfor %}
+		
+		</p>
 		{% endif %}
 		
 		{% if tweet.non_public_metrics %}
@@ -74,10 +119,10 @@
 
 		
 		<p class="w-100">
-			impressions: {{ tweet.non_public_metrics.impression_count }},
-			clicks: {{ tweet.non_public_metrics.user_profile_clicks }},
-			link clicks: {{ tweet.non_public_metrics.url_link_clicks | default('n/a') }},
-			profile clicks: {{ tweet.non_public_metrics.user_profile_clicks | default('n/a') }}
+			{% for k, v in tweet.non_public_metrics.items() %}
+				{{ k.replace('_count', 's').replace('ys', 'ies').replace('_', ' ') }}: {{ v }}, 
+			{% endfor %}
+			
 		</p>
 		{% endif %}
 	

templates/partial/tweets-carousel.html

@@ -0,0 +1,110 @@
+<script src="{{ url_for('.static', filename='tweets-ui.js') }}"></script>
+<script>
+
+{% if notes_app_url %}
+var notesAppUrl = {{ notes_app_url | tojson }}
+{% endif %}
+
+
+	if (!window['dataset']) {
+		window.dataset = {
+			items: [],
+			update: function (items) {
+				dataset.items = dataset.items.concat(items);
+			}
+		}
+	}
+</script>
+<ul id="tweets" class="tweets w-75 center z-0">
+
+{% for tweet in tweets %}
+
+<li class="tweet w-100 dt">
+<script>
+	dataset.update([
+		{{ tweet | tojson }}
+	]);
+
+</script>
+
+
+	{% if tweet.retweeted_by %}
+	<div class="dt-row moon-gray">
+		<p class="dtc w-10 tr pa1">RT</p>
+		<p class="dtc w-90"><a class="moon-gray" href="{{ tweet.retweeted_by_url }}">{{ tweet.retweeted_by }} Retweeted</a></p>
+	</div>
+	{% endif %}
+	<div class="dt-row">
+		{% include "partial/timeline-tweet.html" %}
+		
+				
+
+		
+		
+	</div>
+	<div class="dt-row">
+		<div class="dtc"></div>
+		<div class="dtc ">
+		
+		
+		<p class="tweet-actions-box">
+		
+		<a hx-post="{{ url_for('.post_tweet_retweet', tweet_id=tweet.id) }}">retweet</a>
+		|
+		<a hx-get="/twitter/data/thread/{{ tweet.conversation_id }}?me={{ me }}" hx-target="#tweets" hx-swap="outerHTML">view author thread</a>
+		|
+		<a hx-get="/twitter/data/conversation/{{ tweet.conversation_id }}?me={{ me }}" hx-target="#tweets" hx-swap="outerHTML">view full convo</a>
+		|
+		<a class="tweet-action bookmark" href="#">bookmark</a>
+		|
+		<a class="tweet-action copy-formatted" href="javascript:copyTweetToClipboard('{{ tweet.id }}')">copy formatted</a>
+		{% if notes_app_url %}
+		|
+		<a class="tweet-action swipe-to-note" href="javascript:swipeTweetToNotesApp('{{ tweet.id }}')">swipe to note</a>
+		{% endif %}
+		</p>
+		</div>
+	</div>
+	
+
+	</li>
+
+{% endfor %}
+
+
+
+{% if query.next_data_url %}
+
+	<li style="height: 50px; vertical-align: middle"
+		hx-get="{{ query.next_data_url }}"
+		hx-trigger="revealed"
+		hx-swap="outerHTML"
+		hx-select="ul#tweets > li"
+		>
+		<center style="height: 100%">
+
+		<span class="js-only">
+		Loading more tweets...
+		</span>
+		
+		{% if query.next_page_url %}
+		<a href="{{ query.next_page_url }}">
+		Go to Next Page
+		</a>
+		{% endif %}
+		
+		<script>
+			var profileDataEl = document.querySelector('#profile-data');
+			
+			if (window['dataset'] && profileDataEl) {
+				profileDataEl.innerHTML = dataset.items.filter(i => 'public_metrics' in i).map(i => i.public_metrics.like_count).join(', ');
+			}
+
+
+		</script>
+		</center>
+	</li>
+
+{% endif %}
+
+</ul>

templates/partial/tweets-timeline.html

@@ -1,4 +1,10 @@
+<script src="{{ url_for('.static', filename='tweets-ui.js') }}"></script>
 <script>
+
+{% if notes_app_url %}
+var notesAppUrl = {{ notes_app_url | tojson }}
+{% endif %}
+
 	if (!window['dataset']) {
 		window.dataset = {
 			items: [],
@@ -8,11 +14,12 @@
 		}
 	}
 </script>
+
 <ul id="tweets" class="tweets w-75 center z-0">
 
 {% for tweet in tweets %}
 
-<li class="tweet w-100 dt">
+<li class="tweet w-100 dt {% if tweet.is_marked %}marked{% endif %}">
 <script>
 	dataset.update([
 		{{ tweet | tojson }}
@@ -30,9 +37,6 @@
 	<div class="dt-row">
 		{% include "partial/timeline-tweet.html" %}
 		
-				
-
-		
 		
 	</div>
 	<div class="dt-row">
@@ -42,14 +46,30 @@
 		
 		<p class="tweet-actions-box">
 		
-		<a hx-get="/twitter/data/thread/{{ tweet.conversation_id }}?me={{ me }}" hx-target="#tweets" hx-swap="outerHTML">view author thread</a>
+		<a href="{{ url_for('.get_tweet_html', tweet_id=tweet.id, view='replies') }}">replies</a>
+		|
+		{% if show_thread_controls %}
+		<a href="{{ url_for('.get_tweet_html', tweet_id=tweet.conversation_id, view='thread') }}">author thread</a>
+		|
+		<a href="{{ url_for('.get_tweet_html', tweet_id=tweet.conversation_id, view='conversation') }}">full convo</a>
+		|
+		{% endif %}
+		{% if twitter_user %}
+		<a hx-post="{{ url_for('.post_tweet_retweet', tweet_id=tweet.id) }}">retweet</a>
 		|
-		<a hx-get="/twitter/data/conversation/{{ tweet.conversation_id }}?me={{ me }}" hx-target="#tweets" hx-swap="outerHTML">view full convo</a>
+		{% endif %}
+		{% if twitter_user or mastodon_user %}
+		<a hx-post="{{ url_for('.post_tweet_bookmark', tweet_id=tweet.id, me=me) }}">bookmark</a>
+		[
+		<a hx-delete="{{url_for('.delete_tweet_bookmark', tweet_id=tweet.id, me=me)}}">-</a>
+		]
 		|
-		<a class="tweet-action bookmark" href="#">bookmark</a>
-		<a class="tweet-action copy-formatted" href="#">copy formatted</a>
+		{% endif %}
+		<a class="tweet-action copy-formatted" href="javascript:copyTweetToClipboard('{{ tweet.id }}')">copy formatted</a>
+		{% if notes_app_url %}
 		|
-		<a class="tweet-action swipe-to-note" href="#">swipe to note</a>
+		<a class="tweet-action swipe-to-note" href="javascript:swipeTweetToNotesApp('{{ tweet.id }}')">swipe to note</a>
+		{% endif %}
 		</p>
 		</div>
 	</div>
@@ -74,12 +94,26 @@
 		<span class="js-only">
 		Loading more tweets...
 		</span>
-		
-		{% if query.next_page_url %}
+
+		<script>
+			var profileDataEl = document.querySelector('#profile-data');
+			
+			if (window['dataset'] && profileDataEl) {
+				profileDataEl.innerHTML = dataset.items.filter(i => 'public_metrics' in i).map(i => i.public_metrics.like_count).join(', ');
+			}
+
+
+		</script>
+		</center>
+	</li>
+	
+{% elif query.next_page_url %}
+	<li style="height: 50px; vertical-align: middle"
+		>
+		<center style="height: 100%">
 		<a href="{{ query.next_page_url }}">
 		Go to Next Page
 		</a>
-		{% endif %}
 		
 		<script>
 			var profileDataEl = document.querySelector('#profile-data');
@@ -91,18 +125,9 @@
 
 		</script>
 		</center>
+	
 	</li>
-
-{% endif %}
-
-</ul>
-		
-<script>
-	var profileDataEl = document.querySelector('#profile-data');
 	
-	if (window['dataset'] && profileDataEl) {
-		profileDataEl.innerHTML = dataset.items.filter(i => 'public_metrics' in i).map(i => i.public_metrics.like_count).join(', ');
-	}
-
+{% endif %}
 
-</script>
+</ul>

templates/partial/user-picker.html

@@ -1,10 +1,16 @@
+<h2>Accounts</h2>
+
 <ul>
 	{% for k, v in session.items() %}
 		{% if k.startswith('twitter:') %}
-			<li><a href="{{ url_for('twitter_v2_facade.get_profile_html', me=k, user_id=v.id) }}">{{ k }}</a>
+			<li><a href="{{ url_for('twitter_v2_facade.get_timeline_home_html', me=k) }}">{{ k }}</a>
+		{% endif %}
+		{% if mastodon_enabled and k.startswith('mastodon:') %}
+			<li><a href="{{ url_for('mastodon_facade.get_timeline_home_html', me=k) }}">{{ k }}</a>
 		{% endif %}
 	{% endfor %}
 	{% if archive_enabled %}
-		<li><a href="{{ url_for('twitter_archive_facade.get_profile_html', user_id=0) }}">Archive</a>
+		<li><a href="{{ url_for('twitter_archive_facade.get_profile_html', user_id=0) }}">Twitter archive</a>
 	{% endif %}
+	
 </ul>

templates/tweet-collection.html

@@ -7,6 +7,18 @@
 
 {% block content %}
 
-	{% include "partial/tweets-timeline.html" %}
+{% if show_parent_tweet_controls %}
+<div>
+	<a href="{{ url_for('.get_tweet_html', tweet_id=tweets[0].conversation_id, view='thread') }}">author thread</a>
+	|
+	<a href="{{ url_for('.get_tweet_html', tweet_id=tweets[0].conversation_id, view='conversation') }}">full convo</a>
+</div>
+{% endif %}
 
+	{% with show_thread_controls=True %}
+	
+	{% include "partial/tweets-timeline.html" %}
+	
+	{% endwith %}
+	
 {% endblock %}

templates/user-profile.html

@@ -21,10 +21,44 @@
 
 	<div class="w-100" style="margin-top: 80px">
 		<div class="w-100" style="height: 40px">
-			<a href="{{ url_for('.get_profile_html', user_id=user.id, exclude_replies = 1) }}">Without replies</a>
+			<p class="w-100" style="text-align: right">
+			<a href="#">Timeline</a> 
 			|
-			<a href="{{ url_for('.get_profile_html', user_id=user.id, exclude_replies = 0) }}">With replies</a>
+			<a href="#">Likes</a> 
+			|
+			<a href="#">Following</a>
+			|
+			<a href="#">Followers</a>
+			|
+			<a href="#">Activity</a>
+			</p>
+			<form action="{{ url_for('.get_profile_html', user_id=user.id) }}" method="GET">
+			
+			<input type="hidden" name="me" value="{{ me }}">
+			<input type="hidden" name="user_id" value="{{ user.id }}">
+			
+			<input type="checkbox" name="exclude_replies" value="1" {% if request.args.exclude_replies %}checked{% endif %}> No replies 
+			|
+			<input type="checkbox" name="only_media" value="1" {%if request.args.only_media %}checked{% endif %}> Only media
+			<br>
+			<button type="submit">Filter</button>
+			</form>
 		</div>
-		{% include "partial/tweets-timeline.html" %}
+		
+		{% block tab_content %}
+		
+		{% if tab == "media" %}
+			media
+		
+		{% else %}
+			{% with show_thread_controls=True %}
+			
+			{% include "partial/tweets-timeline.html" %}
+			
+			{% endwith %}
+			
+		{% endif %}
+		
+		{% endblock %}
 	</div>
-{% endblock %}
+{% endblock %}

test/unit/twitter_v2_facade_test/test_tweet_source.py

@@ -0,0 +1,44 @@
+import responses
+
+from tweet_source import ApiV2TweetSource
+
+@responses.activate
+def test_create_tweet ():
+    
+    fake_res = {
+        'id': 1234,
+        'text': 'Test Text'
+    }
+    
+    responses.add(responses.POST, 'https://api.twitter.com/2/tweets',
+                  json=fake_res, status=200)
+    
+    tweet_source = ApiV2TweetSource('FAKE TOKEN')
+    
+    res = tweet_source.create_tweet('Test Text')
+    
+    assert len(responses.calls) == 1
+    
+    assert(res == fake_res)
+
+@responses.activate
+def test_get_timeline ():
+    
+    fake_res = {
+        'meta': 1234,
+        'text': 'Test Text'
+    }
+    
+    responses.add(responses.GET, 'https://api.twitter.com/2/timeline/test',
+                  json=fake_res, status=200)
+    
+    tweet_source = ApiV2TweetSource('FAKE TOKEN')
+    
+    res = tweet_source.get_timeline('timeline/test', pagination_token='1234')
+    
+    assert len(responses.calls) == 1
+    
+    req = responses.calls[0].request
+    assert req.params.get('pagination_token') == '1234'
+    
+    assert(res == fake_res)

test/unit/twitter_v2_facade_test/test_view_model.py

@@ -0,0 +1,53 @@
+import twitter_v2_facade as t2f
+
+"""
+
+The big item to test here is tweet_model() which converts a Tweet into a UI view and action links.
+
+"""
+
+def mock_url_for (route, *args, **kwargs):
+    return 'URL:' + route
+
+
+def test_tweet_model ():
+    tweet_api_resp = {
+        'data': [{
+            'id': 12345,
+            'text': 'A tweet!',
+            'created_at': 'a time',
+            'author_id': 456,
+            
+            'conversation_id': 12345
+            
+        }
+        
+        ],
+        'includes': {
+            'tweets': [],
+            'users': [{
+                'id': 456,
+                'username': 'user456',
+                'name': 'User 456',
+                'verified': False,
+                'profile_image_url': 'pfp_url'
+            }],
+            'media': []
+        },
+        'meta': {
+            'result_count': 1,
+            # 'next_token': 'AAA'
+        }
+    }
+    
+    includes = tweet_api_resp.get('includes')
+    tweet_data = tweet_api_resp.get('data')[0]
+    
+    tweet = t2f.tweet_model (includes, tweet_data, 'TEST', my_url_for = mock_url_for)
+    
+    assert(tweet['id'] == 12345)
+    assert(tweet['text'] == 'A tweet!')
+    
+    assert(tweet['author_id'] == 456)
+    assert(tweet['display_name'] == 'User 456')
+    

tweet_source.py

@@ -6,7 +6,7 @@ class ArchiveTweetSource:
     """
     id, created_at, retweeted, favorited, retweet_count, favorite_count, full_text, in_reply_to_status_id_str, in_reply_to_user_id, in_reply_to_screen_nam
     """
-    def __init__ (self, archive_path, db_path = "data/tweet.db", archive_user_id = None):
+    def __init__ (self, archive_path, db_path = ".data/tweet.db", archive_user_id = None):
         self.archive_path = archive_path
         self.user_id = archive_user_id
         self.db_path = db_path
@@ -129,15 +129,68 @@ class TwitterApiV2SocialGraph:
         # GET /2/users?ids=
         return
         
-    def get_following (user_id, 
-                        max_results = 10, pagination_token = None):
+    def get_following (self, user_id, 
+                        max_results = 1000, pagination_token = None):
         # GET /2/users/:id/following
+        
+        url = "https://api.twitter.com/2/users/{}/following".format(user_id)
+        
+        
+        user_fields = ["created_at", "name", "username", "location", "profile_image_url", "verified"]
+        
+        params = {
+            'user.fields' : ','.join(user_fields),
+            
+            
+            
+            'max_results': max_results
+        }
+        
+        if pagination_token:
+            params['pagination_token'] = pagination_token
+        
+        headers = {
+            'Authorization': 'Bearer {}'.format(self.token),
+            'Content-Type': 'application/json'
+        }
+        
+        response = requests.get(url, params=params, headers=headers)
+        result = json.loads(response.text)
+        
+        return result
+        
         return
         
-    def get_followers (user_id,
-                        max_results = 10, pagination_token = None):
+    def get_followers (self, user_id,
+                        max_results = 1000, pagination_token = None):
         # GET /2/users/:id/followers
-        return
+        
+        url = "https://api.twitter.com/2/users/{}/followers".format(user_id)
+        
+        
+        user_fields = ["created_at", "name", "username", "location", "profile_image_url", "verified"]
+        
+        params = {
+            'user.fields' : ','.join(user_fields),
+            
+            
+            
+            'max_results': max_results
+        }
+        
+        if pagination_token:
+            params['pagination_token'] = pagination_token
+        
+        headers = {
+            'Authorization': 'Bearer {}'.format(self.token),
+            'Content-Type': 'application/json'
+        }
+        
+        response = requests.get(url, params=params, headers=headers)
+        result = json.loads(response.text)
+        
+        return result
+        
         
     def follow_user (user_id, target_user_id):
         # POST /2/users/:id/following
@@ -181,8 +234,8 @@ class ApiV2TweetSource:
         result = json.loads(response.text)
         
         return result
-    
-    def retweet_tweet( self, user_id, tweet_id ):
+        
+    def retweet (self, tweet_id, user_id):
         
         url = "https://api.twitter.com/2/users/{}/retweets".format(user_id)
         
@@ -201,7 +254,40 @@ class ApiV2TweetSource:
         result = json.loads(response.text)
         
         return result
+    
+    def bookmark (self, tweet_id, user_id):
+        
+        url = "https://api.twitter.com/2/users/{}/bookmarks".format(user_id)
+        
+        bookmark = {
+            'tweet_id': tweet_id
+        }
+        
+        body = json.dumps(bookmark)
+        
+        headers = {
+            'Authorization': 'Bearer {}'.format(self.token),
+            'Content-Type': 'application/json'
+        }
+        
+        response = requests.post(url, data=body, headers=headers)
+        result = json.loads(response.text)
+        
+        return result
+        
+    def delete_bookmark (self, tweet_id, user_id):
+        
+        url = "https://api.twitter.com/2/users/{}/bookmarks/{}".format(user_id, tweet_id)
+   
+        headers = {
+            'Authorization': 'Bearer {}'.format(self.token)
+        }
+        
+        response = requests.delete(url, headers=headers)
+        result = json.loads(response.text)
         
+        return result
+    
     
     def get_home_timeline (self, user_id, variant = 'reverse_chronological', max_results = 10, pagination_token = None, since_id = None):
         """
@@ -216,7 +302,8 @@ class ApiV2TweetSource:
     def get_timeline (self, path,
         max_results = 10, pagination_token = None, since_id = None,
         non_public_metrics = False,
-        exclude_replies=False):
+        exclude_replies=False,
+        exclude_retweets=False):
         """
         Get any timeline, including custom curated timelines built by Tweet Deck / ApiV11.
         """
@@ -253,8 +340,12 @@ class ApiV2TweetSource:
         if exclude_replies:
             exclude.append('replies')
             
+        
+        if exclude_retweets:
+            exclude.append('retweets')
+            
         if len(exclude):
-            params['exclude'] = exclude
+            params['exclude'] = ','.join(exclude)
         
         
         if pagination_token:
@@ -283,7 +374,8 @@ class ApiV2TweetSource:
     def get_user_timeline (self, user_id,
                           max_results = 10, pagination_token = None, since_id = None,
                           non_public_metrics=False,
-                          exclude_replies=False):
+                          exclude_replies=False,
+                          exclude_retweets=False):
         """
         Get a user's Tweets as viewed by another.
         """
@@ -292,7 +384,7 @@ class ApiV2TweetSource:
         return self.get_timeline(path, 
             max_results=max_results, pagination_token=pagination_token, since_id=since_id,
             non_public_metrics = non_public_metrics,
-            exclude_replies=exclude_replies)
+            exclude_replies=exclude_replies, exclude_retweets=exclude_retweets)
     
     
     def get_tweet (self, id_, non_public_metrics = False):
@@ -329,6 +421,8 @@ class ApiV2TweetSource:
         }
         headers = {"Authorization": "Bearer {}".format(token)}
         
+        #print(params)
+        
         response = requests.get(url, params=params, headers=headers)
         response_json = json.loads(response.text)
         
@@ -424,6 +518,7 @@ class ApiV2TweetSource:
 
     def get_thread (self, tweet_id,
                        author_id = None,
+                       only_replies = False, 
                        pagination_token = None,
                        since_id = None,
                        max_results = 10,
@@ -431,10 +526,17 @@ class ApiV2TweetSource:
                        ):
         
         # FIXME author_id can be determined from a Tweet object
-        query = "conversation_id:{}".format(tweet_id)
+        query = ""
         if author_id:
             query += " from:{}".format(author_id)
             
+        if only_replies:
+            query += " in_reply_to_tweet_id:{}".format(tweet_id)
+        else:
+            query += " conversation_id:{}".format(tweet_id)
+        
+        print("get_thread query=" + query)
+        
         return self.search_tweets(query, 
             pagination_token = pagination_token, since_id = since_id, max_results = max_results, sort_order = sort_order)
     
@@ -451,6 +553,8 @@ class ApiV2TweetSource:
                    has_links = None,
                    has_images = None,
                    has_videos = None,
+                   is_reply = None,
+                   is_retweet = None,
                    pagination_token = None,
                    since_id = None,
                    max_results = 10,
@@ -480,7 +584,17 @@ class ApiV2TweetSource:
                 query += " -"
             query += "has:videos "
             
-        
+        if is_reply != None:
+            if not is_reply:
+                query += " -"
+            query += "is:reply "
+            
+        if is_retweet != None:
+            if not is_retweet:
+                query += " -"
+            query += "is:retweet "
+            
+            
         if author_id:
             query += "from:{} ".format(author_id)
             
@@ -496,13 +610,17 @@ class ApiV2TweetSource:
         # GET /2/tweets/:id/quote_tweets
         return 
         
-    def get_likes (self, tweet_id):
-        # GET /2/tweets/:id/liking_users
-        return 
+    def get_likes (self, user_id,
+                     max_results = 10, pagination_token = None, since_id = None):
+        # GET /2/users/:id/liked_tweets
+        path = "users/{}/liked_tweets".format(user_id)
         
-    def get_liked_by (self, user_id):
-        #  GET /2/users/:id/liked_tweets
-        return 
+        return self.get_timeline(path, 
+            max_results=max_results, pagination_token=pagination_token, since_id=since_id)
+        
+    def get_liked_by (self, tweet_id):
+        #  GET /2/tweets/:id/liking_users
+        return
     
     def get_list_tweets (self, list_id):
         # GET /2/lists/:id/tweets

twitter_app.py

@@ -1,12 +1,34 @@
 import os
+from importlib.util import find_spec
 from configparser import ConfigParser
 
-from flask import Flask, g, redirect, url_for
+from flask import Flask, g, redirect, url_for, render_template, jsonify
 from flask_cors import CORS
 
-from twitter_v2_facade import twitter_app as twitter_v2
-from twitter_archive_facade import twitter_app as twitter_archive
 
+if find_spec('twitter_v2_facade'):
+    from twitter_v2_facade import twitter_app as twitter_v2
+    import oauth2_login
+    twitter_enabled = True
+else:
+    print('twitter module not found.')
+    twitter_enabled = False
+
+if find_spec('twitter_archive_facade'):
+    from twitter_archive_facade import twitter_app as twitter_archive
+    archive_enabled = True
+else:
+    print('twitter archive module not found.')
+    archive_enabled = False
+
+if find_spec('mastodon_facade'):
+    from mastodon_facade import twitter_app as mastodon
+    mastodon_enabled = True
+else:
+    print('mastodon module not found.')
+    mastodon_enabled = False
+
+add_account_enabled = True
 
 def import_env ():
     cp = ConfigParser()
@@ -15,6 +37,8 @@ def import_env ():
             cp.read_string('[default]\n' + stream.read())
             os.environ.update(dict(cp['default']))
 
+
+
 if __name__ == '__main__':
     import_env()
 
@@ -24,24 +48,60 @@ if __name__ == '__main__':
     archive_enabled = os.environ.get('ARCHIVE_TWEETS_PATH') and True
     glitch_enabled = os.environ.get('PROJECT_DOMAIN') and True
     
+    notes_app_url = os.environ.get('NOTES_APP_URL')
+    
+    if not os.path.exists('.data'):
+        os.mkdir('.data')
+    
+    if not os.path.exists('.data/cache'):
+        os.mkdir('.data/cache')
+    
     api = Flask(__name__, static_url_path='')
     
     
+    # HACK - environ from .env isn't set yet when the import happens. We should call an init function somewhere.
+    oauth2_login.app_access_token = os.environ.get("BEARER_TOKEN")
+    oauth2_login.app_consumer_key = os.environ.get("TWITTER_CONSUMER_KEY")
+    oauth2_login.app_secret_key = os.environ.get("TWITTER_CONSUMER_SECRET")
+
 
     @api.before_request
     def add_config ():
+        g.twitter_enabled = twitter_enabled
+        g.archive_enabled = archive_enabled
+        g.mastodon_enabled = mastodon_enabled
+        
+        g.add_account_enabled = add_account_enabled
+        
+        
         g.glitch_enabled = glitch_enabled
         
         if glitch_enabled:
             g.app_url = 'https://{}.glitch.me'.format( os.environ.get('PROJECT_DOMAIN') )
         else:
             g.app_url = 'http://{}:{}'.format('localhost', PORT)
+            
+        if notes_app_url:
+            g.notes_app_url = notes_app_url
         
 
     
     @api.context_processor
     def inject_config ():
-        return {'archive_enabled': archive_enabled}
+        config = {}
+        config['twitter_enabled'] = twitter_enabled
+        config['archive_enabled'] = archive_enabled
+        config['mastodon_enabled'] = mastodon_enabled
+        
+        config['add_account_enabled'] = add_account_enabled
+        
+        config['glitch_enabled'] = glitch_enabled
+
+        if notes_app_url:
+            config['notes_app_url'] = notes_app_url
+        
+        
+        return config
     
     api.secret_key = os.environ.get('FLASK_SECRET')
     
@@ -52,11 +112,30 @@ if __name__ == '__main__':
     if archive_enabled:
         api.register_blueprint(twitter_archive, url_prefix='/twitter-archive')
     
+    if mastodon_enabled:
+        api.register_blueprint(mastodon, url_prefix='/mastodon')
+    
     CORS(api)
     
-    @api.route('/')
-    def index ():
-        return redirect(url_for('.twitter_v2_facade.get_login_html'))
+    @api.get('/login.html')
+    def get_login_html ():
+        return render_template('login.html')
     
+    @api.get('/')
+    def index ():
+        return redirect(url_for('.get_login_html'))
     
+    @api.get('/brand/<brand_id>.html')
+    def get_brand_html (brand_id):
+        brand = {
+            'id': 'ispoogedaily',
+            'display_name': 'iSpooge Daily',
+            'accounts': [
+                'twitter:14520320',
+                'mastodon:mastodon.cloud:109271381872332822'
+            ]
+        }
+        
+        return jsonify(brand)
+        
     api.run(port=PORT, host=HOST)

twitter_archive_facade.py

@@ -20,7 +20,7 @@ import requests
 
 from tweet_source import ArchiveTweetSource
 
-ARCHIVE_TWEETS_PATH=os.environ.get('ARCHIVE_TWEETS_PATH', 'data/tweets.json')
+ARCHIVE_TWEETS_PATH=os.environ.get('ARCHIVE_TWEETS_PATH', '.data/tweets.json')
 
 
 twitter_app = Blueprint('twitter_archive_facade', 'twitter_archive_facade',
@@ -33,14 +33,15 @@ twitter_app = Blueprint('twitter_archive_facade', 'twitter_archive_facade',
 @twitter_app.before_request
 def add_me ():
     #if me.startswith('twitter') and me in session:
-    g.twitter_user = {'id': '0'}
-    
+    #g.twitter_user = {'id': '0'}
+    return 
 
 
 @twitter_app.context_processor
 def inject_me():
     
-    return {'twitter_user': g.twitter_user}
+    #return {'twitter_user': g.twitter_user}
+    return {}
     
     
     
@@ -146,7 +147,7 @@ def post_tweets_compressed ():
   if not db_exists:
     db = sqlite3.connect("tweets.db")
     db.execute("create table tweet (id, full_text_length, date, reply)")
-    populate_tweetsdb_from_compressed_json(db, "data/tweet-items.json")
+    populate_tweetsdb_from_compressed_json(db, ".data/tweet-items.json")
     db.commit()
     db.close()
   
@@ -325,9 +326,18 @@ def get_profile_html (user_id):
     profile_user = {
             'id': user_id
         }
+    theme = {
+        'name': None,
+        'body': {'background': 'floralwhite'},
+        'timeline': {'tweet': {'background': 'white',
+                               'border': '1px solid silver'}}
+    }
     
-    return render_template('user-profile.html', user = profile_user, tweets = tweets, query = query)
+    return render_template('user-profile.html', user = profile_user, tweets = tweets, query = query, theme=theme)
 
+@twitter_app.get('/tweet.html')
+def get_tweet_html (tweet_id):
+    return ''
 
 @twitter_app.route('/latest.html', methods=['GET'])
 def get_timeline_home_html (variant = "reverse_chronological", pagination_token=None):
@@ -361,7 +371,7 @@ def get_tweets_search (response_format='json'):
     
     in_reply_to_user_id = int(request.args.get('in_reply_to_user_id', 0))
     
-    db = sqlite3.connect('data/tweet.db')
+    db = sqlite3.connect('.data/tweet.db')
     
     sql = """
 select
@@ -438,7 +448,7 @@ def post_tweets ():
     tweets_file = open(tweets_path, 'rt', encoding='utf-8')
     tweets_data = json_stream.load(tweets_file)
     
-    db = sqlite3.connect('data/tweet.db')
+    db = sqlite3.connect('.data/tweet.db')
     
     db.execute('create table tweet (id, created_at, retweeted, favorited, retweet_count, favorite_count, full_text, in_reply_to_status_id_str, in_reply_to_user_id, in_reply_to_screen_name)')
     db.commit()