Add in age restriction feature to mitigate snooping

README.md

@@ -110,12 +110,24 @@ request payloads that were pushed before they subscribed, using an `X-Fetch-Sinc
 header denoting from when (in seconds since the UNIX epoch) they wish to receive events.
 
 If there are any events in the backlog (private or public) that match this, they will be
-delivered to the client. It is worth noting here, for pseudo security reasons, that if the 
-backlog maximum is set sufficiently low, this feature could be used to deduce whether or not 
-private events have happened, as a client can request everything in the backlog and potentially 
-gauge whether the size of the backlog differs from time to time. Clients without authorization 
-cannot see private payloads this way, but it is theoretically possible to deduce _that they happened_.
-So...keep that in mind.
+delivered to the client, assuming they are younger than the backlog maximum age requirement. 
+
+*It is worth noting here*, for pseudo security reasons, that if the backlog maximum is set 
+sufficiently low (or the age requirement is omitted), this feature could be used to deduce 
+whether or not private events have happened, as a client can request everything in the backlog 
+and potentially gauge whether the size of the backlog differs from time to time. Clients without 
+authorization cannot see private payloads this way, but it is theoretically possible to deduce 
+_that they happened_. A sane approach is to always set the maximum age configuration to a 
+value that is sufficiently low compared to the backlog max size. If you expect 1,000 payloads 
+per day, you could set your max age to 48h and the backlog size to 5,000 to easily mitigate 
+any potential deduction.
+
+The backlog maximum age configuration expect a number with a time unit after it, for instance:
+- `45s`: 45 seconds
+- `30m`: 30 minutes
+- `12h`: 12 hours
+- `7d`: 7 days
+
 
 ## Access-Control-List and private events
 PyPubSub supports private events that only authenticated clients can receive.

pypubsub.py

@@ -33,6 +33,7 @@ PUBSUB_VERSION = '0.4.6'
 PUBSUB_CONTENT_TYPE = 'application/vnd.pypubsub-stream'
 PUBSUB_DEFAULT_MAX_PAYLOAD_SIZE = 102400
 PUBSUB_DEFAULT_BACKLOG_SIZE = 0
+PUBSUB_DEFAULT_BACKLOG_AGE = 0
 PUBSUB_BAD_REQUEST = "I could not understand your request, sorry! Please see https://pubsub.apache.org/api.html \
 for usage documentation.\n"
 PUBSUB_PAYLOAD_RECEIVED = "Payload received, thank you very much!\n"
@@ -53,6 +54,20 @@ class Server:
         self.last_ping = time.time()
         self.server = None
 
+        # Backlog age calcs
+        bma = self.config['clients'].get('payload_backlog_max_age', PUBSUB_DEFAULT_BACKLOG_AGE)
+        if isinstance(bma, str):
+            bma = bma.lower()
+            if bma.endswith('s'):
+                bma = int(bma.replace('s', ''))
+            elif bma.endswith('m'):
+                bma = int(bma.replace('m', '') * 60)
+            elif bma.endswith('h'):
+                bma = int(bma.replace('h', '') * 3600)
+            elif bma.endswith('d'):
+                bma = int(bma.replace('d', '') * 86400)
+        self.backlog_max_age = bma
+
         if 'ldap' in self.config.get('clients', {}):
             self.lconfig = self.config['clients']['ldap']
             plugins.ldap.vet_settings(self.lconfig)
@@ -148,6 +163,9 @@ class Server:
                         backlog_ts = int(backlog)
                     except ValueError:  # Default to 0 if we can't parse the epoch
                         backlog_ts = 0
+                    # If max age is specified, force the TS to minimum that age
+                    if self.backlog_max_age and self.backlog_max_age > 0:
+                        backlog_ts = max(backlog_ts, time.time() - self.backlog_max_age)
                     # For each item, publish to client if new enough.
                     for item in self.backlog:
                         if item.timestamp >= backlog_ts:

pypubsub.yaml

@@ -9,8 +9,9 @@ clients:
   payloaders:
     - 127.0.0.1/24
     - 10.0.0.1/24
-  max_payload_size:     102400   # Max size of each JSON payload
-  payload_backlog_size: 0        # Max number of payloads to keep in backlog cache (set to 0 to disable)
+  max_payload_size:          102400   # Max size of each JSON payload
+  payload_backlog_size:           0   # Max number of payloads to keep in backlog cache (set to 0 to disable)
+  payload_backlog_max_age:      48h   # Maximum age of a backlog item before culling it (set to 0 to never prune on age)
   # Oldschoolers denotes clients expecting binary events, such as svnwcsub
   oldschoolers:
     - svnwcsub