Report Groups (#38)

* New reporting format, improved feature detection and error handling
* Minor updates, migrate wifi report
* Normalize output, update report
* Migrate files to new report type
* Report type conversion
* Migrate more reports, fix issue with backup.info report not attempting parsing binary plist
* Improve notes reporting - move to a single report.
* Fix files report not extracting files properly
* Remove oldnotes report
* Add some docs, work on migrating final reports
* Move thirdparty reports into other folder
* Add geofences
* Add migrated calendars report
* Fixes #32
* Decode call type
* Replace <<RecentsNumberLocationNotFound>> with null
* Update README
* Bump to v4
* Update README
* Migrate address book report
* bugfixes in apps, geofences, pushstore
* Make table formatter more efficient if there is nested data.
* Bugfix voicemail, call stats, bluetooth devices.
* Bugfix formatters output to file
* Minor bugfixes in reports
* Bugfix list report
* Rename `Backup3` to `Backup`.
* Export backup lib
* Bugfix for when gmail contact or avatar does not exist
* Update code style to match eslint rules
* Update spotify report code style
* Plugin modules support
* Split module and cli into separate files
* Documentation / rename functions
* Add log-level, add support for third party module running
* Add init-test for cli
* Update comment / error strings
* Add ability to pass single module into registerModule()
* Fix issue where default reports are not defined until a module is registered.
* Code cleanup: use async functions for reports
* Temporarily drop support for current node version
* Pass ibackuptool module as parameter, bugfix unwrapping backup instances for nested modules
* Update sqlite3
* Rename spotify report
* update name string
* Update Dependencies
* Bugfix for calls report when it has no result
* Fix CSV Output
* Move report list behind a command line flag

.travis.yml

@@ -2,6 +2,7 @@ language: node_js
 node_js:
   - '8'
   - '9'
-  - 'node'
+  - '10'
+  #- 'node'
 before_script:
-  - npm install -g tap
+  - npm install -g tap

Readme.md

@@ -12,10 +12,35 @@ Check out my recently updated post about my work on backups here: [Reverse Engin
 
 Currently works on macOS, not tested on windows but should work on windows by setting the attribute `--dir` to the backups directory location.
 
+## Documentation
+This readme is intended to be an overview of features. Please read the [wiki](https://github.com/richinfante/iphonebackuptools/wiki) for more up-to-date and in-depth examples, and examples of how to make and use reports.
+
 ## iOS Support
-iOS Support depends on the individual reporting types and which files are specifically present inside of the backup. When a report type is not supported, an error message is outputted to the terminal. Some reports, such as the manifest report, support iOS versions as early as iOS 5 to the latest iOS (iOS 11 as of this writing).
+iOS Support depends on the individual reporting types and which files are specifically present inside of the backup. When a report type is not supported, an error message is outputted to the terminal. Some reports, will output an error message if files that are required are not present in the backup.
+
+## Reports List
+the full report list is available [on the wiki](https://github.com/richinfante/iphonebackuptools/wiki/V4-Reports-List)
+
+## Installing (as a module)
+```bash
+npm i ibackuptool --save
+```
+
+You can then import the module to run reports and get javascript objects as results:
+
+```js
+const bt = require('ibackuptool')
+
+// Call the backups.list report.
+bt.run('backups.list')
+  .then(backups => {
+    // Gives you a list of backups.
+    console.log(backups)
+  }
+```
+
 
-## Installing
+## Installing (as a global command line tool)
 
 ```bash
 # Install directly from NPM
@@ -32,7 +57,7 @@ npm install
 run `node tools/index.js` # use this instead of ibackuptool
 ```
 
-### Usage
+### CLI Quickstart
 ```bash
 # List all the backups on the system
 ibackuptool -l 
@@ -40,82 +65,24 @@ ibackuptool -l
 # I'm using "0c1bc52c50016933679b0980ccff3680e5831162" as a placeholder.
 # The list of backups contains the different UDIDs in the first column.
 UDID="0c1bc52c50016933679b0980ccff3680e5831162"
-```
-
-### Reports
-- run using `ibackuptool -b <udid> --report <type>`
-- Current types:
-    - `apps` - List all installed applications and container IDs.
-    - `calls` - List all call records contained in the backup.
-    - `calls-statistics` List call statistics
-    - `conversations` - List all SMS and iMessage conversations
-    - `conversations_full` - Output raw message contents along with conversations. Requires `-f raw-json` or `--dump`.
-    - `cookies` - List cookie domains for all apps
-    - `list` - List of all backups. alias for `-l`. Does not require the `-b` flag.
-    - `manifest` - List all the files contained in the backup (iOS 10+)
-    - `messages` - List all SMS and iMessage messages in a conversation. This requires using the `--id` flag to specify a conversation to inspect.
-    - `notes` - List all iOS notes
-    - `oldnotes` - List all iOS notes (from older unused database)
-    - `photolocations` - List all geolocation information for iOS photos (iOS 10+)
-    - `voicemail-files` - List all or extract voicemail files (iOS 10+)
-    - `voicemail` - List all or extract voicemails on device
-    - `webhistory` - List all web history
-    - `wifi` - List associated wifi networks and their usage information
-
-```bash
-# Using a UDID from the previous step, now you can run:
-# List Installed Apps
-ibackuptool -b $UDID --report apps
-
-# List Recent Web History
-ibackuptool -b $UDID --report webhistory
-
-# List Recent Photos Geolocations (iOS 10+)
-ibackuptool -b $UDID --report photolocations
-
-# List iOS Notes
-ibackuptool -b $UDID --report notes
-
-# List iOS Notes from old database that may exist
-ibackuptool -b $UDID --report oldnotes
-
-# List calls
-ibackuptool -b $UDID --report calls
 
-# List voicemails 
-ibackuptool -b $UDID --report voicemail
-
-# List voicemail files (iOS 10+)
-ibackuptool -b $UDID --report voicemail-files
-
-# Export voicemail files (iOS 10+)
-ibackuptool -b $UDID --report voicemail-files --export ./ExportedVoicemails
-
-# List wifi networks 
-ibackuptool -b $UDID --report wifi
-
-# Extract all files into a new directory called "BACKUP"
-ibackuptool -b $UDID --report manifest --extract BACKUP --filter all
-
-# Extract all Camera Roll data into a new directory called "BACKUP".
-# See the wiki for additonal info about filtering.
-ibackuptool -b $UDID --report manifest --extract BACKUP --filter CameraRollDomain
+# Run ibackuptool --help to get a list of reports that are available
+ibackuptool -b $UDID --report '$TYPE'
 ```
 
-### Messages Access
+#### Multiple-Reporting 
+You can also provide a comma separated list of reports to generate. Additionally, there is a special `all` report type which will run all available reports. This is best paired with the `-o` option for saving to disk and the `-f` option for selecting a format such as CSV, or JSON.
 
 ```bash
-# List of all conversations, indexed by ID.
-# Each row starts with an ID number, which is needed for the next step.
-ibackuptool -b $UDID -r conversations
+# Run all phone reports and wifi report.
+ibackuptool -b $UDID --report 'phone.*,system.wifi'
 
-# Now, Fetch the messages with the following command
-# Replace $CONVERSATION_ID with a row ID from `ibackuptool -b $UDID -r conversations`
-ibackuptool -b $UDID -r messages --id $CONVERSATION_ID
+# Report all possible
+ibackuptool -b $UDID --report all
 ```
 
 ### Reporting formats
-iBackupTool now supports multiple kinds of data export:
+iBackupTool now supports multiple kinds of data export, which can be selected using the `-f` flag.
 - `table` - Selected data columns in an ascii table
 - `json` - Selected data columns for display (same data as `table`)
 - `csv` - CSV file containing selected columns (same data as `table`)
@@ -124,23 +91,12 @@ Additionally, there are more comprehensive export functions that will export ALL
 - `raw-csv` - Full-data CSV export from each of the tables.
 - `raw`, `raw-json` - Full-data JSON export from each of the tables. This output can be quite large.
 
-#### Multiple-Reporting 
-You can also provide a comma separated list of reports to generate. Additionally, there is a special `all` report type which will run all available reports. This is best paired with the `-o` option for saving to disk and the `-f` option for selecting a format such as CSV, or JSON.
-
-```bash
-# Report wifi, calls, voicemail
-ibackuptool -b $UDID --report wifi,calls,voicemail
-
-# Report all possible
-ibackuptool -b $UDID --report all
-```
-
 #### Joined Reports
 Additionally, for the `json` and `raw-json` types, there's a `--join-reports` flag which will merge all of the data into a single JSON file, where the top level object has a key for each report type that is selected.
 
 ```bash
 # Generate both wifi and calls reports, joined as JSON
-ibackuptool -b $UDID -r wifi,calls -f json --join-reports
+ibackuptool -b $UDID -r systme.wifi,phone.calls -f json --join-reports
 ```
 
 ### Output to disk
@@ -148,7 +104,7 @@ the `-o <path>` (`--output <path>`option specifies a folder to export reports to
 
 ```bash
 # Export wifi, calls, voicemail as CSV files to disk in a folder named "exported/"
-ibackuptool -b $UDID --report wifi,calls,voicemail -f csv -o exported
+ibackuptool -b $UDID --report system.wifi,phone.calls,phone.voicemail -f csv -o exported
 ```
 ## Running Tests
 first, install [tap](https://www.npmjs.com/package/tap)

package.json

@@ -1,14 +1,14 @@
 {
   "name": "ibackuptool",
-  "version": "3.0.0",
+  "version": "4.0.0",
   "description": "Read Messages and info from iOS Backups",
   "main": "tools/index.js",
   "scripts": {
     "test": "tap tests",
-    "start": "node tools/index.js"
+    "start": "node tools/cli.js"
   },
   "bin": {
-    "ibackuptool": "tools/index.js"
+    "ibackuptool": "tools/cli.js"
   },
   "preferGlobal": true,
   "repository": "https://github.com/richinfante/iphonebackuptools",
@@ -18,20 +18,19 @@
     "bplist-parser": "^0.1.1",
     "buffer-reader": "^0.1.0",
     "chalk": "^1.1.3",
-    "commander": "^2.12.2",
+    "commander": "^2.15.1",
     "fs-extra": "^4.0.3",
     "json2csv": "^3.11.5",
-    "plist": "^2.0.1",
-    "sqlite3": "^3.1.8",
-    "strip-ansi": "^4.0.0",
-    "zpad": "^0.5.0"
+    "plist": "^2.1.0",
+    "sqlite3": "^4.0.0",
+    "strip-ansi": "^4.0.0"
   },
   "devDependencies": {
-    "eslint": "^4.18.1",
-    "eslint-plugin-import": "^2.9.0",
+    "eslint": "^4.19.1",
+    "eslint-plugin-import": "^2.11.0",
     "eslint-plugin-node": "^6.0.1",
-    "eslint-plugin-promise": "^3.6.0",
-    "eslint-plugin-standard": "^3.0.1",
-    "tap": "^11.1.0"
+    "eslint-plugin-promise": "^3.7.0",
+    "eslint-plugin-standard": "^3.1.0",
+    "tap": "^11.1.4"
   }
 }

tests/test_init.js

@@ -0,0 +1,8 @@
+const tap = require('tap')
+
+tap.test('version', function (childTest) {
+  // Require it to check if we can load the modules
+  require('../tools/index')
+  require('../tools/cli')
+  childTest.end()
+})

tools/backup.js

@@ -0,0 +1,133 @@
+const fs = require('fs')
+const sqlite3 = require('sqlite3')
+const path = require('path')
+const log = require('./util/log')
+const filehash = require('./util/backup_filehash')
+
+/**
+ * Backup3 is the version 4 of the backup library.
+ * It focuses on file lookups, and better error handling.
+ */
+class Backup {
+  /**
+   * Create a new backup instance.
+   * @param {*} base path to backups folder.. Defaults to '~/Library/Application Support/MobileSync/Backup/'
+   * @param {*} id directory name of the backup.
+   */
+  constructor (base, id) {
+    log.verbose(`create backup with base=${base}, id=${id}`)
+    id = id || ''
+    base = base || ''
+
+    // Very wierd, but unwrap from existing backup instance.
+    if (id.constructor === Backup) {
+      id = id.id
+      log.verbose(`unwrapping backup to id=${id}`)
+    }
+
+    this.id = id
+    this.base = base
+
+    // Get the path of the folder.
+    if (base) {
+      this.path = path.join(base, id)
+    } else {
+      this.path = path.join(process.env.HOME, '/Library/Application Support/MobileSync/Backup/', id)
+    }
+  }
+
+  /**
+ * Derive a file's ID from it's filename and domain.
+ * @param {string} file the path to the file in the domain
+ * @param {string=} domain (optional) the file's domain. Default: HomeDomain
+ */
+  getFileID (path, domain) {
+    return Backup.getFileID(path, domain)
+  }
+
+  /**
+ * Derive a file's ID from it's filename and domain.
+ * @param {string} file the path to the file in the domain
+ * @param {string=} domain (optional) the file's domain. Default: HomeDomain
+ */
+  static getFileID (path, domain) {
+    return filehash(path, domain)
+  }
+
+  /**
+   * Get the on-disk filename of a fileID.
+   * You shouldn't really ever need to use the isAbsolute flag at all.
+   * By default, it searches both possibile paths.
+   *
+   * @param {*} fileID the file ID. derive using getFileID()
+   * @param {boolean=} isAbsoulte (optional) default: false. should we check other file locations?.
+   * @throws Throws an error if no file is found
+   */
+  getFileName (fileID, isAbsoulte) {
+    // Default to non-absolute paths.
+    isAbsoulte = isAbsoulte || false
+
+    // Possible file locations for an ID
+    let possibilities
+
+    if (isAbsoulte) {
+      // We must only check in the root folder of the backup.
+      possibilities = [
+        path.join(this.path, fileID)
+      ]
+    } else {
+      // Check in both /abcdefghi and /ab/abcdefghi
+      possibilities = [
+        path.join(this.path, fileID),
+        path.join(this.path, fileID.substr(0, 2), fileID)
+      ]
+    }
+
+    // Return first path that works.
+    for (let path of possibilities) {
+      log.verbose('trying', path, fs.existsSync(path))
+
+      // Check if the path exists
+      if (fs.existsSync(path)) {
+        log.verbose('trying', path, '[found]')
+
+        // If it does, return it.
+        return path
+      }
+    }
+
+    // Throw an error.
+    throw new Error(`Could not find a file needed for this report. It may not be compatibile with this specific backup or iOS Version.`)
+  }
+
+  /**
+   * Open a database referenced by a fileID
+   * It uses getFileName(), so it tries both v2 and v3 paths.
+   * @param {string} fileID ihe file id
+   * @param {boolean=} isAbsoulte is this an absolute path? default: false.
+   * @returns {Promise.<sqlite3.Database>} database instance.
+   */
+  openDatabase (fileID, isAbsoulte) {
+    return new Promise((resolve, reject) => {
+      try {
+        // Lookup the filename
+        let file = this.getFileName(fileID, isAbsoulte)
+
+        // Open as read only
+        let db = new sqlite3.Database(file, sqlite3.OPEN_READONLY, (err) => {
+          if (err) { return reject(err) }
+
+          if (db != null) {
+            resolve(db)
+          } else {
+            reject(new Error('did not get a database instance.'))
+          }
+        })
+      } catch (e) {
+        return reject(e)
+      }
+    })
+  }
+}
+
+module.exports = Backup

tools/cli.js

@@ -0,0 +1,268 @@
+#!/usr/bin/env node
+
+const program = require('commander')
+const chalk = require('chalk')
+const log = require('./util/log')
+const report = require('./reports')
+const matcher = require('./util/matcher')
+const Group = report.Group
+const core = require('./index')
+const packageJSON = require('../package.json')
+
+const { runSingleReport, runSwitchedReport } = require('./util/report_runner')
+
+var formatters = {
+  'json': require('./formatters/json'),
+  'table': require('./formatters/table'),
+  'raw': require('./formatters/raw-json'),
+  'raw-json': require('./formatters/raw-json'),
+  'csv': require('./formatters/csv'),
+  'raw-csv': require('./formatters/raw-csv')
+}
+
+process.on('unhandledRejection', (e) => {
+  console.log('[cli.js] unhandled rejection', e)
+  process.exit(1)
+})
+
+program
+  .version(packageJSON.version)
+  .option('-l, --list', 'List Backups')
+  .option(`-b, --backup <backup>`, 'Backup ID')
+  .option(`-d, --dir <directory>`, `Backup Directory (default: ${core.base})`)
+  .option('-r, --report <report_type>', 'Select a report type. see below for a full list.')
+  .option('-i, --id <id>', 'Specify an ID for filtering certain reports')
+  .option('-f, --formatter <type>', 'Specify output format. default: table')
+  .option(`-e, --extract <dir>`, 'Extract data for commands. supported by: voicemail-files, manifest')
+  .option('-o, --output <path>', 'Specify an output directory for files to be written to.')
+  .option(`-v, --verbose`, 'Verbose debugging output')
+  .option(`    --plugins <plugins>`, 'List of pluging modules to use')
+  .option(`    --filter <filter>`, 'Filter output fo r individual reports. See the README for usage.')
+  .option('    --join-reports', 'Join JSON reports together. (available for -f json or -f raw only!)')
+  .option(`    --no-color`, 'Disable colorized output')
+  .option(`    --dump`, 'alias for "--formatter raw"')
+  .option(`    --quiet`, 'quiet all messages, except for errors and raw output')
+  .option(`    --available`, 'output a list of available reports')
+
+program.on('--help', function () {
+  console.log('')
+  console.log(`Version: ${packageJSON.version}`)
+  console.log('')
+  console.log('Run ibackuptool --available for a listing of report types.')
+
+  console.log('')
+  console.log("If you're interested to know how this works, check out my post:")
+  console.log('https://www.richinfante.com/2017/3/16/reverse-engineering-the-ios-backup')
+  console.log('')
+  console.log('Issue tracker:')
+  console.log('https://github.com/richinfante/iphonebackuptools/issues')
+  console.log('')
+})
+
+function printReportList () {
+  // Print a report group and recursively print children
+  function printGroup (group, i, pn) {
+    i = i || 0
+    pn = pn || ''
+
+    for (let [name, report] of Object.entries(group)) {
+      // Ignore groups
+      if (name === '__group') { continue }
+
+      if (report instanceof Group || report.__group === true) {
+        // console.log(`${' '.repeat(i * 2)}- ${pn}${name}`.padStart(i * 2))
+        printGroup(report, i + 1, `${pn}${name}.`)
+      } else {
+        console.log(`- ${chalk.green(pn + name)}${report.deprecated ? chalk.gray(' [deprecated]') : ''}: ${report.description} `)
+      }
+    }
+  }
+
+  printGroup(core.modules)
+}
+
+process.on('unhandledRejection', (e) => {
+  console.log('[index.js] unhandled rejection', e)
+  process.exit(1)
+})
+
+// If we're the main module, run some things.
+if (require.main === module) {
+  init()
+}
+
+// Initialize the tool.
+function init () {
+  program.parse(process.argv)
+
+  if (program.available) {
+    printReportList()
+    process.exit(0)
+  }
+
+  // Set the log level
+  log.setVerbose(program.quiet ? 0 : (program.verbose ? 2 : 1))
+
+  // Parse plugins
+  program.plugins = program.plugins || process.env.IBACKUPTOOL_PLUGINS || ''
+  program.plugins = program.plugins.split(',')
+    .map(name => name.trim())
+    .filter(name => name !== '')
+    .map(path => require(path))
+
+  // Register witht the core module
+  core.registerModule(program.plugins)
+
+  log.verbose('Plugins:', program.plugins)
+
+  // Save the formatter
+  program.formatter = formatters[program.formatter] || formatters.table
+
+  // Legacy support for `--dump` flag.
+  if (program.dump) {
+    program.formatter = formatters.raw
+  }
+
+  // Disable color for non-ttys.
+  if (!process.stdout.isTTY) { program.color = false }
+
+  // Find the base if it is set
+  if (program.dir) {
+    core.base = program.dir
+  }
+
+  log.verbose('Base Directory:', core.base)
+
+  main().then(() => {})
+}
+
+/**
+ * Main CLI function
+ */
+async function main () {
+  // Legacy support for --list (-l) flag
+  if (program.list) {
+    program.report = 'backups.list'
+  }
+
+  log.verbose('Top-level modules', Object.keys(core.modules))
+
+  if (program.report) {
+    var reportContents = []
+
+    // Turn the report argument into an array of report type names
+    var selectedTypes = program.report
+      .split(',')
+      .map(el => el.trim())
+      .filter(el => el !== '')
+
+    let selectedReports = []
+
+    // Match an array of reports.
+    for (let query of selectedTypes) {
+      selectedReports = [
+        ...selectedReports,
+        ...matcher(core.modules, query, (el) => !(el instanceof Group || el.__group === true))
+      ]
+    }
+
+    let set = new Set(selectedReports)
+
+    log.verbose('selected set', set)
+
+    // If the set's size is 0, throw an error.
+    if (set.size === 0) {
+      log.error(`Couldn't run reports specified by: '${program.report}'.`)
+      log.error(`No matching reports were found.`)
+      process.exit(1)
+    }
+
+    // Iterate over each item in the set.
+    for (let report of set.values()) {
+      if (report.version && report.version >= 3) {
+        try {
+          log.begin('run', report.name)
+
+          // Create some parameters to send by default.
+          let params = {
+            backup: program.backup,
+            extract: program.extract,
+            filter: program.filter,
+            id: program.id,
+            raw: !!program.formatter.isRaw
+          }
+
+          // Run a v3 report.
+          let contents = await core.runReport(report, params)
+
+          // Possibly symbolicate
+          contents = await core.compileReport(report, contents, params)
+
+          // Format the v3 report's result.
+          let formattedContent = program.formatter.format(contents, {
+            program
+          })
+
+          // Push onto the list to be compiled.
+          reportContents.push({
+            name: report.name,
+            contents: formattedContent
+          })
+
+          log.end()
+        } catch (e) {
+          log.end()
+          log.error(`Couldn't run '${report.name}'.`)
+          log.error(e)
+        }
+      } else {
+        // Older reports still work for file and screen output
+        // They just may not be as compatible
+        // They cannot be called from the module's api.
+
+        if (selectedReports.length > 1 && !report.usesPromises) {
+          log.warning('the report', report.name, 'does not utilize promises.')
+          log.warning('this may not work')
+        }
+
+        log.begin('run', report.name)
+
+        // Check if there's a backup specified and one is required.
+        if (report.requiresBackup) {
+          if (!program.backup) {
+            log.error('use -b or --backup <id> to specify backup.')
+            process.exit(1)
+          }
+        }
+        try {
+          if (report.func) {
+            let contents = await runSingleReport(report, program)
+            if (contents == null) { log.end(); continue }
+
+            reportContents.push({
+              name: report.name,
+              contents: contents
+            })
+          } else if (report.functions) {
+            let contents = await runSwitchedReport(report, program)
+            if (contents == null) { log.end(); continue }
+
+            reportContents.push({
+              name: report.name,
+              contents: contents
+            })
+          }
+        } catch (e) {
+          log.error(`Couldn't run '${report.name}'.`)
+          log.error(e)
+        }
+
+        log.end()
+      }
+    }
+
+    program.formatter.finalReport(reportContents, program)
+  } else {
+    program.outputHelp()
+  }
+}

tools/formatters/csv.js

@@ -4,25 +4,47 @@ const path = require('path')
 const log = require('../util/log')
 
 module.exports.format = function (data, options) {
-  var processedData = data.map(el => {
+  // Default columns to an empty object
+  options.colums = options.columns || {}
+
+  // Check if the array is empty. If so, return an empty string.
+  if (data instanceof Array && data.length === 0) {
+    return ''
+  }
+
+  // If we didn't get a data object, make it an array for ease of use.
+  if (!(data instanceof Array) && typeof data === 'object') {
+    data = [data]
+  }
+
+  // Do some preprocessing to find the columns.
+  if ((!options.columns || Object.keys(options.colums).length === 0) && data.length > 0) {
+    // Extract the fields from the first object.
+    options.columns = Object.keys(data[0])
+    log.verbose('assigning csv columns to', options.columns)
+  }
+
+  function processRow (el) {
     var row = {}
 
     // Iterate over the columns and add each item to the new row.
-    for (var key in options.columns) {
-      row[key] = options.columns[key](el)
+    for (var key of options.columns) {
+      if (typeof options.columns[key] === 'function') {
+        row[key] = options.columns[key](el)
+      } else {
+        row[key] = el[key]
+      }
     }
 
     return row
-  })
+  }
 
-  const csv = json2csv({ data: processedData })
+  var processedData = data.map(processRow)
 
-  if (options.program) {
-    // If reporting output is defined, ignore console log here.
-    if (options.program.output === undefined) {
-      log.raw(csv)
-    }
-  } else {
+  const csv = json2csv({ data: processedData, fieldNames: Object.keys(data[0]) })
+
+  // Print the output if required.
+  if (!options.program || options.program.output === undefined) {
     log.raw(csv)
   }
 
@@ -41,6 +63,11 @@ module.exports.finalReport = async function (reports, program) {
   for (var report of reports) {
     var outPath = path.join(program.output, report.name + '.csv')
     log.action('saving', outPath)
-    fs.writeFileSync(outPath, report.contents, 'utf8')
+
+    if (program.output === '-') {
+      console.log(report.contents)
+    } else {
+      fs.writeFileSync(outPath, report.contents, 'utf8')
+    }
   }
 }

tools/formatters/json.js

@@ -3,7 +3,12 @@ const path = require('path')
 const log = require('../util/log')
 
 module.exports.format = function (data, options) {
-  var processedData = data.map(el => {
+  function convertRow (el) {
+    // No Columns defined, just return raw object.
+    if (!options.columns || Object.keys(options.columns).length === 0) {
+      return el
+    }
+
     var row = {}
 
     // Iterate over the columns and add each item to the new row.
@@ -12,7 +17,14 @@ module.exports.format = function (data, options) {
     }
 
     return row
-  })
+  }
+
+  var processedData
+  if (data instanceof Array) {
+    processedData = data.map(convertRow)
+  } else {
+    processedData = convertRow(data)
+  }
 
   // Strigify the output, using 2 space indent.
   var output = JSON.stringify(processedData, null, 2)
@@ -59,7 +71,12 @@ module.exports.finalReport = async function (reports, program) {
     for (let report of reports) {
       let outPath = path.join(program.output, report.name + '.json')
       log.action('saving', outPath)
-      fs.writeFileSync(outPath, JSON.stringify(report.contents, null, 2), 'utf8')
+
+      if (program.output === '-') {
+        console.log(JSON.stringify(report.contents))
+      } else {
+        fs.writeFileSync(outPath, JSON.stringify(report.contents), 'utf8')
+      }
     }
   }
 }

tools/formatters/raw-csv.js

@@ -1,36 +1,47 @@
 const json2csv = require('json2csv')
 const log = require('../util/log')
 
-module.exports.format = function (data, options) {
-  const csv = json2csv({ data })
+const fs = require('fs-extra')
+const path = require('path')
+
+module.exports = {
 
-  if (options.program) {
-    // If reporting output is defined, ignore console log here.
-    if (options.program.output === undefined) {
+  // This report wants raw data.
+  isRaw: true,
+
+  format (data, options) {
+    const csv = json2csv({ data })
+
+    if (options.program) {
+      // If reporting output is defined, ignore console log here.
+      if (options.program.output === undefined) {
+        log.raw(csv)
+      }
+    } else {
       log.raw(csv)
     }
-  } else {
-    log.raw(csv)
-  }
 
-  return csv
-}
+    return csv
+  },
 
-const fs = require('fs-extra')
-const path = require('path')
+  finalReport (reports, program) {
+    if (program.output === undefined) {
+      return
+    }
 
-module.exports.finalReport = async function (reports, program) {
-  if (program.output === undefined) {
-    return
-  }
+    // Ensure the output directory exists.
+    fs.ensureDirSync(program.output)
 
-  // Ensure the output directory exists.
-  fs.ensureDirSync(program.output)
+    // Write each report to the disk
+    for (var report of reports) {
+      var outPath = path.join(program.output, report.name + '.csv')
+      log.action('saving', outPath)
 
-  // Write each report to the disk
-  for (var report of reports) {
-    var outPath = path.join(program.output, report.name + '.csv')
-    log.action('saving', outPath)
-    fs.writeFileSync(outPath, report.contents, 'utf8')
+      if (program.output === '-') {
+        console.log(JSON.stringify(report.contents))
+      } else {
+        fs.writeFileSync(outPath, report.contents, 'utf8')
+      }
+    }
   }
 }

tools/formatters/raw-json.js

@@ -1,5 +1,7 @@
 const log = require('../util/log')
 
+module.exports.isRaw = true
+
 module.exports.format = function (data, options) {
   var output = JSON.stringify(data)
 
@@ -50,7 +52,12 @@ module.exports.finalReport = async function (reports, program) {
     for (let report of reports) {
       let outPath = path.join(program.output, report.name + '.json')
       log.action('saving', outPath)
-      fs.writeFileSync(outPath, JSON.stringify(report.contents), 'utf8')
+
+      if (program.output === '-') {
+        console.log(JSON.stringify(report.contents))
+      } else {
+        fs.writeFileSync(outPath, JSON.stringify(report.contents), 'utf8')
+      }
     }
   }
 }

tools/formatters/table.js

@@ -2,7 +2,13 @@ const stripAnsi = require('strip-ansi')
 const log = require('../util/log')
 
 function keyValueArray (columns, keys, obj) {
-  return keys.map(el => columns[el](obj))
+  return keys.map(el => {
+    if (typeof columns[el] === 'function') {
+      return columns[el](obj)
+    } else {
+      return obj[el]
+    }
+  })
 }
 
 function findMaxLengths (rows) {
@@ -93,7 +99,7 @@ function createTable (rows, fitWidth) {
 
   // Store the output rows
   var outputRows = []
-  
+
   // Cursors for each item in the current row.
   var cursors = []
 
@@ -105,9 +111,14 @@ function createTable (rows, fitWidth) {
 
     for (let j = 0; j < rows[i].length; j++) {
       cursors[j] = cursors[j] || 0
-      
+
       // Extract item
-      let rawItem = rows[i][j] + ''
+      let rawItem = ''
+      if (typeof rawItem === 'object') {
+        rawItem = '[Object]'
+      } else {
+        rawItem = rows[i][j] + ''
+      }
 
       // Slice for this row.
       let inputItem = rawItem.substr(cursors[j], widths[j])
@@ -152,6 +163,22 @@ module.exports.format = function (data, options) {
   // Separators for each column
   var separators = []
 
+  // If data is not an array,
+  // Turn it into one.
+  if (!(data instanceof Array)) {
+    data = [data]
+  }
+
+  // Ensure we have a column list.
+  // If there are no items, grab the keys from the data object.
+  if ((!options.columns || Object.keys(options.columns).length === 0) && data.length > 0) {
+    options.columns = {}
+
+    for (let item of Object.keys(data[0])) {
+      options.columns[item] = true
+    }
+  }
+
   // Add to collection of keys
   for (var key in options.columns) {
     keys.push(key)
@@ -206,6 +233,11 @@ module.exports.finalReport = async function (reports, program) {
   for (var report of reports) {
     var outPath = path.join(program.output, report.name + '.txt')
     log.action('saving', outPath)
-    fs.writeFileSync(outPath, report.contents, 'utf8')
+
+    if (program.output === '-') {
+      console.log(report.contents)
+    } else {
+      fs.writeFileSync(outPath, report.contents, 'utf8')
+    }
   }
 }

tools/index.js

@@ -1,301 +1,218 @@
-#!/usr/bin/env node
-
-const program = require('commander')
 const path = require('path')
-const chalk = require('chalk')
-const version = require('./util/version_compare')
-const iPhoneBackup = require('./util/iphone_backup.js').iPhoneBackup
 const log = require('./util/log')
+const report = require('./reports')
+const matcher = require('./util/matcher')
+const Group = report.Group
+const Backup = require('./backup')
+
+// Backup source directory
+var backupDirectory = path.join(process.env.HOME, '/Library/Application Support/MobileSync/Backup/')
+
+// Object containing all report modules
+var moduleCache = report.types
+
+// Array of plugin modules
+var plugins = []
+
+/**
+ * Add an array of plugins to the plugins list.
+ * @param {Array<Object>} array contains array of plugin objects.
+ */
+function registerModule (array) {
+  if (!(array instanceof Array)) {
+    array = [ array ]
+  }
 
-var base = path.join(process.env.HOME, '/Library/Application Support/MobileSync/Backup/')
-
-var reportTypes = {
-  'apps': require('./reports/apps'),
-  'calls': require('./reports/calls'),
-  'conversations': require('./reports/conversations'),
-  'conversations_full': require('./reports/conversations_full'),
-  'cookies': require('./reports/cookies'),
-  'list': require('./reports/list'),
-  'manifest': require('./reports/manifest'),
-  'messages': require('./reports/messages'),
-  'notes': require('./reports/notes'),
-  'oldnotes': require('./reports/oldnotes'),
-  'photolocations': require('./reports/photolocations'),
-  'voicemail-files': require('./reports/voicemail-files'),
-  'voicemail': require('./reports/voicemail'),
-  'webhistory': require('./reports/webhistory'),
-  'calls_statistics': require('./reports/calls_statistics'),
-  'wifi': require('./reports/wifi'),
-  'address_book': require('./reports/address_book'),
-  'safari_bookmarks': require('./reports/safari_bookmarks'),
-  'pushstore': require('./reports/pushstore'),
-  'calendar': require('./reports/calendar'),
-  'facebook_profile': require('./reports/facebook_profile'),
-  'facebook_messenger_friends': require('./reports/facebook_messenger_friends'),
-  'spotify': require('./reports/spotify'),
-  'instagram_profile': require('./reports/instagram_profile'),
-  'instagram_recent_searches': require('./reports/instagram_recent_searches'),
-  'instagram_following_users_coded': require('./reports/instagram_following_users_coded'),
-  'instagram_fb_friends': require('./reports/instagram_fb_friends'),
-  'speed_dial': require('./reports/speed_dial'),
-  'bluetooth_devices': require('./reports/bluetooth_devices'),
-  'safari_open_tabs': require('./reports/safari_open_tabs'),
-  'safari_recent_searches': require('./reports/safari_recent_searches'),
-  'gmail_accounts': require('./reports/gmail_accounts'),
-  'gmail_shared_contacts': require('./reports/gmail_shared_contacts'),
-  'waze_favorites': require('./reports/waze_favorites'),
-  'waze_places': require('./reports/waze_places'),
-  'waze_recents': require('./reports/waze_recents'),
-  'skype_accounts': require('./reports/skype_accounts'),
-  'skype_calls': require('./reports/skype_calls'),
-  'viber_contacts': require('./reports/viber_contacts'),
-  'viber_calls': require('./reports/viber_calls'),
-  'viber_messages': require('./reports/viber_messages')
+  plugins.push(...array)
+  moduleCache = getCompleteModuleList()
 }
 
-var formatters = {
-  'json': require('./formatters/json'),
-  'table': require('./formatters/table'),
-  'raw': require('./formatters/raw-json'),
-  'raw-json': require('./formatters/raw-json'),
-  'csv': require('./formatters/csv'),
-  'raw-csv': require('./formatters/raw-csv')
+/**
+ * Remove all non-default plugins.
+ */
+function clearModules () {
+  plugins = []
+  moduleCache = getCompleteModuleList()
 }
 
-program
-  .version('3.0.0')
-  .option('-l, --list', 'List Backups')
-  .option(`-b, --backup <backup>`, 'Backup ID')
-  .option(`-d, --dir <directory>`, `Backup Directory (default: ${base})`)
-  .option('-r, --report <report_type>', 'Select a report type. see below for a full list.')
-  .option('-i, --id <id>', 'Specify an ID for filtering certain reports')
-  .option('-f, --formatter <type>', 'Specify output format. default: table')
-  .option(`-e, --extract <dir>`, 'Extract data for commands. supported by: voicemail-files, manifest')
-  .option('-o, --output <path>', 'Specify an output directory for files to be written to.')
-  .option(`-v, --verbose`, 'Verbose debugging output')
-  .option(`    --filter <filter>`, 'Filter output fo r individual reports. See the README for usage.')
-  .option('    --join-reports', 'Join JSON reports together. (available for -f json or -f raw only!)')
-  .option(`    --no-color`, 'Disable colorized output')
-  .option(`    --dump`, 'alias for "--formatter raw"')
-  .option(`    --quiet`, 'quiet all messages, except for errors and raw output')
-
-program.on('--help', function () {
-  console.log('')
-  console.log('Supported Report Types:')
-
-  // Generate a list of report types.
-  for (var i in reportTypes) {
-    var r = reportTypes[i]
-    console.log('  ', chalk.green(r.name), (r.supportedVersions ? chalk.gray('(iOS ' + r.supportedVersions + ') ') : '') + '-', r.description)
+/**
+ * Get all modules in a top-down manner.
+ */
+function getCompleteModuleList () {
+  let allModules = {}
+
+  // Add all of the require()'d modules into the plugins list.
+  plugins.forEach(function (plugin) {
+    allModules = { ...allModules, ...plugin }
+  })
+
+  // Add all of the modules to a single object.
+  // JS's behavior dictates that the items are added sequentially
+  // So, the default reports overwrite any third party plugin.
+  let result = {
+    ...allModules,
+    ...report.types
   }
-  console.log('')
-  console.log("If you're interested to know how this works, check out my post:")
-  console.log('https://www.richinfante.com/2017/3/16/reverse-engineering-the-ios-backup')
-  console.log('')
-  console.log('Issue tracker:')
-  console.log('https://github.com/richinfante/iphonebackuptools/issues')
-  console.log('')
-})
-
-process.on('unhandledRejection', (e) => {
-  console.log('unhandled', e)
-  process.exit(1)
-})
-
-// Parse argv.
-program.parse(process.argv)
-
-log.setVerbose(program.quiet ? 0 : (program.verbose ? 2 : 1))
-
-// Save the formatter
-program.formatter = formatters[program.formatter] || formatters.table
-
-// Legacy support for `--dump` flag.
-if (program.dump) {
-  program.formatter = formatters.raw
-}
-
-// Disable color for non-ttys.
-if (!process.stdout.isTTY) { program.color = false }
-
-// Find the base
-base = program.dir || base
-
-log.verbose('Using source:', base)
 
-// Run the main function
-main()
+  moduleCache = result
 
-async function main () {
-  if (program.list) {
-    // Run the list report standalone
-    await new Promise((resolve, reject) => {
-      reportTypes.list.func(program, base, resolve, reject)
-    })
-  } else if (program.report) {
-    var reportContents = []
+  return result
+}
 
-    // Turn the report argument into an array of report type names
-    var selectedTypes = program.report
-      .split(',')
-      .map(el => el.trim())
-      .filter(el => el !== '')
+/**
+ * Try to find a single report.
+ * @param {string} query name to find.
+ */
+function findReport (query) {
+  return new Promise((resolve, reject) => {
+    // Check there is no wildcard in the query.
+    if (query.indexOf('*') > -1) {
+      return reject(new Error('Cannot run a wildcard match here.'))
+    }
 
-    // Add all types if type is 'all'
-    if (program.report === 'all') {
-      selectedTypes = []
+    // Run matches.
+    let matches = matcher(moduleCache, query, (el) => !(el instanceof Group))
 
-      for (var key in reportTypes) {
-        if (reportTypes[key].requiresInteractivity === true) {
-          continue
-        }
+    // If no report found, fail.
+    if (matches.length === 0) {
+      return reject(new Error(`No report found with name "${query}"`))
+    }
 
-        selectedTypes.push(key)
-      }
+    // If multiple matches, fail.
+    if (matches.length > 1) {
+      return reject(new Error(`Multiple report matches for name "${query}", not allowed.`))
     }
 
-    for (var reportName of selectedTypes) {
-      // If the report is valid
-      if (reportTypes[reportName]) {
-        var report = reportTypes[reportName]
+    // Resolve match
+    resolve(matches[0])
+  })
+}
 
-        if (selectedTypes.length > 1 && !report.usesPromises) {
-          log.warning('the report', report.name, 'does not utilize promises.')
-          log.warning('this may not work')
-        }
+/**
+ * Translate the raw output of a report to the correct result, based on the "raw" parameter.
+ * @param {Object} report The report module
+ * @param {Object} result Raw data output from the aforementioned report
+ * @param {Object} params parameters object.
+ */
+function compileReport (report, result, { raw }) {
+  return new Promise((resolve, reject) => {
+    if (!raw && report.output) {
+      if (result instanceof Array) {
+        log.verbose('compiling report (array)...')
+        // if it's an array, translate each item.
+        result = result.map(item => {
+          // For each item, run the functions on the entry.
+          let editedResult = {}
+          for (let [key, value] of Object.entries(report.output)) {
+            editedResult[key] = value(item)
+          }
 
-        log.begin('run', report.name)
+          return editedResult
+        })
 
-        // Check if there's a backup specified and one is required.
-        if (report.requiresBackup) {
-          if (!program.backup) {
-            log.error('use -b or --backup <id> to specify backup.')
-            process.exit(1)
-          }
-        }
-        try {
-          if (report.func) {
-            let contents = await runSingleReport(report, program)
-            if (contents == null) { log.end(); continue }
-
-            reportContents.push({
-              name: reportName,
-              contents: contents
-            })
-          } else if (report.functions) {
-            let contents = await runSwitchedReport(report, program)
-            if (contents == null) { log.end(); continue }
-
-            reportContents.push({
-              name: reportName,
-              contents: contents
-            })
-          }
-        } catch (e) {
-          log.error(`Couldn't run '${report.name}'.`)
-          log.error(e)
+        resolve(result)
+      } else {
+        log.verbose('compiling report (single)...')
+        // Otherwise, translate the object returned.
+        let editedResult = {}
+        for (let [key, value] of Object.entries(report.output)) {
+          editedResult[key] = value(result)
         }
 
-        log.end()
-      } else {
-        log.error('Unknown report type:', reportName)
-        log.error(`It's possible this tool is out-of date.`)
-        log.error(`https://github.com/richinfante/iphonebackuptools/issues`)
-        program.outputHelp()
+        resolve(editedResult)
       }
+    } else {
+      resolve(result)
     }
-
-    program.formatter.finalReport(reportContents, program)
-  } else {
-    program.outputHelp()
-  }
+  })
 }
 
-async function runSwitchedReport (report, program) {
-  async function createPromise (key, program, backup) {
-    log.verbose('resolving using promises.')
-
-    return new Promise((resolve, reject) => {
-      report.functions[key](program, backup, resolve, reject)
-    })
-  }
-
-  // New type of reports
-  var backup = iPhoneBackup.fromID(program.backup, base)
-
-  var flag = false
-  var value
-  // Check for a compatible reporting tool.
-  for (var key in report.functions) {
-    if (version.versionCheck(backup.iOSVersion, key)) {
-      if (!report.usesPromises) {
-        log.verbose('using synchronous call.')
-
-        value = report.functions[key](program, backup)
-      } else {
-        // Use promises to resolve synchronously
-        value = await createPromise(key, program, backup)
-      }
-      flag = true
-      break
+/**
+ * Run a named report and resolve to it's output.
+ * The output is formatted based on the `report.output` key, if the params.raw option is NOT set to true.
+ * @param {string} query report name
+ * @param {Object=} params parameters.
+ */
+function run (query, params) {
+  params = params || {}
+  return new Promise(async (resolve, reject) => {
+    try {
+      let report = await findReport(query)
+      let result = await runReport(report, params)
+      let compiled = await compileReport(report, result, params)
+
+      resolve(compiled)
+    } catch (e) {
+      reject(e)
     }
-  }
+  })
+}
 
-  if (!flag) {
-    log.error(`Couldn't run '${report.name}'.`)
-    log.error(`The report generator '${report.name}' does not support iOS`, backup.iOSVersion)
-    log.error(`If you think it should, file an issue here:`)
-    log.error(`https://github.com/richinfante/iphonebackuptools/issues`)
-    return null
-  }
+/**
+ * Run a report
+ * @param {object} report report module
+ * @param {object=} params parameters
+ */
+function runReport (report, params) {
+  params = params || {}
 
-  return value
-}
+  return new Promise((resolve, reject) => {
+    var backup
 
-async function runSingleReport (report, program) {
-  async function createPromise (program, backup, base) {
-    log.verbose('resolving using promises.')
+    // Cannot run < v3 backups in this manner.
+    if (!report.version || report.version < 3) {
+      return reject(new Error(`Cannot call ${report.name} as a module, it is not updated to the v3 api`))
+    }
 
-    return new Promise((resolve, reject) => {
-      if (report.requiresBackup) {
-        report.func(program, backup, resolve, reject)
-      } else {
-        report.func(program, base, resolve, reject)
+    // If it requires a backup and none is provided, reject.
+    if (report.requiresBackup) {
+      if (!params.backup) {
+        return reject(new Error('Please specify the `backup` parameter to run this report.'))
       }
-    })
-  }
 
-  async function runReport (backup, base) {
-    if (!report.usesPromises) {
-      log.verbose('using synchronous call.')
+      backup = new Backup(backupDirectory, params.backup)
+    }
 
-      // Old-style non-promise based report.
-      if (report.requiresBackup) {
-        return report.func(program, backup)
-      } else {
-        return report.func(program, base)
-      }
-    } else {
-      // Create a promise to resolve this function
-      // Use promises to resolve synchronously
-      return createPromise(program, backup, base)
+    // Input params to func
+    let inputParams = {
+      ...params,
+      backup
     }
-  }
 
-  // New type of reports
-  var backup = iPhoneBackup.fromID(program.backup, base)
+    report.run(module.exports, inputParams)
+      .then(resolve)
+      .catch(reject)
+  })
+}
 
-  if (report.supportedVersions !== undefined) {
-    if (version.versionCheck(backup.iOSVersion, report.supportedVersions)) {
-      return runReport(backup, base)
-    } else {
-      log.error(`Couldn't run '${report.name}'.`)
-      log.error(`The report generator '${report.name}' does not support iOS`, backup.iOSVersion)
-      log.error(`If you think it should, file an issue here:`)
-      log.error(`https://github.com/richinfante/iphonebackuptools/issues`)
-      return null
-    }
-  } else {
-    return runReport(backup, base)
+module.exports = {
+  // Exported Libraries
+  Backup,
+  Group,
+
+  // Module management
+  registerModule,
+  clearModules,
+  get modules () {
+    return moduleCache
+  },
+
+  // Source directory
+  set base (value) {
+    backupDirectory = value
+  },
+  get base () {
+    return backupDirectory
+  },
+
+  // Runners
+  run,
+  findReport,
+  runReport,
+  compileReport,
+
+  // misc
+  setLogLevel (lvl) {
+    log.setVerbose(lvl)
   }
 }

tools/reports.js

@@ -0,0 +1,120 @@
+
+class Group {
+  constructor (children) {
+    for (let [ key, value ] of Object.entries(children)) {
+      this[key] = value
+    }
+  }
+}
+
+module.exports.types = {
+  // Global utilities.
+  backups: new Group({
+    list: require('./reports/backups/list')
+  }),
+
+  // Per-backup utilities
+  backup: new Group({
+    status: require('./reports/backup/status'),
+    manifest: require('./reports/backup/manifest'),
+    info: require('./reports/backup/info'),
+    files: require('./reports/backup/files')
+  }),
+
+  // Phone Data
+  phone: new Group({
+    calls: require('./reports/phone/calls'),
+    calls_statistics: require('./reports/phone/calls_statistics'),
+    address_book: require('./reports/phone/address_book'),
+    speed_dial: require('./reports/phone/speed_dial'),
+    voicemail: require('./reports/phone/voicemail')
+  }),
+
+  // Notes report
+  notes: require('./reports/notes/notes'),
+
+  // Camera report
+  photos: new Group({
+    locations: require('./reports/photos/locations')
+  }),
+
+  // Calendar
+  calendar: new Group({
+    events: require('./reports/calendar/events')
+  }),
+
+  // Reports related to messaging.
+  messages: new Group({
+    all: require('./reports/messages/all'),
+    conversations: require('./reports/messages/conversations'),
+    messages: require('./reports/messages/messages'),
+    conversations_full: require('./reports/messages/conversations_full')
+  }),
+
+  // Safari Data
+  safari: new Group({
+    history: require('./reports/safari/webhistory'),
+    bookmarks: require('./reports/safari/bookmarks'),
+    open_tabs: require('./reports/safari/open_tabs'),
+    recent_searches: require('./reports/safari/recent_searches'),
+    cookies: require('./reports/safari/cookies')
+  }),
+
+  // System level reports, such as wifi
+  system: new Group({
+    wifi: require('./reports/system/wifi'),
+    bluetooth_devices: require('./reports/system/bluetooth_devices'),
+    pushstore: require('./reports/system/pushstore'),
+    apps: require('./reports/system/apps'),
+    geofences: require('./reports/system/geofences')
+  }),
+
+  // Facebook Data
+  facebook: new Group({
+    profile: require('./reports/thirdparty/facebook/profile'),
+    messenger: new Group({
+      friends: require('./reports/thirdparty/facebook/messenger')
+    })
+  }),
+
+  // Instagram data
+  instagram: new Group({
+    profile: require('./reports/thirdparty/instagram/profile'),
+    recent_searches: require('./reports/thirdparty/instagram/recent_searches'),
+    following_users_coded: require('./reports/thirdparty/instagram/following_users_coded'),
+    fb_friends: require('./reports/thirdparty/instagram/fb_friends')
+  }),
+
+  // Gmail
+  gmail: new Group({
+    accounts: require('./reports/thirdparty/gmail/accounts'),
+    shared_contacts: require('./reports/thirdparty/gmail/shared_contacts')
+  }),
+
+  // Spotify
+  spotify: new Group({
+    searches: require('./reports/thirdparty/spotify/searches')
+  }),
+
+  // Waze
+  waze: new Group({
+    favorites: require('./reports/thirdparty/waze/favorites'),
+    places: require('./reports/thirdparty/waze/places'),
+    recents: require('./reports/thirdparty/waze/recents')
+  }),
+
+  // Skype
+  skype: new Group({
+    accounts: require('./reports/thirdparty/skype/accounts'),
+    calls: require('./reports/thirdparty/skype/calls')
+  }),
+
+  // Viber
+  viber: new Group({
+    contacts: require('./reports/thirdparty/viber/contacts'),
+    calls: require('./reports/thirdparty/viber/calls'),
+    messages: require('./reports/thirdparty/viber/messages')
+  })
+}
+
+module.exports.Group = Group

tools/reports/_example.js

@@ -1,68 +1,104 @@
-// First, give it a name!
-module.exports.name = 'example module'
 
-// Provide a description.
-module.exports.description = 'An example module to show how it works'
+module.exports = {
+  /**
+   * The reporting API version. This should be set to 3.
+   */
+  version: 4,
 
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-// Most reporting types should use this.
-module.exports.requiresBackup = true
+  /**
+   * Report name. This should match the nesting found in the reports.js file.
+   */
+  name: 'ibackuptool.example',
 
-// Should this report be skipped in automated reports?
-// This is used when the 'all' report type is specified, and all possible reports are generated.
-// with this set to true, the report WILL NOT run when report type = 'all'
-// Most reporting types shouldn't need this.
-module.exports.requiresInteractivity = true
+  /**
+   * Human readable description of the file
+   */
+  description: `Example Report module.`,
 
-// Specify this reporter supports the promises API for allowing chaining of reports.
-// All modules should use this.
-module.exports.usesPromises = true
+  /**
+   * Optional flag requiring a backup parameter to be present in order to run this report.
+   */
+  requiresBackup: true,
 
-// Specify this only works for iOS 10+
-// If it is iOS-version specific, you can specify version information here.
-// You may provide a comma separated string such as ">=6.0,<11.0" to indicate ranges.
-module.exports.supportedVersions = '>=10.0'
+  /**
+   * Run on a v3 lib / backup object.
+   * The run() function must return a promise, which always resolves to valid data.
+   * If the files aren't in the backup or aren't formatted in a known way, we reject
+   * and print the error message for the user.
+   * @param {object} lib standard lib, contains lib.run() function.
+   * @param {object} options options object.
+   */
+  run (lib, { backup }) {
+    return new Promise((resolve, reject) => {
+      // resolve to valid data.
+      // Typically, this would be "raw" data containing as much info as possible.
+      // Se below for data formatting.
+      resolve([{
+        name: 'example1',
+        data: {
+          code: 33,
+          values: [1, 2, 3, 4, 5]
+        }
+      }])
+    })
+  },
+
+  /**
+   * The "output" property declares the public interface for most operations.
+   * This provides a level of abstraction from the datatypes that are stored in the
+   * backups since they may vary between versions, or need normalization.
+   *
+   * This collection of functions allows that to occur.
+   */
+  output: {
+    name: el => el.name,
+    code: el => el.data.code
+  }
 
-// Most reports will use this pattern.
-// Reporting function (for usesPromises = true)
-module.exports.func = function (program, backup, resolve, reject) {
-  // This function will be called with the `commander` program, and the iPhoneBackup instance as arguments
-  // It MUST resolve() the final result, or reject() if there's an error
+  /*
+    For the above example, if run() resolved to:
+    [{
+      name: 'example1',
+      data: {
+        code: 33,
+        values: [1, 2, 3, 4, 5]
+      }
+    }]
 
-  // First, fetch some data. This variable should be an array of objects representing each row in a report.
-  // This would be replaced with a function from the backup object.
-  let data = backup.getData()
+    The actual module output when using a normal json, csv, table formatter would be the following,
+    due to the output declaration:
 
-  // Next, pass it to the user-selected formatter.
-  var result = program.formatter.format(data, {
-    // Provide the program options
-    program: program,
+    [{
+      name: 'example1',
+      code: 33
+    }]
 
-    // A dictionary of items to be displayed as formatted data.
-    // The key is the column name, the value is a function that returns the value given an object representing a row.
-    columns: {
-      'Column-Name': row => row.ROWID
+    // We can also output a single raw object:
+    {
+      Name: 'test',
+      Version: '1.0',
+      BackupData: [104, 101, 108, 108, 111, 044, 119, 111, 114, 108, 100]
     }
-  })
 
-  // Resolve the promise with the result.
-  // This ensures proper file output and multi-reporting.
-  resolve(result)
-}
+    // And map it using the following output declaration:
+    output: {
+      name: el => el.Name,
+      version: el => el.Version
+    }
 
-// --- OR ---
+    To the following:
+    {
+      name: 'test',
+      version: '1.0'
+    }
 
-// You can also provide an array of functions instead of using `module.exports.func`.
-// These functions *should* be independent ranges to ensure reliable execution.
-module.exports.functions = {
-  '>=10.0': function (program, backup, resolve, reject) {
-    // This function would be called for iOS 10+.
-    // format and resolve() in the same manner as the example above.
+    IF we specify a "raw" formatter, or run using the { raw: true } parameter to lib.run(),
+    We'd get back the original raw object.
 
-  },
-  '>=9.0,<10.0': function (program, backup) {
-    // This function would be called for all iOS 9.
-    // format and resolve() in the same manner as the example above.
-  }
+    {
+      Name: 'test',
+      Version: '1.0',
+      BackupData: [104, 101, 108, 108, 111, 044, 119, 111, 114, 108, 100]
+    }
+  */
 }

tools/reports/address_book.js

@@ -1,40 +0,0 @@
-module.exports.name = 'address_book'
-module.exports.description = 'List all address book records contained in the backup.'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getAddressBook()
-    .then((items) => {
-    // Use the configured formatter to print the rows.
-      const result = program.formatter.format(items, {
-      // Color formatting?
-        program: program,
-
-        // Columns to be displayed in human-readable printouts.
-        // Some formatters, like raw or CSV, ignore these.
-        columns: {
-          'ID': el => el.ROWID,
-          'First': el => el.First ? el.First.substring(0, 10) + '' : '',
-          'Last': el => el.Last ? el.Last.substring(0, 10) + '' : '',
-          'Organization': el => el.organization ? el.organization.substring(0, 10) + '' : '',
-          'Phone Work': el => el.phone_work ? el.phone_work.substring(0, 14) + '' : '',
-          'Phone Mobile': el => el.phone_mobile ? el.phone_mobile.substring(0, 14) + '' : '',
-          'Phone Home': el => el.phone_home ? el.phone_home.substring(0, 14) + '' : '',
-          'Email': el => el.email ? el.email.substring(0, 28) + '' : '',
-          'Created Date': el => el.created_date ? el.created_date.substring(0, 28) + '' : '',
-          'Note': el => el.note ? el.note.substring(0, 28) + '' : '',
-          'Picture': el => el.profile_picture ? 1 : 0
-        }
-      })
-
-      // If using promises, we must call resolve()
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/apps.js

@@ -1,32 +0,0 @@
-module.exports.name = 'apps'
-module.exports.description = 'List all installed applications and container IDs.'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  // Fail if manifest does not exist
-  if (!backup.manifest) return reject(new Error('Manifest does not exist in this version'))
-
-  // Enumerate the apps in the backup
-  var apps = []
-  for (var key in backup.manifest.Applications) {
-    var app = backup.manifest.Applications[key]
-
-    apps.push({ bundleID: app.CFBundleIdentifier, path: app.Path })
-  }
-
-  var result = program.formatter.format(apps, {
-    program: program,
-    columns: {
-      'Bundle ID': el => el.bundleID,
-      'Bundle Path': el => el.path
-    }
-  })
-
-  resolve(result)
-}

tools/reports/backup/files.js

@@ -0,0 +1,134 @@
+const fs = require('fs-extra')
+const path = require('path')
+const log = require('../../util/log')
+const manifestMBDBParse = require('../../util/manifest_mbdb_parse')
+
+module.exports = {
+  version: 4,
+  name: 'backup.files',
+  description: `Gets a backup's file list`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup, extract, filter }) {
+    return new Promise(async (resolve, reject) => {
+      getManifest(backup)
+        .then(files => {
+          // Possibly extract objects.
+          if (extract) {
+            extractFiles(backup, extract, filter, files)
+          }
+
+          resolve(files)
+        })
+        .catch(reject)
+    })
+  },
+
+  // Available fields.
+  output: {
+    id: el => el.fileID,
+    domain: el => el.domain,
+    path: el => el.filename,
+    size: el => el.filelen || 0
+  }
+}
+
+/// Get the manifest for an sqlite database if available
+function getSqliteFileManifest (backup) {
+  return new Promise(async (resolve, reject) => {
+    backup.openDatabase('Manifest.db', true)
+      .then(db => {
+        db.all('SELECT fileID, domain, relativePath as filename from FILES', async function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+/// Get the manifest from the mbdb file
+function getMBDBFileManifest (backup) {
+  return new Promise((resolve, reject) => {
+    let mbdbPath = backup.getFileName('Manifest.mbdb', true)
+    manifestMBDBParse.process(mbdbPath, resolve, reject)
+  })
+}
+
+/// Try to load both of the manifest files
+function getManifest (backup) {
+  return new Promise(async (resolve, reject) => {
+    // Try the new sqlite file database.
+    try {
+      log.verbose('Trying sqlite manifest...')
+      let item = await getSqliteFileManifest(backup)
+      return resolve(item)
+    } catch (e) {
+      log.verbose('Trying sqlite manifest... [failed]', e)
+    }
+
+    // Try the mbdb file database
+    try {
+      log.verbose('Trying mbdb manifest...')
+      let item = await getMBDBFileManifest(backup)
+      return resolve(item)
+    } catch (e) {
+      log.verbose('Trying mbdb manifest...[failed]', e)
+    }
+
+    reject(new Error('Could not find a manifest.'))
+  })
+}
+
+/// Filter exclusion check
+function isIncludedByFilter (filter, item) {
+  return filter === 'all' ||
+    filter === undefined ||
+    (filter && item.domain.indexOf(filter) > -1) ||
+    (filter && item.filename.indexOf(filter) > -1)
+}
+
+/// Extract files
+/// - backup: the backup api object
+/// - destination: file system location
+/// - filter: contains check filter for files
+/// - items: list of files.
+function extractFiles (backup, destination, filter, items) {
+  for (var item of items) {
+    // Filter by the domain.
+    // Simple "Contains" Search
+    if (!isIncludedByFilter(filter, item)) {
+      // Skip to the next iteration of the loop.
+      log.action('skipped', item.filename)
+      continue
+    }
+
+    try {
+      let sourceFile = backup.getFileName(item.fileID)
+      var stat = fs.lstatSync(sourceFile)
+
+      // Only process files that exist.
+      if (stat.isFile() && fs.existsSync(sourceFile)) {
+        log.action('export', item.filename)
+
+        // Calculate the output dir.
+        var outDir = path.join(destination, item.domain, item.filename)
+
+        // Create the directory and copy
+        fs.ensureDirSync(path.dirname(outDir))
+        fs.copySync(sourceFile, outDir)
+
+        // Save output info to the data item.
+        item.output_dir = outDir
+      } else if (stat.isDirectory()) {
+      // Do nothing..
+      } else {
+        log.error('not found', sourceFile)
+      }
+    } catch (e) {
+      log.error(item.fileID, item.filename, e.toString())
+    }
+  }
+}

tools/reports/backup/info.js

@@ -0,0 +1,63 @@
+const fs = require('fs')
+const path = require('path')
+const plist = require('plist')
+const bplist = require('bplist-parser')
+
+const log = require('../../util/log')
+
+module.exports = {
+  version: 4,
+  name: 'backup.info',
+  description: `Gets a backup's info`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  async run (lib, { backup }) {
+    // Get the path for the info plist.
+    let infoPath = path.join(backup.path, 'Info.plist')
+
+    let fd = fs.openSync(infoPath, 'r')
+    let buffer = Buffer.alloc(7)
+    // Read the first 7 bytes into the buffer.
+    fs.readSync(fd, buffer, 0, 7, 0)
+    fs.closeSync(fd)
+
+    var data
+    // Binary plists have the marker 'bplist0'
+    if (buffer.toString('ascii') === 'bplist0') {
+      // Parse as binary plist
+      log.verbose('parsing manifest', infoPath)
+      data = bplist.parseBuffer(fs.readFileSync(infoPath))[0]
+
+      // Remove this data, it's kind of useless.
+      delete data['iTunes Files']
+    } else {
+      // Parse as normal plist.
+      log.verbose('parsing info', infoPath)
+      data = plist.parse(fs.readFileSync(infoPath, 'utf8'))
+
+      delete data['iTunes Files']
+    }
+
+    return data
+  },
+
+  // Public facing properties
+  output: {
+    buildVersion: el => el['Build Version'],
+    deviceName: el => el['Device Name'],
+    displayName: el => el['Display Name'],
+    guid: el => el['GUID'],
+    installedApplications: el => el['Installed Applications'],
+    lastBackupDate: el => el['Last Backup Date'],
+    productName: el => el['Product Name'],
+    productType: el => el['Product Type'],
+    productVersion: el => el['Product Version'],
+    serialNumber: el => el['Serial Number'],
+    targetIdentifier: el => el['Target Identifier'],
+    targetType: el => el['Target Type'],
+    uniqueIdentifier: el => el['Unique Identifier'],
+    iTunesSettings: el => el['iTunes Settings'],
+    iTunesVersion: el => el['iTunes Version']
+  }
+}

tools/reports/backup/manifest.js

@@ -0,0 +1,46 @@
+const fs = require('fs')
+const path = require('path')
+
+const log = require('../../util/log')
+const bplist = require('bplist-parser')
+
+module.exports = {
+  version: 4,
+  name: 'backup.manifest',
+  description: `Gets a backup's manifest plist`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  async run (lib, { backup }) {
+    // Load and parse the maniest for the backup.
+    log.verbose('parsing manifest', backup.path)
+    let data = bplist.parseBuffer(fs.readFileSync(path.join(backup.path, 'Manifest.plist')))[0]
+
+    // Remove this data, it's kind of useless.
+    delete data['BackupKeyBag']
+
+    return data
+  },
+
+  // Manifest fields.
+  output: {
+    SystemDomainsVersion: el => el.SystemDomainsVersion,
+    Applications: el => el.Applications,
+    Lockdown: el => {
+      el = el.Lockdown
+      return {
+        ProductVersion: el.ProductVersion,
+        BuildVersion: el.BuildVersion,
+        DeviceName: el.DeviceName,
+        SerialNumber: el.SerialNumber,
+        ProductType: el.ProductType,
+        UniqueDeviceID: el.UniqueDeviceID,
+        ...el
+      }
+    },
+    Version: el => el.Version,
+    IsEncrypted: el => el.IsEncrypted,
+    WasPasscodeSet: el => el.WasPasscodeSet,
+    Date: el => el.Date
+  }
+}

tools/reports/backup/status.js

@@ -0,0 +1,31 @@
+const fs = require('fs')
+const path = require('path')
+
+const log = require('../../util/log')
+const bplist = require('bplist-parser')
+
+module.exports = {
+  version: 4,
+  name: 'backup.status',
+  description: `Gets a backup's status`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  async run (lib, { backup }) {
+    // Load and parse status for the backup.
+    log.verbose('parsing status', backup.path)
+    let data = bplist.parseBuffer(fs.readFileSync(path.join(backup.path, 'Status.plist')))[0]
+
+    return data
+  },
+
+  // Status fields.
+  output: {
+    uuid: el => el.UUID,
+    isFullBackup: el => el.IsFullBackup,
+    version: el => el.Version, // backup version
+    backupState: el => el.BackupState,
+    date: el => el.Date,
+    snapshotState: el => el.SnapshotState
+  }
+}

tools/reports/backups/list.js

@@ -0,0 +1,39 @@
+const fs = require('fs-extra')
+
+module.exports = {
+  version: 4,
+  name: 'backups.list',
+  description: 'List of all backups',
+
+  run (lib) {
+    return new Promise(async (resolve, reject) => {
+      let files = fs.readdirSync(lib.base, { encoding: 'utf8' })
+        .filter(el => (el !== '.DS_Store'))
+
+      var results = []
+
+      // Iterate over the file list and try to get statuses for each backup.
+      for (let id of files) {
+        var result = { id }
+
+        result.status = await lib.run('backup.status', { backup: id }).catch(() => {}) || {}
+        result.info = await lib.run('backup.info', { backup: id }).catch(() => {}) || {}
+        result.manifest = await lib.run('backup.manifest', { backup: id }).catch(() => {}) || {}
+
+        results.push(result)
+      }
+
+      resolve(results)
+    })
+  },
+
+  output: {
+    udid: el => el.id,
+    encrypted: el => el.manifest ? (!!el.manifest.IsEncrypted) : false,
+    date: el => el.status ? new Date(el.status.date).toLocaleString() : '',
+    deviceName: el => el.manifest ? el.manifest.Lockdown.DeviceName : 'Unknown Device',
+    serialNumber: el => el.manifest.Lockdown.SerialNumber,
+    iOSVersion: el => el.manifest ? el.manifest.Lockdown.ProductVersion : '?',
+    backupVersion: el => el.status ? el.status.version : '?'
+  }
+}

tools/reports/bluetooth_devices.js

@@ -1,72 +0,0 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
-// Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
-
-const database_paired = fileHash('Library/Database/com.apple.MobileBluetooth.ledevices.paired.db', 'SysSharedContainerDomain-systemgroup.com.apple.bluetooth')
-const database_other  = fileHash('Library/Database/com.apple.MobileBluetooth.ledevices.other.db', 'SysSharedContainerDomain-systemgroup.com.apple.bluetooth')
-
-module.exports.name = 'bluetooth_devices'
-module.exports.description = 'List known bluetooth devices'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  bluetoothReport(backup)
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'Name': el => el.Name ? el.Name : 'N/A',
-          'Mac Address': el => {
-            let address = el.ResolvedAddress ? el.ResolvedAddress : el.Address ? el.Address : 'N/A'
-            address = address.indexOf(' ') !== -1 ? address.split(' ')[1] : address
-            return address
-          },
-          'Paired': el => el.Paired ? el.Paired : 'No'
-        }
-      })
-      resolve(result)
-    })
-    .catch(reject)
-}
-
-const bluetoothReport = (backup) => {
-  return new Promise((resolve, reject) => {
-    var paireddb = backup.getDatabase(database_paired)
-      try {
-        const query = `
-        select * from PairedDevices
-        `
-        paireddb.all(query, async function (err, rows) {
-          if (err) reject(err)
-          rows.forEach(row => row.Paired = 'Yes')
-          var otherdb = backup.getDatabase(database_other)
-          try {
-            const query = `
-            select * from OtherDevices
-            `
-            otherdb.all(query, async function (err, rows_other) {
-              if (err) reject(err)
-              rows_other.forEach(row_other => rows.push(row_other))
-              resolve(rows)
-            })
-          } catch (e) {
-            reject(e)
-          }
-        })
-      } catch (e) {
-        reject(e)
-      }
-  })
-}

tools/reports/calendar.js

@@ -1,56 +0,0 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
-// Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
-
-const database = fileHash('Library/Calendar/Calendar.sqlitedb')
-
-module.exports.name = 'calendar'
-module.exports.description = 'List calendar entries'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  calendarReport(backup)
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'Timestamp': el => (new Date((el.start_date + 978307200) * 1000).toDateString()) + ' ' + (new Date((el.start_date + 978307200) * 1000).toTimeString()) ,
-          'Title': el => el.summary,
-          'Content': el => el.description
-        }
-      })
-      resolve(result)
-    })
-    .catch(reject)
-}
-
-const calendarReport = (backup) => {
-  return new Promise((resolve, reject) => {
-    var calendardb = backup.getDatabase(database)
-      try {
-        const query = `
-        select * from CalendarItem
-        order by start_date
-        `
-        calendardb.all(query, async function (err, rows) {
-          if (err) reject(err)
-
-          resolve(rows)
-        })
-      } catch (e) {
-        reject(e)
-      }
-  })
-}

tools/reports/calendar/events.js

@@ -0,0 +1,48 @@
+// Derive filenames based on domain + file path
+const fileHash = require('../../util/backup_filehash')
+
+const CAL_DB = fileHash('Library/Calendar/Calendar.sqlitedb')
+
+module.exports = {
+  version: 4,
+  name: 'calendar.events',
+  description: `List all calendar entries`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return calendarReport(backup)
+  },
+
+  // Fields for apps report
+  output: {
+    timestamp: el => (new Date((el.start_date + 978307200) * 1000).toDateString()) + ' ' + (new Date((el.start_date + 978307200) * 1000).toTimeString()),
+    title: el => el.summary,
+    content: el => el.description,
+    calendarId: el => el.calendar_id,
+    calendarTitle: el => el.calendar_title
+  }
+}
+
+function calendarReport (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(CAL_DB)
+      .then(db => {
+        const query = `
+        SELECT
+          CalendarItem.*,
+          Calendar.title as calendar_title
+        FROM CalendarItem
+        LEFT JOIN Calendar ON
+          Calendar.ROWID = CalendarItem.calendar_id
+        ORDER BY start_date
+        `
+        db.all(query, async function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/calls.js

@@ -1,42 +0,0 @@
-module.exports.name = 'calls'
-module.exports.description = 'List all call records contained in the backup.'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works for iOS 10+
-module.exports.supportedVersions = '>=10.0'
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getCallsList()
-    .then((items) => {
-    // Use the configured formatter to print the rows.
-      const result = program.formatter.format(items, {
-      // Color formatting?
-        program: program,
-
-        // Columns to be displayed in human-readable printouts.
-        // Some formatters, like raw or CSV, ignore these.
-        columns: {
-          'ID': el => el.Z_PK,
-          'Date': el => el.XFORMATTEDDATESTRING,
-          'Answered': el => el.ZANSWERED + '',
-          'Originated': el => el.ZORIGINATED + '',
-          'Call Type': el => el.ZCALLTYPE + '',
-          'Duration': el => el.ZDURATION + '',
-          'Location': el => el.ZLOCATION + '',
-          'Country': el => el.ZISO_COUNTRY_CODE + '',
-          'Service': el => el.ZSERVICE_PROVIDER + '',
-          'Address': el => (el.ZADDRESS || '').toString()
-        }
-      })
-
-      // If using promises, we must call resolve()
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/calls_statistics.js

@@ -1,46 +0,0 @@
-module.exports.name = 'calls_statistics'
-module.exports.description = 'Get statistics about all calls'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// You can also provide an array of functions instead of using `module.exports.func`.
-// These functions *should* be independent ranges to ensure reliable execution
-module.exports.functions = {
-  '>=9.0': function (program, backup, resolve, reject) {
-    // This function would be called for iOS 10+
-    backup.getCallsStatistics()
-      .then((items) => {
-        var result = program.formatter.format(Object.entries(items[0]), {
-          program: program,
-          columns: {
-            'Key': el => el[0] + '',
-            'Value': el => el[1] + ''
-          }
-        })
-
-        resolve(result)
-      })
-      .catch(reject)
-  },
-  '>=1.0,<9.0': function (program, backup, resolve, reject) {
-    // This function would be called for all iOS 9.
-    backup.getCallsStatisticsiOS7()
-      .then((items) => {
-        var result = program.formatter.format(items, {
-          program: program,
-          columns: {
-            'Key': el => el.key + '',
-            'Value': el => el.value + ''
-          }
-        })
-
-        resolve(result)
-      })
-      .catch(reject)
-  }
-}

tools/reports/conversations.js

@@ -1,28 +0,0 @@
-module.exports.name = 'conversations'
-module.exports.description = 'List all SMS and iMessage conversations'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getConversations()
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'ID': el => el.ROWID,
-          'Date': el => el.XFORMATTEDDATESTRING || '??',
-          'Service': el => el.service_name + '',
-          'Chat Name': el => el.chat_identifier + '',
-          'Display Name': el => el.display_name + ''
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/cookies.js

@@ -1,38 +0,0 @@
-module.exports.name = 'cookies'
-module.exports.description = 'List all iOS cookies'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works for iOS 10+
-module.exports.supportedVersions = '>=10.0'
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getCookies()
-    .then((items) => {
-    // Use the configured formatter to print the rows.
-      const result = program.formatter.format(items, {
-      // Color formatting?
-        program: program,
-
-        // Columns to be displayed in human-readable printouts.
-        // Some formatters, like raw or CSV, ignore these.
-        columns: {
-          'domain': el => el.domain,
-          'url': el => el.cookie.url,
-          'path': el => el.cookie.name,
-          'value': el => el.cookie.value,
-          'creation': el => el.cookie.creation,
-          'expiration': el => el.cookie.expiration,
-          'flags': el => el.cookie.flags
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/list.js

@@ -1,35 +0,0 @@
-const iPhoneBackup = require('../util/iphone_backup.js').iPhoneBackup
-const fs = require('fs-extra')
-const chalk = require('chalk')
-
-module.exports.name = 'list'
-module.exports.description = 'List of all backups. alias for -l'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = false
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, base, resolve, reject) {
-  var items = fs.readdirSync(base, { encoding: 'utf8' })
-    .filter(el => (el !== '.DS_Store'))
-    .map(file => iPhoneBackup.fromID(file, base))
-    .filter(el => el.manifest && el.status)
-
-  var output = program.formatter.format(items, {
-    program: program,
-    columns: {
-      'UDID': el => el.id,
-      'Encryption': el => el.manifest ? (el.manifest.IsEncrypted ? 'encrypted' : 'not encrypted') : 'unknown',
-      'Date': el => el.status ? new Date(el.status.Date).toLocaleString() : '',
-      'Device Name': el => el.manifest ? el.manifest.Lockdown.DeviceName : 'Unknown Device',
-      'Serial #': el => el.manifest.Lockdown.SerialNumber,
-      'iOS Version': el => el.manifest ? el.manifest.Lockdown.ProductVersion : '?',
-      'Backup Version': el => el.status ? el.status.Version : '?'
-    }
-  })
-
-  resolve(output)
-}

tools/reports/manifest.js

@@ -1,150 +0,0 @@
-const fs = require('fs-extra')
-const log = require('../util/log')
-const path = require('path')
-module.exports.name = 'manifest'
-module.exports.description = 'List all the files contained in the backup (iOS 5+)'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works for iOS 5+
-module.exports.supportedVersions = '>=5.0'
-
-function isIncludedByFilter (program, item) {
-  return program.filter === 'all' ||
-    program.filter === undefined ||
-    (program.filter && item.domain.indexOf(program.filter) > -1)
-}
-
-// You can also provide an array of functions instead of using `module.exports.func`.
-// These functions *should* be independent ranges to ensure reliable execution
-module.exports.functions = {
-
-  '>=10.0': function (program, backup, resolve, reject) {
-    // This function would be called for iOS 10+
-
-    backup.getFileManifest()
-      .then((items) => {
-      // Extract items for analysis on-disk.
-        if (program.extract) {
-          for (var item of items) {
-          // Filter by the domain.
-          // Simple "Contains" Search
-            if (isIncludedByFilter(program, item)) {
-            // Do nothing, we'll process later.
-            } else {
-            // Skip to the next iteration of the loop.
-              log.action('skipped', item.relativePath)
-              continue
-            }
-
-            try {
-              var sourceFile = backup.getFileName(item.fileID)
-              var stat = fs.lstatSync(sourceFile)
-
-              // Only process files that exist.
-              if (stat.isFile() && fs.existsSync(sourceFile)) {
-                log.action('export', item.relativePath)
-
-                // Calculate the output dir.
-                var outDir = path.join(program.extract, item.domain, item.relativePath)
-
-                // Create the directory and copy
-                fs.ensureDirSync(path.dirname(outDir))
-                fs.copySync(sourceFile, outDir)
-
-                // Save output info to the data item.
-                item.output_dir = outDir
-              } else if (stat.isDirectory()) {
-              // Do nothing..
-              } else {
-                log.error('not found', item.relativePath)
-              }
-            } catch (e) {
-              log.error(item.relativePath, e.toString())
-            }
-          }
-
-          resolve([])
-        } else {
-          var result = program.formatter.format(items, {
-            program: program,
-            columns: {
-              'ID': el => el.fileID,
-              'Domain/Path': el => el.domain + ': ' + el.relativePath
-            }
-          })
-
-          resolve(result)
-        }
-      })
-      .catch((e) => {
-        console.log('[!] Encountered an Error:', e)
-      })
-  },
-
-  '>=5.0,<10.0': function (program, backup, resolve, reject) {
-    // This function would be called for all iOS 5 up to iOS 9.x.
-    backup.getOldFileManifest()
-      .then((items) => {
-      // Extract items for analysis on-disk.
-        if (program.extract) {
-          for (var item of items) {
-          // Filter by the domain.
-          // Simple "Contains" Search
-            if (isIncludedByFilter(program, item)) {
-            // Do nothing, we'll process later.
-            } else {
-            // Skip to the next iteration of the loop.
-              log.action('skipped', item.filename)
-              continue
-            }
-
-            try {
-              var sourceFile = backup.getFileName(item.fileID)
-              var stat = fs.lstatSync(sourceFile)
-
-              // Only process files that exist.
-              if (stat.isFile() && fs.existsSync(sourceFile)) {
-                log.action(item.filename)
-
-                // Calculate the output dir.
-                var outDir = path.join(program.extract, item.domain, item.filename)
-
-                // Create the directory and copy
-                fs.ensureDirSync(path.dirname(outDir))
-                fs.copySync(sourceFile, outDir)
-
-                // Save output info to the data item.
-                item.output_dir = outDir
-              } else if (stat.isDirectory()) {
-              // Do nothing..
-              } else {
-                log.error('not found', item.filename)
-              }
-            } catch (e) {
-              log.error(item.filename, e.toString())
-            }
-          }
-
-          resolve([])
-        } else {
-          var result = program.formatter.format(items, {
-            program: program,
-            columns: {
-              'ID': el => el.fileID,
-              'Domain/Path': el => (el.domain + ': ' + el.filename).substr(0, 70),
-              'Size': el => el.filelen
-            }
-          })
-
-          resolve(result)
-        }
-      })
-      .catch(reject)
-  }
-}

tools/reports/messages.js

@@ -1,39 +0,0 @@
-const log = require('../util/log')
-
-module.exports.name = 'messages'
-module.exports.description = 'List all SMS and iMessage messages in a conversation'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Should this report be skipped in automated reports?
-// This is used when the 'all' report type is specified, and all possible reports are generated.
-// with this set to true, the report WILL NOT run when report type = 'all'
-module.exports.requiresInteractivity = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  if (!program.id) {
-    log.error('use -i or --id <id> to specify conversation ID.')
-    process.exit(1)
-  }
-
-  backup.getMessages(program.id)
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'ID': el => el.ROWID,
-          'Date': el => el.XFORMATTEDDATESTRING,
-          'Sender': el => el.x_sender,
-          'Text': el => (el.text || '').trim()
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/messages/all.js

@@ -0,0 +1,37 @@
+
+module.exports = {
+  version: 4,
+  name: 'messages.all',
+  description: `List all SMS and iMessage conversations, nesting items. This may be SLOW, it is recommended you use a JSON formatter along with this.`,
+  requiresBackup: true,
+
+  // Available fields.
+  output: {
+    id: el => el.id,
+    date: el => el.date,
+    service: el => el.service,
+    chatName: el => el.chatName,
+    displayName: el => el.displayName,
+    messages: el => el.messages // see messages.messages report!
+  },
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return new Promise(async (resolve, reject) => {
+      try {
+        let conversations = await lib.run('messages.conversations', { backup })
+
+        for (var conversation of conversations) {
+          conversation.messages = await lib.run('messages.messages', {
+            backup,
+            id: conversation.id
+          })
+        }
+
+        resolve(conversations)
+      } catch (e) {
+        reject(e)
+      }
+    })
+  }
+}

tools/reports/messages/conversations.js

@@ -0,0 +1,109 @@
+const bplist = require('bplist-parser')
+
+const fileHash = require('../../util/backup_filehash')
+const log = require('../../util/log')
+
+const SMS_DB = fileHash('Library/SMS/sms.db')
+
+module.exports = {
+  version: 4,
+  name: 'messages.conversations',
+  description: `List all SMS and iMessage conversations`,
+  requiresBackup: true,
+
+  // Available fields.
+  output: {
+    id: el => el.ROWID,
+    date: el => el.XFORMATTEDDATESTRING || '??',
+    service: el => el.service_name + '',
+    chatName: el => el.chat_identifier + '',
+    displayName: el => el.display_name + ''
+  },
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return getConversations(backup)
+  }
+}
+
+function getConversationsiOS9 (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(SMS_DB)
+      .then(db => {
+        db.all(`SELECT *  FROM chat ORDER BY ROWID ASC`, async function (err, rows) {
+          if (err) return reject(err)
+          rows = rows || []
+
+          // We need to do some manual parsing of these records.
+          // The timestamp information is stored in a binary blob named `properties`
+          // Which is formatted as a binary PLIST.
+          for (var el of rows) {
+            if (el.properties) el.properties = bplist.parseBuffer(el.properties)[0]
+
+            // Interestingly, some of these do not have dates attached.
+            if (el.properties) {
+              el.date = new Date(el.properties.CKChatWatermarkTime * 1000)
+            } else {
+              el.date = new Date(0)
+            }
+
+            // Format as YY-MM-DD HH:MM:SS
+            try {
+              el.XFORMATTEDDATESTRING = el.date.toISOString()
+                .split('T')
+                .join(' ')
+                .split('Z')
+                .join(' ')
+                .split('.')[0]
+                .trim()
+            } catch (e) {
+              el.XFORMATTEDDATESTRING = ''
+            }
+          }
+
+          // Sort by the date.
+          rows = rows.sort(function (a, b) {
+            return (a.date.getTime() || 0) - (b.date.getTime() || 0)
+          })
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+function getConversationsiOS10iOS11 (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(SMS_DB)
+      .then(db => {
+        db.all(`SELECT *, datetime(last_read_message_timestamp / 1000000000 + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING FROM chat ORDER BY last_read_message_timestamp ASC`, async function (err, rows) {
+          if (err) return reject(err)
+          rows = rows || []
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+function getConversations (backup) {
+  return new Promise(async (resolve, reject) => {
+    try {
+      let conversations = await getConversationsiOS10iOS11(backup)
+      return resolve(conversations)
+    } catch (e) {
+      log.verbose('failed to read sms conversations as iOS10/11 format', e)
+    }
+
+    try {
+      let conversations = await getConversationsiOS9(backup)
+      return resolve(conversations)
+    } catch (e) {
+      log.verbose('failed to read sms conversations as iOS9 format', e)
+    }
+
+    reject(new Error('No suitable SMS database found. Use -v to see error informaton.'))
+  })
+}

tools/reports/conversations_full.js → tools/reports/messages/conversations_full.js

@@ -4,6 +4,7 @@ module.exports.description = 'List all SMS and iMessage conversations and their
 // Specify this reporter requires a backup.
 // The second parameter to func() is now a backup instead of the path to one.
 module.exports.requiresBackup = true
+module.exports.deprecated = true
 
 // Specify this reporter supports the promises API for allowing chaining of reports.
 module.exports.usesPromises = true

tools/reports/messages/messages.js

@@ -0,0 +1,142 @@
+const fileHash = require('../../util/backup_filehash')
+const log = require('../../util/log')
+
+const SMS_DB = fileHash('Library/SMS/sms.db')
+
+module.exports = {
+  version: 4,
+  name: 'messages.messages',
+  description: `List all SMS and iMessage messages in a conversation`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup, id }) {
+    return new Promise((resolve, reject) => {
+      if (id === undefined) {
+        return reject(new Error('You must specify an id for usage with the message report'))
+      }
+
+      resolve(getMessages(backup, id))
+    })
+  },
+
+  // Available fields.
+  output: {
+    id: el => el.ROWID,
+    date: el => el.XFORMATTEDDATESTRING,
+    sender: el => el.x_sender,
+    text: el => (el.text || '').trim(),
+    dateRead: el => el.date_read + '',
+    dateDelivered: el => el.date_delivered + '',
+    isDelivered: el => !!el.is_delivered,
+    isFinished: el => !!el.is_finished,
+    isFromMe: el => !!el.is_from_me,
+    isRead: el => !!el.is_read,
+    isSent: el => !!el.is_sent,
+    attachments: el => (el.attachments || []).map((at) => at.filename)
+  }
+}
+
+function getMessages (backup, chatId) {
+  return new Promise(async (resolve, reject) => {
+    try {
+      let messages = await getMessagesiOS10iOS11(backup, chatId)
+      resolve(messages)
+    } catch (e) {
+      log.verbose('iOS 10/11 messages lookup failed', e)
+    }
+
+    try {
+      let messages = await getMessagesiOS9(backup, chatId)
+      resolve(messages)
+    } catch (e) {
+      log.verbose('iOS 9 messages lookup failed', e)
+    }
+
+    reject(new Error('No Suitable messages database or query found. Use -v to see error information'))
+  })
+}
+
+function getMessagesiOS9 (backup, chatId) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(SMS_DB)
+      .then(db => {
+        db.all(`
+      SELECT 
+        message.*,
+        handle.id as sender_name,
+        datetime(date + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING
+      FROM chat_message_join 
+      INNER JOIN message 
+        ON message.rowid = chat_message_join.message_id 
+      INNER JOIN handle
+        ON handle.rowid = message.handle_id
+      WHERE chat_message_join.chat_id = ?`, [parseInt(chatId)],
+        async function (err, chats) {
+          if (err) return reject(err)
+
+          chats = chats || []
+
+          // Compute the user's name
+          for (var i in chats) {
+            var el = chats[i]
+            el.x_sender = el.is_from_me ? 'Me' : el.sender_name
+
+            // if (!el.is_from_me) {
+            //   var contact = await backup.getName(el.sender_name)
+
+            //   if (contact) {
+            //     el.x_sender = `${contact.name} <${contact.query}>`
+            //   }
+            // }
+          }
+
+          resolve(chats)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+function getMessagesiOS10iOS11 (backup, chatId) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(SMS_DB)
+      .then(db => {
+        db.all(`
+      SELECT 
+        message.*,
+        handle.id as sender_name,
+        datetime((date_read + 978307200), 'unixepoch') as date_read,
+        datetime((date_delivered + 978307200), 'unixepoch') as date_delivered,
+        datetime(date / 1000000000 + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING
+      FROM chat_message_join 
+      INNER JOIN message 
+        ON message.rowid = chat_message_join.message_id 
+      INNER JOIN handle
+        ON handle.rowid = message.handle_id
+      WHERE chat_message_join.chat_id = ?`, [parseInt(chatId)],
+        async function (err, chats) {
+          if (err) return reject(err)
+
+          chats = chats || []
+
+          // Compute the user's name
+          for (var i in chats) {
+            var el = chats[i]
+            el.x_sender = el.is_from_me ? 'Me' : el.sender_name
+
+            // if (!el.is_from_me) {
+            //   var contact = await backup.getName(el.sender_name)
+
+            //   if (contact) {
+            //     el.x_sender = `${contact.name} <${contact.query}>`
+            //   }
+            // }
+          }
+
+          resolve(chats)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/notes.js

@@ -1,31 +0,0 @@
-module.exports.name = 'notes'
-module.exports.description = 'List all iOS notes'
-
-// Specify this only works for iOS 9+
-module.exports.supportedVersions = '>=9.0'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getNotes(program.dump)
-    .then((items) => {
-      // Format the output
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'Modified': el => (el.XFORMATTEDDATESTRING || el.XFORMATTEDDATESTRING1) + '',
-          'ID': el => el.Z_PK,
-          'Title2': el => (el.ZTITLE2 + '').trim().substring(0, 128),
-          'Title1': el => (el.ZTITLE1 + '').trim() || ''
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/notes/notes.js

@@ -0,0 +1,109 @@
+const fileHash = require('../../util/backup_filehash')
+const log = require('../../util/log')
+
+const NOTES_DB = fileHash('Library/Notes/notes.sqlite')
+const NOTES2_DB = fileHash('NoteStore.sqlite', 'AppDomainGroup-group.com.apple.notes')
+
+module.exports = {
+  version: 4,
+  name: 'notes',
+  description: `List all iOS notes`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return getAllNotes(backup)
+  },
+
+  // Public facing properties
+  output: {
+    id: el => el.Z_PK,
+    identifier: el => el.ZIDENTIFIER,
+    modified: el => (el.XFORMATTEDDATESTRING || el.XFORMATTEDDATESTRING1) + '',
+    passwordProtected: el => !!el.ZISPASSWORDPROTECTED,
+    title: el => (el.ZTITLE || el.ZTITLE1 || el.ZTITLE2 || '').trim() || null,
+    content: el => el.ZCONTENT || null
+  }
+}
+
+function getAllNotes (backup) {
+  return new Promise(async (resolve, reject) => {
+    var newNotes
+
+    // Try iOS 10/11 query.
+    try {
+      newNotes = await getNewNotesiOS10iOS11(backup)
+    } catch (e) {
+      log.verbose(`couldn't query notes as iOS10/11, trying iOS9`, e)
+    }
+
+    // If iOS 10/11 query fails, try iOS 9.
+    if (newNotes == null) {
+      try {
+        newNotes = await getNewNotesiOS9(backup)
+      } catch (e) {
+        log.verbose(`couldn't query notes as iOS9`, e)
+      }
+    }
+
+    // Try to fetch old notes database
+    try {
+      var oldNotes = await getOldNotes(backup)
+    } catch (e) {
+      log.verbose(`couldn't query old notes`, e)
+    }
+
+    // If we didn't get anything successfully, reject.
+    if (newNotes == null && oldNotes == null) {
+      return reject(new Error(`Couldn't find any known notes database in the system.`))
+    }
+
+    // Join the notes together.
+    let result = [...(newNotes || []), ...(oldNotes || [])]
+    // console.log(result)
+    log.verbose(result)
+
+    resolve(result)
+  })
+}
+
+function getNewNotesiOS9 (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(NOTES2_DB)
+      .then(db => {
+        db.all(`SELECT ZICCLOUDSYNCINGOBJECT.*, ZICNOTEDATA.ZDATA as X_CONTENT_DATA, datetime(ZCREATIONDATE + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING FROM ZICCLOUDSYNCINGOBJECT LEFT JOIN ZICNOTEDATA ON ZICCLOUDSYNCINGOBJECT.ZNOTE = ZICNOTEDATA.ZNOTE`, async function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+function getOldNotes (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(NOTES_DB)
+      .then(db => {
+        db.all(`SELECT *, datetime(ZCREATIONDATE + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING from ZNOTE LEFT JOIN ZNOTEBODY ON ZBODY = ZNOTEBODY.Z_PK`, function (err, rows) {
+          if (err) reject(err)
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+function getNewNotesiOS10iOS11 (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(NOTES2_DB)
+      .then(db => {
+        db.all(`SELECT ZICCLOUDSYNCINGOBJECT.*, ZICNOTEDATA.ZDATA as X_CONTENT_DATA, datetime(ZCREATIONDATE + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING, datetime(ZCREATIONDATE1 + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING1 FROM ZICCLOUDSYNCINGOBJECT LEFT JOIN ZICNOTEDATA ON ZICCLOUDSYNCINGOBJECT.ZNOTE = ZICNOTEDATA.ZNOTE`, function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/oldnotes.js

@@ -1,26 +0,0 @@
-module.exports.name = 'oldnotes'
-module.exports.description = 'List all iOS notes (from older unused database)'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getOldNotes(program.dump)
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'Modified': el => el.XFORMATTEDDATESTRING,
-          'ID': el => el.Z_PK,
-          'Title': el => (el.ZTITLE + '').substring(0, 128)
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/phone/address_book.js

@@ -0,0 +1,109 @@
+module.exports = {
+  version: 4,
+  name: 'phone.address_book',
+  description: `List all address book records contained in the backup.`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return getAddressBook(backup)
+  },
+
+  // Manifest fields.
+  output: {
+    id: el => el.ROWID,
+    first: el => el.First || null,
+    last: el => el.Last || null,
+    organization: el => el.organization || null,
+    phoneWork: el => el.phone_work || null,
+    phoneMobile: el => el.phone_mobile || null,
+    phoneHome: el => el.phone_home || null,
+    email: el => el.email || null,
+    createdDate: el => el.created_date || null,
+    note: el => el.note || null,
+    picture: el => !!el.profile_picture
+  }
+}
+
+function getAddressBook (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(backup.getFileID('Library/AddressBook/AddressBook.sqlitedb'))
+      .then(db => {
+        // Query basic Address Book fields
+        const query = `
+      select ABPerson.ROWID
+          , ABPerson.first
+          , ABPerson.middle
+          , ABPerson.last
+          , ABPerson.Organization as organization
+          , ABPerson.Department as department
+          , ABPerson.Birthday as birthday
+          , ABPerson.JobTitle as jobtitle
+          , datetime(ABPerson.CreationDate + 978307200, 'unixepoch') as created_date 
+          , datetime(ABPerson.ModificationDate + 978307200, 'unixepoch') as updated_date 
+          , (select value from ABMultiValue where property = 3 and record_id = ABPerson.ROWID and label = (select ROWID from ABMultiValueLabel where value = '_$!<Work>!$_')) as phone_work
+          , (select value from ABMultiValue where property = 3 and record_id = ABPerson.ROWID and label = (select ROWID from ABMultiValueLabel where value = '_$!<Mobile>!$_')) as phone_mobile
+          , (select value from ABMultiValue where property = 3 and record_id = ABPerson.ROWID and label = (select ROWID from ABMultiValueLabel where value = '_$!<Home>!$_')) as phone_home
+
+          , (select value from ABMultiValue where property = 4 and record_id = ABPerson.ROWID) as email
+          
+          , (select value from ABMultiValueEntry where parent_id in (select ROWID from ABMultiValue where record_id = ABPerson.ROWID) and key = (select ROWID from ABMultiValueEntryKey where lower(value) = 'street')) as address
+          , (select value from ABMultiValueEntry where parent_id in (select ROWID from ABMultiValue where record_id = ABPerson.ROWID) and key = (select ROWID from ABMultiValueEntryKey where lower(value) = 'city')) as city
+          , ABPerson.Note as note
+        from ABPerson
+      order by ABPerson.ROWID
+      `
+        db.all(query, async function (err, rows) {
+          if (err) reject(err)
+          const iterateElements = (elements, index, callback) => {
+            if (index === elements.length) { return callback() }
+            // do parse call with element
+            let ele = elements[index]
+            // Query username and profile links for other services (facebook etc)
+            const query = `
+          select (select value from ABMultiValue where property = 22 and record_id = ABPerson.ROWID and label = (select ROWID from ABMultiValueLabel where value = 'PROFILE')) as google_profile
+              , (select value from ABMultiValue where property = 22 and record_id = ABPerson.ROWID and label = (select ROWID from ABMultiValueLabel where value = 'profile')) as google_profile1
+              , (select value from ABMultiValue where property = 4 and record_id = ABPerson.ROWID and label = (select ROWID from ABMultiValueLabel where value = 'iCloud')) as icloud
+              
+              , (select value from ABMultiValueEntry where parent_id in (select ROWID from ABMultiValue where record_id = ABPerson.ROWID) and key = (select ROWID from ABMultiValueEntryKey where lower(value) = 'service')) as service
+              , (select value from ABMultiValueEntry where parent_id in (select ROWID from ABMultiValue where record_id = ABPerson.ROWID) and key = (select ROWID from ABMultiValueEntryKey where lower(value) = 'username')) as username
+              , (select value from ABMultiValueEntry where parent_id in (select ROWID from ABMultiValue where record_id = ABPerson.ROWID) and key = (select ROWID from ABMultiValueEntryKey where lower(value) = 'url')) as url
+            from ABPerson
+            where ABPerson.ROWID = ${ele.ROWID}
+          order by ABPerson.ROWID
+          `
+            db.all(query, async function (err, rows1) {
+              if (err) return reject(err)
+              rows1[0].google_profile = rows1[0].google_profile || rows1[0].google_profile1
+              delete rows1[0].google_profile1
+              ele.services = rows1[0]
+
+              backup.openDatabase(backup.getFileID('Library/AddressBook/AddressBookImages.sqlitedb'))
+                .then(imageDB => {
+                  // Query profile picture extraction from /Library/AddressBook/AddressBookImages.sqlitedb
+                  const query = `
+                  select data
+                  from ABFullSizeImage
+                    where ABFullSizeImage.record_id = ${ele.ROWID}
+                  `
+                  imageDB.get(query, async function (err, row) {
+                    if (err) return reject(err)
+                    ele.profile_picture = null
+                    if (row) {
+                      ele.profile_picture = (row.data || '').toString('base64')
+                    }
+
+                    iterateElements(elements, index + 1, callback)
+                  })
+                })
+                .catch(reject)
+            })
+          }
+          iterateElements(rows, 0, () => {
+            resolve(rows)
+          })
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/phone/calls.js

@@ -0,0 +1,116 @@
+const fileHash = require('../../util/backup_filehash')
+const log = require('../../util/log')
+
+const CALLS_DB = '2b2b0084a1bc3a5ac8c27afdf14afb42c61a19ca'
+const CALLS2_DB = fileHash('Library/CallHistoryDB/CallHistory.storedata')
+
+module.exports = {
+  version: 4,
+  name: 'phone.calls',
+  description: `List all call records contained in the backup.`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return getCallsList(backup)
+  },
+
+  // Manifest fields.
+  output: {
+    id: el => el.Z_PK,
+    date: el => el.XFORMATTEDDATESTRING,
+    answered: el => !!el.ZANSWERED,
+    originated: el => !!el.ZORIGINATED,
+    callType: el => {
+      if (el.ZCALLTYPE === 1) {
+        return 'Cellular'
+      } else if (el.ZCALLTYPE === 16) {
+        return 'FacetimeAudio'
+      } else if (el.ZCALLTYPE === 8) {
+        return 'FacetimeVideo'
+      }
+
+      return 'Unknown'
+    },
+    duration: el => el.ZDURATION + '',
+    location: el => {
+      // console.log(el.ZLOCATION)
+      if (el.ZLOCATION === '<<RecentsNumberLocationNotFound>>') {
+        return null
+      }
+
+      return el.ZLOCATION + ''
+    },
+    country: el => el.ZISO_COUNTRY_CODE + '',
+    service: el => el.ZSERVICE_PROVIDER || null,
+    address: el => (el.ZADDRESS || '').toString()
+  }
+}
+
+function getCallsList (backup) {
+  return new Promise(async (resolve, reject) => {
+    try {
+      var ios7log = await getCallsListiOS7(backup)
+    } catch (e) {
+      log.verbose('tried ios7 calls', e)
+    }
+
+    try {
+      var newerIOS = await getCallsListLater(backup)
+    } catch (e) {
+      log.verbose('tried ios7+ calls', e)
+    }
+
+    // Check if they both are not found.
+    if (ios7log == null && newerIOS == null) {
+      return reject(new Error('no call logs found'))
+    }
+
+    // Resolve call logs.
+    resolve([...(ios7log || []), ...(newerIOS || [])])
+  })
+}
+
+// Try older databases.
+function getCallsListiOS7 (backup) {
+  return new Promise((resolve, reject) => {
+    // Attempt to open database.
+    backup.openDatabase(CALLS_DB)
+      .then(db => {
+        db.all(`
+        SELECT 
+          ROWID as Z_PK, 
+          datetime(date, 'unixepoch') AS XFORMATTEDDATESTRING, 
+          answered as ZANSWERED,
+          duration as ZDURATION,
+          address as ZADDRESS,
+          country_code as ZISO_COUNTRY_CODE, 
+          country_code as ZISO_COUNTRY_CODE, 
+          *
+        FROM call
+        ORDER BY date ASC`,
+        function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+// iOS 7+ moves to a new database.
+// Try that.
+function getCallsListLater (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(CALLS2_DB)
+      .then(db => {
+        db.all(`SELECT *, datetime(ZDATE + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING from ZCALLRECORD ORDER BY ZDATE ASC`, async function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/phone/calls_statistics.js

@@ -0,0 +1,126 @@
+const fileHash = require('../../util/backup_filehash')
+const log = require('../../util/log')
+
+const CALLS_DB = '2b2b0084a1bc3a5ac8c27afdf14afb42c61a19ca'
+const CALLS2_DB = fileHash('Library/CallHistoryDB/CallHistory.storedata')
+
+module.exports = {
+  version: 4,
+  name: 'phone.calls_statistics',
+  description: `Get statistics about all calls`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return getCallsStatistics(backup)
+  },
+
+  // Manifest fields.
+  // We need to find a value, so OR both of the data sources.
+  output: {
+    timerAll: el => el.timer_all || el.ZTIMER_ALL || 0,
+    timerIncoming: el => el.timer_incoming || el.ZTIMER_INCOMING || 0,
+    timerLast: el => el.timer_last || el.ZTIMER_LAST || 0,
+    timerOutgoing: el => el.timer_outgoing || el.ZTIMER_OUTGOING || 0,
+    timerLifetime: el => el.timer_lifetime || el.ZTIMER_LIFETIME || 0
+  }
+}
+
+function getCallsStatistics (backup) {
+  return new Promise(async (resolve, reject) => {
+    try {
+      var ios7stats = await getCallsStatisticsiOS7(backup)
+    } catch (e) {
+      log.verbose('tried ios7 stats', e)
+    }
+
+    try {
+      var newerIOS = await getCallsStatisticsLater(backup)
+    } catch (e) {
+      log.verbose('tried ios7+ stats', e)
+    }
+
+    // Check if they both are not found.
+    if (ios7stats == null && newerIOS == null) {
+      return reject(new Error('no call stats found'))
+    }
+
+    // Resolve call logs.
+    resolve({...(ios7stats || {}), ...(newerIOS || {})})
+  })
+}
+
+function getCallsStatisticsiOS7 (backup) {
+  /*
+    This resolves to a data object, similar to this:
+
+    {
+      call_history_limit: 100,
+      timer_last: 0,
+      timer_outgoing: 0,
+      timer_incoming: 0,
+      timer_all: 0,
+      timer_lifetime: 0,
+      timer_last_reset: 0,
+      kCallDBHasMigratedToCoreDataProperty: 0,
+      _ClientVersion: 13,
+      _UniqueIdentifier: '<uuid>'
+    }
+  */
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(CALLS_DB)
+      .then(db => {
+        db.all(`SELECT * from _SqliteDatabaseProperties`, function (err, rows) {
+          if (err) reject(err)
+
+          var result = {}
+          rows = rows || []
+
+          for (var item of rows) {
+            // Try to convert numbers to strings.
+            if (/^[+-]?\d+$/.test(item.value)) {
+              // Matches int regex
+              result[item.key] = parseInt(item.value)
+            } else if (/^[+-]?\d+\.\d+$/.test(item.value)) {
+              // Matches float regex
+              result[item.key] = parseFloat(item.value)
+            } else {
+              // Use existing value
+              result[item.key] = item.value
+            }
+          }
+
+          resolve(result)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+function getCallsStatisticsLater (backup) {
+  /*
+  This resolves to a data object, similar to:
+
+  {
+    Z_PK: 1,
+    Z_ENT: 1,
+    Z_OPT: 14,
+    ZTIMER_ALL: 0,
+    ZTIMER_INCOMING: 2135,
+    ZTIMER_LAST: 0,
+    ZTIMER_LIFETIME: 6590,
+    ZTIMER_OUTGOING: 4455
+  }
+  */
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(CALLS2_DB)
+      .then(db => {
+        db.all(`SELECT * from ZCALLDBPROPERTIES`, function (err, rows) {
+          if (err) reject(err)
+          rows = rows || []
+          resolve(rows[0])
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/phone/speed_dial.js

@@ -0,0 +1,51 @@
+
+const bplist = require('bplist-parser')
+const fs = require('fs')
+
+// Derive filenames based on domain + file path
+const fileHash = require('../../util/backup_filehash')
+
+const file = fileHash('Library/Preferences/com.apple.mobilephone.speeddial.plist')
+
+module.exports = {
+  version: 4,
+  name: 'phone.speed_dial',
+  description: `Show Speed dial contact information`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return speedDialReport(backup)
+  },
+
+  // Public facing properties
+  output: {
+    actionType: el => {
+      // Preprocess action type
+      if (el.ActionType) {
+        if (el.ActionType.indexOf('ActionType') !== -1) {
+          return el.ActionType.split('ActionType')[0]
+        } else {
+          return el.ActionType
+        }
+      } else {
+        return 'N/A'
+      }
+    },
+    contactName: el => el.Name,
+    value: el => el.Value
+  }
+}
+
+const speedDialReport = (backup) => {
+  return new Promise((resolve, reject) => {
+    try {
+      var filename = backup.getFileName(file)
+      let speeddialPlist = bplist.parseBuffer(fs.readFileSync(filename))[0]
+
+      resolve(speeddialPlist)
+    } catch (e) {
+      reject(e)
+    }
+  })
+}

tools/reports/phone/voicemail.js

@@ -0,0 +1,87 @@
+const path = require('path')
+const fs = require('fs-extra')
+
+// Derive filenames based on domain + file path
+const fileHash = require('../../util/backup_filehash')
+const log = require('../../util/log')
+
+const VOICEMAIL_DB = fileHash('Library/Voicemail/voicemail.db')
+
+module.exports = {
+  version: 4,
+  name: 'phone.voicemail',
+  description: `List all or extract voicemails on device`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup, extract }) {
+    return new Promise(async (resolve, reject) => {
+      try {
+        let voicemails = await getVoicemailsList(backup)
+
+        if (extract) {
+          extractVoicemails(voicemails, backup, extract)
+        }
+
+        resolve(voicemails)
+      } catch (e) {
+        reject(e)
+      }
+    })
+  },
+
+  // Manifest fields.
+  output: {
+    id: el => el.ROWID,
+    date: el => el.XFORMATTEDDATESTRING,
+    sender: el => el.sender,
+    token: el => el.token,
+    duration: el => el.duration,
+    expiration: el => el.expiration,
+    trashed: el => el.trashed_date,
+    flags: el => el.flags
+  }
+}
+
+// Extract a list of voicemails from a backup to a folder on disk.
+// - voicemails: list of voicemail data objects.
+// - backup: backup api object
+// - extractDest: extract location on disk
+function extractVoicemails (voicemails, backup, extractDest) {
+  for (let voicemail of voicemails) {
+    try {
+      // Get sender number
+      let sender = (voicemail.sender || '').replace(/[ +()-]/g, '')
+      var outDir = path.join(extractDest, `voicemail_${sender}_${voicemail.ROWID}.amr`)
+
+      // Get file hash of matching voicemail
+      let id = backup.getFileID(`Library/Voicemail/${voicemail.ROWID}.amr`)
+      var srcDir = backup.getFileName(id)
+
+      // Log the export
+      log.action('extract', srcDir)
+
+      // Ensure output dir exists
+      fs.ensureDirSync(path.dirname(outDir))
+
+      // Create a stream to the output.
+      fs.createReadStream(srcDir).pipe(fs.createWriteStream(outDir))
+    } catch (e) {
+      log.error(`Couldn't extract: ${srcDir}`, e)
+    }
+  }
+}
+
+// Get all voicemails from the db.
+function getVoicemailsList (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(VOICEMAIL_DB)
+      .then(db => {
+        db.all(`SELECT *, datetime(date, 'unixepoch') AS XFORMATTEDDATESTRING from voicemail ORDER BY date ASC`, function (err, rows) {
+          if (err) reject(err)
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/photolocations.js

@@ -1,32 +0,0 @@
-module.exports.name = 'photolocations'
-module.exports.description = 'List all geolocation information for iOS photos (iOS 10+)'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works for iOS 10+
-module.exports.supportedVersions = '>=10.0'
-
-// Reporting function
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getPhotoLocationHistory()
-    .then((history) => {
-      // Format the output according to the configured formatter.
-      var output = program.formatter.format(history, {
-        program: program,
-        columns: {
-          'Time': el => el.XFORMATTEDDATESTRING,
-          'Latitude': el => el.ZLATITUDE,
-          'Longitude': el => el.ZLONGITUDE,
-          'File': el => el.ZFILENAME
-        }
-      })
-
-      resolve(output)
-    })
-    .catch(reject)
-}

tools/reports/photos/locations.js

@@ -0,0 +1,43 @@
+const fileHash = require('../../util/backup_filehash')
+
+const PHOTOS_DB = fileHash('Media/PhotoData/Photos.sqlite', 'CameraRollDomain')
+
+module.exports = {
+  version: 4,
+  name: 'photos.locations',
+  description: `List all photo geotag's GPS locations`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return getPhotoLocationHistory(backup)
+  },
+
+  // Manifest fields.
+  output: {
+    time: el => el.XFORMATTEDDATESTRING,
+    latitude: el => el.ZLATITUDE,
+    longitude: el => el.ZLONGITUDE,
+    file: el => el.ZFILENAME
+  }
+}
+
+function getPhotoLocationHistory (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(PHOTOS_DB)
+      .then(db => {
+        db.all(`SELECT 
+          ZDATECREATED, 
+          ZLATITUDE, 
+          ZLONGITUDE,
+          ZFILENAME,
+          datetime(ZDATECREATED + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING 
+          FROM ZGENERICASSET ORDER BY ZDATECREATED ASC`, function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/pushstore.js

@@ -1,57 +0,0 @@
-module.exports.name = 'pushstore'
-module.exports.description = 'List pushstore files'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works up to iOS 10
-module.exports.supportedVersions = '<11.0'
-
-// You can also provide an array of functions instead of using `module.exports.func`.
-// These functions *should* be independent ranges to ensure reliable execution
-module.exports.functions = {
-
-  '>=10.0': function (program, backup, resolve, reject) {
-    // This function would be called for iOS 10+
-    backup.getPushstore()
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'AppNotificationCreationDate': el => el.AppNotificationCreationDate,
-          'AppNotificationTitle': el => el.AppNotificationTitle,
-          'AppNotificationMessage': el => el.AppNotificationMessage,
-          'RequestedDate': el => el.RequestedDate,
-          'TriggerDate': el => el.TriggerDate
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-  },
-
-  '>=5.0,<10.0': function (program, backup, resolve, reject) {
-    // This function would be called for all iOS 5 up to iOS 9.x.
-    backup.getOldPushstore()
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'AppNotificationCreationDate': el => el.AppNotificationCreationDate,
-          'AppNotificationTitle': el => el.AppNotificationTitle,
-          'AppNotificationMessage': el => el.AppNotificationMessage,
-          'RequestedDate': el => el.RequestedDate,
-          'TriggerDate': el => el.TriggerDate
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-  }
-}

tools/reports/safari/bookmarks.js

@@ -0,0 +1,146 @@
+const fileHash = require('../../util/backup_filehash')
+const log = require('../../util/log')
+
+const BOOKMARKS_DB = fileHash('Library/Safari/Bookmarks.db')
+
+module.exports = {
+  version: 4,
+  name: 'safari.bookmarks',
+  description: `List all Safari bookmarks`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object
+  run (lib, { backup }) {
+    return getAllBookmarks(backup)
+  },
+
+  // Available fields.
+  output: {
+    id: el => el.id,
+    title: el => el.title ? el.title.trim() : '',
+    url: el => el.url ? el.url.trim() : '',
+    parent: el => el.parent_title
+  }
+}
+
+function getAllBookmarks (backup) {
+  return new Promise(async (resolve, reject) => {
+    // Try to run newer version. If it fails, try older.
+    try {
+      let iOSBookmarks = await getSafariBookmarksLater(backup)
+      return resolve(iOSBookmarks)
+    } catch (e) {
+      log.verbose(`Couldn't load iOS7+ bookmarks`, e)
+    }
+
+    // Run older.
+    try {
+      let iOS7Bookmarks = await getSafariBookmarksiOS7(backup)
+      return resolve(iOS7Bookmarks)
+    } catch (e) {
+      log.verbose(`Couldn't load iOS7+ bookmarks`, e)
+    }
+
+    // Fail
+    reject(new Error('Could not find any bookmarks. Use -v to see error info.'))
+  })
+}
+
+// Get iOS Bookmarks for newer versions
+function getSafariBookmarksLater (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(BOOKMARKS_DB)
+      .then(db => {
+        const query = `
+        select bookmarks.id
+          , bookmarks.title
+          , bookmarks.url
+          , bookmarks.parent as parent_id
+          , bookmarks.special_id
+          , bookmarks.type
+          , bookmarks.num_children
+          , bookmarks.editable
+          , bookmarks.deletable
+          , bookmarks.hidden
+          , bookmarks.hidden_ancestor_count
+          , bookmarks.order_index
+          , bookmarks.external_uuid
+          , bookmarks.read
+          , bookmarks.last_modified
+          , bookmarks.server_id
+          , bookmarks.sync_key
+          , bookmarks.added
+          , bookmarks.deleted
+          , bookmarks.fetched_icon
+          , bookmarks.dav_generation
+          , bookmarks.locally_added
+          , bookmarks.archive_status
+          , bookmarks.syncable
+          , bookmarks.web_filter_status
+          , bookmarks.modified_attributes
+          , parent_bookmarks.title as parent_title
+        from bookmarks
+        left join bookmarks as parent_bookmarks on parent_bookmarks.id = bookmarks.parent
+        where bookmarks.type = 0
+        order by bookmarks.id
+      `
+        db.all(query, async function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+// Run older bookmark report
+function getSafariBookmarksiOS7 (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(BOOKMARKS_DB)
+      .then(db => {
+        const query = `
+        select bookmarks.id
+          , bookmarks.special_id
+          , bookmarks.parent as parent_id
+          , bookmarks.type
+          , bookmarks.title
+          , bookmarks.url
+          , bookmarks.num_children
+          , bookmarks.editable
+          , bookmarks.deletable
+          , bookmarks.hidden
+          , bookmarks.hidden_ancestor_count
+          , bookmarks.order_index
+          , bookmarks.external_uuid
+          , bookmarks.read
+          , bookmarks.last_modified
+          , bookmarks.server_id
+          , bookmarks.sync_key
+          , bookmarks.sync_data
+          , bookmarks.added
+          , bookmarks.deleted
+          , bookmarks.extra_attributes
+          , bookmarks.local_attributes
+          , bookmarks.fetched_icon
+          , bookmarks.icon
+          , bookmarks.dav_generation
+          , bookmarks.locally_added
+          , bookmarks.archive_status
+          , bookmarks.syncable
+          , bookmarks.web_filter_status
+          , parent_bookmarks.title as parent_title
+        from bookmarks
+        left join bookmarks as parent_bookmarks on parent_bookmarks.id = bookmarks.parent
+        where bookmarks.type = 0
+        order by bookmarks.id
+      `
+        db.all(query, async function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/safari/cookies.js

@@ -0,0 +1,63 @@
+// Cookie Parser
+const cookieParser = require('../../util/cookies.js')
+
+module.exports = {
+  version: 4,
+  name: 'safari.cookies',
+  description: `List all iOS cookies`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object
+  run (lib, { backup }) {
+    return new Promise(async (resolve, reject) => {
+      let files = await lib.run('backup.files', { backup, raw: true })
+
+      files = files.filter(el => {
+        return el.filename.indexOf('Library/Cookies/Cookies.binarycookies') > -1
+      })
+
+      resolve(getCookies(backup, files))
+    })
+  },
+
+  // Available fields.
+  output: {
+    domain: el => el.domain,
+    url: el => el.cookie.url,
+    path: el => el.cookie.name,
+    value: el => el.cookie.value,
+    creation: el => el.cookie.creation,
+    expiration: el => el.cookie.expiration,
+    flags: el => el.cookie.flags
+  }
+}
+
+// Find all the cookies in a set of files in a backup
+function getCookies (backup, files) {
+  return new Promise(async (resolve, reject) => {
+    // Cookies result
+    let cookiesResult = []
+
+    const iterateElements = (elements, index, callback) => {
+      if (index === elements.length) { return callback() }
+      // do parse call with element
+      var ele = elements[index]
+
+      cookieParser.parse(backup.getFileName(ele.fileID))
+        .then(cookies => {
+          // Map to include domain
+          let formatted = cookies.map(el => { return { domain: ele.domain, cookie: el } })
+
+          // Append result
+          cookiesResult = [...cookiesResult, ...formatted]
+
+          // Next file.
+          iterateElements(elements, index + 1, callback)
+        })
+    }
+
+    iterateElements(files, 0, () => {
+      resolve(cookiesResult)
+    })
+  })
+}

tools/reports/safari/open_tabs.js

@@ -0,0 +1,39 @@
+const fileHash = require('../../util/backup_filehash')
+
+const TABS_DB = fileHash('Library/Safari/BrowserState.db', 'AppDomain-com.apple.mobilesafari')
+
+module.exports = {
+  version: 4,
+  name: 'safari.open_tabs',
+  description: `List open Safari tabs when backup was made`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object
+  run (lib, { backup }) {
+    return openTabsReport(backup)
+  },
+
+  // Available fields.
+  output: {
+    title: el => el.title,
+    url: el => el.url,
+    lastViewedTime: el => (new Date((el.last_viewed_time + 978307200) * 1000).toDateString()) + ' ' + (new Date((el.last_viewed_time + 978307200) * 1000).toTimeString())
+  }
+}
+
+const openTabsReport = (backup) => {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(TABS_DB)
+      .then(db => {
+        db.all(`
+          select * from tabs
+          order by last_viewed_time DESC
+          `, function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/safari/recent_searches.js

@@ -0,0 +1,38 @@
+const fs = require('fs')
+const bplist = require('bplist-parser')
+const fileHash = require('../../util/backup_filehash')
+
+const SAFARI_PLIST = fileHash('Library/Preferences/com.apple.mobilesafari.plist', 'AppDomain-com.apple.mobilesafari')
+
+module.exports = {
+  version: 4,
+  name: 'safari.recent_searches',
+  description: `Show Safari recent searches`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object
+  run (lib, { backup }) {
+    return safariRecentSearches(backup)
+  },
+
+  // Available fields.
+  output: {
+    searchString: el => el.SearchString,
+    date: el => el.Date
+  }
+}
+
+// Pull the recent searches out of the file
+const safariRecentSearches = (backup) => {
+  return new Promise((resolve, reject) => {
+    try {
+      // Get the filename of the ID
+      var filename = backup.getFileName(SAFARI_PLIST)
+      let mobilesafariPlist = bplist.parseBuffer(fs.readFileSync(filename))[0]
+
+      resolve(mobilesafariPlist['RecentWebSearches'])
+    } catch (e) {
+      reject(e)
+    }
+  })
+}

tools/reports/safari/webhistory.js

@@ -0,0 +1,38 @@
+const { URL } = require('url')
+const fileHash = require('../../util/backup_filehash')
+const HISTORY_DB = fileHash('Library/Safari/History.db', 'AppDomain-com.apple.mobilesafari')
+
+module.exports = {
+  version: 4,
+  name: 'safari.webhistory',
+  description: `List all web history`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object
+  run (lib, { backup }) {
+    return getWebHistory(backup)
+  },
+
+  // Available fields.
+  output: {
+    timestamp: el => el.XFORMATTEDDATESTRING,
+    origin: el => new URL(el.url || '').origin || '',
+    url: el => el.url,
+    title: el => (el.title || '')
+  }
+}
+
+/// Get all web history entries.
+function getWebHistory (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(HISTORY_DB)
+      .then(db => {
+        db.all(`SELECT *, datetime(visit_time + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING from history_visits LEFT JOIN history_items ON history_items.ROWID = history_visits.history_item`, function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/safari_bookmarks.js

@@ -1,57 +0,0 @@
-module.exports.name = 'safari_bookmarks'
-module.exports.description = 'List all Safari bookmarks'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works for iOS 7+
-module.exports.supportedVersions = '>=7.0'
-
-// You can also provide an array of functions instead of using `module.exports.func`.
-// These functions *should* be independent ranges to ensure reliable execution
-module.exports.functions = {
-  '>=11.0': function (program, backup, resolve, reject) {
-    // This function would be called for iOS 10+
-    backup.getSafariBookmarks()
-      .then((items) => {
-        var result = program.formatter.format(items, {
-          program: program,
-          // Columns to be displayed in human-readable printouts.
-          // Some formatters, like raw or CSV, ignore these.
-          columns: {
-            'id': el => el.id,
-            'title': el => el.title ? el.title.trim() : '',
-            'url': el => el.url ? el.url.trim() : '',
-            'parent': el => el.parent_title
-          }
-        })
-
-        resolve(result)
-      })
-      .catch(reject)
-  },
-  '>=7.0,<11.0': function (program, backup, resolve, reject) {
-    // This function would be called for all iOS 7+.
-    backup.getSafariBookmarksiOS7()
-      .then((items) => {
-        var result = program.formatter.format(items, {
-          program: program,
-          // Columns to be displayed in human-readable printouts.
-          // Some formatters, like raw or CSV, ignore these.
-          columns: {
-            'id': el => el.id,
-            'title': el => el.title ? el.title.trim() : '',
-            'url': el => el.url ? el.url.trim() : '',
-            'parent': el => el.parent_title
-          }
-        })
-
-        resolve(result)
-      })
-      .catch(reject)
-  }
-}

tools/reports/safari_open_tabs.js

@@ -1,61 +0,0 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
-// Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
-
-const database = fileHash('Library/Safari/BrowserState.db', 'AppDomain-com.apple.mobilesafari')
-
-module.exports.name = 'safari_open_tabs'
-module.exports.description = 'List open Safari tabs when backup was made'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works for iOS 10+
-// If it is iOS-version specific, you can specify version information here.
-// You may provide a comma separated string such as ">=6.0,<11.0" to indicate ranges.
-module.exports.supportedVersions = '>=10.0'
-
-module.exports.func = function (program, backup, resolve, reject) {
-  openTabsReport(backup)
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'Title': el => el.title,
-          'URL': el => el.url,
-          'Last Viewed Time': el => (new Date((el.last_viewed_time + 978307200) * 1000).toDateString()) + ' ' + (new Date((el.last_viewed_time + 978307200) * 1000).toTimeString())
-        }
-      })
-      resolve(result)
-    })
-    .catch(reject)
-}
-
-const openTabsReport = (backup) => {
-  return new Promise((resolve, reject) => {
-    var browserStatedb = backup.getDatabase(database)
-      try {
-        const query = `
-        select * from tabs
-        order by last_viewed_time DESC
-        `
-        browserStatedb.all(query, async function (err, rows) {
-          if (err) reject(err)
-
-          resolve(rows)
-        })
-      } catch (e) {
-        reject(e)
-      }
-  })
-}

tools/reports/safari_recent_searches.js

@@ -1,56 +0,0 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
-
-// Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
-
-const file = fileHash('Library/Preferences/com.apple.mobilesafari.plist', 'AppDomain-com.apple.mobilesafari')
-
-module.exports.name = 'safari_recent_searches'
-module.exports.description = 'Show Safari recent searches'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works for iOS 10+
-// If it is iOS-version specific, you can specify version information here.
-// You may provide a comma separated string such as ">=6.0,<11.0" to indicate ranges.
-module.exports.supportedVersions = '>=10.0'
-
-module.exports.func = function (program, backup, resolve, reject) {
-  safariRecentSearches(backup)
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: { 
-          'SearchString': el => el.SearchString,
-          'Date': el => el.Date
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}
-
-const safariRecentSearches = (backup) => {
-  return new Promise((resolve, reject) => {
-    var filename = backup.getFileName(file)
-    try {
-      let mobilesafariPlist = bplist.parseBuffer(fs.readFileSync(filename))[0]
-      
-      resolve(mobilesafariPlist['RecentWebSearches'])
-    } catch (e) {
-      reject(e)
-    }
-  })
-}

tools/reports/speed_dial.js

@@ -1,52 +0,0 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
-
-// Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
-
-const file = fileHash('Library/Preferences/com.apple.mobilephone.speeddial.plist')
-
-module.exports.name = 'speed_dial'
-module.exports.description = 'Show Speed dial contact information'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  speedDialReport(backup)
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: { 
-          'Action Type': el => el.ActionType ? el.ActionType.indexOf('ActionType') !== -1 ? el.ActionType.split('ActionType')[0] : el.ActionType : 'N/A',
-          'Contact Name': el => el.Name,
-          'Value': el => el.Value
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}
-
-const speedDialReport = (backup) => {
-  return new Promise((resolve, reject) => {
-    var filename = backup.getFileName(file)
-    try {
-      let speeddialPlist = bplist.parseBuffer(fs.readFileSync(filename))[0]
-      
-      resolve(speeddialPlist)
-    } catch (e) {
-      reject(e)
-    }
-  })
-}

tools/reports/system/apps.js

@@ -0,0 +1,35 @@
+module.exports = {
+  version: 4,
+  name: 'system.apps',
+  description: `List all installed applications and container IDs.`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return new Promise(async (resolve, reject) => {
+      try {
+        // This report directly depends on manifest report.
+        // If it fails, so do we.
+        let manifest = await lib.run('backup.manifest', { backup, raw: true })
+
+        // Fetch each app in the manifest.
+        var apps = []
+        for (var key in manifest.Applications) {
+          var app = manifest.Applications[key]
+
+          apps.push({ bundleID: app.CFBundleIdentifier, path: app.Path })
+        }
+
+        resolve(apps)
+      } catch (e) {
+        reject(e)
+      }
+    })
+  },
+
+  // Fields for apps report
+  output: {
+    bundleID: el => el.bundleID || null,
+    path: el => el.path || null
+  }
+}

tools/reports/system/bluetooth_devices.js

@@ -0,0 +1,85 @@
+const fileHash = require('../../util/backup_filehash')
+const log = require('../../util/log')
+
+const PAIRED_DB = fileHash('Library/Database/com.apple.MobileBluetooth.ledevices.paired.db', 'SysSharedContainerDomain-systemgroup.com.apple.bluetooth')
+const OTHER_DB = fileHash('Library/Database/com.apple.MobileBluetooth.ledevices.other.db', 'SysSharedContainerDomain-systemgroup.com.apple.bluetooth')
+
+module.exports = {
+  version: 4,
+  name: 'system.bluetooth_devices',
+  description: `List known bluetooth devices`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object
+  run (lib, { backup }) {
+    return getBluetoothDevices(backup)
+  },
+
+  // Available fields.
+  output: {
+    uuid: el => el.Uuid || null,
+    name: el => el.Name ? el.Name : 'N/A',
+    macAddress: el => {
+      let address = el.ResolvedAddress ? el.ResolvedAddress : el.Address ? el.Address : 'N/A'
+      address = address.indexOf(' ') !== -1 ? address.split(' ')[1] : address
+      return address
+    },
+    lastConnected: el => el.LastConnectionTime || 0,
+    lastSeen: el => el.LastSeenTime || 0,
+    paired: el => el.Paired ? el.Paired : 'No'
+  }
+}
+
+// Get the bluetooth devices in a backup.
+function getBluetoothDevices (backup) {
+  return new Promise(async (resolve, reject) => {
+    // Get paired devices
+    try {
+      var paired = await getPairedDevices(backup)
+    } catch (e) {
+      log.verbose(`couldn't get paired devices`, e)
+    }
+
+    // Get other devices
+    try {
+      var other = await getOtherDevices(backup)
+    } catch (e) {
+      log.verbose(`couldn't get paired devices`, e)
+    }
+
+    // console.log(paired, other)
+    resolve([...(paired || []), ...(other || [])])
+  })
+}
+
+// Get devies we've paired with
+function getPairedDevices (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(PAIRED_DB)
+      .then(paired => {
+        const query = `SELECT *, 'Yes' as Paired FROM PairedDevices`
+        paired.all(query, function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}
+
+// Get other devices we've seen.
+function getOtherDevices (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(OTHER_DB)
+      .then(paired => {
+        const query = `SELECT * FROM OtherDevices`
+        paired.all(query, function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/system/geofences.js

@@ -0,0 +1,37 @@
+const fileHash = require('../../util/backup_filehash')
+
+const GEO_DB = fileHash('Library/Caches/locationd/consolidated.db', 'RootDomain')
+
+module.exports = {
+  version: 4,
+  name: 'system.geofences',
+  description: `List local geofences used for triggers`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return getGeoFences(backup)
+  },
+
+  // Wifi Report Fields.
+  output: {
+    date: el => el.XFORMATTEDDATESTRING,
+    latitude: el => el.Latitude,
+    longitude: el => el.Longitude,
+    distance: el => el.Distance
+  }
+}
+
+function getGeoFences (backup) {
+  return new Promise((resolve, reject) => {
+    backup.openDatabase(GEO_DB)
+      .then(db => {
+        db.all(`SELECT datetime(Timestamp + 978307200, 'unixepoch') AS XFORMATTEDDATESTRING, Latitude, Longitude, Distance FROM Fences ORDER BY Timestamp ASC`, async function (err, rows) {
+          if (err) reject(err)
+
+          resolve(rows)
+        })
+      })
+      .catch(reject)
+  })
+}

tools/reports/system/pushstore.js

@@ -0,0 +1,49 @@
+const fs = require('fs')
+const bplist = require('bplist-parser')
+const pushstoreParse = require('../../util/pushstore_parse')
+
+module.exports = {
+  version: 4,
+  name: 'system.pushstore',
+  description: `List pushstore contents`,
+  requiresBackup: true,
+
+  // Available fields.
+  output: {
+    appNotificationCreationDate: el => el.AppNotificationCreationDate,
+    appNotificationTitle: el => el.AppNotificationTitle,
+    appNotificationMessage: el => el.AppNotificationMessage,
+    requestedDate: el => el.RequestedDate,
+    triggerDate: el => el.TriggerDate
+  },
+
+  // Run on a v3 lib / backup object
+  run (lib, { backup }) {
+    return new Promise(async (resolve, reject) => {
+      try {
+        // Run files report as a sub-report.
+        let files = await lib.run('backup.files', { backup })
+
+        files = files.filter((file) => {
+          if (file.path) {
+            return ~file.path.indexOf('Library/SpringBoard/PushStore/')
+          }
+          return false
+        })
+
+        // Collect the push stores
+        const pushstores = []
+
+        // For each file, run a parse on the plist.
+        files.forEach((file) => {
+          let plist = bplist.parseBuffer(fs.readFileSync(backup.getFileName(file.id)))[0]
+          pushstores.push(...pushstoreParse.run(plist))
+        })
+
+        resolve(pushstores)
+      } catch (e) {
+        reject(e)
+      }
+    })
+  }
+}

tools/reports/system/wifi.js

@@ -0,0 +1,51 @@
+const bplist = require('bplist-parser')
+const fs = require('fs')
+
+// Normalize mac addresses in wifi output
+const macParse = require('../../util/mac_address_parse')
+
+// Derive filenames based on domain + file path
+const fileHash = require('../../util/backup_filehash')
+
+const WIFI_PLIST = fileHash('SystemConfiguration/com.apple.wifi.plist', 'SystemPreferencesDomain')
+
+module.exports = {
+  version: 4,
+  name: 'system.wifi',
+  description: `List associated wifi networks and their usage information`,
+  requiresBackup: true,
+
+  // Run on a v3 lib / backup object.
+  run (lib, { backup }) {
+    return new Promise((resolve, reject) => {
+      try {
+        // Get the fifi file
+        var filename = backup.getFileName(WIFI_PLIST)
+
+        // Attempt to parse it
+        let wifiList = bplist.parseBuffer(fs.readFileSync(filename))[0]
+        let result = wifiList['List of known networks']
+          .map(el => {
+            if (el.BSSID) {
+              el.BSSID = macParse.pad_zeros(el.BSSID) + ''
+            }
+            return el
+          })
+        resolve(result)
+      } catch (e) {
+        reject(e)
+      }
+    })
+  },
+
+  // Wifi Report Fields.
+  output: {
+    lastJoined: el => el.lastJoined,
+    lastAutoJoined: el => el.lastAutoJoined || '',
+    ssid: el => el.SSID_STR,
+    bssid: el => el.BSSID,
+    security: el => el.SecurityMode || '',
+    hidden: el => !!el.HIDDEN_NETWORK,
+    enabled: el => !!el.enabled
+  }
+}

tools/reports/facebook_messenger_friends.js → tools/reports/thirdparty/facebook/messenger.js

@@ -1,12 +1,7 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
+const log = require('../../../util/log')
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const domain = 'AppDomainGroup-group.com.facebook.Messenger'
 
@@ -34,8 +29,7 @@ module.exports.functions = {
       .then((items) => {
         let filename = 'fbomnistore.db'
         let fileitem = items.find((file) => {
-          if (file && file.relativePath)
-            return ~file.relativePath.indexOf(filename)
+          if (file && file.relativePath) { return ~file.relativePath.indexOf(filename) }
           return false
         })
         if (fileitem) {
@@ -47,7 +41,7 @@ module.exports.functions = {
       .then((items) => {
         var result = program.formatter.format(items, {
           program: program,
-          columns: { 
+          columns: {
             'Facebook Friend Usernames': el => el.field_value
           }
         })
@@ -55,25 +49,25 @@ module.exports.functions = {
         resolve(result)
       })
       .catch((e) => {
-        console.log('[!] Encountered an Error:', e)
+        log.error('[!] Encountered an Error:', e)
       })
   },
 
   '>=5.0,<10.0': function (program, backup, resolve, reject) {
     // This function would be called for all iOS 5 up to iOS 9.x.
     // TODO
-    /*backup.getOldFileManifest()
+    /* backup.getOldFileManifest()
       .then((items) => {
         var result = program.formatter.format(items, {
           program: program,
-          columns: { 
+          columns: {
             'Facebook Friend Username': el => el.field_value
           }
         })
 
         resolve(result)
       })
-      .catch(reject)*/
+      .catch(reject) */
   }
 }
 
@@ -88,20 +82,21 @@ const facebookMessengerFriendsReport = (backup, file) => {
       AND name LIKE 'collection_index#messenger_contacts_ios%' 
       LIMIT 1
       `,
-      (err, table_name) => {
-        table_name = table_name.name
-        console.log("Table", table_name)
+      (err, tableName) => {
+        if (err) return reject(err)
+        tableName = tableName.name
+        log.verbose('Table', tableName)
         database.all(`
         SELECT field_value 
-        FROM '${table_name}' 
+        FROM '${tableName}' 
         WHERE field_name='username'
         `, (err, rows) => {
+          if (err) return reject(err)
           resolve(rows)
         })
       })
-
     } catch (e) {
       reject(e)
     }
   })
-}
+}

tools/reports/facebook_profile.js → tools/reports/thirdparty/facebook/profile.js

@@ -1,13 +1,8 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
 const fs = require('fs')
-const plist = require('plist')
-
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const file = fileHash('Library/Preferences/com.facebook.Messenger.plist', 'AppDomain-com.facebook.Messenger')
 
@@ -26,7 +21,7 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       var result = program.formatter.format(items, {
         program: program,
-        columns: { 
+        columns: {
           'Facebook User ID': el => el.fbid
         }
       })
@@ -51,4 +46,4 @@ const facebookProfileReport = (backup) => {
       reject(e)
     }
   })
-}
+}

tools/reports/gmail_accounts.js → tools/reports/thirdparty/gmail/accounts.js

@@ -1,13 +1,8 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
 const fs = require('fs')
-const plist = require('plist')
-
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const file = fileHash('Library/Preferences/group.com.google.Gmail.plist', 'AppDomainGroup-group.com.google.Gmail')
 
@@ -26,10 +21,10 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       var result = program.formatter.format(items, {
         program: program,
-        columns: { 
+        columns: {
           'Id': el => el.id,
           'Email': el => el.email,
-          'Avatar': el => el.avatar
+          'Avatar': el => el.avatar || null
         }
       })
 
@@ -43,8 +38,8 @@ const gmailAccountsReport = (backup) => {
     var filename = backup.getFileName(file)
     try {
       let gmailPlist = bplist.parseBuffer(fs.readFileSync(filename))[0]
-      let gmailAccountIds = Object.keys(gmailPlist).filter(key => key.indexOf('kIdToEmailMapKey') !== -1);
-      let gmailAvatars = Object.keys(gmailPlist).filter(key => key.indexOf('kCurrentAvatarUrlKey') !== -1);
+      let gmailAccountIds = Object.keys(gmailPlist).filter(key => key.indexOf('kIdToEmailMapKey') !== -1)
+      let gmailAvatars = Object.keys(gmailPlist).filter(key => key.indexOf('kCurrentAvatarUrlKey') !== -1)
       gmailAvatars = gmailAvatars.map(avatarKey => {
         let id = avatarKey.split('kCurrentAvatarUrlKey')[1].split('-')
         id = id[id.length - 1]
@@ -56,16 +51,17 @@ const gmailAccountsReport = (backup) => {
 
       gmailAccountIds = gmailAccountIds.map(key => {
         const split = key.split('-')
+        let avatar = gmailAvatars.find(avatar => avatar.accountId === split[split.length - 1])
         return {
           id: split[split.length - 1],
           email: gmailPlist[key],
-          avatar: gmailPlist[gmailAvatars.find(avatar => avatar.accountId === split[split.length - 1]).avatarKey]
+          avatar: gmailPlist[(avatar || {}).avatarKey]
         }
-      });
+      })
 
       resolve(gmailAccountIds)
     } catch (e) {
       reject(e)
     }
   })
-}
+}

tools/reports/gmail_shared_contacts.js → tools/reports/thirdparty/gmail/shared_contacts.js

@@ -1,13 +1,8 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
 const fs = require('fs')
-const plist = require('plist')
-
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const file = fileHash('Library/Preferences/group.com.google.Gmail.plist', 'AppDomainGroup-group.com.google.Gmail')
 
@@ -26,7 +21,7 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       var result = program.formatter.format(items, {
         program: program,
-        columns: { 
+        columns: {
           'Account': el => el.account,
           'Name': el => el.name,
           'Email': el => el.email,
@@ -44,8 +39,8 @@ const gmailAccountsReport = (backup) => {
     var filename = backup.getFileName(file)
     try {
       let gmailPlist = bplist.parseBuffer(fs.readFileSync(filename))[0]
-      let gmailAccountIds = Object.keys(gmailPlist).filter(key => key.indexOf('kIdToEmailMapKey') !== -1);
-      let gmailContactsByAccount = Object.keys(gmailPlist).filter(key => key.indexOf('kInboxSharedStorageContacts') !== -1);
+      let gmailAccountIds = Object.keys(gmailPlist).filter(key => key.indexOf('kIdToEmailMapKey') !== -1)
+      let gmailContactsByAccount = Object.keys(gmailPlist).filter(key => key.indexOf('kInboxSharedStorageContacts') !== -1)
       gmailContactsByAccount = gmailContactsByAccount.map(contactsKey => {
         let id = contactsKey.split('kInboxSharedStorageContacts')[1].split('_')
         id = id[id.length - 1]
@@ -57,15 +52,18 @@ const gmailAccountsReport = (backup) => {
 
       gmailAccountIds = gmailAccountIds.map(key => {
         const split = key.split('kIdToEmailMapKey-')
+        let contacts = gmailContactsByAccount.find(contacts => contacts.accountId === split[split.length - 1])
         return {
           id: split[split.length - 1],
           email: gmailPlist[key],
-          contacts: gmailPlist[gmailContactsByAccount.find(contacts => contacts.accountId === split[split.length - 1]).contactsKey]
+          contacts: gmailPlist[(contacts || {}).contactsKey]
         }
-      });
+      })
 
       let contacts = []
       gmailAccountIds.forEach(gmailAccount => {
+        gmailAccount.contacts = gmailAccount.contacts || []
+
         gmailAccount.contacts.forEach(contact => {
           contacts.push({
             account: gmailAccount.email,
@@ -81,4 +79,4 @@ const gmailAccountsReport = (backup) => {
       reject(e)
     }
   })
-}
+}

tools/reports/instagram_fb_friends.js → tools/reports/thirdparty/instagram/fb_friends.js

@@ -1,13 +1,8 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
 const fs = require('fs')
-const plist = require('plist')
-
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const file = fileHash('Library/Preferences/group.com.burbn.instagram.plist', 'AppDomainGroup-group.com.burbn.instagram')
 
@@ -26,7 +21,7 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       var result = program.formatter.format(items, {
         program: program,
-        columns: { 
+        columns: {
           'Fb_id': el => el.fb_id,
           'Name': el => el.full_name,
           'Profile Pic': el => el.profile_pic_url,
@@ -45,7 +40,7 @@ const instagramRecentSearchesReport = (backup) => {
     var filename = backup.getFileName(file)
     try {
       let instagramPlist = bplist.parseBuffer(fs.readFileSync(filename))[0]
-      let regex = /[0-9]*-fb-friends$/g;
+      let regex = /[0-9]*-fb-friends$/g
       let fbFriendsKey = Object.keys(instagramPlist).filter(key => regex.test(key))
       console.log(fbFriendsKey)
       fbFriendsKey.forEach(key => {
@@ -58,4 +53,4 @@ const instagramRecentSearchesReport = (backup) => {
       reject(e)
     }
   })
-}
+}

tools/reports/instagram_following_users_coded.js → tools/reports/thirdparty/instagram/following_users_coded.js

@@ -1,13 +1,8 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
 const fs = require('fs')
-const plist = require('plist')
-
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const file = fileHash('Library/Preferences/group.com.burbn.instagram.plist', 'AppDomainGroup-group.com.burbn.instagram')
 
@@ -26,7 +21,7 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       var result = program.formatter.format(items, {
         program: program,
-        columns: { 
+        columns: {
           'Identifier': el => el
         }
       })
@@ -53,4 +48,4 @@ const instagramRecentSearchesReport = (backup) => {
       reject(e)
     }
   })
-}
+}

tools/reports/instagram_profile.js → tools/reports/thirdparty/instagram/profile.js

@@ -1,13 +1,8 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
 const fs = require('fs')
-const plist = require('plist')
-
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const file = fileHash('Library/Preferences/com.burbn.instagram.plist', 'AppDomain-com.burbn.instagram')
 
@@ -26,7 +21,7 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       var result = program.formatter.format(items, {
         program: program,
-        columns: { 
+        columns: {
           'Key': el => el.key,
           'Value': el => el.value
         }
@@ -48,7 +43,7 @@ const instagramProfileReport = (backup) => {
     var filename = backup.getFileName(file)
     try {
       let instagramPlist = bplist.parseBuffer(fs.readFileSync(filename))[0]
-      
+
       results.push(new KeyValue('last-logged-in-username', instagramPlist))
       results.push(new KeyValue('prefill_fb_email', instagramPlist))
       results.push(new KeyValue('prefill_fb_phone', instagramPlist))
@@ -58,4 +53,4 @@ const instagramProfileReport = (backup) => {
       reject(e)
     }
   })
-}
+}

tools/reports/instagram_recent_searches.js → tools/reports/thirdparty/instagram/recent_searches.js

@@ -1,13 +1,8 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
 const fs = require('fs')
-const plist = require('plist')
-
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const file = fileHash('Library/Preferences/group.com.burbn.instagram.plist', 'AppDomainGroup-group.com.burbn.instagram')
 
@@ -26,7 +21,7 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       var result = program.formatter.format(items, {
         program: program,
-        columns: { 
+        columns: {
           'Type': el => el.type,
           'Identifier': el => el.identifier
         }
@@ -54,4 +49,4 @@ const instagramRecentSearchesReport = (backup) => {
       reject(e)
     }
   })
-}
+}

tools/reports/skype_accounts.js → tools/reports/thirdparty/skype/accounts.js

@@ -1,12 +1,5 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const domain = 'AppDomain-com.skype.skype'
 
@@ -25,8 +18,9 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       let filename = 'main.db'
       let fileitem = items.find((file) => {
-        if (file && file.relativePath)
-          return ~file.relativePath.indexOf(filename) && file.domain == domain
+        if (file && file.relativePath) {
+          return ~file.relativePath.indexOf(filename) && file.domain === domain
+        }
         return false
       })
       if (fileitem) {
@@ -38,7 +32,7 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       var result = program.formatter.format(items, {
         program: program,
-        columns: { 
+        columns: {
           'Skype Name': el => el.skypename
         }
       })
@@ -59,11 +53,11 @@ const skypeAccountsReport = (backup, file) => {
       FROM Accounts 
       `,
       (err, rows) => {
+        if (err) resolve(err)
         resolve(rows)
       })
-
     } catch (e) {
       reject(e)
     }
   })
-}
+}

tools/reports/skype_calls.js → tools/reports/thirdparty/skype/calls.js

@@ -1,12 +1,5 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const domain = 'AppDomain-com.skype.skype'
 
@@ -25,8 +18,9 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       let filename = 'main.db'
       let fileitem = items.find((file) => {
-        if (file && file.relativePath)
-          return ~file.relativePath.indexOf(filename) && file.domain == domain
+        if (file && file.relativePath) {
+          return ~file.relativePath.indexOf(filename) && file.domain === domain
+        }
         return false
       })
       if (fileitem) {
@@ -38,7 +32,7 @@ module.exports.func = function (program, backup, resolve, reject) {
     .then((items) => {
       var result = program.formatter.format(items, {
         program: program,
-        columns: { 
+        columns: {
           'Begin Timestamp': el => (new Date((el.begin_timestamp) * 1000).toDateString()) + ' ' + (new Date((el.begin_timestamp) * 1000).toTimeString()),
           'Host Identity': el => el.host_identity,
           'Duration': el => el.duration ? el.duration : 'N/A',
@@ -63,11 +57,11 @@ const skypeAccountsReport = (backup, file) => {
       FROM Calls 
       `,
       (err, rows) => {
+        if (err) return reject(err)
         resolve(rows)
       })
-
     } catch (e) {
       reject(e)
     }
   })
-}
+}

tools/reports/spotify.js → tools/reports/thirdparty/spotify/searches.js

@@ -1,16 +1,12 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
 const fs = require('fs')
-const plist = require('plist')
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const database = fileHash('Library/Preferences/com.spotify.client.plist', 'AppDomain-com.spotify.client')
 
-module.exports.name = 'spotify'
+module.exports.name = 'spotify.searches'
 module.exports.description = 'List associated Spotify account and its usage information'
 
 // Specify this reporter requires a backup.
@@ -43,24 +39,17 @@ const spotifyReport = (backup) => {
     try {
       let spotifyData = bplist.parseBuffer(fs.readFileSync(filename))[0]
       let spotifyResult = []
-      /*
-      wifiList['List of known networks'] = wifiList['List of known networks']
-        .map(el => {
-          if (el.BSSID) {
-            el.BSSID = macParse.pad_zeros(el.BSSID) + ''
-          }
-          return el
-        })*/
-      //console.log('spotifyData', spotifyData)
-      //Get spotify username
-      if (Object.keys(spotifyData).some((key) => ~key.indexOf(".com.spotify"))) {
-        const keys = Object.keys(spotifyData).filter((key) => ~key.indexOf(".com.spotify"))
-        const username = keys[0].split(".com.spotify")[0]
-        //Get spotify search history
+
+      // console.log('spotifyData', spotifyData)
+      // Get spotify username
+      if (Object.keys(spotifyData).some((key) => ~key.indexOf('.com.spotify'))) {
+        const keys = Object.keys(spotifyData).filter((key) => ~key.indexOf('.com.spotify'))
+        const username = keys[0].split('.com.spotify')[0]
+        // Get spotify search history
         const searchHistory = spotifyData[username + '.com.spotify.feature.search.com.spotify.search.fancyRecents']
         searchHistory.forEach(element => {
           element.username = username
-        });
+        })
         spotifyResult = searchHistory
       }
       resolve(spotifyResult)
@@ -68,4 +57,4 @@ const spotifyReport = (backup) => {
       reject(e)
     }
   })
-}
+}

tools/reports/viber_calls.js → tools/reports/thirdparty/viber/calls.js

@@ -1,12 +1,5 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const database = fileHash('com.viber/database/Contacts.data', 'AppDomainGroup-group.viber.share.container')
 
@@ -41,20 +34,20 @@ module.exports.func = function (program, backup, resolve, reject) {
 const viberCallsReport = (backup) => {
   return new Promise((resolve, reject) => {
     var vibercallsdb = backup.getDatabase(database)
-      try {
-        const query = `
+    try {
+      const query = `
         SELECT * FROM ZRECENTSLINE
         INNER JOIN ZABCONTACT
         ON ZABCONTACT.Z_PK = ZRECENTSLINE.ZCONTACT
         ORDER BY ZABCONTACT.Z_PK;
         `
-        vibercallsdb.all(query, async function (err, rows) {
-          if (err) reject(err)
-          
-          resolve(rows)
-        })
-      } catch (e) {
-        reject(e)
-      }
+      vibercallsdb.all(query, async function (err, rows) {
+        if (err) reject(err)
+
+        resolve(rows)
+      })
+    } catch (e) {
+      reject(e)
+    }
   })
-}
+}

tools/reports/viber_contacts.js → tools/reports/thirdparty/viber/contacts.js

@@ -1,12 +1,5 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const database = fileHash('com.viber/database/Contacts.data', 'AppDomainGroup-group.viber.share.container')
 
@@ -39,20 +32,20 @@ module.exports.func = function (program, backup, resolve, reject) {
 const viberContactsReport = (backup) => {
   return new Promise((resolve, reject) => {
     var vibercontactsdb = backup.getDatabase(database)
-      try {
-        const query = `
+    try {
+      const query = `
         SELECT * FROM ZMEMBER
         INNER JOIN ZPHONENUMBER
         ON ZPHONENUMBER.ZMEMBER = ZMEMBER.Z_PK
         ORDER BY ZMEMBER.Z_PK;
         `
-        vibercontactsdb.all(query, async function (err, rows) {
-          if (err) reject(err)
-          
-          resolve(rows)
-        })
-      } catch (e) {
-        reject(e)
-      }
+      vibercontactsdb.all(query, async function (err, rows) {
+        if (err) reject(err)
+
+        resolve(rows)
+      })
+    } catch (e) {
+      reject(e)
+    }
   })
-}
+}

tools/reports/viber_messages.js → tools/reports/thirdparty/viber/messages.js

@@ -1,12 +1,5 @@
-const log = require('../util/log')
-const path = require('path')
-const sqlite3 = require('sqlite3')
-const bplist = require('bplist-parser')
-const fs = require('fs')
-const plist = require('plist')
-
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const database = fileHash('com.viber/database/Contacts.data', 'AppDomainGroup-group.viber.share.container')
 
@@ -41,8 +34,8 @@ module.exports.func = function (program, backup, resolve, reject) {
 const viberMessagesReport = (backup) => {
   return new Promise((resolve, reject) => {
     var vibermessagesdb = backup.getDatabase(database)
-      try {
-        const query = `
+    try {
+      const query = `
         SELECT * FROM ZVIBERMESSAGE
         INNER JOIN ZCONVERSATION
         ON ZCONVERSATION.Z_PK = ZVIBERMESSAGE.ZCONVERSATION
@@ -50,13 +43,13 @@ const viberMessagesReport = (backup) => {
         ON ZCONVERSATION.ZINTERLOCUTOR = ZMEMBER.Z_PK
         ORDER BY ZVIBERMESSAGE.Z_PK DESC;
         `
-        vibermessagesdb.all(query, async function (err, rows) {
-          if (err) reject(err)
-          
-          resolve(rows)
-        })
-      } catch (e) {
-        reject(e)
-      }
+      vibermessagesdb.all(query, async function (err, rows) {
+        if (err) reject(err)
+
+        resolve(rows)
+      })
+    } catch (e) {
+      reject(e)
+    }
   })
-}
+}

tools/reports/waze_favorites.js → tools/reports/thirdparty/waze/favorites.js

@@ -1,4 +1,4 @@
-const log = require('../util/log')
+const log = require('../../../util/log')
 const path = require('path')
 const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
@@ -6,7 +6,7 @@ const fs = require('fs')
 const plist = require('plist')
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const database = fileHash('Documents/user.db', 'AppDomain-com.waze.iphone')
 

tools/reports/waze_places.js → tools/reports/thirdparty/waze/places.js

@@ -1,4 +1,4 @@
-const log = require('../util/log')
+const log = require('../../../util/log')
 const path = require('path')
 const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
@@ -6,7 +6,7 @@ const fs = require('fs')
 const plist = require('plist')
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const database = fileHash('Documents/user.db', 'AppDomain-com.waze.iphone')
 

tools/reports/waze_recents.js → tools/reports/thirdparty/waze/recents.js

@@ -1,4 +1,4 @@
-const log = require('../util/log')
+const log = require('../../../util/log')
 const path = require('path')
 const sqlite3 = require('sqlite3')
 const bplist = require('bplist-parser')
@@ -6,7 +6,7 @@ const fs = require('fs')
 const plist = require('plist')
 
 // Derive filenames based on domain + file path
-const fileHash = require('../util/backup_filehash')
+const fileHash = require('../../../util/backup_filehash')
 
 const database = fileHash('Documents/user.db', 'AppDomain-com.waze.iphone')
 

tools/reports/voicemail-files.js

@@ -1,45 +0,0 @@
-const log = require('../util/log')
-const path = require('path')
-const fs = require('fs-extra')
-
-module.exports.name = 'voicemail-files'
-module.exports.description = 'List all or extract voicemail files (iOS 10+)'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getVoicemailFileList()
-    .then((list) => {
-      // Extract to the specified location
-      if (program.extract) {
-        for (var item of list) {
-          try {
-            var outDir = path.join(program.extract, path.basename(item.relativePath))
-            fs.ensureDirSync(path.dirname(outDir))
-            fs.createReadStream(backup.getFileName(item.fileID)).pipe(fs.createWriteStream(outDir))
-            item.output_dir = outDir
-          } catch (e) {
-            log.error(`Couldn't Export: ${item.relativePath}`, e)
-          }
-        }
-      }
-
-      // Generate report.
-      var result = program.formatter.format(list, {
-        program: program,
-        columns: {
-          'ID': el => el.fileID,
-          'Path': el => el.relativePath,
-          'Export Path': el => el.output_dir || '<not exported>'
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/voicemail.js

@@ -1,34 +0,0 @@
-module.exports.name = 'voicemail'
-module.exports.description = 'List all or extract voicemails on device'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works for iOS 9+
-module.exports.supportedVersions = '>=9.0'
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getVoicemailsList()
-    .then((items) => {
-      var result = program.formatter.format(items, {
-        program: program,
-        columns: {
-          'ID': el => el.ROWID,
-          'Date': el => el.XFORMATTEDDATESTRING,
-          'Sender': el => el.sender,
-          'Token': el => el.token,
-          'Duration': el => el.duration,
-          'Expiration': el => el.expiration,
-          'Trashed': el => el.trashed_date,
-          'Flags': el => el.flags
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/webhistory.js

@@ -1,31 +0,0 @@
-const { URL } = require('url')
-
-module.exports.name = 'webhistory'
-module.exports.description = 'List all web history'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-// Specify this only works for iOS 9+
-module.exports.supportedVersions = '>=9.0'
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getWebHistory(program.dump)
-    .then((history) => {
-      var result = program.formatter.format(history, {
-        program: program,
-        columns: {
-          'Time': el => el.XFORMATTEDDATESTRING,
-          'URL': el => new URL(el.url || '').origin || '',
-          'Title': el => (el.title || '').substring(0, 64)
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/reports/wifi.js

@@ -1,30 +0,0 @@
-module.exports.name = 'wifi'
-module.exports.description = 'List associated wifi networks and their usage information'
-
-// Specify this reporter requires a backup.
-// The second parameter to func() is now a backup instead of the path to one.
-module.exports.requiresBackup = true
-
-// Specify this reporter supports the promises API for allowing chaining of reports.
-module.exports.usesPromises = true
-
-module.exports.func = function (program, backup, resolve, reject) {
-  backup.getWifiList()
-    .then((items) => {
-      var result = program.formatter.format(items['List of known networks'], {
-        program: program,
-        columns: {
-          'Last Joined': el => el.lastJoined,
-          'Last AutoJoined': el => el.lastAutoJoined,
-          'SSID': el => el.SSID_STR,
-          'BSSID': el => el.BSSID,
-          'Security': el => el.SecurityMode || '',
-          'Hidden': el => el.HIDDEN_NETWORK || '',
-          'Enabled': el => el.enabled
-        }
-      })
-
-      resolve(result)
-    })
-    .catch(reject)
-}

tools/util/backup_filehash.js

@@ -1,6 +1,12 @@
 const crypto = require('crypto')
 
-/// Derive the name of the file inside of the backup from it's domain and file name.
+/**
+ * Derive a file's ID from it's filename and domain.
+ * @deprecated use backup3.js -> getFileID(file, domain) instead.
+ *
+ * @param {string} file the path to the file in the domain
+ * @param {string=} domain (optional) the file's domain. Default: HomeDomain
+ */
 module.exports = function fileHash (file, domain) {
   domain = domain || 'HomeDomain'
   let shasum = crypto.createHash('sha1')

tools/util/iphone_backup.js

@@ -44,6 +44,8 @@ var cache = {}
 
 class IPhoneBackup {
   constructor (id, status, info, manifest, base) {
+    log.warning('v3 reporting API is deprecated, this report may need to be updated to ensure stability with all iOS versions')
+    log.warning('https://github.com/richinfante/iphonebackuptools/wiki/V4-API-Migration-Notes')
     this.id = id
     this.status = status
     this.info = info
@@ -54,6 +56,8 @@ class IPhoneBackup {
   // Open a backup with a specified ID
   // base is optional and will be computed if not used.
   static fromID (id, base) {
+    id = id || ''
+    
     // Get the path of the folder.
     if (base) {
       base = path.join(base, id)
@@ -683,7 +687,7 @@ class IPhoneBackup {
                 if (err) reject(err)
                 ele.profile_picture = null
                 if (row) {
-                  ele.profile_picture = row.data.toString('base64')
+                  ele.profile_picture = (row.data || '').toString('base64')
                 }
                 iterateElements(elements, index + 1, callback)
               })
@@ -859,4 +863,4 @@ module.exports.availableBackups = function () {
 }
 
 module.exports.iPhoneBackup = IPhoneBackup
-module.exports.IPhoneBackup = IPhoneBackup
+module.exports.IPhoneBackup = IPhoneBackup

tools/util/mac_address_parse.js

@@ -1,8 +1,12 @@
-const zpad = require('zpad')
 
 module.exports = {
   pad_zeros: (macAddress) => {
-    if (macAddress) { return macAddress.split(':').map((hex) => zpad(hex)).join(':') }
+    if (macAddress) {
+      return macAddress.split(':')
+        .map((hex) => hex.padStart(2, '0'))
+        .join(':') 
+    }
+
     return macAddress
   }
 }

tools/util/matcher.js

@@ -0,0 +1,55 @@
+const log = require('./log')
+
+// Match an object with a query.
+// Ex: Querying a.* on { a: { b: 1, c: 2 }}} -> [1, 2]
+// We stop recusively attempting if isLeaf(node) returns true.
+module.exports = function match (object, query, isLeaf) {
+  isLeaf = isLeaf || function () { return true }
+
+  query = query || '*'
+  query = query.split('.')
+
+  return doMatch(object, query, isLeaf)
+}
+
+function nameMatches (query, name) {
+  /// __group is a reserved name.
+  if (name === '__group') {
+    return false
+  }
+
+  if (query === '*') {
+    return true
+  } else if (query === name) {
+    return true
+  }
+
+  return false
+}
+
+function doMatch (object, query, isLeaf) {
+  query = query || []
+
+  let result = []
+  let level = query.shift() || '*'
+
+  for (let [ key, value ] of Object.entries(object)) {
+    // If the name doesn't match, continue.
+    if (!nameMatches(level, key)) {
+      continue
+    }
+
+    
+    if (isLeaf(value)) {
+      // If it's a leaf, add a result.
+      result.push(value)
+    } else {
+      // Otherwise, add child results.
+      // We must slice(0) the query, so that it is duplicated.
+      result = [...result, ...doMatch(value, query.slice(0), isLeaf)]
+    }
+
+  }
+
+  return result
+}

tools/util/report_runner.js

@@ -0,0 +1,97 @@
+const version = require('./version_compare')
+const iPhoneBackup = require('./iphone_backup.js').iPhoneBackup
+const log = require('./log')
+
+async function runSwitchedReport (report, program) {
+  log.verbose('runner got path', program.base)
+  async function createPromise (key, program, backup) {
+    log.verbose('resolving using promises.')
+
+    return new Promise((resolve, reject) => {
+      report.functions[key](program, backup, resolve, reject)
+    })
+  }
+
+  // New type of reports
+  var backup = iPhoneBackup.fromID(program.backup, program.base)
+
+  var flag = false
+  var value
+  // Check for a compatible reporting tool.
+  for (var key in report.functions) {
+    if (version.versionCheck(backup.iOSVersion, key)) {
+      if (!report.usesPromises) {
+        log.verbose('using synchronous call.')
+
+        value = report.functions[key](program, backup)
+      } else {
+        // Use promises to resolve synchronously
+        value = await createPromise(key, program, backup)
+      }
+      flag = true
+      break
+    }
+  }
+
+  if (!flag) {
+    log.error(`Couldn't run '${report.name}'.`)
+    log.error(`The report generator '${report.name}' does not support iOS`, backup.iOSVersion)
+    log.error(`If you think it should, file an issue here:`)
+    log.error(`https://github.com/richinfante/iphonebackuptools/issues`)
+    return null
+  }
+
+  return value
+}
+
+async function runSingleReport (report, program) {
+  log.verbose('runner got path', program.base)
+  async function createPromise (program, backup, base) {
+    log.verbose('resolving using promises.')
+
+    return new Promise((resolve, reject) => {
+      if (report.requiresBackup) {
+        report.func(program, backup, resolve, reject)
+      } else {
+        report.func(program, base, resolve, reject)
+      }
+    })
+  }
+
+  async function runReport (backup, base) {
+    if (!report.usesPromises) {
+      log.verbose('using synchronous call.')
+
+      // Old-style non-promise based report.
+      if (report.requiresBackup) {
+        return report.func(program, backup)
+      } else {
+        return report.func(program, base)
+      }
+    } else {
+      // Create a promise to resolve this function
+      // Use promises to resolve synchronously
+      return createPromise(program, backup, base)
+    }
+  }
+
+  // New type of reports
+  var backup = iPhoneBackup.fromID(program.backup, program.base)
+
+  if (report.supportedVersions !== undefined) {
+    if (version.versionCheck(backup.iOSVersion, report.supportedVersions)) {
+      return runReport(backup, program.base)
+    } else {
+      log.error(`Couldn't run '${report.name}'.`)
+      log.error(`The report generator '${report.name}' does not support iOS`, backup.iOSVersion)
+      log.error(`If you think it should, file an issue here:`)
+      log.error(`https://github.com/richinfante/iphonebackuptools/issues`)
+      return null
+    }
+  } else {
+    return runReport(backup, program.base)
+  }
+}
+
+module.exports.runSwitchedReport = runSwitchedReport
+module.exports.runSingleReport = runSingleReport