B: Technology Stack

This project used a variety of tools, applications, programming languages, and so on. Below is a high-level record of the ‘tech stack’ - the what and why:

Programming

  1. Python - Main programming language.
  2. SQL - SELECT queries to extract source data from relational database.
  3. Cypher - Querying language for Neo4j Graph Databases.
  4. Batch - Windows command language to handle yaml files and multi-format rendering.
  5. VSCode - Main IDE (Integrated Development Environment).

Documentation

  • Quarto - Open-source technical publishing system.
  • Jupyter - Open-source application for interactive notebooks.
  • Zotero - Open-source reference management system.

Visualisation

  • Graphviz - Open-source graph visualisation application.
  • Mermaid - Open-source JavaScript diagramming tool.
  • Arrows - Neo4j Labs diagramming tool.

Versioning

  • Github - Web-based platform for version control and collaboration using Git.

Python Libraries

Several Python libraries were explored in the development of this prototype. The below libraries are the ones used in the current implementation.

Directory/File Handling

  • os - Interacting with the operating system for tasks like creating, deleting, and navigating directories and files.
  • glob - Finding files and directories based on pattern matching.
  • io - Working with input/output streams for reading and writing data.

Data Handling

  • pandas - Handling tabular data for analysis and manipulation.
  • json - Encoding and decoding JSON (JavaScript Object Notation) data.

Typing and Logging

  • typing - Adding type hints to code for better code readability, maintainability, and static type checking.
  • logging - Configuring and managing logging for application.
  • time - Working with time-related functions, potentially used for logging timestamps.

Database Connectivity

  • keyring - Securely storing and retrieving passwords and other sensitive information.
  • pyodbc - Connecting to and interacting with SQL databases using the Open Database Connectivity (ODBC) standard.
  • neo4j - Interacting with Neo4j graph databases.

Google API Integration

  • googleapiclient - Interacting with various Google APIs.
  • google.oauth2 - Handling OAuth 2.0 authentication for accessing Google services securely.

Anonymisation

  • random - Generating random numbers and making random choices.
  • hashlib - Implementing various secure hash and message digest algorithms.
  • Faker - Generating fake data for testing and development purposes.