Neo4j Load

With accessible csv files, the final module of the ETL pipeline creates (or updates) nodes and relationships in the Neo4j instance.

neo4j_load gdrive_files Google Drive Files (nodes, relationships folders) gdrive_api Connect to Google Drive API gdrive_files->gdrive_api Connect via API keyring Keyring (Credentials) neo4j_connection Connect to Neo4j (with credentials) keyring->neo4j_connection list_files List Files (from folders) gdrive_api->list_files determine_nodes Determine Nodes (to create) list_files->determine_nodes determine_relationships Determine Relationships (to create) list_files->determine_relationships create_schema Create Schema (dynamic or custom) list_files->create_schema Optionally Create create_nodes Create Nodes neo4j_connection->create_nodes create_relationships Create Relationships neo4j_connection->create_relationships determine_nodes->create_nodes determine_relationships->create_relationships create_schema->create_nodes create_schema->create_relationships set_node_properties Set Node Properties (with datatypes) create_nodes->set_node_properties set_relationship_properties Set Relationship Properties create_relationships->set_relationship_properties process_done Process Complete set_node_properties->process_done set_relationship_properties->process_done

There are two authentication requirements:

  1. Google Drive to get node and relationship files and data.
  2. Neo4j Aura instance is connected to with Keyring encrypted credentials.

Nodes and relationships are dynamically processed by using a file-pattern matching approach. However, this can be overridden within configuration, if needed.

Also in configuration is the option to create a database schema. There are three options:

  1. No schema
  2. Dynamic (default) - creates unique constraints based on nodes
  3. Custom - allows the user to specify specific constraints prior to loading.

At this point, the ETL loads data on a row-by-row basis, reading the public csv files. Columns become properties with data types cross-referenced from a data-mapping dictionary in the configuration.

If there have been no errors - we should have data in our Neo4j Aura instance!

Data successfully loaded for “MSc Artificial Intelligence (I400)”