Fortnightly Update - 2024-06-24

Summary

Significant progress in the backend and more proof-of-concept in front end.

Accomplishments

• Project Management
  • post supervisor meeting notes
• Data Collection:
  • Anonymisation function for staff/student personal data developed and tested.
  • MSc Data Science cohort isolated into separate csv files
  • Data extraction pipeline designed, developed, tested. This allows for extracting data filtered by programme on demand.
    • modularised, scalable, configurable, efficient pipeline
  • Transformation pipeline (preprocessing, anonymising)
    • More than half way completed - same principles
    • need to add staff, student, activity nodes and test outcomes
    • need to add relationships and test.
  • Loading to Neo4j pipeline developed - suitable for version 1
• Analysis / Wrangling:
  • More advanced cypher queries - constraint violation queries developed using cypher for version 1. some are very complex. more testing needed.
  • list of insights which can be derived
• Model Development:
  • Comparing different representations of time
• Results:
• Pipeline development
• Cypher queries for version 1
• conversation with business owners to validate work

Next Steps

Weekly Goal: What is goal of next fortnight?

• write up and benchmark v1 notes
• finish developing and testing pipeline:
  • extract
  • transform
  • load
• document pipeline - written and visual (mermaid?)
• develop cypher queries for version 2
• design (theory) timetable quality index
• consider scaling dataset
• start writing project notes

Issues/Blockers

• Technical:
  • Digital certificates on machines preventing load
  • Neo4j Aura (free) limitations - loading issues
• Methodological: Concerns about approach or analysis?
  • Spending a lot of time getting ETL ‘right’
  • Not sure about balance of project and what I will deliver at the end
• Data-Related: Issues with data quality, access, or quantity?
  • Working on pipelines which will mean I can scale accordingly (within constraints of free instance)

Post-Meeting Notes

My supervisor and I spoke about where I am at in the project at the moment and next steps. We discussed what I am attempting to do and why, including graph data structures, proof-of-concept, etc. and that it is becoming a data engineering project. I stated that my timeline looks to complete a robust data ETL pipeline by the middle of July, with a view to shifting towards more work within Neo4j from the end of July to the middle of August.

We agreed to meet in two weeks.