Fortnightly Update - 10 June 2024

Summary

This week has been about getting back on track after an extended period away from the dissertation. It has mainly revolved around refocusing, rescoping, reacquainting and rekindling motivation.

Accomplishments

• Results:

  • recreated single student graph timetable - see poc-1-basic
  • created SQL to csv pipeline
  • created cypher queries

• Project Management

  • new github repo (currently private) https://github.com/zoonalink/graph-project
    • draft readme
    • folder/file structure
  • weekly update template

• Data Collection:

  • SQL to CSV files for single student
  • raw SQL data tables

• Analysis / Wrangling:

  • Transformation in SQL query

• Model Development:

• Investigation

  • pipelines to automate data flow - prefect
  • how to represent time in graph

Issues/Blockers

• Technical:
  • Original servers were deleted and access to new servers was gone
  • Some original work no longer works
• Methodological:
  • I need to ensure that scope does not creep.
  • I need to stay focused and not start solving problems which are out of scope or are interesting.
• Data-Related:
  • develop a robust anonymisation process
  • representing time - see representing time

Next Steps

Weekly Goals

1. Bigger cohort of data (MSc DataScience?)
2. Explore time in Graph
3. Data pipeline documentation and steps (anonymisation)

• Project Management Project tasks - planning, admin.
  • add supervisor to github repo
• Data:
  • Anonymisation or generation
  • MSc Data Science cohort isolated into separate csv files
  • splitting data into single rows
  • Pipeline steps plotted
• Analysis / Wrangling:
  • More advanced cypher queries
• Modeling:
  • Comparing different representations of time
• Validation:
• Deadlines:

Post-Meeting Notes

We met on 13 June 2024 for approximately 45 minutes. I showed Xiaodong what I have been up to referencing my working files, poc files, and some code. I also showed what I currently have in my free instance of neo4j aura - which was MSc DS students activities and rooms. Staff and Students to be loaded but I want to ensure my anonymisation function is tested first.

We discussed what my aims are with this project, clarifying that it is not building a timetable schedule based on graph structure; it is also not a timetable optimiser. Instead it is a data engineering project that ultimately explores whether representing timetables in graph format can bring opportunities for reporting and insight which is currently difficult to produce using the current system.

We agreed to meet in two weeks’ time. I will schedule a meeting for us.

Additional Notes

Summary from intial meeting

I met with Xiaodong Li my supervisor on Wednesday 01 May 2024 where we introduced ourselves and our backgrounds. We discussed my proposal which Xiaodong had read in advance of our meeting.

I showed Xiaodong my work so far, most of it completed in January which included proof-of-concept data engineering steps to extract, transform and load data from a relational database to a graph structure. I showed the steps I had taken, the challenges encountered and the possible opportunities of graph data structures which I am hoping to explore in more detail.

We discussed project scope and outcomes and recognised that I need to ensure that I keep within scope.

The next few weeks will require attention on other matters (work, taught modules, etc.) but I will look to pick up project work soon.

If possible, Xiaodong was going to investigate Neo4j and graph to get a bit of context.