Consumer Milestone: Airline Dataset Ingestion

Your team will use the existing backend API service you already built and deployed as the consumer for the new airline datasets published by nexus. Your service should receive SQS messages, parse the dataset, and persist the contents into your database.

Due: Friday, June 19 · 11:59 PM

What You Must Build

Consume dataset messages from your existing SQS integration.
Detect the dataset type from meta.dataset.
Persist each dataset into your database.
Support replay/seed for all datasets.
Support schedule updates for the rolling 30-day window.

Datasets

Dataset	Purpose	Persisted Data
`airports`	Airport lookup for routes and search	Airport code, city, country, countryCode, region
`airlines`	Airline lookup	Airline code and name
`aircrafts`	Seat capacity per aircraft	Aircraft id, airline code, type, seat capacity
`locations`	Tourism-focused semantic search content	Document text and metadata
`flight-policies`	Flight-only travel policy content for RAG/search	Document text and metadata
`schedules`	Bookable flights in the active 30-day window	Flight instances, dates, times, airline, aircraft, price

How Publishing Works

Static Datasets

Static datasets are published only when a team requests a seed/replay. They are not published daily.

Schedules

Schedules use a rolling 30-day window. Teams may seed a range within the current window. After the initial seed, the system assumes your service already has days 1 through 29. Each new day, nexus publishes only the new 30th day.

Initial seed: request a range inside the current 30-day window.
Daily publish: one day only, the newest day in the window.
Consumer behavior: drop the oldest day and ingest the new day.

Travel vs Airline Catalog

catalog: "travel": all airline schedules are included in one message and delivered only to the requesting team queue.
catalog: "airline": only that airline's schedules are included. Supported airline teams are blue, green, and red.

Sample Publish Requests

Request URL: POST https://nexus.incs1008.com/publish

1. Airports

Request body

{
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "airports"
  }
}

Response

{
  "status": "published",
  "message": "Dataset published successfully",
  "eventId": "94b892d0-5f60-632d-a87c-461e341d14b0",
  "failedEntryCount": 0,
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "airports",
    "environment": "stage",
    "publisher": "nexus",
    "version": "1.0",
    "recordCount": 12,
    "dataHash": "...",
    "timestamp": "2026-06-07T13:22:51.456Z",
    "reason": "on-demand"
  }
}

2. Airlines

Request body

{
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "airlines"
  }
}

Response

{
  "status": "published",
  "message": "Dataset published successfully",
  "eventId": "5d72d830-6d55-4abc-9d4e-cd73a14b4f22",
  "failedEntryCount": 0,
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "airlines",
    "environment": "stage",
    "publisher": "nexus",
    "version": "1.0",
    "recordCount": 3,
    "dataHash": "...",
    "timestamp": "2026-06-07T13:24:10.104Z",
    "reason": "on-demand"
  }
}

3. Aircrafts

Request body

{
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "aircrafts"
  }
}

Response

{
  "status": "published",
  "message": "Dataset published successfully",
  "eventId": "c1a6a223-07a2-46cc-92a4-a4ae012e74d4",
  "failedEntryCount": 0,
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "aircrafts",
    "environment": "stage",
    "publisher": "nexus",
    "version": "1.0",
    "recordCount": 3,
    "dataHash": "...",
    "timestamp": "2026-06-07T13:25:32.991Z",
    "reason": "on-demand"
  }
}

4. Locations

Request body

{
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "locations"
  }
}

Response

{
  "status": "published",
  "message": "Dataset published successfully",
  "eventId": "2f43b91b-3ec0-4517-b07e-6408df0921c9",
  "failedEntryCount": 0,
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "locations",
    "environment": "stage",
    "publisher": "nexus",
    "version": "1.0",
    "recordCount": 10,
    "dataHash": "...",
    "timestamp": "2026-06-07T13:26:58.512Z",
    "reason": "on-demand"
  }
}

5. Flight Policies

Request body

{
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "flight-policies"
  }
}

Response

{
  "status": "published",
  "message": "Dataset published successfully",
  "eventId": "a45f2ba0-7a96-4a29-a00b-f548537a4532",
  "failedEntryCount": 0,
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "flight-policies",
    "environment": "stage",
    "publisher": "nexus",
    "version": "1.0",
    "recordCount": 5,
    "dataHash": "...",
    "timestamp": "2026-06-07T13:28:14.220Z",
    "reason": "on-demand"
  }
}

6. Flight Schedules - Airline

Request body

{
  "meta": {
    "scope": "team",
    "team": "blue",
    "dataset": "schedules",
    "catalog": "airline",
    "startDate": "2026-06-07",
    "endDate": "2026-06-20"
  }
}

Response

{
  "status": "published",
  "message": "Dataset published successfully",
  "eventId": "1ce53117-449a-49a4-ac7a-3f1f3b116e67",
  "failedEntryCount": 0,
  "meta": {
    "scope": "team",
    "team": "blue",
    "dataset": "schedules",
    "catalog": "airline",
    "startDate": "2026-06-07",
    "endDate": "2026-06-20",
    "environment": "stage",
    "publisher": "nexus",
    "version": "1.0",
    "recordCount": 168,
    "dataHash": "...",
    "timestamp": "2026-06-07T13:30:09.802Z",
    "reason": "on-demand"
  },
  "requestedRange": {
    "startDate": "2026-06-07",
    "endDate": "2026-06-20"
  },
  "effectiveRange": {
    "startDate": "2026-06-07",
    "endDate": "2026-06-20"
  }
}

7. Flight Schedules - Travel

Request body

{
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "schedules",
    "catalog": "travel",
    "startDate": "2026-06-07",
    "endDate": "2026-06-20"
  }
}

Response

{
  "status": "published",
  "message": "Dataset published successfully",
  "eventId": "d1c38a7f-3881-4671-aad2-59c5fca65e95",
  "failedEntryCount": 0,
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "schedules",
    "catalog": "travel",
    "startDate": "2026-06-07",
    "endDate": "2026-06-20",
    "environment": "stage",
    "publisher": "nexus",
    "version": "1.0",
    "recordCount": 504,
    "dataHash": "...",
    "timestamp": "2026-06-07T13:31:44.110Z",
    "reason": "on-demand"
  },
  "requestedRange": {
    "startDate": "2026-06-07",
    "endDate": "2026-06-20"
  },
  "effectiveRange": {
    "startDate": "2026-06-07",
    "endDate": "2026-06-20"
  }
}

The actual dataset is not returned here. Your service must consume it from SQS.

Sample SQS Message Shape

Airports

{
  "meta": {
    "scope": "team",
    "team": "white",
    "dataset": "airports",
    "environment": "stage",
    "publisher": "nexus",
    "version": "1.0",
    "recordCount": 12,
    "dataHash": "...",
    "timestamp": "2026-06-07T13:22:51.456Z",
    "reason": "on-demand"
  },
  "data": {
    "airports": [
      {
        "code": "DXB",
        "name": "Dubai International Airport",
        "city": "Dubai",
        "country": "United Arab Emirates",
        "countryCode": "AE",
        "region": "Middle East"
      }
    ]
  }
}

Schedules

{
  "meta": {
    "scope": "team",
    "team": "blue",
    "dataset": "schedules",
    "catalog": "airline",
    "startDate": "2026-06-07",
    "endDate": "2026-06-10",
    "environment": "stage",
    "publisher": "nexus",
    "version": "1.0",
    "recordCount": 48,
    "dataHash": "...",
    "timestamp": "2026-06-07T13:22:51.456Z",
    "reason": "on-demand"
  },
  "data": {
    "schedules": [
      {
        "flightInstanceId": "BL101-2026-06-07",
        "flightNumber": "BL101",
        "airlineCode": "BL",
        "aircraftId": "BL-A320-01",
        "originAirportCode": "DXB",
        "destinationAirportCode": "CAI",
        "departureDate": "2026-06-07",
        "arrivalDate": "2026-06-07",
        "departureLocalTime": "08:00",
        "arrivalLocalTime": "10:45",
        "status": "SCHEDULED",
        "price": 220,
        "currency": "USD"
      }
    ]
  }
}

How To Consume Each Dataset

airports

Read data.airports.
Upsert by airport code.

airlines

Read data.airlines.
Upsert by airline code.

aircrafts

Read data.aircrafts.
Upsert by aircraftId.
Use seatCapacity for booking inventory logic.

locations

Read data.documents.
Persist documents and metadata for semantic search.

flight-policies

Read data.documents.
Persist documents and metadata for embedding and policy lookup.

schedules

Read data.schedules.
Upsert by flightInstanceId.
Join with aircrafts using aircraftId.
Join with airports using airport codes.
For airline catalog, keep only your airline's rows. This is already filtered for you by nexus.

Simple Grading Rubric

Area	Points	Expectation
Airports ingestion	10	Consumed from SQS and persisted correctly
Airlines ingestion	10	Consumed from SQS and persisted correctly
Aircrafts ingestion	10	Consumed from SQS and persisted correctly
Locations ingestion	15	Consumed from SQS and stored for semantic search use
Flight policies ingestion	15	Consumed from SQS and stored for semantic search use
Schedule ingestion	20	Consumed from SQS and persisted using flightInstanceId
30-day rolling window handling	10	Initial seed plus daily ingestion of the newest day only
Travel vs airline handling	10	Travel site supports all airlines, airline site supports own airline