Your First AWS Data Pipeline: S3 to Athena, Glue in Between

A logistics company's finance team drops a daily CSV export of shipment costs into an S3 bucket, the same way they've done for years. Someone used to open it in Excel every morning. Now the file is 40,000 rows and growing, and finance wants "just query it" without waiting for someone to email back a pivot table. That's the job AWS Data Engineering exists to solve, and it's a smaller build than it sounds.

Landing the File

S3 is the starting point for nearly every AWS pipeline, and the structure matters more than people expect on a first project. A flat bucket with every file dropped at the root works for a demo and breaks the moment two departments both name a file export.csv on the same day. The fix is a prefix convention that encodes the source and the date directly into the path, something like raw/shipping-costs/2026/09/03/export.csv. Glue and Athena both read S3 paths as implicit partitions later, so getting this right on day one saves a painful re-load down the line.

Letting Glue Find the Schema

Once the file lands, an AWS Glue Crawler points at that S3 prefix, inspects the file, and writes a table definition into the Glue Data Catalog, column names, inferred types, the works, without anyone hand-writing a CREATE TABLE statement. Glue ships built-in classifiers for CSV, JSON, and a handful of other common formats, and it tries them in order until one confidently matches the file. A CSV with a genuinely ambiguous delimiter is one of the few cases where the crawler guesses wrong and a custom classifier earns its keep.

The Transform Step Nobody Skips

Raw exports are never actually clean. Currency columns arrive as text with a stray dollar sign, dates come in three different formats depending on which system generated the row, and a shipment occasionally has no recorded cost at all. A Glue ETL job, PySpark under the hood, is where that gets fixed: cast columns, drop or flag nulls, and write the result back to S3 as Parquet rather than CSV, which shrinks both storage and every Athena query that reads it later.

Re-running that job every night against the same growing file would waste money and time re-processing rows that were already handled. Glue's answer is job bookmarks: pass a transformation_ctx string to the relevant read or write step, and Glue persists which data it has already seen between runs, so a nightly job only touches what actually arrived since yesterday.

Bad rows shouldn't just vanish silently, either. A shipment with no cost value is still worth routing somewhere finance can see it, rather than dropping it or letting it corrupt an average.

Querying Without a Server

This is the part that surprises people coming from an on-prem warehouse background: there's no database to provision. Athena queries the Parquet table straight out of S3 using standard SQL, and AWS bills per query rather than per hour of a running server. The rate is $5 per terabyte of data actually scanned (AWS, 2026), which is exactly why the Parquet conversion and the dated S3 prefixes from step one matter economically, not just architecturally. A query that Athena can restrict to one day's partition scans a few megabytes. The same query against an unpartitioned pile of CSV scans the whole bucket every time, and the bill follows.

When to Bring In Redshift

Athena is the right tool for exactly this kind of ad hoc, finance-runs-their-own-queries use case. It stops being enough once dozens of analysts are running complex joins against the same tables all day, or the business wants dashboards refreshing constantly. That's the point where the pipeline graduates to Redshift, either loading the cleaned Parquet data with a COPY command or querying it in place through Redshift Spectrum.

The shipment-cost example above is a small pipeline, but it's the same four-stage shape as far bigger ones: land it, catalog it, clean it, query it. SkyTrainings' AWS Data Engineering course builds exactly this sequence across its Data Processing and Data Analytics modules, then adds the real-time layer with Kinesis and MSK once batch pipelines like this one feel routine.

Build one yourself before deciding whether the course is worth it. The AWS Data Engineering course walks through this same S3-to-Glue-to-Athena path hands-on, then goes further into Redshift and streaming.